Skip to content

feat(config): type secrets as such and read config from several files - #857

Draft
cjorge-thegraph wants to merge 1 commit into
mainfrom
cjorge/typed-secrets
Draft

cjorge-thegraph wants to merge 1 commit into
mainfrom
cjorge/typed-secrets

Conversation

@cjorge-thegraph

Copy link
Copy Markdown
Contributor

Replaces #848. Instead of _file references and DIPPER__ environment overrides, dipper now knows which fields are secrets by their type, and its configuration can come from several files merged in order.

The problem: dipper read its whole configuration from 1 JSON file, so every secret sat next to settings that are not secret. The file could then only be kept as a secret, out of review. The secret fields were also typed unevenly:

  • The RPC provider URLs, which carry API keys, were plain Urls, and the Kafka sasl_password was a plain String with hand-written redaction.
  • serde quoted a value it could not read in its error, so a mistyped secret ended up in a log: invalid type: string "hunter2-secret", expected u32.

Secrets known by type

  • Hidden<T> (dipper-core) leaves a value it can't deserialise out of the error, and so does the signer key's hex adapter. The error keeps its position.
  • Kafka: the sasl_passwords in dipper-producer are now Hidden<String>. dipper-producer depends on dipper-core for this; it already depended on thegraph-core.
  • SecretUrl (new) is used for chain_client.providers and alerts.slack_webhook_url.
    • Shown: the URL's scheme, host, port and the shape of its path.
    • Masked: the user and password, every query value (its name stays), the fragment, and every path segment but a version such as v2.
    • Fingerprint: when anything was masked, a short keccak fingerprint of the whole URL follows: https://mainnet.infura.io/v3/*** (fingerprint e1d789). 2 log lines can then be told to use the same key or not, without showing it.
    • Why mask every segment: a key can look like any word (/v2/rpc-key), so dipper doesn't guess which segment holds it; the fingerprint is what tells keys apart.
    • Errors: a URL that fails to parse is left out of the error.
    • Code: ChainClientConfig::provider_urls() exposes the real URLs for connecting.
    • Visible change: the startup config dump now shows the webhook's host and its path masked. Before, the whole webhook was hidden, and the providers showed by host only.

Config from several files

  • Usage: dipper-service config.json [secrets.json ...]. Every argument is a file, merged in order: objects key by key, anything else (a string, a number, a list, null) replaced by the later file.
  • Read once: the merged result is deserialised once, so deny_unknown_fields still refuses a misspelt field in any file.
  • Errors name the field through serde_path_to_error (db.password: …). A single file still reports its line and column.
  • Failed loads: a missing or unreadable file, or no file at all, exits with code 2 and a usage line rather than a panic.
  • Where secrets live:
    • A test setup keeps everything inline in 1 file.
    • Production keeps config.json free of secrets in the ConfigMap, reviewable in git, and adds a secrets.json holding only the secret fields. That file lives in a Secret, for example one External Secrets renders from the secret store.
  • Docs: k8s/README.md shows the shape of secrets.json and the extra mount and args. It notes that a list (chain_client.providers) is replaced whole, and that a changed secret needs a restart. The main README lists the secret fields.

Dropped from #848

  • _file references
  • the DIPPER__ environment overrides
  • the guessing between JSON and string values

That removes the review's findings on #848: ambiguous types, values in errors, map defaults wiped by an override, unreachable list items, and the UTF-8 panic.

Not done: a startup warning when a secret field sits inline in the 1st file while a 2nd is given, which would catch a secret committed to the ConfigMap. It is easy to add if wanted.

Tests

  • a_rejected_secret_is_left_out_of_the_error (dipper-core) failed on main with the quoted value above, and passes now.
  • SecretUrl: hosted RPC URLs (Alchemy, Infura, Ankr, QuickNode), a word-like key, Slack, user-info, query and fragment, a URL with nothing to mask, fingerprints, Debug/Display, and rejected values.
  • Loader:
    • a secrets file merging into the example config;
    • the same config from 1 file or 2;
    • merge rules;
    • an unknown field in a later file;
    • errors naming db.password and alerts.slack_webhook_url without their values;
    • line numbers for a single file;
    • no file or a missing file;
    • the logged config masking keys.
  • The existing dump test still finds no rpc-key in the output. The webhook test now checks that the host shows with the path masked.
  • cargo +nightly fmt, cargo clippy --workspace --all-targets --all-features -D warnings and the workspace tests (950 in dipper-service) pass locally.

🤖 Generated with Claude Code

Dipper read its whole configuration from 1 JSON file, so every secret had to sit next to settings
that are not secret, and the file could then only be kept as a secret, out of review. And the
secret fields were typed unevenly: the RPC provider URLs, which carry API keys, and the Kafka
password were plain values, and serde quoted a secret it could not read in the error, which ends
up in a log.

Secrets are now known by type:

- `Hidden<T>` leaves a value it cannot deserialise out of the error, as does the signer key's
  hex adapter, and the Kafka `sasl_password`s become `Hidden<String>`.
- `SecretUrl`, for `chain_client.providers` and `alerts.slack_webhook_url`, shows a URL with its
  scheme, host, port and the shape of its path, masking the user and password, query values,
  the fragment and every path segment but a version such as `v2`, with a short fingerprint
  when anything was masked: `https://mainnet.infura.io/v3/*** (fingerprint e1d789)`. A key can
  look like any word, so dipper does not guess which segment holds it; the fingerprint tells 2
  keys apart. A URL that fails to parse is left out of the error.

The config now comes from every file on the command line, merged in order: objects key by key,
anything else replaced by the later file. The merged result is read once, so an unknown field is
refused wherever it came from, and an error names the field (`db.password: ...`); a single file
still reports its line and column. Secrets can stay inline in 1 file, as a test setup wants, or
go in a later `secrets.json` that holds only the secret fields, kept in a Secret, so the
ConfigMap holds none and can be reviewed. The READMEs and the manifest describe both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant