Skip to content

fix(kms): name the cause of AWS KMS envelope failures - #400

Draft
aka-bo wants to merge 9 commits into
mainfrom
kms-decrypt-error-visibility-v2
Draft

aka-bo wants to merge 9 commits into
mainfrom
kms-decrypt-error-visibility-v2

Conversation

@aka-bo

@aka-bo aka-bo commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Problem

When every regional Decrypt attempt in AwsKmsEnvelope failed, the error and its error-level log line read:

all KMS backends failed to decrypt (tried regions: us-west-2)

Any cause produced that string: a Decrypt service error, a response without plaintext, a data key that failed authentication. The per-region cause was logged only at debug, as the SdkError's Display, which is the bare variant name (service error). AccessDenied, missing credentials, throttling and an unreachable endpoint were indistinguishable.

GenerateDataKey and Encrypt failures had a different fault. The SdkError was kept as the error source, so {:#} (used by the core's error log) printed the AWS error message after it. That message embeds principal and key ARNs and account ids. Separately, a region map with key and value swapped put a key ARN where the region name belongs, and it was printed as the region.

Change

  • Each attempted decrypt region records the failed step, the SDK failure kind, a fixed label for client-side causes (credentials error, invalid endpoint parameters, I/O error kind, no region configured), the KMS error code and the AWS request id:
    all KMS backends failed to decrypt (tried regions: us-west-2 [Decrypt service error AccessDeniedException, request id 4f6c8a1e-…])
    The all KMS backends failed to decrypt ( prefix and the tried regions: / no matching regions found tokens stay; a no-match failure also lists the configured and envelope regions.
  • GenerateDataKey and Encrypt failures use the same formatter and name the operation and region: KMS GenerateDataKey failed in region us-west-2: dispatch failure (other: no credentials available).
  • No error's Display, Debug or message text is used, so the AWS message, key and envelope ARNs, credential-provider text and connector addresses never reach the output. Server-supplied and configured values pass only in a narrow shape and otherwise print as a placeholder: error codes [A-Za-z]{1,64}, request ids in UUID form, regions [a-z][a-z0-9-]{0,31}.
  • Where callers see the detail: the aggregated decrypt failure is logged at error level and lists every attempted region with its cause. The exception or error a binding raises carries only the outermost context; the bindings' default log hooks deliver warn and error, which is where an application gets the detail.
  • docs/kms-errors.md documents the message shapes and labels once; the KMS section of the six binding troubleshooting guides is rewritten to match and links to it. aws-credential-types becomes a direct dependency of asherah for the credentials-error downcast (already in Cargo.lock).

Behaviour changes

  • Decrypt and encrypt error text differs as described above. Code matching the all KMS backends failed to decrypt ( prefix keeps working; exact-string comparison does not.
  • A MultiKms composed over AwsKmsEnvelope backends now treats AccessDenied, Disabled and InvalidKeyUsage failures from the preferred backend as terminal and stops instead of trying the next backend. is_terminal_kms_error in asherah/src/kms_multi.rs (unchanged) substring-matches the error text, which now carries the KMS error code; before, it carried only region names, so the heuristic never fired for this backend. Nothing in the repository composes the two this way.
  • MultiKms over the plain single-region AwsKms (what kms_builders::AwsKmsBuilder builds) is unchanged and still never short-circuits, because that type formats the SDK error without its code. That inconsistency is left for a separate change.
  • GenerateDataKey and Encrypt errors no longer carry the SdkError as their source, so callers cannot downcast to it. This is deliberate: the source chain carries the AWS message text.

Scope

Error formatting in AwsKmsEnvelope, with its docs and tests. The stored envelope format and the configuration are unchanged. Out of scope:

  • Error text of the plain AwsKms and of MultiKms itself.
  • DynamoDB metastore errors, which still carry the AWS message.
  • A TLS-specific label; TLS failures read dispatch failure (io: other error).
  • A min_level option for the binding log hooks (Python's is the PR stacked on this one).

Trade-off

The credentials-ordering test waits a few seconds on the SDK's credentials provider in the default run. The connect-timeout scenario fills a listener's accept backlog, which depends on the host, so it is an ignored test: cargo test -p asherah --test kms_envelope_transport_errors -- --ignored.

Tests

  • A fake KMS on loopback, after every AWS configuration source (config and credentials files, profile, IMDS, proxies) is removed and with a refusal to run unless the SDK resolves a loopback endpoint. It asserts the exact message for each decrypt and encrypt failure mode, including through a full session decrypt, and pins the MultiKms interaction above, sync and async.
  • Checks that no returned error or log record, at any level, carries the key ARN, envelope ARN, AWS message or data-key bytes, including with the region map swapped.
  • Unit tests cover the code, request-id and region shapes (real values, the exact maximum length, rejected boundary cases) and the labels of SdkErrors built without sockets.
  • Scenarios that need their own process-wide environment run in separate test binaries: no credentials, refused connection, connect timeout, no region, and a credentials error above an I/O error (the label must name the credentials failure).

🤖 Generated with Claude Code

aka-bo and others added 9 commits October 27, 2026 16:54
AwsKmsEnvelope reported every decrypt failure as "all KMS backends
failed to decrypt (tried regions: us-west-2)". A Decrypt service error,
a response without plaintext and a data key that failed AEAD produced
the same string, and the per-region debug lines printed the SdkError's
Display, which is only the variant name ("service error"), so an
AccessDeniedException could not be told apart from throttling or a
network failure.

Each attempted region now records the step that failed: the Decrypt
call by SdkError kind (with the KMS error code for a service error, the
connector error kind for a dispatch failure, and the HTTP status where
no code is available), "Decrypt returned no plaintext", or "data key
failed envelope authentication", plus the AWS request id when a
response carried one. The aggregate error and its error-level log line
keep the "all KMS backends failed to decrypt (" prefix and the
"tried regions: " / "no matching regions found" tokens and append the
per-region detail; a decrypt with no matching region also lists the
configured and envelope regions.

GenerateDataKey and Encrypt failures on the encrypt path go through the
same classifier into errors that name the operation and the region.
Before, `?` kept the SdkError as the anyhow source, and `{:#}` (which
the Python binding logs at warn) printed the AWS error message, which
embeds principal and key ARNs and account ids.

AWS error messages, configured and envelope ARNs, and AEAD error text
are never formatted. Error codes and request ids are server-supplied,
so they pass only in a bounded charset and length; anything else
becomes a fixed placeholder.

kms_envelope_errors.rs covers each failure mode against an offline fake
KMS on 127.0.0.1. Before any client is built it removes every AWS
configuration source (config and credentials files, profile, IMDS,
proxies), installs dummy credentials, disables retries, and refuses to
run unless the SDK resolves a loopback endpoint and the dummy
credentials.

Co-Authored-By: Claude <noreply@anthropic.com>
KMS calls that failed before reaching an endpoint all read "dispatch
failure (other)": missing credentials and a missing region could not be
told apart from each other or from any other client-side failure, and
the encrypt path no longer named the cause. A region map with key and
value swapped printed the key ARN, account id included, as a region in
the decrypt error, its Error log line and the encrypt error.

describe_sdk_error now appends a fixed label for the first recognised
cause on the SdkError's source chain, identified by downcasting: the
CredentialsError variant ("other: no credentials available"), invalid
endpoint parameters (a region that is not a valid host label), or the
std::io::ErrorKind ("io: connection refused"; "response error (HTTP
400, unexpected end of file)"). No cause's Display or Debug is used:
credential providers can carry STS or SSO messages with role ARNs, and
connector errors carry addresses. A client whose configuration resolved
no region is reported as "not sent: no region configured" before any
SDK call, because the SDK's own cause for that case is a crate-private
type.

Every region name this module prints, configured or read from an
envelope, at every log level, must match [a-z][a-z0-9-]{0,31} and is
otherwise printed as <unrecognized region>. Error codes now pass only as
[A-Za-z]{1,64} and request ids only in the UUID form.

aws-credential-types becomes a direct dependency of asherah (already in
Cargo.lock at 1.3.0) for the CredentialsError downcast.

Behaviour changes:
- A MultiKms composed over AwsKmsEnvelope backends now treats
  AccessDenied, Disabled, (KMS)InvalidKeyUsage and NotAuthorized
  failures from the preferred backend as terminal and does not try the
  other backends: is_terminal_kms_error matches those codes, and they
  now appear in the envelope's error text. Other failures still fall
  back. A test pins this.
- Errors from the encrypt path (GenerateDataKey, Encrypt) no longer
  carry the SDK error as their source, so they cannot be downcast to
  SdkError.

Two new test binaries cover failures that need their own environment:
kms_envelope_nocreds.rs (no credentials) and
kms_envelope_transport_errors.rs (connection refused, connect timeout,
no region). They apply the same loopback-only fence as
kms_envelope_errors.rs, now shared in tests/support/kms_fence.rs.

Co-Authored-By: Claude <noreply@anthropic.com>
The troubleshooting guides for every binding now separate DynamoDB
metastore errors, which still carry the AWS message, from KMS errors,
which carry fixed labels, and say which KMS labels mean configuration
(credentials, region) and which mean an unreachable endpoint.

The Python guide gains a KMS section listing those labels and where a
Python caller sees them: the RuntimeError carries only the outermost
context, while the KMS error event and the binding's warn event, both
delivered by the default log hook, carry the per-region detail. Its
diagnostic recipe called set_log_hook_sync(..., min_level="trace"),
which asherah-py does not provide, and set "Verbose", which the Python
binding does not read; it now installs set_log_hook(callback) and
states that the hook delivers warn and error events.

Co-Authored-By: Claude <noreply@anthropic.com>
The troubleshooting pages filed every `dispatch failure (io: …)` under
an unreachable endpoint, but TLS failures (an untrusted certificate, a
non-TLS listener, a handshake the server closes) also read
`io: other error`, and retrying does not fix them. The pages also
listed `dispatch failure: ResolveError`, which the HTTP stack in use
cannot produce, named only one of the five credentials labels, and
showed DynamoDB ValidationException in a form aws-sdk-dynamodb does
not print for an exception it does not model.

All six pages now separate connection-level I/O kinds from
`io: other error`, list every credentials label with its known causes,
show DynamoDB's unmodelled exceptions as `service error: unhandled
error (<Code>)`, and describe DNS failures only as a possible
`dispatch failure (io: …)`. The Node, Java, .NET, Ruby and Go pages
now say that the error returned to application code carries only the
outermost context, and that the KMS detail is in the log events the
log hook receives at its default (warn) level.

The module comment in kms_aws_envelope.rs now states the remaining
limits of the token shapes: values that fit a shape are printed
verbatim, so an endpoint answering as KMS can return arbitrary
letters-only words as a code, and a region-shaped name carrying an
account id or key-id prefix is printed whether it comes from the
stored envelope or from configuration.

kms_envelope_credentials_order.rs points the container credentials
provider at a closed loopback port. The resulting chain holds a
CredentialsError above an io::Error (connection refused); the test
pins that the label names the credentials failure, which an
"I/O kind anywhere on the chain wins" ordering would report as a
refused connection.

Co-Authored-By: Claude <noreply@anthropic.com>
The Ruby and .NET troubleshooting pages said every async method returns
the outermost context alone and logs no warn event. That holds only for
the methods that use the native async calls (Ruby
Session#encrypt_bytes_async / #decrypt_bytes_async; .NET
AsherahSession.*BytesAsync and AsherahApi.EncryptAsync / DecryptAsync).
Ruby's module-level Asherah.encrypt_async / decrypt_async and the .NET
Compat package's Session.EncryptAsync / DecryptAsync run the
synchronous call on another thread, so they raise the synchronous
`<op> failed: …` shape and log the warn event. Both pages now describe
both families.

All six pages listed a `credentials provider timed out` label that no
locked credential provider produces. A credentials endpoint that never
answers shows as a bare `dispatch failure (other)` after a delay, and
the pages now say so; `invalid credentials configuration` is described
by its observed cause, an invalid AWS_CONTAINER_AUTHORIZATION_TOKEN.
The DNS bullets regain the VPC endpoint / network ACL / AWS_REGION
advice where the pages had it before.

Co-Authored-By: Claude <noreply@anthropic.com>
The module comment in kms_aws_envelope.rs listed every label source,
every metadata shape and each shape's limits, and the decrypt-loop
comment repeated the inventory of output fields and exclusions. Each
choice is now explained once, beside the code responsible for it: the
shape limits on the validators, the credentials-before-I/O ordering on
source_label, and why the SdkError is not kept as a source on
kms_call_error. The decrypt-loop comment keeps the preferred-region
ordering and the envelope arn/AEAD explanation and drops its inventory
of what the per-region reasons exclude.

Comments only; no code or error text changes.

Co-Authored-By: Claude <noreply@anthropic.com>
…-timeout scenario opt-in

The shape tests exercised character-and-length rules with a hand-copied
snapshot of aws-sdk-kms 1.114.0's modeled error codes and partition
regions, plus a length assert against the snapshot itself. The lists
proved nothing a few representative values do not, could not detect SDK
drift, and had to be kept in step by hand. They now use a few real codes
and regions, the exact maximum length, and the rejected boundary cases,
plus uppercase inside a region name and a trailing newline in a code.

A new unit test builds SdkErrors without sockets (connector timeout,
I/O and user errors, a credentials error above an I/O error, an
operation timeout) and checks the exact label for each.

The connect-timeout scenario in kms_envelope_transport_errors.rs fills
a listener's accept backlog, which depends on the host and takes
several seconds. It is now its own ignored test, run with --ignored;
the connection-refused and no-region scenarios stay in the default run.
Both tests take one lock, so --include-ignored keeps their
process-wide environment changes apart.

Co-Authored-By: Claude <noreply@anthropic.com>
…troubleshooting guides

The six binding troubleshooting guides each carried the same 60-100
line inventory of AWS KMS failure labels. That inventory now lives once
in docs/kms-errors.md: the encrypt and decrypt message shapes, where the
detail is logged, each label with its meaning and next step, the
placeholders that stand in for values outside their expected shape, and
how a request id lets AWS Support or CloudTrail find the call.

Each guide keeps a short KMS section: one representative message, the
binding's own statement of what the exception carries and which log
events hold the detail (now naming the hook API), one line of guidance
each for permissions, credentials, region, network and envelope
failures, and a link to the shared page.

Test-environment observations (call timings, unreproduced cases,
examples tying credential labels to particular endpoint behaviour) and
the unconditional retry verdict on `dispatch failure (io: other error)`
are removed; that label is now described as one that can indicate TLS
trouble. The Python guide no longer suggests PreferredRegion for a
missing region, since it applies only together with RegionMap.

Co-Authored-By: Claude <noreply@anthropic.com>
…imeout label

The six troubleshooting guides showed a DynamoDB DNS failure as
`dispatch failure: io error: client error (Connect): dns error: …`. That
string was never observed; the guides' remaining network advice covers
the case without quoting a message that may not match.

docs/kms-errors.md now says that `credentials provider timed out` is
defined for the SDK's timeout variant but not produced by any of the
locked credential providers, and that a source that never answers shows
as a bare `dispatch failure (other)`.

Co-Authored-By: Claude <noreply@anthropic.com>
@aka-bo
aka-bo force-pushed the kms-decrypt-error-visibility-v2 branch from 1d778dc to 05a9c10 Compare October 27, 2026 23:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant