Repository navigation
Conversation
AwsKmsEnvelope reported every decrypt failure as "all KMS backends
failed to decrypt (tried regions: us-west-2)". A Decrypt service error,
a response without plaintext and a data key that failed AEAD produced
the same string, and the per-region debug lines printed the SdkError's
Display, which is only the variant name ("service error"), so an
AccessDeniedException could not be told apart from throttling or a
network failure.
Each attempted region now records the step that failed: the Decrypt
call by SdkError kind (with the KMS error code for a service error, the
connector error kind for a dispatch failure, and the HTTP status where
no code is available), "Decrypt returned no plaintext", or "data key
failed envelope authentication", plus the AWS request id when a
response carried one. The aggregate error and its error-level log line
keep the "all KMS backends failed to decrypt (" prefix and the
"tried regions: " / "no matching regions found" tokens and append the
per-region detail; a decrypt with no matching region also lists the
configured and envelope regions.
GenerateDataKey and Encrypt failures on the encrypt path go through the
same classifier into errors that name the operation and the region.
Before, `?` kept the SdkError as the anyhow source, and `{:#}` (which
the Python binding logs at warn) printed the AWS error message, which
embeds principal and key ARNs and account ids.
AWS error messages, configured and envelope ARNs, and AEAD error text
are never formatted. Error codes and request ids are server-supplied,
so they pass only in a bounded charset and length; anything else
becomes a fixed placeholder.
kms_envelope_errors.rs covers each failure mode against an offline fake
KMS on 127.0.0.1. Before any client is built it removes every AWS
configuration source (config and credentials files, profile, IMDS,
proxies), installs dummy credentials, disables retries, and refuses to
run unless the SDK resolves a loopback endpoint and the dummy
credentials.
Co-Authored-By: Claude <noreply@anthropic.com>
KMS calls that failed before reaching an endpoint all read "dispatch
failure (other)": missing credentials and a missing region could not be
told apart from each other or from any other client-side failure, and
the encrypt path no longer named the cause. A region map with key and
value swapped printed the key ARN, account id included, as a region in
the decrypt error, its Error log line and the encrypt error.
describe_sdk_error now appends a fixed label for the first recognised
cause on the SdkError's source chain, identified by downcasting: the
CredentialsError variant ("other: no credentials available"), invalid
endpoint parameters (a region that is not a valid host label), or the
std::io::ErrorKind ("io: connection refused"; "response error (HTTP
400, unexpected end of file)"). No cause's Display or Debug is used:
credential providers can carry STS or SSO messages with role ARNs, and
connector errors carry addresses. A client whose configuration resolved
no region is reported as "not sent: no region configured" before any
SDK call, because the SDK's own cause for that case is a crate-private
type.
Every region name this module prints, configured or read from an
envelope, at every log level, must match [a-z][a-z0-9-]{0,31} and is
otherwise printed as <unrecognized region>. Error codes now pass only as
[A-Za-z]{1,64} and request ids only in the UUID form.
aws-credential-types becomes a direct dependency of asherah (already in
Cargo.lock at 1.3.0) for the CredentialsError downcast.
Behaviour changes:
- A MultiKms composed over AwsKmsEnvelope backends now treats
AccessDenied, Disabled, (KMS)InvalidKeyUsage and NotAuthorized
failures from the preferred backend as terminal and does not try the
other backends: is_terminal_kms_error matches those codes, and they
now appear in the envelope's error text. Other failures still fall
back. A test pins this.
- Errors from the encrypt path (GenerateDataKey, Encrypt) no longer
carry the SDK error as their source, so they cannot be downcast to
SdkError.
Two new test binaries cover failures that need their own environment:
kms_envelope_nocreds.rs (no credentials) and
kms_envelope_transport_errors.rs (connection refused, connect timeout,
no region). They apply the same loopback-only fence as
kms_envelope_errors.rs, now shared in tests/support/kms_fence.rs.
Co-Authored-By: Claude <noreply@anthropic.com>
The troubleshooting guides for every binding now separate DynamoDB metastore errors, which still carry the AWS message, from KMS errors, which carry fixed labels, and say which KMS labels mean configuration (credentials, region) and which mean an unreachable endpoint. The Python guide gains a KMS section listing those labels and where a Python caller sees them: the RuntimeError carries only the outermost context, while the KMS error event and the binding's warn event, both delivered by the default log hook, carry the per-region detail. Its diagnostic recipe called set_log_hook_sync(..., min_level="trace"), which asherah-py does not provide, and set "Verbose", which the Python binding does not read; it now installs set_log_hook(callback) and states that the hook delivers warn and error events. Co-Authored-By: Claude <noreply@anthropic.com>
The troubleshooting pages filed every `dispatch failure (io: …)` under an unreachable endpoint, but TLS failures (an untrusted certificate, a non-TLS listener, a handshake the server closes) also read `io: other error`, and retrying does not fix them. The pages also listed `dispatch failure: ResolveError`, which the HTTP stack in use cannot produce, named only one of the five credentials labels, and showed DynamoDB ValidationException in a form aws-sdk-dynamodb does not print for an exception it does not model. All six pages now separate connection-level I/O kinds from `io: other error`, list every credentials label with its known causes, show DynamoDB's unmodelled exceptions as `service error: unhandled error (<Code>)`, and describe DNS failures only as a possible `dispatch failure (io: …)`. The Node, Java, .NET, Ruby and Go pages now say that the error returned to application code carries only the outermost context, and that the KMS detail is in the log events the log hook receives at its default (warn) level. The module comment in kms_aws_envelope.rs now states the remaining limits of the token shapes: values that fit a shape are printed verbatim, so an endpoint answering as KMS can return arbitrary letters-only words as a code, and a region-shaped name carrying an account id or key-id prefix is printed whether it comes from the stored envelope or from configuration. kms_envelope_credentials_order.rs points the container credentials provider at a closed loopback port. The resulting chain holds a CredentialsError above an io::Error (connection refused); the test pins that the label names the credentials failure, which an "I/O kind anywhere on the chain wins" ordering would report as a refused connection. Co-Authored-By: Claude <noreply@anthropic.com>
The Ruby and .NET troubleshooting pages said every async method returns the outermost context alone and logs no warn event. That holds only for the methods that use the native async calls (Ruby Session#encrypt_bytes_async / #decrypt_bytes_async; .NET AsherahSession.*BytesAsync and AsherahApi.EncryptAsync / DecryptAsync). Ruby's module-level Asherah.encrypt_async / decrypt_async and the .NET Compat package's Session.EncryptAsync / DecryptAsync run the synchronous call on another thread, so they raise the synchronous `<op> failed: …` shape and log the warn event. Both pages now describe both families. All six pages listed a `credentials provider timed out` label that no locked credential provider produces. A credentials endpoint that never answers shows as a bare `dispatch failure (other)` after a delay, and the pages now say so; `invalid credentials configuration` is described by its observed cause, an invalid AWS_CONTAINER_AUTHORIZATION_TOKEN. The DNS bullets regain the VPC endpoint / network ACL / AWS_REGION advice where the pages had it before. Co-Authored-By: Claude <noreply@anthropic.com>
The module comment in kms_aws_envelope.rs listed every label source, every metadata shape and each shape's limits, and the decrypt-loop comment repeated the inventory of output fields and exclusions. Each choice is now explained once, beside the code responsible for it: the shape limits on the validators, the credentials-before-I/O ordering on source_label, and why the SdkError is not kept as a source on kms_call_error. The decrypt-loop comment keeps the preferred-region ordering and the envelope arn/AEAD explanation and drops its inventory of what the per-region reasons exclude. Comments only; no code or error text changes. Co-Authored-By: Claude <noreply@anthropic.com>
…-timeout scenario opt-in The shape tests exercised character-and-length rules with a hand-copied snapshot of aws-sdk-kms 1.114.0's modeled error codes and partition regions, plus a length assert against the snapshot itself. The lists proved nothing a few representative values do not, could not detect SDK drift, and had to be kept in step by hand. They now use a few real codes and regions, the exact maximum length, and the rejected boundary cases, plus uppercase inside a region name and a trailing newline in a code. A new unit test builds SdkErrors without sockets (connector timeout, I/O and user errors, a credentials error above an I/O error, an operation timeout) and checks the exact label for each. The connect-timeout scenario in kms_envelope_transport_errors.rs fills a listener's accept backlog, which depends on the host and takes several seconds. It is now its own ignored test, run with --ignored; the connection-refused and no-region scenarios stay in the default run. Both tests take one lock, so --include-ignored keeps their process-wide environment changes apart. Co-Authored-By: Claude <noreply@anthropic.com>
…troubleshooting guides The six binding troubleshooting guides each carried the same 60-100 line inventory of AWS KMS failure labels. That inventory now lives once in docs/kms-errors.md: the encrypt and decrypt message shapes, where the detail is logged, each label with its meaning and next step, the placeholders that stand in for values outside their expected shape, and how a request id lets AWS Support or CloudTrail find the call. Each guide keeps a short KMS section: one representative message, the binding's own statement of what the exception carries and which log events hold the detail (now naming the hook API), one line of guidance each for permissions, credentials, region, network and envelope failures, and a link to the shared page. Test-environment observations (call timings, unreproduced cases, examples tying credential labels to particular endpoint behaviour) and the unconditional retry verdict on `dispatch failure (io: other error)` are removed; that label is now described as one that can indicate TLS trouble. The Python guide no longer suggests PreferredRegion for a missing region, since it applies only together with RegionMap. Co-Authored-By: Claude <noreply@anthropic.com>
…imeout label The six troubleshooting guides showed a DynamoDB DNS failure as `dispatch failure: io error: client error (Connect): dns error: …`. That string was never observed; the guides' remaining network advice covers the case without quoting a message that may not match. docs/kms-errors.md now says that `credentials provider timed out` is defined for the SDK's timeout variant but not produced by any of the locked credential providers, and that a source that never answers shows as a bare `dispatch failure (other)`. Co-Authored-By: Claude <noreply@anthropic.com>
aka-bo
force-pushed
the
kms-decrypt-error-visibility-v2
branch
from
October 27, 2026 23:54
1d778dc to
05a9c10
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When every regional Decrypt attempt in
AwsKmsEnvelopefailed, the error and its error-level log line read:Any cause produced that string: a Decrypt service error, a response without plaintext, a data key that failed authentication. The per-region cause was logged only at debug, as the
SdkError's Display, which is the bare variant name (service error). AccessDenied, missing credentials, throttling and an unreachable endpoint were indistinguishable.GenerateDataKey and Encrypt failures had a different fault. The
SdkErrorwas kept as the error source, so{:#}(used by the core's error log) printed the AWS error message after it. That message embeds principal and key ARNs and account ids. Separately, a region map with key and value swapped put a key ARN where the region name belongs, and it was printed as the region.Change
all KMS backends failed to decrypt (tried regions: us-west-2 [Decrypt service error AccessDeniedException, request id 4f6c8a1e-…])The
all KMS backends failed to decrypt (prefix and thetried regions:/no matching regions foundtokens stay; a no-match failure also lists the configured and envelope regions.KMS GenerateDataKey failed in region us-west-2: dispatch failure (other: no credentials available).[A-Za-z]{1,64}, request ids in UUID form, regions[a-z][a-z0-9-]{0,31}.docs/kms-errors.mddocuments the message shapes and labels once; the KMS section of the six binding troubleshooting guides is rewritten to match and links to it.aws-credential-typesbecomes a direct dependency ofasherahfor the credentials-error downcast (already inCargo.lock).Behaviour changes
all KMS backends failed to decrypt (prefix keeps working; exact-string comparison does not.MultiKmscomposed overAwsKmsEnvelopebackends now treats AccessDenied, Disabled and InvalidKeyUsage failures from the preferred backend as terminal and stops instead of trying the next backend.is_terminal_kms_errorinasherah/src/kms_multi.rs(unchanged) substring-matches the error text, which now carries the KMS error code; before, it carried only region names, so the heuristic never fired for this backend. Nothing in the repository composes the two this way.MultiKmsover the plain single-regionAwsKms(whatkms_builders::AwsKmsBuilderbuilds) is unchanged and still never short-circuits, because that type formats the SDK error without its code. That inconsistency is left for a separate change.SdkErroras their source, so callers cannot downcast to it. This is deliberate: the source chain carries the AWS message text.Scope
Error formatting in
AwsKmsEnvelope, with its docs and tests. The stored envelope format and the configuration are unchanged. Out of scope:AwsKmsand ofMultiKmsitself.dispatch failure (io: other error).min_leveloption for the binding log hooks (Python's is the PR stacked on this one).Trade-off
The credentials-ordering test waits a few seconds on the SDK's credentials provider in the default run. The connect-timeout scenario fills a listener's accept backlog, which depends on the host, so it is an ignored test:
cargo test -p asherah --test kms_envelope_transport_errors -- --ignored.Tests
MultiKmsinteraction above, sync and async.SdkErrors built without sockets.🤖 Generated with Claude Code