Skip to content
Draft
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

62 changes: 51 additions & 11 deletions asherah-dotnet/docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,20 +192,60 @@ architecture in your publish profile if needed.

## AWS-specific errors

Forwarded from the AWS SDK for Rust running in the native core. Common
shapes:

- `dispatch failure: unhandled error: ResolveError` — DNS resolution
failed. Check VPC endpoints / network ACLs / `AWS_REGION` agreement.
- `service error: AccessDeniedException` — IAM. The error usually
names the missing action; cross-reference against the policy in
DynamoDB metastore errors (`Metastore: "dynamodb"`) carry the AWS
SDK's error chain, including the AWS message. Exceptions that
aws-sdk-dynamodb does not model, `AccessDeniedException` and
`ValidationException` among them, print as `service error: unhandled
error (<Code>)` followed by the AWS message. Common shapes:

- `service error: unhandled error (AccessDeniedException)` — IAM. The
AWS message that follows usually names the missing `dynamodb:`
action; cross-reference against the policy in
[`aws-production-setup.md`](./aws-production-setup.md#step-3-iam-policy-for-the-application).
- `service error: ValidationException: ...AttributeName...` — DynamoDB
table schema doesn't match. Asherah expects partition key `Id`
- `service error: unhandled error (ValidationException)` — for example
a table schema mismatch. Asherah expects partition key `Id`
(string), sort key `Created` (number); recreate the table with the
correct schema if it was created manually with a different one.
- `service error: KMSInvalidStateException` — the KMS key is
`PendingDeletion` / `Disabled`. Re-enable or use a different key.

KMS errors (`KMS: "aws"`) name the step that failed, the region and a
fixed label for the cause, plus the AWS request id when AWS answered:

```
all KMS backends failed to decrypt (tried regions: us-west-2 [Decrypt service error AccessDeniedException, request id 4f6c8a1e-…])
```

The exception carries only the outermost context. Synchronous calls,
and the `GoDaddy.Asherah.Encryption.Compat` package's
`Session.EncryptAsync` / `DecryptAsync` (which run the synchronous call
with `Task.Run`), throw `<op>: <op> failed: …` and also log the full
chain, with the KMS detail, as a warn-level `<op> failed: …` event.
`AsherahSession.EncryptBytesAsync` / `DecryptBytesAsync` and
`AsherahApi.EncryptAsync` / `DecryptAsync` use the native async calls:
their exception carries the outermost context alone, and the native
layer logs no warn-level event for them. For every call style, a failed
KMS decrypt also produces an error-level `asherah::kms_aws_envelope`
event and a failed KMS encrypt an error-level `asherah::session`
event. The log hook (`AsherahHooks.SetLogHook`) receives these at its
default level (warn).

- Permissions or key state: `service error AccessDeniedException` (IAM
or key policy), `DisabledException`, `KMSInvalidStateException`,
`NotFoundException` (key disabled, pending deletion or missing).
- Credentials: `dispatch failure (other: no credentials available)` or
another credentials label; the request never reached KMS.
- Region: `not sent: no region configured` — set `AWS_REGION` or
`RegionMap`; `no matching regions found` — no configured region has
a KEK in the stored envelope.
- Network: `dispatch failure (io: …)`, `(timeout)`, `(other: transient
error)` — KMS could not be reached; `io: other error` can indicate TLS
trouble. Check VPC endpoints / network ACLs / `AWS_REGION` agreement.
- Envelope: `data key failed envelope authentication` — the envelope
is corrupt or the KEK was replaced out of band.

Unlike the DynamoDB errors above, KMS errors omit the AWS message,
which embeds ARNs and account ids; the request id lets AWS support (or
CloudTrail) find the call. Every label is explained in
[`docs/kms-errors.md`](../../docs/kms-errors.md).

## Diagnostic recipe

Expand Down
55 changes: 46 additions & 9 deletions asherah-go/docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,15 +155,52 @@ introduced a CGO import path; check your imports for `// #cgo` lines.

## AWS-specific errors

Forwarded from the AWS SDK for Rust running in the native FFI:

- `dispatch failure: ResolveError` — DNS resolution failed.
- `service error: AccessDeniedException` — IAM. The error names the
missing action.
- `service error: ValidationException: ...AttributeName...` —
DynamoDB schema mismatch.
- `service error: KMSInvalidStateException` — KMS key is
`PendingDeletion`/`Disabled`.
DynamoDB metastore errors (`Metastore: "dynamodb"`) carry the AWS
SDK's error chain, including the AWS message. Exceptions that
aws-sdk-dynamodb does not model, `AccessDeniedException` and
`ValidationException` among them, print as `service error: unhandled
error (<Code>)` followed by the AWS message:

- `service error: unhandled error (AccessDeniedException)` — IAM. The
AWS message that follows usually names the missing `dynamodb:`
action.
- `service error: unhandled error (ValidationException)` — for example
a table schema mismatch. Asherah expects partition key `Id`
(string), sort key `Created` (number).

KMS errors (`KMS: "aws"`) name the step that failed, the region and a
fixed label for the cause, plus the AWS request id when AWS answered:

```
all KMS backends failed to decrypt (tried regions: us-west-2 [Decrypt service error AccessDeniedException, request id 4f6c8a1e-…])
```

The returned error carries only the outermost context (`<op>
failed: …`). The full chain, with the KMS detail, is in the native
layer's warn-level `<op> failed: …` log event; a failed KMS decrypt
also produces an error-level `asherah::kms_aws_envelope` event and a
failed KMS encrypt an error-level `asherah::session` event. The log
hook (`asherah.SetLogHook` or `asherah.SetSlogLogger`) receives all of
these at its default level (warn).

- Permissions or key state: `service error AccessDeniedException` (IAM
or key policy), `DisabledException`, `KMSInvalidStateException`,
`NotFoundException` (key disabled, pending deletion or missing).
- Credentials: `dispatch failure (other: no credentials available)` or
another credentials label; the request never reached KMS.
- Region: `not sent: no region configured` — set `AWS_REGION` or
`RegionMap`; `no matching regions found` — no configured region has
a KEK in the stored envelope.
- Network: `dispatch failure (io: …)`, `(timeout)`, `(other: transient
error)` — KMS could not be reached; `io: other error` can indicate TLS
trouble. Check VPC endpoints / network ACLs / `AWS_REGION` agreement.
- Envelope: `data key failed envelope authentication` — the envelope
is corrupt or the KEK was replaced out of band.

Unlike the DynamoDB errors above, KMS errors omit the AWS message,
which embeds ARNs and account ids; the request id lets AWS support (or
CloudTrail) find the call. Every label is explained in
[`docs/kms-errors.md`](../../docs/kms-errors.md).

The `aws-sdk-go-v2` package's credential cache is **not** consulted
— Asherah uses the AWS SDK for Rust's chain.
Expand Down
58 changes: 48 additions & 10 deletions asherah-java/docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,17 +127,55 @@ retry.

## AWS-specific errors

Forwarded from the AWS SDK for Rust running in the native core:

- `dispatch failure: ResolveError` — DNS resolution failed. VPC
endpoint / network ACL / `AWS_REGION` mismatch.
- `service error: AccessDeniedException` — IAM. The error names the
missing action.
- `service error: ValidationException: ...AttributeName...` —
DynamoDB schema mismatch. Asherah expects partition key `Id`
DynamoDB metastore errors (`Metastore: "dynamodb"`) carry the AWS
SDK's error chain, including the AWS message. Exceptions that
aws-sdk-dynamodb does not model, `AccessDeniedException` and
`ValidationException` among them, print as `service error: unhandled
error (<Code>)` followed by the AWS message:

- `service error: unhandled error (AccessDeniedException)` — IAM. The
AWS message that follows usually names the missing `dynamodb:`
action.
- `service error: unhandled error (ValidationException)` — for example
a table schema mismatch. Asherah expects partition key `Id`
(string), sort key `Created` (number).
- `service error: KMSInvalidStateException` — KMS key is
`PendingDeletion`/`Disabled`.

KMS errors (`KMS: "aws"`) name the step that failed, the region and a
fixed label for the cause, plus the AWS request id when AWS answered:

```
all KMS backends failed to decrypt (tried regions: us-west-2 [Decrypt service error AccessDeniedException, request id 4f6c8a1e-…])
```

The `RuntimeException` carries only the outermost context:
`<op> failed: …` for synchronous calls, and the outermost context
alone when an async call's `CompletableFuture` completes
exceptionally. The full chain, with the KMS detail, is in a
warn-level log event (`<op> failed: …`, or `java async operation
failed: …` for async calls); a failed KMS decrypt also produces an
error-level `asherah::kms_aws_envelope` event and a failed KMS encrypt
an error-level `asherah::session` event. The log hook
(`Asherah.setLogHook`) receives all of these at its default level
(warn).

- Permissions or key state: `service error AccessDeniedException` (IAM
or key policy), `DisabledException`, `KMSInvalidStateException`,
`NotFoundException` (key disabled, pending deletion or missing).
- Credentials: `dispatch failure (other: no credentials available)` or
another credentials label; the request never reached KMS.
- Region: `not sent: no region configured` — set `AWS_REGION` or
`RegionMap`; `no matching regions found` — no configured region has
a KEK in the stored envelope.
- Network: `dispatch failure (io: …)`, `(timeout)`, `(other: transient
error)` — KMS could not be reached; `io: other error` can indicate TLS
trouble. Check for a VPC endpoint / network ACL / `AWS_REGION` mismatch.
- Envelope: `data key failed envelope authentication` — the envelope
is corrupt or the KEK was replaced out of band.

Unlike the DynamoDB errors above, KMS errors omit the AWS message,
which embeds ARNs and account ids; the request id lets AWS support (or
CloudTrail) find the call. Every label is explained in
[`docs/kms-errors.md`](../../docs/kms-errors.md).

The AWS Java SDK's `DefaultCredentialsProvider` is **not** consulted —
Asherah's native core uses the AWS SDK for Rust's chain. They have
Expand Down
54 changes: 44 additions & 10 deletions asherah-node/docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,18 +141,52 @@ Reinstall Node for the matching architecture and `npm install`.

## AWS-specific errors

Forwarded from the AWS SDK for Rust running in the native core:

- `dispatch failure: ResolveError` — DNS resolution failed. VPC
endpoint / network ACL / `AWS_REGION` mismatch.
- `service error: AccessDeniedException` — IAM. The error names the
missing action; cross-reference against
DynamoDB metastore errors (`Metastore: "dynamodb"`) carry the AWS
SDK's error chain, including the AWS message. Exceptions that
aws-sdk-dynamodb does not model, `AccessDeniedException` and
`ValidationException` among them, print as `service error: unhandled
error (<Code>)` followed by the AWS message:

- `service error: unhandled error (AccessDeniedException)` — IAM. The
AWS message that follows usually names the missing `dynamodb:`
action; cross-reference against
[`aws-production-setup.md`](./aws-production-setup.md#step-3-iam-policy).
- `service error: ValidationException: ...AttributeName...` —
DynamoDB schema mismatch. Asherah expects partition key `Id`
- `service error: unhandled error (ValidationException)` — for example
a table schema mismatch. Asherah expects partition key `Id`
(string), sort key `Created` (number).
- `service error: KMSInvalidStateException` — KMS key is
`PendingDeletion`/`Disabled`. Re-enable or use a different key.

KMS errors (`KMS: "aws"`) name the step that failed, the region and a
fixed label for the cause, plus the AWS request id when AWS answered:

```
all KMS backends failed to decrypt (tried regions: us-west-2 [Decrypt service error AccessDeniedException, request id 4f6c8a1e-…])
```

The `Error` thrown to your code carries only the outermost context
(`<op> failed: …`). The full chain, with the KMS detail, is in the
binding's warn-level `<op> failed: …` log event; a failed KMS decrypt
also produces an error-level `asherah::kms_aws_envelope` event and a
failed KMS encrypt an error-level `asherah::session` event. The log
hook (`setLogHook`) receives all of these at its default level (warn).

- Permissions or key state: `service error AccessDeniedException` (IAM
or key policy), `DisabledException`, `KMSInvalidStateException`,
`NotFoundException` (key disabled, pending deletion or missing).
- Credentials: `dispatch failure (other: no credentials available)` or
another credentials label; the request never reached KMS.
- Region: `not sent: no region configured` — set `AWS_REGION` or
`RegionMap`; `no matching regions found` — no configured region has
a KEK in the stored envelope.
- Network: `dispatch failure (io: …)`, `(timeout)`, `(other: transient
error)` — KMS could not be reached; `io: other error` can indicate TLS
trouble. Check for a VPC endpoint / network ACL / `AWS_REGION` mismatch.
- Envelope: `data key failed envelope authentication` — the envelope
is corrupt or the KEK was replaced out of band.

Unlike the DynamoDB errors above, KMS errors omit the AWS message,
which embeds ARNs and account ids; the request id lets AWS support (or
CloudTrail) find the call. Every label is explained in
[`docs/kms-errors.md`](../../docs/kms-errors.md).

## Diagnostic recipe

Expand Down
71 changes: 56 additions & 15 deletions asherah-py/docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,18 +131,55 @@ Reinstall Python for the matching architecture and `pip install`.

## AWS-specific errors

Forwarded from the AWS SDK for Rust running in the native core:
### DynamoDB metastore

- `dispatch failure: ResolveError` — DNS resolution failed. VPC
endpoint / network ACL / `AWS_REGION` mismatch.
- `service error: AccessDeniedException` — IAM. The error names the
missing action; cross-reference against
DynamoDB metastore errors (`Metastore: "dynamodb"`) carry the AWS
SDK's error chain, including the AWS message. Exceptions that
aws-sdk-dynamodb does not model, `AccessDeniedException` and
`ValidationException` among them, print as `service error: unhandled
error (<Code>)` followed by the AWS message:

- `service error: unhandled error (AccessDeniedException)` — IAM. The
AWS message that follows usually names the missing `dynamodb:`
action; cross-reference against
[`aws-production-setup.md`](./aws-production-setup.md#step-3-iam-policy).
- `service error: ValidationException: ...AttributeName...` —
DynamoDB schema mismatch. Asherah expects partition key `Id`
- `service error: unhandled error (ValidationException)` — for example
a table schema mismatch. Asherah expects partition key `Id`
(string), sort key `Created` (number).
- `service error: KMSInvalidStateException` — KMS key is
`PendingDeletion`/`Disabled`.

### KMS

KMS failures (`KMS: "aws"`) name the step that failed, the region and a
fixed label for the cause, plus the AWS request id when AWS answered:

```
all KMS backends failed to decrypt (tried regions: us-west-2 [Decrypt service error AccessDeniedException, request id 4f6c8a1e-…])
```

Decrypt failures are logged as an `asherah::kms_aws_envelope` error
event, and both decrypt and encrypt failures appear in full in the
binding's `python binding operation failed: …` warn event; both reach
the log hook (`asherah.set_log_hook`) at the default `warn` level. The
`RuntimeError` message carries only the outermost context.

- Permissions or key state: `service error AccessDeniedException` (IAM
or key policy), `DisabledException`, `KMSInvalidStateException`,
`NotFoundException` (key disabled, pending deletion or missing).
- Credentials: `dispatch failure (other: no credentials available)` or
another credentials label; the request never reached KMS.
- Region: `not sent: no region configured` — set `AWS_REGION` or
`RegionMap`; `no matching regions found` — no configured region has
a KEK in the stored envelope.
- Network: `dispatch failure (io: …)`, `(timeout)`, `(other: transient
error)` — KMS could not be reached; `io: other error` can indicate TLS
trouble. Check for a VPC endpoint / network ACL / `AWS_REGION` mismatch.
- Envelope: `data key failed envelope authentication` — the envelope
is corrupt or the KEK was replaced out of band.

Unlike the DynamoDB errors above, KMS errors omit the AWS message,
which embeds ARNs and account ids; the request id lets AWS support (or
CloudTrail) find the call. Every label is explained in
[`docs/kms-errors.md`](../../docs/kms-errors.md).

## asyncio caveats

Expand All @@ -165,15 +202,19 @@ that's still consuming Asherah resources. Ensure all in-flight encrypt

When a problem isn't covered above:

1. **Set verbose logging:**
1. **Install a log hook:**
```python
def trace_log(event):
def on_log(event):
print(f"[asherah {event['level']}] {event['target']}: {event['message']}")
asherah.set_log_hook_sync(trace_log, min_level="trace")
factory = asherah.SessionFactory.from_config({**config, "Verbose": True})
asherah.set_log_hook(on_log)
factory = asherah.SessionFactory.from_config(config)
```
Trace records cover every metastore call, KMS call, and
key-cache decision.
The hook receives `warn` and `error` events; `info`, `debug` and
`trace` records are not delivered. An operation that fails in the
Rust core logs a `python binding operation failed: …` warn event
with the full error chain, and a failed KMS decrypt logs an
`asherah::kms_aws_envelope` error event that lists every attempted
region with its cause (see [KMS](#kms)).

2. **Inspect the metastore directly.** RDBMS: query
`encryption_key`. DynamoDB: scan `AsherahKeys`. Confirm rows for
Expand Down
Loading
Loading