Skip to content

perf(dispatch): resolve a run's authority once before launch - #1469

Open
MTG-Thomas wants to merge 2 commits into
gobifrost:mainfrom
MTG-Thomas:t3/dispatch-authority-once
Open

MTG-Thomas wants to merge 2 commits into
gobifrost:mainfrom
MTG-Thomas:t3/dispatch-authority-once

Conversation

@MTG-Thomas

@MTG-Thomas MTG-Thomas commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem And Result

Before a workflow run reaches its worker, the consumer reads the run's authority twice and the run's user row twice:

  • issue_run_credentials mints claims from the run's RuntimeAuthority in a read-only snapshot, then the consumer opens another session and loads the same authority again for provenance.
  • load_run_user reads the users row for identity/active state, then build_authorization_context reads the same row again.

Each statement is cheap in Postgres (0.01-0.05 ms), but they run one after another on the dispatch path at about 1-1.5 ms per round trip.

Now:

  • Claims credentials carry the authority they were minted from (IssuedAccessToken.authority, RunCredentials.authority), and the consumer reuses it. Legacy-format tokens are signed without reading the authority, so that route still loads it, as before.
  • load_run_user reads the user row once and passes it to the new authorization_context_for_user_row; build_authorization_context keeps its signature and delegates to it.

Provenance is still read in its own short session, so transactions per run are unchanged. No protocol, schema, settings, DTO or UI change.

One promoter test simulated the initiator being deleted between the two user-row reads. That window no longer exists, so the test now covers an initiator already gone when it is read (test_an_initiator_deleted_before_the_recheck_fails_that_row_only).

Measurements

Sync POST /api/workflows/execute against a remote worker, one request in flight, alternating builds (before, fix, before, fix), spans exported to a local sink. Illustrative numbers from one host:

p50 Before Fix
Claimed → worker receives the launch command 41.3 / 41.4 ms 30.8 / 30.9 ms
Workflow-consumer span 48.3 ms 38.7 ms
Whole request 249.6 ms 231.0 ms
SQL statements per execute (statement log, 20 runs) 97.6 90.0

Contribution Basis And Tracking

Actual use: our production instance runs tens of thousands of short, mostly webhook-driven workflow executions a day, and per-run dispatch overhead is a large share of its CPU. We found these repeated reads while tracing dispatch with the OpenTelemetry work discussed in #1386. No issue was filed for this change; happy to open one if you prefer to track it there.

Verification

  • New tests failed on main for the expected reasons (two users reads; no authority carried on credentials) and pass with the change.
  • ./test.sh tests/unit/authorization tests/unit/jobs/consumers tests/unit/jobs/schedulers/test_deferred_execution_promoter.py tests/unit/services/test_access_check_policies.py tests/unit/services/test_access_check_writer.py tests/unit/core/test_execution_policy_authority.py tests/unit/services/test_legacy_broker_drain.py: 866 passed.
  • ./test.sh tests/e2e/api/test_executions.py tests/e2e/api/test_access_check_attribution.py tests/e2e/api/test_agent_workflow_tool_execution.py tests/e2e/platform/test_workflow_conversion.py: 29 passed.
  • ./test.sh quality api: pyright 0 errors; Ruff passed.
  • ./test.sh pr-preflight on the rebased head: repository checks, lint/types, 13,444 backend unit tests and 663 client test files passed. The local CodeQL step did not run (no CodeQL CLI on this host); CI runs it.
  • The unit and E2E sets above ran before rebasing onto e22df7d, which touches none of the changed files or their callers.
  • Full backend E2E and browser suites were not run locally; the merge queue is the complete-suite gate.

Product Updates

Omit: routine performance repair, with canonical disposition in product-updates/dispositions/pr-1469.json, following current authoring policy.

🤖 Generated with Claude Code

The workflow consumer minted a run's claims credentials, which loads the
run's authority inside the minter's read-only snapshot, and then opened a
second session to load the same authority again only to read the run
user and provenance. Claims credentials now carry the authority they were
minted from, and the consumer reuses it. A legacy-format token is signed
without reading the authority, so that route still loads it as before.

load_run_user also read the user's row twice: once for identity kind and
active state, then again inside build_authorization_context. It now reads
the row once and builds the context from it. The promoter test that
simulated a deletion between those two reads now simulates the user being
gone when the recheck reads them, the only window left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant