Skip to content

fix(shutdown): stop taking jobs first and fit the teardown in its budget - #854

Draft
cjorge-thegraph wants to merge 2 commits into
mainfrom
cjorge/shut-down-in-order
Draft

cjorge-thegraph wants to merge 2 commits into
mainfrom
cjorge/shut-down-in-order

Conversation

@cjorge-thegraph

@cjorge-thegraph cjorge-thegraph commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Makes dipper's shutdown finish inside its own time budget, and stops the worker taking new jobs at the start of a shutdown rather than near the end.

Why:

  • main's stop sequence no longer fits its budget. It stops 15 services one at a time, each capped at 5s, then flushes events and closes the pool. Its worst case is about 85s, past the 75s TEARDOWN_GRACE. A slow shutdown then ends with the supervisor aborting whatever is left, instead of finishing in order: the DB pool close is skipped, and lifecycle events still in the in-memory queue may never reach Kafka.
  • The worker is 13th in line. It keeps taking new jobs for most of a shutdown, then gets 5s to finish them. A job started during shutdown is usually cut off part-way, which wastes its IISA and RPC calls and gives it a retry on the next start.

What changes:

  1. The worker drains while its feeders stop. It stops taking jobs and gets 45s (WORKER_DRAIN) to finish the ones it is running. Meanwhile the services that feed it or call it stop together: the admin RPC, reassignment, expiration, the liveness checker, the DIPs info watcher, the chain listener, cancel retry, the escrow reconciler and the indexing request consumer. A job queued meanwhile stays in the database queue for the next start.
  2. Then the services the jobs read from stop together: indexer URLs, the RCA domain refresh, the entity count cache and the suggested minimums.
  3. The health endpoint, the event flush and the pool close follow as before.

The ceiling is 45s plus 4 steps of 5s, 65s. A compile-time assert keeps it under TEARDOWN_GRACE, so adding a service can't silently push shutdown past its budget again.

What this is not: a safety mechanism. The drain saves work, but a job can be cut off at any point anyway: by a crash, a kill, or simply by running longer than any sensible grace period (jobs may run up to 420s). Re-running a job that was cut off must therefore be safe however shutdown behaves. That belongs in the jobs themselves, through recorded intents and transactional state in Postgres, not in shutdown ordering.

Tests:

  • services_stopped_together_take_as_long_as_the_slowest
  • wedged_services_stopped_together_cost_1_cap
  • the_worker_gets_its_own_limit
  • The budget itself is checked at compile time.
  • cargo +nightly fmt, cargo clippy --all-targets -D warnings and the dipper-service unit tests pass locally.

The tests cover the stop helpers, not which wave each service is in.

🤖 Generated with Claude Code

Shutdown stopped the services 1 at a time, each given up to 5s, and the worker 13th: until its
turn it kept taking new jobs, and then it had 5s to finish the one it was running, which a job
seldom does, so the job was dropped part-way and the DB pool close skipped. With the services
added since, the sequence could take 85s, past the 75s after which the supervisor stops waiting
for it and aborts what is left.

Shutdown now goes in 2 waves. First the worker stops taking jobs and gets 45s to finish the ones
it is running, while the services that feed it work or call it (the admin RPC, reassignment,
expiration, the liveness checker, the DIPs info watcher, the chain listener, cancel retry, the
escrow reconciler and the indexing request consumer) stop alongside. The queue is in the
database, so a job queued meanwhile waits there for the next start. Then the services the
running jobs read from (indexer URLs, the RCA domain refresh, the entity count cache and the
suggested minimums) stop together. The health endpoint, the event flush and the pool close
follow as before. The ceiling is the 45s drain and 4 steps of 5s, 65s, and a compile-time check
keeps it under the supervisor's 75s.

A job still running after 45s is dropped as before, rolling its queue transaction back so it runs
again on the next start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re-runs safe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cjorge-thegraph cjorge-thegraph changed the title fix(shutdown): stop taking jobs first and stop the rest alongside fix(shutdown): stop taking jobs first and fit the teardown in its budget Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant