Repository navigation
fix(shutdown): stop taking jobs first and fit the teardown in its budget - #854
Draft
cjorge-thegraph wants to merge 2 commits into
Draft
cjorge-thegraph wants to merge 2 commits into
cjorge-thegraph wants to merge 2 commits into
Conversation
Shutdown stopped the services 1 at a time, each given up to 5s, and the worker 13th: until its turn it kept taking new jobs, and then it had 5s to finish the one it was running, which a job seldom does, so the job was dropped part-way and the DB pool close skipped. With the services added since, the sequence could take 85s, past the 75s after which the supervisor stops waiting for it and aborts what is left. Shutdown now goes in 2 waves. First the worker stops taking jobs and gets 45s to finish the ones it is running, while the services that feed it work or call it (the admin RPC, reassignment, expiration, the liveness checker, the DIPs info watcher, the chain listener, cancel retry, the escrow reconciler and the indexing request consumer) stop alongside. The queue is in the database, so a job queued meanwhile waits there for the next start. Then the services the running jobs read from (indexer URLs, the RCA domain refresh, the entity count cache and the suggested minimums) stop together. The health endpoint, the event flush and the pool close follow as before. The ceiling is the 45s drain and 4 steps of 5s, 65s, and a compile-time check keeps it under the supervisor's 75s. A job still running after 45s is dropped as before, rolling its queue transaction back so it runs again on the next start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re-runs safe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes dipper's shutdown finish inside its own time budget, and stops the worker taking new jobs at the start of a shutdown rather than near the end.
Why:
main's stop sequence no longer fits its budget. It stops 15 services one at a time, each capped at 5s, then flushes events and closes the pool. Its worst case is about 85s, past the 75sTEARDOWN_GRACE. A slow shutdown then ends with the supervisor aborting whatever is left, instead of finishing in order: the DB pool close is skipped, and lifecycle events still in the in-memory queue may never reach Kafka.What changes:
WORKER_DRAIN) to finish the ones it is running. Meanwhile the services that feed it or call it stop together: the admin RPC, reassignment, expiration, the liveness checker, the DIPs info watcher, the chain listener, cancel retry, the escrow reconciler and the indexing request consumer. A job queued meanwhile stays in the database queue for the next start.The ceiling is 45s plus 4 steps of 5s, 65s. A compile-time assert keeps it under
TEARDOWN_GRACE, so adding a service can't silently push shutdown past its budget again.What this is not: a safety mechanism. The drain saves work, but a job can be cut off at any point anyway: by a crash, a kill, or simply by running longer than any sensible grace period (jobs may run up to 420s). Re-running a job that was cut off must therefore be safe however shutdown behaves. That belongs in the jobs themselves, through recorded intents and transactional state in Postgres, not in shutdown ordering.
Tests:
services_stopped_together_take_as_long_as_the_slowestwedged_services_stopped_together_cost_1_capthe_worker_gets_its_own_limitcargo +nightly fmt,cargo clippy --all-targets -D warningsand the dipper-service unit tests pass locally.The tests cover the stop helpers, not which wave each service is in.
🤖 Generated with Claude Code