Skip to content

fix(k8s): roll dipper out one pod at a time - #846

Merged
cjorge-thegraph merged 3 commits into
mainfrom
cjorge/roll-out-one-pod-at-a-time
Nov 19, 2026
Merged

cjorge-thegraph merged 3 commits into
mainfrom
cjorge/roll-out-one-pod-at-a-time

Conversation

@cjorge-thegraph

@cjorge-thegraph cjorge-thegraph commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

A rollout with the Deployment's default RollingUpdate strategy runs the old and the new dipper side by side until the new pod is ready. Each pod keeps its own nonce for the one signer (nonce: AtomicU64 under submit_lock in chain_client/client.rs) and its own in-process locks, the global reassess lock among them, while both pull jobs from the same queue. During that overlap the 2 can send transactions on the same nonce and offer or cancel the same agreement twice. testnet-v2 runs this way today (maxSurge: 25% rounds up to a second pod).

  • k8s/deployment.yaml now sets strategy: Recreate, and k8s/README.md says to run exactly 1 replica and why. The price is a gap while the new pod starts.
  • The example config asked for 10 database connections, which dipper's own startup check refuses (the default 8 worker loops need 3 × 8 + 2 = 26), so an unedited copy did not start. The check moves from main into Config::validate_db_pool, run with the other config checks before anything connects, so a test holds the example to it; the example now uses the default pool of 32.

A Postgres advisory lock (#841, on the Redpanda stack) keeps 2 pods from consuming Studio's requests at once. Recreate orders rollouts only: on an eviction, a node drain or a pod deletion, Kubernetes starts the replacement while the old pod is still terminating, so this does not rule out 2 signers on its own. That needs a lock dipper holds for as long as it runs, which is a follow-up.

Tests: example_config_has_db_connections_for_its_worker_loops failed on main ("needs at least 26 db connections … db.max_connections is 10") and passes now; 3 more pin the rule. just fmt-check, just check and just test-unit pass locally.

Changes after review

  • The manifest comment, the k8s README and this description no longer claim Recreate prevents 2 pods acting at once: it orders rollouts only. The README says to keep evictions and node drains to planned windows until dipper holds a lock of its own.
  • rollingUpdate: null next to type: Recreate, so a client-side apply onto an existing Deployment drops the defaulted rolling-update settings the API refuses with Recreate.
  • A pool too small for the worker loops is reported as an invalid config, not a db config.

🤖 Generated with Claude Code

cjorge-thegraph and others added 3 commits November 18, 2026 12:22
A rollout with the Deployment's default RollingUpdate strategy runs the old and the new pod side
by side. Each keeps its own nonce for the one signer and its own in-process locks, the global
reassess lock among them, while both pull jobs from the same queue, so the 2 could send
transactions on the same nonce and offer or cancel the same agreement twice. The manifest now
uses Recreate, and the k8s README says to run exactly 1 replica.

The example config also asked for 10 database connections, which dipper's own startup check
refuses: the default 8 worker loops need 26. The check moves from main into
Config::validate_db_pool, which now runs with the other config checks before anything
connects, so a test can hold the example to it. The example now uses the default pool of 32.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review found the change claimed more than it does. `Recreate` only orders a rollout: when a pod
is evicted, deleted or its node drained, Kubernetes starts the replacement while the old pod is
still shutting down, and the old pod's worker keeps taking jobs until its turn in the stop
sequence. The manifest and the k8s README now say so, and to keep such disruptions to planned
windows until dipper holds a lock of its own for as long as it runs.

`rollingUpdate: null` drops the defaulted rolling-update settings that a client-side apply onto
an existing Deployment would otherwise keep, which the API refuses next to `Recreate`. A pool
too small for the worker loops is now reported as an invalid config, not a db config, as
`worker_concurrency` is part of the problem.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…od-at-a-time

# Conflicts:
#	bin/dipper-service/src/config.rs
@cjorge-thegraph
cjorge-thegraph marked this pull request as ready for review November 19, 2026 20:08
@cjorge-thegraph
cjorge-thegraph merged commit de57e46 into main Nov 19, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant