Skip to content

feat(service): run 1 dipper at a time on a database - #856

Closed
cjorge-thegraph wants to merge 1 commit into
cjorge/shut-down-in-orderfrom
cjorge/hold-the-singleton-lock
Closed

cjorge-thegraph wants to merge 1 commit into
cjorge/shut-down-in-orderfrom
cjorge/hold-the-singleton-lock

Conversation

@cjorge-thegraph

Copy link
Copy Markdown
Contributor

Stacked on #854 (it adds 1 step to that shutdown sequence); review that first.

Dipper keeps its signer's nonce, and the locks that keep 2 jobs from acting on the same agreement, in memory. So 2 processes on one database could send transactions on the same nonce, or offer or cancel the same agreement twice. strategy: Recreate keeps a rollout from overlapping them. It does not cover a pod that is evicted or deleted, or a node that is drained: Kubernetes starts the replacement while the old pod is still shutting down.

Dipper now takes a database lock before it migrates or starts anything, and holds it for as long as it runs:

  • A session advisory lock on a connection of its own. It reuses the AdvisoryLockConnection from feat: apply Studio's indexing requests from the Redpanda topic #753 (the indexing request consumer's lock) and has its own key, dipper/singleton. The connection is named dipper-singleton <pod> in pg_stat_activity. The lock is freed when dipper closes the connection or the process dies. If the holder vanishes without closing its socket, the lock is freed after the connection has been idle for 30s (idle_session_timeout, PG14+; a warning is logged where the server refuses it).
  • A 2nd dipper waits for the lock, retrying every 2s and logging singleton_lock_waiting every 30s. It does nothing else until it has the lock.
  • Keepalive. The holder checks its connection every 5s. If the connection doesn't answer within 5s, dipper logs singleton_lock_lost and exits at once with code 3. That is 10s at most, well before the 30s after which the database would give the lock to another. A compile-time assert holds KEEPALIVE + PING_TIMEOUT < IDLE_TIMEOUT.
  • Shutdown releases it last, after the DB pool, and only if every service stopped in time. If one didn't, the lock stays held until the process exits, since the service may still act. This adds 1 step to the teardown ceiling: 70s, still under the 75s budget.
  • Metric: dipper_singleton_lock_held.

The LockConnection trait and the stable FNV-1a key hash move from the consumer into a shared lock module. The consumer's keys are unchanged, and its pinned-value tests still pass.

Deployment notes (now in k8s/README.md and the manifest):

  • Keep Recreate. With a rolling update, the new pod would wait for a lock that the old pod gives up only once the new pod is ready, until the startup probe gave up on it.
  • The lock needs a real session. Reach Postgres directly, or through a proxy that keeps sessions (the Cloud SQL Auth Proxy does). Never use a pooler in transaction mode.

Tests:

  • Paused-clock tests with a scripted connection:
    • A 2nd dipper waits through 3 refusals.
    • A failed check runs on_lost on the 3rd tick.
    • A hanging check counts as lost after KEEPALIVE + PING_TIMEOUT.
    • release closes the connection and stops the checks.
  • it_the_lock_goes_to_the_2nd_dipper_once_the_1st_lets_go uses 2 holders on a real pgtemp Postgres: the 2nd blocks until the 1st releases.
  • it_a_connection_ended_by_the_database_loses_the_lock terminates the lock's backend with pg_terminate_backend, and the holder notices within a check.
  • the_key_is_fixed_and_apart_from_the_consumers.
  • cargo +nightly fmt, cargo clippy --all-targets -D warnings, the unit tests and dipper-service's it_ tests pass locally.

🤖 Generated with Claude Code

Dipper keeps its signer's nonce and the locks that keep 2 jobs from acting on the same agreement
in memory, so 2 processes on 1 database could send transactions on the same nonce, or offer or
cancel the same agreement twice. `Recreate` keeps a rollout from overlapping them, but when a
pod is evicted, deleted or its node drained, Kubernetes starts the replacement while the old pod
is still shutting down.

Dipper now takes a database lock before it migrates or starts anything, waiting for as long as
another dipper on the same database holds it, and holds it for as long as it runs:

- It is a Postgres session advisory lock on a connection of its own (the
  `AdvisoryLockConnection` the indexing request consumer uses), named in `pg_stat_activity`
  after the pod, so it is freed when dipper closes it, when the process dies, or, should the
  holder vanish without closing its socket, after the connection has been idle for 30s.
- The holder checks the connection every 5s. If it does not answer within 5s, dipper logs
  `singleton_lock_lost` and exits at once with code 3, well before the database could give
  the lock to another; a waiting dipper logs `singleton_lock_waiting` every 30s.
- Shutdown gives the lock up last, after the DB pool, and only when every service stopped in
  time; otherwise it stays held until the process exits. The teardown ceiling grows by 1 step
  to 70s, still under the supervisor's 75s.
- `dipper_singleton_lock_held` reports it.

The `LockConnection` trait and the stable key hash move from the consumer to a shared `lock`
module; the consumer's keys are unchanged. The k8s README and the manifest explain the lock, why
`Recreate` must stay (a rolling update would deadlock on it), and that Postgres must be reached
directly or through a proxy that keeps sessions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cjorge-thegraph

Copy link
Copy Markdown
Contributor Author

Closing: a session lock can't guarantee one dipper at a time. When its connection drops, Postgres frees the lock at once while this process may still act, and without a fencing token a holder can't know it lost the lock before its next action. Each fix around it narrowed that window without closing it.

The underlying problem is that correctness depends on state outside the database: the in-memory nonce, in-memory mutexes, and sends that aren't recorded. The better direction is to make Postgres the only coordinator: chain transactions recorded as intents with nonces assigned in the database, decisions about a request serialised in its transaction, and Kafka offsets and events written in the same transactions as their effects. That holds with any number of processes and through crashes, so no singleton is needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant