Repository navigation
feat(service): run 1 dipper at a time on a database - #856
cjorge-thegraph wants to merge 1 commit into
Conversation
Dipper keeps its signer's nonce and the locks that keep 2 jobs from acting on the same agreement in memory, so 2 processes on 1 database could send transactions on the same nonce, or offer or cancel the same agreement twice. `Recreate` keeps a rollout from overlapping them, but when a pod is evicted, deleted or its node drained, Kubernetes starts the replacement while the old pod is still shutting down. Dipper now takes a database lock before it migrates or starts anything, waiting for as long as another dipper on the same database holds it, and holds it for as long as it runs: - It is a Postgres session advisory lock on a connection of its own (the `AdvisoryLockConnection` the indexing request consumer uses), named in `pg_stat_activity` after the pod, so it is freed when dipper closes it, when the process dies, or, should the holder vanish without closing its socket, after the connection has been idle for 30s. - The holder checks the connection every 5s. If it does not answer within 5s, dipper logs `singleton_lock_lost` and exits at once with code 3, well before the database could give the lock to another; a waiting dipper logs `singleton_lock_waiting` every 30s. - Shutdown gives the lock up last, after the DB pool, and only when every service stopped in time; otherwise it stays held until the process exits. The teardown ceiling grows by 1 step to 70s, still under the supervisor's 75s. - `dipper_singleton_lock_held` reports it. The `LockConnection` trait and the stable key hash move from the consumer to a shared `lock` module; the consumer's keys are unchanged. The k8s README and the manifest explain the lock, why `Recreate` must stay (a rolling update would deadlock on it), and that Postgres must be reached directly or through a proxy that keeps sessions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Closing: a session lock can't guarantee one dipper at a time. When its connection drops, Postgres frees the lock at once while this process may still act, and without a fencing token a holder can't know it lost the lock before its next action. Each fix around it narrowed that window without closing it. The underlying problem is that correctness depends on state outside the database: the in-memory nonce, in-memory mutexes, and sends that aren't recorded. The better direction is to make Postgres the only coordinator: chain transactions recorded as intents with nonces assigned in the database, decisions about a request serialised in its transaction, and Kafka offsets and events written in the same transactions as their effects. That holds with any number of processes and through crashes, so no singleton is needed. |
Stacked on #854 (it adds 1 step to that shutdown sequence); review that first.
Dipper keeps its signer's nonce, and the locks that keep 2 jobs from acting on the same agreement, in memory. So 2 processes on one database could send transactions on the same nonce, or offer or cancel the same agreement twice.
strategy: Recreatekeeps a rollout from overlapping them. It does not cover a pod that is evicted or deleted, or a node that is drained: Kubernetes starts the replacement while the old pod is still shutting down.Dipper now takes a database lock before it migrates or starts anything, and holds it for as long as it runs:
AdvisoryLockConnectionfrom feat: apply Studio's indexing requests from the Redpanda topic #753 (the indexing request consumer's lock) and has its own key,dipper/singleton. The connection is nameddipper-singleton <pod>inpg_stat_activity. The lock is freed when dipper closes the connection or the process dies. If the holder vanishes without closing its socket, the lock is freed after the connection has been idle for 30s (idle_session_timeout, PG14+; a warning is logged where the server refuses it).singleton_lock_waitingevery 30s. It does nothing else until it has the lock.singleton_lock_lostand exits at once with code 3. That is 10s at most, well before the 30s after which the database would give the lock to another. A compile-time assert holdsKEEPALIVE + PING_TIMEOUT < IDLE_TIMEOUT.dipper_singleton_lock_held.The
LockConnectiontrait and the stable FNV-1a key hash move from the consumer into a sharedlockmodule. The consumer's keys are unchanged, and its pinned-value tests still pass.Deployment notes (now in
k8s/README.mdand the manifest):Recreate. With a rolling update, the new pod would wait for a lock that the old pod gives up only once the new pod is ready, until the startup probe gave up on it.Tests:
on_loston the 3rd tick.KEEPALIVE + PING_TIMEOUT.releasecloses the connection and stops the checks.it_the_lock_goes_to_the_2nd_dipper_once_the_1st_lets_gouses 2 holders on a real pgtemp Postgres: the 2nd blocks until the 1st releases.it_a_connection_ended_by_the_database_loses_the_lockterminates the lock's backend withpg_terminate_backend, and the holder notices within a check.the_key_is_fixed_and_apart_from_the_consumers.cargo +nightly fmt,cargo clippy --all-targets -D warnings, the unit tests and dipper-service'sit_tests pass locally.🤖 Generated with Claude Code