Repository navigation
ref(metrics): Rework how replication lag is computed - #1114
Conversation
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. It's a large redesign of how lag is measured: it replaces five existing gauges with two new ones and adds a pipeline-owned sampler task, so a human look from the pipelines owners would still be worthwhile.
What was reviewed:
- The new
ReplicationLagMonitorandReplicationLagMetricsinreplication/lag.rs: weak worker registrations, taking the progress snapshot before the source query, the max-per-path aggregation, the -1 sentinel, and the reset-on-drop inMetricsLifetime. - How the sampler is spawned and torn down through
AbortOnDropHandleinpipeline.rs, plus the removal of the per-apply-loop sampler task. - The source WAL head query in
etl-postgres/src/lag.rs(flushed WAL on a primary, replayed WAL on a standby) and the matching docs changes.
Extended reasoning...
The change spans 16 files (about 1000 lines added, 485 removed). It replaces the per-apply-loop lag sampler with a single pipeline-level sampler and new etl-postgres queries for the source WAL head. It removes five Prometheus gauges and adds two new ones plus a pipeline_id label, which breaks existing dashboards and alerts. It touches no security-sensitive surface such as auth, injection or data exposure. The diff is large, it changes the meaning of the metrics, and .github/CODEOWNERS assigns every path to the pipelines team, so this was deferred rather than approved.
This PR improves how replication lag is computed by using a new single task that can consistently emit lag metrics of multiple workers.