Skip to content

feat(metrics): serve Prometheus metrics and optionally log as JSON - #851

Merged
cjorge-thegraph merged 3 commits into
mainfrom
cjorge/metrics-and-json-logs
Nov 19, 2026
Merged

cjorge-thegraph merged 3 commits into
mainfrom
cjorge/metrics-and-json-logs

Conversation

@cjorge-thegraph

@cjorge-thegraph cjorge-thegraph commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Dipper had no metrics. Its only signals were plain-text logs, the Slack alerts it raises from some of them, and the liveness probe. Nothing could graph the job queue, count RPC failures or alert on a dependency being down, and log stores had to search text.

This adds Prometheus metrics at GET /metrics on the health port, and an optional JSON log format.

Metrics

All are named dipper_*. Every label takes values from a bounded set: a job kind, an operation, an outcome, a configured provider's host:port, an event tag set in dipper's code. None is an address, a deployment or an ID, so the number of series stays bounded.

Metric Type Labels
dipper_jobs_total counter kind, outcome (success, retry, gave_up, deferred, fatal, settle_failed)
dipper_job_duration_seconds histogram kind
dipper_jobs_in_flight gauge
dipper_job_listener_fallbacks_total counter
dipper_queue_jobs gauge, sampled state (ready, scheduled, failed)
dipper_queue_oldest_ready_job_age_seconds gauge, sampled
dipper_db_up gauge, sampled
dipper_db_pool_connections gauge, sampled state (idle, in_use)
dipper_rpc_requests_total counter operation, provider, outcome (ok, behind, retryable_error, refused, error)
dipper_rpc_calls_failed_total counter operation
dipper_transactions_total counter call (offer, cancel, reconcile_provider, reconcile_agreement), outcome (mined, reverted, dropped, unconfirmed)
dipper_transaction_submit_failures_total counter call, reason (contract_revert, submit_failed, rpc_error, error)
dipper_nonce_resyncs_total counter
dipper_iisa_requests_total counter endpoint (select-indexers, dips-indexers), outcome (ok, rejected, unavailable, error)
dipper_kafka_sends_total counter (attempts) outcome (ok, unreachable, error)
dipper_kafka_events_dropped_total counter reason (queue_full, queue_closed)
dipper_log_events_total counter event, level
  • The recorder: metrics with metrics-exporter-prometheus, default features off. The existing health server renders /metrics; there's no second listener. dipper-iisa and dipper-producer record through the metrics facade, which does nothing when no recorder is installed.
  • Tagged log lines: dipper_log_events_total is a tracing layer counting every log line that carries an event tag, at any level. That makes each event dipper alerts on in Slack a counter as well, without touching the code that logs it.
  • The database sampler runs every 15 s on a connection of its own, only when the health server is on (as is the recorder), runs the exporter's upkeep, and stops in the health endpoint's shutdown step, adding none. The queue's depth comes from a new dipper_pgmq::queue_depth, a single query on the (status, scheduled_for) index. A running job can't be counted in SQL, because its state is only visible inside the transaction holding it, so dipper_jobs_in_flight is tracked in the process.
  • Probes: dependency health lives in metrics, not in the probes. Liveness and readiness are unchanged.
  • Scraping: the k8s README now notes that a scraper reaches /metrics on the pod directly, and that network-policy.yaml refuses it until the deployment admits its scraper to the health port.

JSON logs

"logging": {"format": "json"} writes 1 JSON object per line, with each line's fields (its event tag among them) at the top level next to timestamp, level, target and message. Text stays the default, and RUST_LOG filters both.

Not included

  • A gauge of agreement events waiting to be sent to Kafka. It would duplicate the registry's status logic, and it would grow for ever where event streaming is off.
  • A signer balance gauge. Dipper never reads the balance today, so it would mean a new RPC call.
  • A build_info gauge. CARGO_PKG_VERSION is 0.1.0 while releases are 0.3.x.

Each could follow on its own.

Tests

  • dipper_log_events_total counts tagged lines by event and level, info included, and ignores untagged ones.
  • The worker counts each outcome under its kind (settling_counts_each_job_by_kind_and_outcome).
  • The RPC pool counts each attempt per provider against a 503-ing mock and a healthy one.
  • A reverted reconcile counts as reverted.
  • IISA requests are counted by endpoint and final outcome.
  • /metrics serves the text exposition format, and returns 404 without a recorder.
  • The sampler reports an unreachable database as down, and reports the database and the queue against a real one (it_the_sampler_reports_the_database_and_the_queue, pgtemp).
  • PgQueue::depth counts ready, scheduled and failed jobs (2 new pgmq integration tests).
  • JSON lines carry their fields at the top level, and text stays plain.
  • logging.format defaults to text and refuses unknown formats.
  • Tests use a local recorder only, never the global one.
  • just fmt-check, just check, just test-unit, and the dipper-service and dipper-pgmq integration tests pass locally.

Changes after review

  • The log-event counter had no filter. It asked to see every call site, so every debug and trace line in dipper and its libraries passed the level check and was built (each sqlx query among them). It now has a filter, TaggedEvents, that wants only call sites with an event field, decided once per call site. A test checks that an untagged debug line's call site is never wanted, which fails without the filter.
  • dipper_jobs_total counts settle_failed when the queue could not record the outcome (the job runs again), instead of the outcome that didn't happen. dipper_jobs_in_flight is held by a guard and starts at 0.
  • RPC: a provider answering with a JSON-RPC error (such as a revert) counts as refused, not error. A call the contract refused no longer counts in dipper_rpc_calls_failed_total.
  • Transactions:
    • dipper_transactions_total counts only transactions that reached the chain: mined, reverted, dropped, or unconfirmed when no receipt check got an answer.
    • Failures before sending move to the new dipper_transaction_submit_failures_total{call, reason}.
  • The sampler uses its own connection, so a busy pool no longer reads as the database being down. It runs the exporter's upkeep (histograms otherwise grow until scraped), drops a sample in progress when stopped, and runs only with the health server.
  • The teardown claim was wrong. The branch's stop sequence had 16 steps, not "12 steps / 70 s". The sampler no longer adds a step, and the comments are back to main's.
    • Separately, main's own sequence (up to 15 steps of 5 s) already reaches the 75 s teardown budget. That belongs to a follow-up on shutdown ordering.
  • Help texts now say:
    • the oldest ready job's age counts a job being run;
    • Kafka sends count attempts;
    • labels are bounded rather than fixed.

🤖 Generated with Claude Code

cjorge-thegraph and others added 3 commits November 18, 2026 14:15
Dipper had no metrics: its only signals were plain-text logs, the Slack alerts it raises from a
few of them, and a liveness probe. Nothing could graph the job queue, count RPC failures or alert
on a dependency being down, and log stores had to search the text.

Metrics, served at `GET /metrics` on the health port, all named `dipper_*` with labels limited to
fixed sets of values:

- jobs by kind and outcome, how long each ran, and how many run now; the queue's ready,
  scheduled and failed jobs and its oldest ready job's age; worker loops falling back to polling
- the database's reachability (`dipper_db_up`) and its pool's connections, sampled every 15s
- RPC requests by operation, provider (host:port) and outcome, and calls every provider failed
- transactions by call (offer, cancel, reconcile_provider, reconcile_agreement) and outcome, and
  nonce resyncs
- IISA requests by endpoint and final outcome; Kafka event sends by outcome and drops by reason
- every log line tagged with an `event`, by event and level, which covers each event dipper
  alerts on without touching the code that logs it

The queue's depth comes from a new `PgQueue::depth`, next to the queue's schema. Dependency
health is reported here rather than in the probes: liveness and readiness stay as they are.

`"logging": {"format": "json"}` writes 1 JSON object per line instead of text, with each line's
fields, its `event` tag among them, at the top level; text stays the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… log lines

Review found several metrics counting the wrong thing, and 1 cost:

- The log-event counter had no filter, so it asked to see every call site, and every debug and
  trace line in dipper and its libraries (each sqlx query among them) passed the level check
  and was built. It now has a filter, `TaggedEvents`, that wants only call sites with an
  `event` field, decided once per call site.
- `dipper_jobs_total` counted success, fatal or deferred when the queue failed to record it, and
  retry when the rescheduling rolled back, though each of those jobs runs again. Those now count
  as `settle_failed`. `dipper_jobs_in_flight` is held by a guard, so a run that ends any other
  way cannot leave it counted, and starts at 0.
- A provider answering with a JSON-RPC error, such as a reverting gas estimate, counted as an
  RPC error, and a call the contract refused counted as a call every provider failed. The first
  is now `refused`, and the second is not counted as failed.
- `dipper_transactions_total` counted transactions that never reached the chain. Those now go
  to `dipper_transaction_submit_failures_total` by call and reason, and a transaction no receipt
  check got an answer for is `unconfirmed` rather than `dropped`.
- The sampler asked the database through the shared pool, so a busy pool read as the database
  being down, and it took a connection the pool check did not reserve. It now uses a connection
  of its own, runs the exporter's upkeep after each sample (histograms otherwise grow until
  scraped), drops a sample in progress when stopped, and runs only with the health server,
  which is also the only time a recorder is installed. It stops in the health endpoint's step,
  so shutdown has no extra step; the stop-step count in the comments is back to what it was.
- Help texts say that the oldest ready job's age counts a job being run, that Kafka sends count
  attempts, and that labels take values from bounded sets rather than fixed ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on-logs

# Conflicts:
#	Cargo.lock
#	bin/dipper-service/Cargo.toml
#	bin/dipper-service/src/config.rs
#	dipper-producer/src/events/subgraph_indexing_agreements_events_emitter.rs
#	dipper-producer/src/lib.rs
@cjorge-thegraph
cjorge-thegraph marked this pull request as ready for review November 19, 2026 20:40
@cjorge-thegraph
cjorge-thegraph merged commit 523819e into main Nov 19, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant