Repository navigation
feat(metrics): serve Prometheus metrics and optionally log as JSON - #851
Merged
Merged
Conversation
Dipper had no metrics: its only signals were plain-text logs, the Slack alerts it raises from a
few of them, and a liveness probe. Nothing could graph the job queue, count RPC failures or alert
on a dependency being down, and log stores had to search the text.
Metrics, served at `GET /metrics` on the health port, all named `dipper_*` with labels limited to
fixed sets of values:
- jobs by kind and outcome, how long each ran, and how many run now; the queue's ready,
scheduled and failed jobs and its oldest ready job's age; worker loops falling back to polling
- the database's reachability (`dipper_db_up`) and its pool's connections, sampled every 15s
- RPC requests by operation, provider (host:port) and outcome, and calls every provider failed
- transactions by call (offer, cancel, reconcile_provider, reconcile_agreement) and outcome, and
nonce resyncs
- IISA requests by endpoint and final outcome; Kafka event sends by outcome and drops by reason
- every log line tagged with an `event`, by event and level, which covers each event dipper
alerts on without touching the code that logs it
The queue's depth comes from a new `PgQueue::depth`, next to the queue's schema. Dependency
health is reported here rather than in the probes: liveness and readiness stay as they are.
`"logging": {"format": "json"}` writes 1 JSON object per line instead of text, with each line's
fields, its `event` tag among them, at the top level; text stays the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… log lines Review found several metrics counting the wrong thing, and 1 cost: - The log-event counter had no filter, so it asked to see every call site, and every debug and trace line in dipper and its libraries (each sqlx query among them) passed the level check and was built. It now has a filter, `TaggedEvents`, that wants only call sites with an `event` field, decided once per call site. - `dipper_jobs_total` counted success, fatal or deferred when the queue failed to record it, and retry when the rescheduling rolled back, though each of those jobs runs again. Those now count as `settle_failed`. `dipper_jobs_in_flight` is held by a guard, so a run that ends any other way cannot leave it counted, and starts at 0. - A provider answering with a JSON-RPC error, such as a reverting gas estimate, counted as an RPC error, and a call the contract refused counted as a call every provider failed. The first is now `refused`, and the second is not counted as failed. - `dipper_transactions_total` counted transactions that never reached the chain. Those now go to `dipper_transaction_submit_failures_total` by call and reason, and a transaction no receipt check got an answer for is `unconfirmed` rather than `dropped`. - The sampler asked the database through the shared pool, so a busy pool read as the database being down, and it took a connection the pool check did not reserve. It now uses a connection of its own, runs the exporter's upkeep after each sample (histograms otherwise grow until scraped), drops a sample in progress when stopped, and runs only with the health server, which is also the only time a recorder is installed. It stops in the health endpoint's step, so shutdown has no extra step; the stop-step count in the comments is back to what it was. - Help texts say that the oldest ready job's age counts a job being run, that Kafka sends count attempts, and that labels take values from bounded sets rather than fixed ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on-logs # Conflicts: # Cargo.lock # bin/dipper-service/Cargo.toml # bin/dipper-service/src/config.rs # dipper-producer/src/events/subgraph_indexing_agreements_events_emitter.rs # dipper-producer/src/lib.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dipper had no metrics. Its only signals were plain-text logs, the Slack alerts it raises from some of them, and the liveness probe. Nothing could graph the job queue, count RPC failures or alert on a dependency being down, and log stores had to search text.
This adds Prometheus metrics at
GET /metricson the health port, and an optional JSON log format.Metrics
All are named
dipper_*. Every label takes values from a bounded set: a job kind, an operation, an outcome, a configured provider'shost:port, aneventtag set in dipper's code. None is an address, a deployment or an ID, so the number of series stays bounded.dipper_jobs_totalkind,outcome(success, retry, gave_up, deferred, fatal, settle_failed)dipper_job_duration_secondskinddipper_jobs_in_flightdipper_job_listener_fallbacks_totaldipper_queue_jobsstate(ready, scheduled, failed)dipper_queue_oldest_ready_job_age_secondsdipper_db_updipper_db_pool_connectionsstate(idle, in_use)dipper_rpc_requests_totaloperation,provider,outcome(ok, behind, retryable_error, refused, error)dipper_rpc_calls_failed_totaloperationdipper_transactions_totalcall(offer, cancel, reconcile_provider, reconcile_agreement),outcome(mined, reverted, dropped, unconfirmed)dipper_transaction_submit_failures_totalcall,reason(contract_revert, submit_failed, rpc_error, error)dipper_nonce_resyncs_totaldipper_iisa_requests_totalendpoint(select-indexers, dips-indexers),outcome(ok, rejected, unavailable, error)dipper_kafka_sends_totaloutcome(ok, unreachable, error)dipper_kafka_events_dropped_totalreason(queue_full, queue_closed)dipper_log_events_totalevent,levelmetricswithmetrics-exporter-prometheus, default features off. The existing health server renders/metrics; there's no second listener. dipper-iisa and dipper-producer record through themetricsfacade, which does nothing when no recorder is installed.dipper_log_events_totalis a tracing layer counting every log line that carries aneventtag, at any level. That makes each event dipper alerts on in Slack a counter as well, without touching the code that logs it.dipper_pgmq::queue_depth, a single query on the(status, scheduled_for)index. A running job can't be counted in SQL, because its state is only visible inside the transaction holding it, sodipper_jobs_in_flightis tracked in the process./metricson the pod directly, and thatnetwork-policy.yamlrefuses it until the deployment admits its scraper to thehealthport.JSON logs
"logging": {"format": "json"}writes 1 JSON object per line, with each line's fields (itseventtag among them) at the top level next totimestamp,level,targetandmessage. Text stays the default, andRUST_LOGfilters both.Not included
build_infogauge.CARGO_PKG_VERSIONis 0.1.0 while releases are 0.3.x.Each could follow on its own.
Tests
dipper_log_events_totalcounts tagged lines by event and level, info included, and ignores untagged ones.settling_counts_each_job_by_kind_and_outcome).reverted./metricsserves the text exposition format, and returns 404 without a recorder.it_the_sampler_reports_the_database_and_the_queue, pgtemp).PgQueue::depthcounts ready, scheduled and failed jobs (2 new pgmq integration tests).logging.formatdefaults to text and refuses unknown formats.just fmt-check,just check,just test-unit, and the dipper-service and dipper-pgmq integration tests pass locally.Changes after review
TaggedEvents, that wants only call sites with aneventfield, decided once per call site. A test checks that an untagged debug line's call site is never wanted, which fails without the filter.dipper_jobs_totalcountssettle_failedwhen the queue could not record the outcome (the job runs again), instead of the outcome that didn't happen.dipper_jobs_in_flightis held by a guard and starts at 0.refused, noterror. A call the contract refused no longer counts indipper_rpc_calls_failed_total.dipper_transactions_totalcounts only transactions that reached the chain: mined, reverted, dropped, orunconfirmedwhen no receipt check got an answer.dipper_transaction_submit_failures_total{call, reason}.main's.main's own sequence (up to 15 steps of 5 s) already reaches the 75 s teardown budget. That belongs to a follow-up on shutdown ordering.🤖 Generated with Claude Code