Repository navigation
Description
Report
In the indexer channel on Discord, 3 to 7 October. Jerry flagged Revert's Aerodrome Base subgraph, and later Request Network's, lagging on Ellipfra. Marc-André (Ellipfra) restarted them and then found the cause in his logs: a network link flapped twice in 6 s and one node dropped its Postgres connections. That node had 210 deployments, and the 58 that were mid-block at the time stopped. They stayed healthy and did nothing until graph-node was restarted the next day:
14:26:57.697 ERRO Subgraph failed with non-deterministic error: failed to process trigger:
block #52080927 … handleMint › loadTransaction: store error: connection closed,
retry_delay_s: 142, attempt: 0
14:26:57.706 ERRO Subgraph instance failed to run: store error: error communicating with the server
14:26:57.707 DEBG Subgraph stopped, WASM runtime thread terminated
mindstyle (pops.one) sees the same on v0.45.0: 2 or 3 of about 15 deployments on a node stop at random, not paused, and graphman restart brings them back. Matthew Darwin asked whether any Prometheus metric fires. Maks noted that nothing in the runner changed between 0.44.0 and 0.45.0, which is correct. The relevant change is older.
What happens
Two separate defects, both in core/src/subgraph/runner/mod.rs. Either one stops a deployment until someone restarts it by hand. Both are reproduced below.
A. The error handler's own store call ends the runner (Marc-André's case)
A handler fails with connection closed. That is non-deterministic, so the runner is supposed to record the error, back off and retry. Before it does that, it checks the deployment's health:
if !self.inputs.store.health().await?.is_failed() {
self.fail_subgraph(message, Some(block_ptr), false).await?;
}health() is retried internally, but only on DatabaseUnavailable, which is the error you get when a new connection is refused. The pool does not validate a connection that was used in the last 30 s (GRAPH_STORE_CONNECTION_VALIDATION_IDLE_SECS, pool/manager.rs:120). On a busy node every connection counts as recently used, so after a link drop health() is handed a dead socket. The query fails with error communicating with the server or connection closed. That error is not retried, and the ? returns it out of run(). instance_manager.rs:600 logs Subgraph instance failed to run, unregisters the deployment's metrics and lets the thread end. Nothing ever starts it again.
fail_subgraph never ran, so the database still says healthy, and indexingStatuses keeps reporting healthy with a frozen latestBlock. This has been the code since at least v0.41.1.
B. A poisoned writer is never replaced (regression in v0.42.0)
If the failure happens inside the async writer (Failed to transact block operations: store error: connection closed), the writer is poisoned. In 2022 this was graph-node #4190, and it was fixed by #4533 (April 2023): every runner restart called store.restart(), which swaps a poisoned writer for a new one.
The runner state-machine refactor (73fabf34c, January 2026, first released in v0.42.0) now calls store.restart() only for RestartReason::StoreError. The retry after a non-deterministic error goes through process_block_state, which tags every Action::Restart as DynamicDataSourceCreated. So the retry reuses the poisoned writer, fails with subgraph writer poisoned by previous error, backs off and tries again. Backoff starts at 2 min and doubles to 1 h, and the deployment never recovers. This one at least shows failed and deployment_status 4. It is probably what mindstyle remembers as "wasn't happening on older versions".
Reproduction
v0.45.0 (5973cd995) built from source on a Linux box. One Postgres reached through toxiproxy, anvil at one block a second, and 11 or 12 block-handler deployments catching up. Each handler does one load of an entity that does not exist, so every block hits the database. Each test cuts the link twice for 3 s, three seconds apart, and checks indexingStatuses 180 s later. GRAPH_SUBGRAPH_ERROR_RETRY_CEIL_SECS=20 keeps the retries short. That cap does not affect either path.
| build | outage | pool | stopped after 180 s |
|---|---|---|---|
| v0.45.0 | proxy disabled | 10 | 1 of 7 (B) |
| v0.45.0 | reset_peer |
10 | 1 of 11 (B) |
| v0.45.0 | timeout |
10 | 1 of 11 (B) |
| v0.45.0 | reset_peer |
40 | 10 of 11 (A), all still healthy |
| v0.45.0 | timeout |
40 | 1 of 11 (A) |
v0.45.0 + VALIDATION_IDLE_SECS=0 |
reset_peer |
40 | 0 of 11, twice |
| patched | reset_peer |
40 | 0 of 11, twice |
| patched | timeout |
40 | 0 of 11 |
| patched | proxy disabled | 40 | 0 of 11 |
Mechanism A needs idle connections in the pool. With a pool of 10 shared by 12 busy deployments there were none, so health() always got a fresh connection attempt and was retried. Marc-André's node had plenty of idle connections.
In the patched runs, 4 deployments logged Failed to record non-deterministic error … connection closed and carried on. One deployment (QmVLCa…) hit B's trigger, Failed to transact block operations: store error: connection closed. It got a new writer on the retry and went on to block 159544.
Fix
Patch against master (6838f4e). It applies cleanly to v0.45.0. One file, +64 −15, in core/src/subgraph/runner/mod.rs:
restart()callsstore.restart()whatever the reason, as it did before v0.42.0. That call does nothing unless the writer is poisoned. Fixes B.- The non-deterministic arm logs a failure to read health or record the error, then backs off and retries anyway. The retry records the error. Fixes A.
- If the deterministic arm cannot record its error, it backs off and retries instead of returning. Otherwise the deployment would sit
healthyand stopped. - A failed
unfail_non_deterministic_errorafter a good block is tried again on the next block instead of ending the runner. - If
initialize()orrestart()fails (for example the store or chain is still down when the retry fires), the runner backs off and tries again. A runner that returns is never restarted. Cancellation is checked first, so an unassigned deployment still stops.
Not done: no runner test injects store faults, so the regression test is the toxiproxy rig above and not CI. cargo clippy adds no new warnings, and the graph-core unit tests pass apart from one that needs a local IPFS daemon.
If you're hitting this now
- To recover:
graphman restart <deployment>, as Marc-André and mindstyle found. - To see it (Matthew's question): the
healthfield will not show A. Look for an assigned deployment whosedeployment_headstops moving, or whosedeployment_statusseries has disappeared, because A unregisters the deployment's metrics. In the logs, look forSubgraph instance failed to run. B showsdeployment_status4 and repeatssubgraph writer poisoned by previous error. - To prevent A until a release ships:
GRAPH_STORE_CONNECTION_VALIDATION_IDLE_SECS=0makes the pool runSELECT 67on every checkout, so dead sockets are thrown away beforehealth()gets one. It only helps when the outage is over within about 3 s, the five retrieshealth()gets ondatabase unavailable; a 10 s outage still stopped 2 of 10 with it (see the correction below). It costs one round trip per checkout. Nothing in config prevents B. - To fix both now: the unofficial build v0.45.0-patch.1 carries core: Keep the subgraph runner alive through transient store errors graphprotocol/graph-node#6730.
Next
- Open the patch upstream on graphprotocol/graph-node and link it here.
- When this closes: add a symptom-index entry ("subgraph stopped indexing but reports healthy"), plus a library entry for the stopgap and the detection metric.
Activity
cargopete commentedon Oct 7, 2026
Opened upstream as graphprotocol/graph-node#6730.
cargopete commentedon Oct 10, 2026
Correction: the stopgap is narrower than stated
mindstyle (pops.one) ran v0.45.0 with
GRAPH_STORE_CONNECTION_VALIDATION_IDLE_SECS=0and still had deployments stop. Reproduced on the rig with the same build and env var and a 10 s outage (toxiproxytimeout0, so packets are dropped and nothing is closed): 2 of 10 stopped.QmPewF…is A. With validation on,health()getsdatabase unavailableand is retried, but only five times, about 3 s in all. Past that the?ends the runner:Subgraph instance failed to run: database query timed out, and it stayshealthy.QmQpY7…is B:subgraph writer poisoned by previous erroron every retry,failed.So the env var only prevents A for outages of a few seconds, and nothing in config prevents B. The section above is corrected.
Patched build
Until a release includes graphprotocol/graph-node#6730 there is an unofficial build: v0.45.0 plus graphprotocol/graph-node#6730, graphprotocol/graph-node#6710 and graphprotocol/graph-node#6729, built in public CI with provenance attestations. https://github.com/cargopete/graph-node/releases/tag/v0.45.0-patch.1, image
ghcr.io/cargopete/graph-node:v0.45.0-patch.1. It reports its version as0.45.0+patch.1.Same rig with patch.1: 0 of 10 stopped under the 10 s outage and 0 of 10 under
reset_peer, and every handler store error was recovered. POIs match v0.45.0 at blocks 1000, 5000, 12345 and 20000. That test subgraph only has block handlers, so graphprotocol/graph-node#6729'seth_getLogspath and graphprotocol/graph-node#6710's firehose path are not covered by it.