Skip to content

[graph-node] "Subgraph instance failed to run: store error: error communicating with the server" after a Postgres blip: the runner ends and the deployment stays "healthy" but frozen; a poisoned writer is never replaced either (regression in v0.42.0) #51

Open

Description

@cargopete

Report

In the indexer channel on Discord, 3 to 7 October. Jerry flagged Revert's Aerodrome Base subgraph, and later Request Network's, lagging on Ellipfra. Marc-André (Ellipfra) restarted them and then found the cause in his logs: a network link flapped twice in 6 s and one node dropped its Postgres connections. That node had 210 deployments, and the 58 that were mid-block at the time stopped. They stayed healthy and did nothing until graph-node was restarted the next day:

14:26:57.697 ERRO Subgraph failed with non-deterministic error: failed to process trigger:
             block #52080927 … handleMint › loadTransaction: store error: connection closed,
             retry_delay_s: 142, attempt: 0
14:26:57.706 ERRO Subgraph instance failed to run: store error: error communicating with the server
14:26:57.707 DEBG Subgraph stopped, WASM runtime thread terminated

mindstyle (pops.one) sees the same on v0.45.0: 2 or 3 of about 15 deployments on a node stop at random, not paused, and graphman restart brings them back. Matthew Darwin asked whether any Prometheus metric fires. Maks noted that nothing in the runner changed between 0.44.0 and 0.45.0, which is correct. The relevant change is older.

What happens

Two separate defects, both in core/src/subgraph/runner/mod.rs. Either one stops a deployment until someone restarts it by hand. Both are reproduced below.

A. The error handler's own store call ends the runner (Marc-André's case)

A handler fails with connection closed. That is non-deterministic, so the runner is supposed to record the error, back off and retry. Before it does that, it checks the deployment's health:

if !self.inputs.store.health().await?.is_failed() {
    self.fail_subgraph(message, Some(block_ptr), false).await?;
}

health() is retried internally, but only on DatabaseUnavailable, which is the error you get when a new connection is refused. The pool does not validate a connection that was used in the last 30 s (GRAPH_STORE_CONNECTION_VALIDATION_IDLE_SECS, pool/manager.rs:120). On a busy node every connection counts as recently used, so after a link drop health() is handed a dead socket. The query fails with error communicating with the server or connection closed. That error is not retried, and the ? returns it out of run(). instance_manager.rs:600 logs Subgraph instance failed to run, unregisters the deployment's metrics and lets the thread end. Nothing ever starts it again.

fail_subgraph never ran, so the database still says healthy, and indexingStatuses keeps reporting healthy with a frozen latestBlock. This has been the code since at least v0.41.1.

B. A poisoned writer is never replaced (regression in v0.42.0)

If the failure happens inside the async writer (Failed to transact block operations: store error: connection closed), the writer is poisoned. In 2022 this was graph-node #4190, and it was fixed by #4533 (April 2023): every runner restart called store.restart(), which swaps a poisoned writer for a new one.

The runner state-machine refactor (73fabf34c, January 2026, first released in v0.42.0) now calls store.restart() only for RestartReason::StoreError. The retry after a non-deterministic error goes through process_block_state, which tags every Action::Restart as DynamicDataSourceCreated. So the retry reuses the poisoned writer, fails with subgraph writer poisoned by previous error, backs off and tries again. Backoff starts at 2 min and doubles to 1 h, and the deployment never recovers. This one at least shows failed and deployment_status 4. It is probably what mindstyle remembers as "wasn't happening on older versions".

Reproduction

v0.45.0 (5973cd995) built from source on a Linux box. One Postgres reached through toxiproxy, anvil at one block a second, and 11 or 12 block-handler deployments catching up. Each handler does one load of an entity that does not exist, so every block hits the database. Each test cuts the link twice for 3 s, three seconds apart, and checks indexingStatuses 180 s later. GRAPH_SUBGRAPH_ERROR_RETRY_CEIL_SECS=20 keeps the retries short. That cap does not affect either path.

build outage pool stopped after 180 s
v0.45.0 proxy disabled 10 1 of 7 (B)
v0.45.0 reset_peer 10 1 of 11 (B)
v0.45.0 timeout 10 1 of 11 (B)
v0.45.0 reset_peer 40 10 of 11 (A), all still healthy
v0.45.0 timeout 40 1 of 11 (A)
v0.45.0 + VALIDATION_IDLE_SECS=0 reset_peer 40 0 of 11, twice
patched reset_peer 40 0 of 11, twice
patched timeout 40 0 of 11
patched proxy disabled 40 0 of 11

Mechanism A needs idle connections in the pool. With a pool of 10 shared by 12 busy deployments there were none, so health() always got a fresh connection attempt and was retried. Marc-André's node had plenty of idle connections.

In the patched runs, 4 deployments logged Failed to record non-deterministic error … connection closed and carried on. One deployment (QmVLCa…) hit B's trigger, Failed to transact block operations: store error: connection closed. It got a new writer on the retry and went on to block 159544.

Fix

Patch against master (6838f4e). It applies cleanly to v0.45.0. One file, +64 −15, in core/src/subgraph/runner/mod.rs:

  1. restart() calls store.restart() whatever the reason, as it did before v0.42.0. That call does nothing unless the writer is poisoned. Fixes B.
  2. The non-deterministic arm logs a failure to read health or record the error, then backs off and retries anyway. The retry records the error. Fixes A.
  3. If the deterministic arm cannot record its error, it backs off and retries instead of returning. Otherwise the deployment would sit healthy and stopped.
  4. A failed unfail_non_deterministic_error after a good block is tried again on the next block instead of ending the runner.
  5. If initialize() or restart() fails (for example the store or chain is still down when the retry fires), the runner backs off and tries again. A runner that returns is never restarted. Cancellation is checked first, so an unassigned deployment still stops.

Not done: no runner test injects store faults, so the regression test is the toxiproxy rig above and not CI. cargo clippy adds no new warnings, and the graph-core unit tests pass apart from one that needs a local IPFS daemon.

If you're hitting this now

  • To recover: graphman restart <deployment>, as Marc-André and mindstyle found.
  • To see it (Matthew's question): the health field will not show A. Look for an assigned deployment whose deployment_head stops moving, or whose deployment_status series has disappeared, because A unregisters the deployment's metrics. In the logs, look for Subgraph instance failed to run. B shows deployment_status 4 and repeats subgraph writer poisoned by previous error.
  • To prevent A until a release ships: GRAPH_STORE_CONNECTION_VALIDATION_IDLE_SECS=0 makes the pool run SELECT 67 on every checkout, so dead sockets are thrown away before health() gets one. It only helps when the outage is over within about 3 s, the five retries health() gets on database unavailable; a 10 s outage still stopped 2 of 10 with it (see the correction below). It costs one round trip per checkout. Nothing in config prevents B.
  • To fix both now: the unofficial build v0.45.0-patch.1 carries core: Keep the subgraph runner alive through transient store errors graphprotocol/graph-node#6730.

Next

  • Open the patch upstream on graphprotocol/graph-node and link it here.
  • When this closes: add a symptom-index entry ("subgraph stopped indexing but reports healthy"), plus a library entry for the stopgap and the detection metric.

Activity

  1. added
    owner/upstreamgraphprotocol upstream repos
    root-cause-foundWe know why. The explanation is in the thread.
    kind/symptomA failure mode the Academy symptom index does not cover yet
    on Oct 7, 2026
  2. cargopete commented on Oct 7, 2026

    @cargopete
    ContributorAuthor

    Opened upstream as graphprotocol/graph-node#6730.

  3. cargopete commented on Oct 10, 2026

    @cargopete
    ContributorAuthor

    Correction: the stopgap is narrower than stated

    mindstyle (pops.one) ran v0.45.0 with GRAPH_STORE_CONNECTION_VALIDATION_IDLE_SECS=0 and still had deployments stop. Reproduced on the rig with the same build and env var and a 10 s outage (toxiproxy timeout 0, so packets are dropped and nothing is closed): 2 of 10 stopped.

    • QmPewF… is A. With validation on, health() gets database unavailable and is retried, but only five times, about 3 s in all. Past that the ? ends the runner: Subgraph instance failed to run: database query timed out, and it stays healthy.
    • QmQpY7… is B: subgraph writer poisoned by previous error on every retry, failed.

    So the env var only prevents A for outages of a few seconds, and nothing in config prevents B. The section above is corrected.

    Patched build

    Until a release includes graphprotocol/graph-node#6730 there is an unofficial build: v0.45.0 plus graphprotocol/graph-node#6730, graphprotocol/graph-node#6710 and graphprotocol/graph-node#6729, built in public CI with provenance attestations. https://github.com/cargopete/graph-node/releases/tag/v0.45.0-patch.1, image ghcr.io/cargopete/graph-node:v0.45.0-patch.1. It reports its version as 0.45.0+patch.1.

    Same rig with patch.1: 0 of 10 stopped under the 10 s outage and 0 of 10 under reset_peer, and every handler store error was recovered. POIs match v0.45.0 at blocks 1000, 5000, 12345 and 20000. That test subgraph only has block handlers, so graphprotocol/graph-node#6729's eth_getLogs path and graphprotocol/graph-node#6710's firehose path are not covered by it.

  4. added
    status/handed-offRaised with the owning party, still watching
    and removed on Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/graph-nodegraph-node behaviourkind/symptomA failure mode the Academy symptom index does not cover yetowner/upstreamgraphprotocol upstream reposroot-cause-foundWe know why. The explanation is in the thread.status/handed-offRaised with the owning party, still watching

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions