Skip to content

PLT-7830: graph-node: subgraph runner exits when the DB is still down at restart time (store.restart / health errors) - #6

Merged
bashalex merged 2 commits into
zerion/v0.43.0from
PLT-7830-runner-restart-db-down
Nov 18, 2026
Merged

bashalex merged 2 commits into
zerion/v0.43.0from
PLT-7830-runner-restart-db-down

Conversation

@bashalex

@bashalex bashalex commented Nov 18, 2026 •

Copy link
Copy Markdown

PLT-7830: graph-node: subgraph runner exits when the DB is still down at restart time

Ports the shape of upstream graphprotocol#6730 (closed unmerged; issue graphprotocol#6741, paths 1 and 3) onto the zerion/v0.43.0 patch line, without graphprotocol#6730's store.restart() change, which this branch already has. It adds the unassign guard that graphprotocol#6730 does not have.

Changes

All in core/src/subgraph/runner/mod.rs:

  • Non-deterministic error arm (handle_action, let already_failed): a store error from health() or fail_subgraph() is logged at warn instead of being returned. The arm still backs off and returns Action::Restart. If health() fails, the error is still recorded with fail_subgraph().
  • Success arm (handle_action): a store error from unfail_non_deterministic_error() is logged at warn and treated as BehindErrorBlock, so the next block tries again.
  • Deterministic arm (handle_action): when fail_subgraph(.., true) fails, the runner logs at error, backs off and retries fail_subgraph in place. It does not restart: the fatal block's PoI-only batch is already written, so a restart would resume past the block. Between attempts it checks is_canceled(), and a canceled runner returns Action::Stop, which process_block_state turns into cancel(). Once the error is recorded, the deterministic error ends the runner as before. I chose this over "log and return" because an unrecorded deterministic error leaves the deployment healthy and stopped. (This changed in review attempt 1. The first version restarted, and that resumed past the fatal block.)
  • restart() failures (run_inner, RunnerState::Restarting, plus the new back_off_before_restart): an Unknown error from restart() (store.restart() or start_block_stream()) backs off with self.state.backoff and retries Restarting { reason }. This happens only when can_observe_unassign() is true, meaning cancel_handle is Some. is_canceled() is checked before and after the sleep, and a canceled runner goes through the existing cancel(), which returns Stopped { Unassigned } or Duplicate.
  • Path 3 (process_block_state → new is_close_to_chain_head): an error from chain_head_ptr() is logged at warn and the block is processed rather than skipped. The runner no longer ends on it.

tests/tests/writer_restart.rs: there are three new tests, serialised by the file's existing ONE_TEST_AT_A_TIME mutex. An Outage helper renames subgraphs.deployment or subgraphs.head and puts a view with the old name in its place. Every statement that reads the view fails with injected outage: database unavailable from the moment the existing injected write failure (fail_first_write_reaching) fires until Outage::end. A sequence counts the failed reads, and Drop restores the table even when a test panics. fail_first_write_reaching now returns the deployment namespace.

Acceptance Criteria

  • 1. Store errors from health(), from fail_subgraph() in the non-deterministic arm and from unfail_non_deterministic_error() no longer end the runner. Tested by store_errors_while_handling_a_non_deterministic_error_keep_the_runner, with the outage on subgraphs.deployment. The test waits for 2 failed reads (health + fail_subgraph), ends the outage, and asserts that the head reaches block 5, past the failing block 2, with no fatal error. Runs are below. The unfail path has no dedicated test; its change is the logged fallback in the success arm.
  • 2. A restart() store error after the first block stream backs off and retries. Tested by store_errors_at_restart_back_off_and_retry, with the outage on subgraphs.head, which store.restart() reads when it builds the new writer. health/fail_subgraph read subgraphs.deployment, so they succeed. The test waits for 2 failed store.restart() reads, asserts the head behind the outage is still 1, ends the outage, and asserts the head reaches 5 with no fatal error.
  • 3. Disposition table below, one row per grep -n '\.await?' core/src/subgraph/runner/mod.rs hit on the branch head.
  • 4. Unassign while backing off: unassigned_runner_stops_while_restart_backs_off uses the subgraphs.head outage and never ends it. It waits for 2 failed restarts, which proves the runner is still in the retry loop, then calls instances.remove(&deployment.id) (the instance manager's stop_subgraph path) and asserts that the runner future completes within 3 s (backoff ceiling 1 s plus jitter) and not with the injected-outage error. The runner log shows Terminating the subgraph runner because subgraph was unassigned and then Stopping subgraph - unassigned. Initializing and the first Restarting are unchanged: RunnerState::Initializing => self.initialize().await? (line 516) still returns the error. In the first Restarting, cancel_handle is None, so the guard if self.can_observe_unassign() (line 527) fails and res => res? (line 530) returns the error as before. No cancel guard is registered early.
  • 5. SubgraphRunnerError::Duplicate is never retried: only Unknown matches, and cancel() still returns Duplicate. Deterministic errors still end the runner once they are recorded, and a failed recording never moves past the fatal block. Tested by deterministic_error_is_recorded_without_passing_its_block: a trigger fails only the fatal-error status update on subgraphs.deployment, and the fatal-error subgraph fails at block 3. The test waits for 2 failed recordings and asserts the head is ≤ 3. It then ends the outage and asserts the runner returns Err, the head is ≤ 3, and fatal_error is deterministic at block 3. At 65c8afa it failed in each of 3 runs (the runner ended during the outage: Ok(()): the runner resumed past block 3). On 23a22e1 it passed in each of 3 runs. Locally, cargo test -p graph-tests --test runner_tests -- --test-threads=2 gave 16 passed, and cargo test -p graph-core passed. CI's Run unit tests and Subgraph Runner integration tests checks are the gate.
  • 6. CI green at the PR head. OUTSTANDING: CI is running now.
  • 7. Manager-only: tag v0.43.0-zerion3 and track the rollout.

AC3: .await? dispositions (branch head)

Line Call Disposition
230 new_block_stream(..) in start_block_stream retried: reached only from restart(), covered by the Restarting back-off once a stream has started
273 triggers_adapter.parent_ptr() in initialize kept: Initializing, where unassign is not observable (AC4)
289 store.unfail_deterministic_error() in initialize kept: Initializing, where unassign is not observable (AC4)
299 store.flush() in initialize (max end block reached) kept: Initializing, where unassign is not observable (AC4)
421 store.restart() in restart retried (AC2) once a block stream has started; kept on the first Restarting (AC4)
427 start_block_stream() in restart retried once a block stream has started; kept on the first Restarting (AC4)
457 store.flush() in finalize kept: the runner is stopping anyway (unassigned, max end block or stream end); there is nothing to retry into
516 self.initialize() in run_inner kept: Initializing, where unassign is not observable (AC4)
528 back_off_before_restart() in run_inner kept: its only error is Duplicate from cancel()
536 await_block() in run_inner kept: its only error is Duplicate from cancel()
546 process_block_state() in run_inner kept: carries the recorded deterministic error and Duplicate; its store and chain-store errors are now handled inside (lines 595, 612)
612 handle_action() in process_block_state kept: after this PR its only Err is the deterministic error after it was recorded
824 maybe_cancel() in transact_block_state kept: ProcessingError inside process_block; handle_action classifies it (non-deterministic → retry)
991 refetch_block() in process_block kept: ProcessingError, classified by handle_action
1188 process_dynamic_data_sources() in process_block kept: ProcessingError, classified by handle_action
1193 process_offchain_triggers() in process_block kept: ProcessingError, classified by handle_action
1206 transact_block_state() in process_block kept: ProcessingError, classified by handle_action (the injected write failure goes this way)
1536 chain.chain_head_ptr() in is_caught_up kept: its caller maps it with .non_deterministic()?, so it is retried by the non-deterministic arm
1649 entity_cache.as_modifications() in handle_offchain_triggers kept: the caller maps it with .non_deterministic()?
1822 store_poi_entity() in update_proof_of_indexing kept: the caller maps it with .non_deterministic()?

These .await? sites are no longer in the grep:

  • chain_head_ptr() in process_block_state (path 3, formerly ~:566) is now logged, through is_close_to_chain_head.
  • unfail_non_deterministic_error() is logged.
  • fail_subgraph(.., true) is retried in place (in the deterministic arm, without a restart).
  • health() and fail_subgraph(.., false) are logged.

Reproduction runs

I ran the tests locally against Postgres 16 and IPFS in Docker. At base, the runner fix was reverted in the worktree (git checkout -- core/src/subgraph/runner/mod.rs, so the runner code is a1a67f93) and the new tests were kept.

At base a1a67f93, three runs gave the same result. Each run printed:

test store_errors_at_restart_back_off_and_retry ... FAILED
test store_errors_while_handling_a_non_deterministic_error_keep_the_runner ... FAILED
test unassigned_runner_stops_while_restart_backs_off ... FAILED
the runner ended during the outage: Err(store error: injected outage: database unavailable)   (x3)
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 2 filtered out

On the branch, three runs of the new tests plus one run of the whole file all passed:

test store_errors_at_restart_back_off_and_retry ... ok
test store_errors_while_handling_a_non_deterministic_error_keep_the_runner ... ok
test unassigned_runner_stops_while_restart_backs_off ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out      (x3)
test result: ok. 5 passed; 0 failed; ...                                         (whole file)

Branch runner log, per test:

  • AC1 test: Failed to read subgraph health ×1, Failed to record non-deterministic error ×1, then Stop block reached.
  • AC2 test: Subgraph failed to restart, retrying ×2, then Stop block reached.
  • AC4 test: Subgraph failed to restart, retrying ×2, then Terminating the subgraph runner because subgraph was unassigned.

Departures

  • The new tests are in the existing tests/tests/writer_restart.rs, not a new file. They share its serialisation and write-failure injection, and CI already runs that file (just test-writer-restart in the runner-test job), so justfile and ci.yml need no change.
  • health() failure: the error is still recorded with fail_subgraph(), as core: Keep the subgraph runner alive through transient store errors graphprotocol/graph-node#6730 does, rather than skipping the record.
  • No dedicated test for path 3's logged fallback. The ACs ask for tests of AC1, AC2 and AC4 only. The deterministic arm now has its own test, added in review.
  • The first local runner_tests run, at default parallelism, failed 6 tests during database setup (database setup failed: database query timed out in pool/coordinator.rs, then cascading PoisonErrors) against my local Docker Postgres. A rerun with --test-threads=2 passed all 16. I count this as local environment, not this change. CI is the gate.
  • I did not run cargo check --release locally. CI's Build in release mode covers it.

Testing

  • cargo fmt --all: clean. cargo clippy -p graph-core -p graph-tests --all-targets: no warnings.
  • New tests in tests/tests/writer_restart.rs: store_errors_while_handling_a_non_deterministic_error_keep_the_runner, store_errors_at_restart_back_off_and_retry, unassigned_runner_stops_while_restart_backs_off, deterministic_error_is_recorded_without_passing_its_block.
  • At 23a22e1 the whole file (6 tests) passed in each of 3 runs.

Resolves PLT-7830

🤖 Generated with Claude Code

Produced by a hive session (PLT-7830)

…estart (PLT-7830)

A store error raised while handling a non-deterministic error, while
unfailing, or by restart() after the first block stream started no longer
ends the runner: health()/fail_subgraph()/unfail errors are logged, a failed
restart backs off with the runner's backoff and retries, and an unreadable
chain head processes the block instead of ending the runner. A failure to
record a deterministic error backs off and retries so the error gets
recorded. Retries happen only once a block stream registered the cancel
guard, so an unassigned runner stops within one backoff interval;
Initializing and the first Restarting keep returning the error.

Tests in tests/tests/writer_restart.rs put subgraphs.deployment or
subgraphs.head behind a view that fails every read from the injected write
failure until the outage ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zerion-hive-worker[bot] <318250561+zerion-hive-worker[bot]@users.noreply.github.com>
@bashalex

Copy link
Copy Markdown
Author

REVIEW REQUEST @ 65c8afa v=1 pr=#6 base=zerion/v0.43.0 patch-id=e402cb3523a386937b1007f6e1c24868bb521382 commits=1 since=none ac=PLT-7830

@linear-code

linear-code Bot commented Nov 18, 2026

Copy link
Copy Markdown

PLT-7830

@bashalex

Copy link
Copy Markdown
Author

Reviewer: reviewer-PLT-7830 @ 65c8afa
REVIEW STARTED @ 65c8afa v=1 pr=#6 by=reviewer-PLT-7830

@bashalex

Copy link
Copy Markdown
Author

Reviewer: reviewer-PLT-7830 @ 65c8afa
VERDICT CHANGES @ 65c8afa v=1 pr=#6 attempt=1/3 blockers=1 class=fatal-block-retry-resumes-past-head repeat=no history=read

Review: PLT-7830

Outcome: CHANGES REQUESTED. One blocker, in the deterministic arm. The other paths are correct, and CI is green at this head.

Blocker 1: the deterministic-arm retry resumes past the failing block (handle_action, ProcessingErrorKind::Deterministic)

The PR body says: "The retry reaches the same error and records it then." The code does not support that. Here is the path:

  1. transact_block_state, has_errors && errors_are_fatal() branch: the PoI-only batch for failing block N goes through transact_block_operations, which sets WritableStore.block_ptr to N, and through store.flush(). Only after that does it return ProcessingError::Deterministic. The batch also inserts the subgraph_error row, but it does not mark the deployment failed. Only fail_subgraph does that.
  2. handle_action → revert_state_to(store.block_ptr()), which is N. Then fail_subgraph(.., true) fails, and the new code sleeps and returns Action::Restart.
  3. restart(): the writer is not poisoned (the flush succeeded), so WritableStore::restart returns Ok(None) and the writer is kept. start_block_stream resumes from store.block_ptr(), which is N, so the next block is N+1.

Result: block N is never processed again. Its entity changes are dropped (only the PoI was stored), the fatal error is never recorded, and the deployment keeps indexing as healthy. At base the runner stops at N (Subgraph instance failed to run, failed metric), so someone can notice. This change makes the skip immediate and silent. It also covers any persistent non-DatabaseUnavailable error from fail_subgraph, not only an outage. (SyncStore::fail_subgraph already retries DatabaseUnavailable forever, so what reaches the runner is everything else, such as a connection terminated mid-query during a CNPG shutdown.) Your re-reach claim holds only when the flush also failed and poisoned the writer.

This is the departure "no dedicated test for the deterministic arm's retry". The missing test would have caught it.

Required change, your choice of shape:

  • (a) The ticket's other option: log at error and return the deterministic error, as base does; or
  • (b) Retry fail_subgraph(.., true) in place with self.state.backoff, checking is_canceled() between attempts, and return Err(err) once it records. Never restart the stream past N.

Either way, add a test in tests/tests/writer_restart.rs that makes fail_subgraph fail at least once after a fatal deterministic error. For example, put the Outage on subgraphs.deployment after the fatal block's flush. It must assert that the deployment ends with fatal_error recorded at block N and a head that never passes N. The test should fail on this head.

Acceptance Criteria

  • ✅ 1. health(), fail_subgraph(.., false) and unfail_non_deterministic_error() are logged, not propagated. The arm still sleeps and returns Action::Restart. Covered by store_errors_while_handling_a_non_deterministic_error_keep_the_runner, which ran green in CI at this head. The fail-at-base result is the engineer's local evidence, three runs; I did not re-run it. The unfail fallback has no test, but it is a one-line logged fallback to BehindErrorBlock and is correct by inspection.
  • ✅ 2. restart() errors back off through back_off_before_restart, which checks is_canceled() before and after sleep_async, and retry Restarting { reason }. Covered by store_errors_at_restart_back_off_and_retry, green in CI. Retrying after store.restart() succeeded but start_block_stream failed is safe: self.inputs is already swapped, so the next store.restart() returns None.
  • ✅ 3. I re-ran grep -n '\.await?' on 65c8afa and got 20 hits: 230, 273, 289, 299, 421, 427, 457, 516, 528, 536, 546, 612, 824, 991, 1188, 1193, 1206, 1530, 1643, 1816. All 20 rows are in the table and none is missing. Path 3 is "logged". One correction is due once Blocker 1 is fixed: the row for 612 and the removed-sites list say fail_subgraph(.., true) is "retried". Update both to match the fix.
  • ✅ 4. can_observe_unassign() is cancel_handle.is_some(). Initializing keeps self.initialize().await?, and the first Restarting falls through to res => res?. Only SubgraphRunnerError::Unknown is matched, so Duplicate passes through. unassigned_runner_stops_while_restart_backs_off is green in CI.
  • ❌ 5. Duplicate handling is unchanged. Deterministic errors do not "still end the runner as before": when recording fails, the runner now continues past the fatal block (Blocker 1). The CI checks pass, but they do not exercise this path.
  • ✅ 6. Check runs at 65c8afa (github-actions): Check rustfmt style=success, Clippy linting=success, Build in release mode=success, Run unit tests=success, Subgraph Runner integration tests=success. The job log shows the three new tests ran and passed (writer-restart: 5 passed).
  • ⬜ 7. manager-only. Its phase (after merge) has not occurred.

Code Quality / other checks

  • Path 3 (is_close_to_chain_head): safe. An unreadable head means the block is processed instead of skipped. If the head is still unreadable, is_caught_up in transact_block_state turns the error into a non-deterministic one, which goes to the back-off arm. A deployment is never marked synced on a guess.
  • Backoff is bounded by subgraph_error_retry_ceil: 3600 s by default, ±20% jitter.
  • The commit carries the zerion-hive-worker[bot] co-author trailer.

Notes (non-blocking)

  • drop: the back-off sleep cannot be interrupted. With the production ceiling of 3600 s, an unassigned runner can wait up to about 72 minutes before it returns. That meets AC4's "within one backoff interval" as written, and it matches the existing non-deterministic arm.
  • drop: if health() fails but fail_subgraph(.., false) then succeeds on a deployment that had already failed, the same error is recorded twice. This matches core: Keep the subgraph runner alive through transient store errors graphprotocol/graph-node#6730 and is cosmetic.
  • drop: retries from back_off_before_restart raise backoff.attempt, which is reset only when an unfail succeeds. A restart failure on a non-error path (DynamicDataSourceCreated or PossibleReorg) therefore leaves a later non-deterministic error starting closer to the ceiling.

Verdict: CHANGES REQUESTED

Required Changes

  1. Fix the deterministic arm so a failed fail_subgraph(.., true) never lets the stream resume past the fatal block, using (a) or (b) above. Add a test that fails on this head.

…ing past its block (PLT-7830)

When fail_subgraph(.., true) fails, the deterministic arm retried with
Action::Restart. The fatal block's PoI-only batch is already written and a
healthy writer's restart() is a no-op, so the stream resumed past the block
and the error was never recorded. Retry fail_subgraph in place with the
backoff instead, and stop when the runner is canceled.

The new test fails the fatal-error status update until the outage ends and
asserts the error is recorded at block 3 and the head never passes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zerion-hive-worker[bot] <318250561+zerion-hive-worker[bot]@users.noreply.github.com>
@bashalex

Copy link
Copy Markdown
Author

REVIEW REQUEST @ 23a22e1 v=1 pr=#6 base=zerion/v0.43.0 patch-id=c130b37ac3150cbccf105b84e8cef59177970249 commits=2 since=65c8afacf850e687a18cc5a9ffdd25acd4449b1a ac=PLT-7830

@bashalex

Copy link
Copy Markdown
Author

Reviewer: reviewer-PLT-7830 @ 23a22e1
REVIEW STARTED @ 23a22e1 v=1 pr=#6 by=reviewer-PLT-7830

@bashalex

Copy link
Copy Markdown
Author

Reviewer: reviewer-PLT-7830 @ 23a22e1
VERDICT APPROVE @ 23a22e1 v=1 pr=#6 patch-id=c130b37ac3150cbccf105b84e8cef59177970249 ci=5/github-actions notes=3

Review: PLT-7830 (re-review; changes since 65c8afa)

Outcome: APPROVE. The fix resolves Blocker 1 from attempt 1 (fatal-block-retry-resumes-past-head).

Blocker 1: resolved

  • In handle_action's ProcessingErrorKind::Deterministic arm, while let Err(e) = self.fail_subgraph(..).await now retries the recording in place and no longer returns Action::Restart, so the stream cannot resume past the fatal block. After each failed attempt, is_canceled() is checked. A canceled runner returns Action::Stop, and process_block_state (the only caller of handle_action) then checks is_canceled() and calls cancel(), which returns Stopped { Unassigned } or Duplicate. Every retry goes through self.state.backoff, so the loop does not spin. When the error is recorded, the arm returns Err(err), as base does.
  • The new test, deterministic_error_is_recorded_without_passing_its_block, injects the failure the right way. A trigger fails only the subgraphs.deployment update that sets fatal_error, so the writer is not poisoned. That is the condition the blocker needs, and the engineer's first attempt, which failed the subgraph_error insert, did not reproduce it. The test asserts head ≤ 3 while recording is failing and again after the outage ends, that the runner returns Err, and that a deterministic fatal_error is recorded at block 3.
  • Pre-fix failure: the engineer's logs (fix2-old{1,2,3}.log in their scratchpad) show 3 of 3 failing runs with the runner ended during the outage: Ok(()), meaning the runner went past block 3 to the stop block. That matches the mechanism I traced. I read the logs but did not run the test myself.

Acceptance Criteria

  • ✅ 1, 2, 4: the code is unchanged since attempt 1. The three tests still pass in CI.
  • ✅ 3: grep -n '\.await?' at 23a22e1 returns 20 hits: 230, 273, 289, 299, 421, 427, 457, 516, 528, 536, 546, 612, 824, 991, 1188, 1193, 1206, 1536, 1649, 1822. Every hit is in the updated table, and fail_subgraph(.., true) is listed as "retried in place".
  • ✅ 5: Duplicate handling is unchanged. A deterministic error still ends the runner once it is recorded, and a failed recording never moves past the fatal block. The new test covers this.
  • ✅ 6: check runs at 23a22e1 (github-actions): Check rustfmt style, Clippy linting, Build in release mode, Run unit tests and Subgraph Runner integration tests all success. The writer-restart step ran 6 tests and all passed, including the new one.
  • ⬜ 7: manager-only. Its phase (after merge) has not occurred.

Verdict: APPROVE

Notes (non-blocking, all drop)

  1. The back-off sleep cannot be interrupted. With the production ceiling of 3600 s plus jitter, an unassigned runner can wait up to about 72 minutes before it stops. This applies to both new retry sites and to the existing non-deterministic arm.
  2. If health() fails but fail_subgraph(.., false) succeeds, a deployment that had already failed can get a duplicate error record. core: Keep the subgraph runner alive through transient store errors graphprotocol/graph-node#6730 behaves the same way.
  3. backoff.attempt is reset only when an unfail succeeds, so a restart failure on a non-error restart path leaves the backoff closer to its ceiling.

Scope

Scoped to 23a22e1 (patch-id c130b37ac3150cbccf105b84e8cef59177970249 from hive-engine patch-id).
Carries over automatically to any later head whose patch-id is unchanged, including a clean merge origin/zerion/v0.43.0, and to nothing else.

@bashalex
bashalex merged commit 1e572ab into zerion/v0.43.0 Nov 18, 2026
5 checks passed
@bashalex
bashalex deleted the PLT-7830-runner-restart-db-down branch November 18, 2026 17:07
@bashalex

Copy link
Copy Markdown
Author

Deployed: v0.43.0-zerion3 (tag at 1e572ab, ECR sha256:aa584d57e13222aa7abf6d49f8828c0b4e754e80d98da63b0fda6244f04dbdce) · image.tag k8s-cluster eksctl 6c8e1bfa5885062b86b69bb934de18a88b7ad99f · production datai, base-bootstrap-graph-node only (user decision "Base-bootstrap only"; fleet stage 2 separate). Ships #5 and #6.
rollout: statefulset/datai-indexer-v1-base-bootstrap-graph-node rolling update complete, ArgoCD datai-indexer-v1-production Synced/Healthy at 6c8e1bfa58 · health: pod Ready on the zerion3 digest, 0 restarts, 0 ERRO/panic/poisoned/store-error lines, sgd5613 committed 282 batches (59162908 → 59164090) over 17:29:22Z–17:33:27Z UTC 2026-11-18; no other datai graph-node restarted. Writer restart covered by CI runner tests at 1e572ab, not exercised live.
Linear: https://linear.app/zerion/issue/PLT-7829 · https://linear.app/zerion/issue/PLT-7830

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant