Skip to content

fix: keep graph setup and settlement jobs alive through backend lag and restarts - #543

Merged
eigmax merged 1 commit into
devfrom
audit/2-graph-liveness
Nov 17, 2026
Merged

eigmax merged 1 commit into
devfrom
audit/2-graph-liveness

Conversation

@KSlashh

@KSlashh KSlashh commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Audit items L2, L4, L5, L8, L9, L10, L11, L12, L15, L68. Open against dev.

Dead slots after a restart (L2). The lowest slot per operator whose Pre-Kickoff can never confirm is kept in memory. Later slots of that operator stay Obsoleted only while the map names it. After a restart the map was rebuilt only when the dead slot itself was fully scanned (every 16 blocks, in a rotating batch), so a later slot scanned first could go back to OperatorDataPushed and be picked for a pegout. Now the Obsoleted graphs are checked once after a start, per operator from the lowest slot up to the first dead one, before any Obsoleted graph is scanned. The check resumes across ticks if the maintenance timeout cuts it short, and an operator whose lookups fail is retried on the next tick while the Obsoleted pass waits. The way back from Obsoleted for a graph whose instance failed and that later shows up posted is unchanged.

Unaffordable Skip-Kickoff (L4). When the Kickoff connector could not pay the Skip-Kickoff fee, only the Pre-Kickoff was broadcast and the step logged a successful skip every block, while the next kickoff stayed blocked. operator_skip_graph now says whether the skip went out. When it did not, the step raises pegout_predecessor_stuck and looks again 10 blocks later, so the skip goes out once fees allow.

Relayer give-up (L5). proceedWithdraw, finishWithdrawHappyPath and finishWithdrawUnhappyPath gave up after a number of job attempts, and every retry during a Goat or Bitcoin outage counted. They now give up by chain depth: 30 blocks after the kickoff, Take-1 or Take-2 confirmed for a withdraw Goat does not show, 360 blocks for a withdraw that stays Initialized. An unconfirmed transaction keeps the job waiting.

Verifier graph validation (L8). The verifier ran the Goat and Bitcoin parameter checks, rebuilt the graph and checked operator signatures for any CreateGraph naming it, before checking it had run the setup. The local setup lookup now comes first.

Setup-state cleanup (L9, L10).

  • With S3 payload storage the cleanup stopped at the first S3 reference and removed nothing. S3 payloads are now skipped and left to the bucket's lifecycle rules (several nodes may share a prefix); the local state files are still removed.

  • A stale state with no graph row was treated as an orphan, although the row only exists once the operator presigns. It is now kept while the graph has a pending or running engine job or an aggregate updated within the retention. A state whose stored graph belongs to another instance is skipped instead of deleted.

Out-of-range candidate index (L11). A SolderingProofReady naming an index outside the frozen membership was an unknown error: 20 retries and a false graph_job_gave_up. It is now rejected as invalid.

Posted graph pull (L12). The event watcher asks for a posted graph's definition once, and the pull gave up after about two days with no way to start again. A pull for a graph posted on Goat now keeps retrying every 10 minutes and raises posted_graph_unavailable about once a day.

Backend lag in committee validation (L15). "Peg-in funding/deposit transaction not found on Bitcoin" is the backend lagging, but it was an unknown error, so the committee's stable-id signing jobs failed for good after about 2.6 hours. It is now a transient BackendLag error. The committee's ingest and nonce jobs check the signing gate before validating, so these retries stop at T0 or when the instance fails.

Local io errors (L68). Every std::io::Error was transient, so a missing or unreadable local file retried a setup job forever without reaching the give-up and its alert. Only network-shaped kinds (timeouts, refused, reset or aborted connections, broken pipes, interrupted or would-block reads, unexpected EOF) stay transient.

Not changed (L14). A rerun of build_graph reuses the funding output already at the address and tops up only the shortfall, so a second funding needs the indexer to miss the first one. Left as is.

Tests: only network io errors are transient; BackendLag is transient; stale setup state with a pending engine job is kept and removed once the job is gone. Node tests, clippy with -D warnings and fmt pass.

The test coverage and validation notes above are from the zkm PR notes. Branch scope and diff formatting were checked locally; builds and tests were not rerun during publication.

@eigmax
eigmax force-pushed the audit/2-graph-liveness branch from cd9f41a to 4be1708 Compare November 16, 2026 07:50
@eigmax
eigmax merged commit 041f814 into dev Nov 17, 2026
4 checks passed
@eigmax
eigmax deleted the audit/2-graph-liveness branch November 17, 2026 02:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants