Skip to content

perf(stages): commit incremental merkle per chunk - #44

Open
0x00101010 wants to merge 7 commits into
releases/v2.8.1from
perf/merkle-incremental-chunk-commits
Open

0x00101010 wants to merge 7 commits into
releases/v2.8.1from
perf/merkle-incremental-chunk-commits

Conversation

@0x00101010

@0x00101010 0x00101010 commented Nov 14, 2026 •

Copy link
Copy Markdown

Incremental MerkleExecute ran every incremental_threshold chunk of a range inside one execute call, so the whole range was one uncommitted MDBX transaction: nothing was saved until the last chunk, and a crash or OOM lost all of it. This commits one chunk per call. The block checkpoint stays at the last validated block until the range's final root is validated, and the last committed chunk is recorded in the execution checkpoint (block_range.to, progress in blocks), so validation-error unwinds, with_max_block and consistency checks still only ever see validated blocks. Motivation: Base's 200ms blocks make 100k+ block catch-up ranges routine (base/base#6504).

Changes in crates/stages/stages/src/stages/merkle.rs:

  • Chunk per call. Resume after the recorded chunk when its range starts at the checkpoint and ends before the target. Only the final chunk is validated: hashing has already moved hashed state to the target, so a chunk's root is not its last block's root. Resuming is still exact, even toward a newer target: the committed trie holds target values for keys changed up to the chunk, and the remaining range's prefix sets cover every key changed after it.
  • Unwind. MerkleUnwind recomputes from min(MerkleExecute checkpoint, unwind_to) + 1, because keys changed after MerkleExecute's checkpoint can have stale nodes. This also fixes unwinding into an interrupted round, which failed root validation before. It clears MerkleExecute's chunk progress, since the recomputed trie no longer matches it. During a rebuild the old range is kept.
  • Interrupted rebuild. If a rebuild's inner checkpoint exists, the rebuild is finished even for a short range. Before, raising rebuild_threshold mid-rebuild sent the wiped trie down the incremental path, which recomputed every missing node in memory (the OOM in Update hive eth test suite in CI paradigmxyz/reth#6504).

Crash and retry coverage. start_round builds a realistic round: trie at block 100, hashed state at 200, 2,000 extra accounts so the trie stores branch hashes, and keys that change only at block 151. Each test drives the real stage and commits between calls like the pipeline. Each row's last column is a deliberate break that makes the test fail:

Scenario Test Fails when
Restart between committed chunks execute_chunked_merkle the block checkpoint advances per chunk
Crash before a chunk commits, then retry incremental_merkle_retries_uncommitted_chunk a chunk skips its first block
Interrupted round (hashing at 150), next round targets 200 incremental_merkle_resumes_toward_newer_target a chunk skips its first block; the block checkpoint advances per chunk
Wrong root at the end of a chunked range, then the pipeline's unwind final_root_error_keeps_checkpoint_at_last_validated_block the block checkpoint advances per chunk
Unwind at or before the last committed chunk, then the next round unwind_before_last_committed_chunk a chunk skips its first block; chunk progress survives the unwind
Unwind past the last committed chunk, then the next round unwind_past_last_committed_chunk the unwind range starts at unwind_to + 1; chunk progress survives the unwind
Short range after an interrupted rebuild interrupted_rebuild_is_finished_on_short_range the rebuild check is removed

cargo test -p reth-stages -p reth-stages-api and cargo clippy -p reth-stages --all-targets -- -D warnings pass.

Not handled: restarting with a pipeline target below the hashing stages' checkpoint (for example --debug.max-block below an interrupted round's committed chunks). The pipeline skips hashing, so MerkleExecute computes the lower block's root from hashed state at the higher one. That root is wrong whether or not chunks were committed, so this change neither causes nor fixes it.

Finish an interrupted rebuild instead of updating its partial trie incrementally, and have MerkleUnwind recompute from MerkleExecute's checkpoint when it stopped short of the unwind target. Tests cover retrying an uncommitted chunk, resuming toward a newer target, an interrupted rebuild, and unwinding on either side of the last committed chunk.
A chunked MerkleExecute round commits checkpoints whose state root was never validated, so a validation error must not unwind to one. Finish only advances after MerkleExecute validated a range, so unwind to the lower of the previous checkpoint and the Finish checkpoint.
Advancing the MerkleExecute block checkpoint per chunk made unvalidated blocks look validated: a root error unwound to one, and with_max_block or a lagging Finish could treat them as done. Keep the block checkpoint at the last validated block, record committed chunks in the execution checkpoint, and have MerkleUnwind clear that progress when it rewrites the trie. Reverts the Finish-based unwind target.
@0x00101010
0x00101010 marked this pull request as ready for review November 14, 2026 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant