Repository navigation
perf(stages): commit incremental merkle per chunk - #44
Open
0x00101010 wants to merge 7 commits into
Open
0x00101010 wants to merge 7 commits into
0x00101010 wants to merge 7 commits into
Conversation
Finish an interrupted rebuild instead of updating its partial trie incrementally, and have MerkleUnwind recompute from MerkleExecute's checkpoint when it stopped short of the unwind target. Tests cover retrying an uncommitted chunk, resuming toward a newer target, an interrupted rebuild, and unwinding on either side of the last committed chunk.
A chunked MerkleExecute round commits checkpoints whose state root was never validated, so a validation error must not unwind to one. Finish only advances after MerkleExecute validated a range, so unwind to the lower of the previous checkpoint and the Finish checkpoint.
Advancing the MerkleExecute block checkpoint per chunk made unvalidated blocks look validated: a root error unwound to one, and with_max_block or a lagging Finish could treat them as done. Keep the block checkpoint at the last validated block, record committed chunks in the execution checkpoint, and have MerkleUnwind clear that progress when it rewrites the trie. Reverts the Finish-based unwind target.
0x00101010
marked this pull request as ready for review
November 14, 2026 05:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incremental
MerkleExecuteran everyincremental_thresholdchunk of a range inside oneexecutecall, so the whole range was one uncommitted MDBX transaction: nothing was saved until the last chunk, and a crash or OOM lost all of it. This commits one chunk per call. The block checkpoint stays at the last validated block until the range's final root is validated, and the last committed chunk is recorded in the execution checkpoint (block_range.to, progress in blocks), so validation-error unwinds,with_max_blockand consistency checks still only ever see validated blocks. Motivation: Base's 200ms blocks make 100k+ block catch-up ranges routine (base/base#6504).Changes in
crates/stages/stages/src/stages/merkle.rs:MerkleUnwindrecomputes frommin(MerkleExecute checkpoint, unwind_to) + 1, because keys changed afterMerkleExecute's checkpoint can have stale nodes. This also fixes unwinding into an interrupted round, which failed root validation before. It clearsMerkleExecute's chunk progress, since the recomputed trie no longer matches it. During a rebuild the old range is kept.rebuild_thresholdmid-rebuild sent the wiped trie down the incremental path, which recomputed every missing node in memory (the OOM in Update hive eth test suite in CI paradigmxyz/reth#6504).Crash and retry coverage.
start_roundbuilds a realistic round: trie at block 100, hashed state at 200, 2,000 extra accounts so the trie stores branch hashes, and keys that change only at block 151. Each test drives the real stage and commits between calls like the pipeline. Each row's last column is a deliberate break that makes the test fail:execute_chunked_merkleincremental_merkle_retries_uncommitted_chunkincremental_merkle_resumes_toward_newer_targetfinal_root_error_keeps_checkpoint_at_last_validated_blockunwind_before_last_committed_chunkunwind_past_last_committed_chunkunwind_to + 1; chunk progress survives the unwindinterrupted_rebuild_is_finished_on_short_rangecargo test -p reth-stages -p reth-stages-apiandcargo clippy -p reth-stages --all-targets -- -D warningspass.Not handled: restarting with a pipeline target below the hashing stages' checkpoint (for example
--debug.max-blockbelow an interrupted round's committed chunks). The pipeline skips hashing, soMerkleExecutecomputes the lower block's root from hashed state at the higher one. That root is wrong whether or not chunks were committed, so this change neither causes nor fixes it.