feat(test-fill): make engine_x fixtures consumable via devp2p sync by danceratopz · Pull Request #3364 · ethereum/execution-specs

danceratopz · 2026-08-12T11:30:41Z

Description

This PR lets the filler append an empty sync payload above each chain represented in a blockchain_test_engine_x fixture and store it separately in a new optional syncPayloads field:

G → T₁ … Tₙ → S*

* marks the extra sync payload announced by a sync-based consumer.

The sync payload's parent is unknown to the client, so the client starts syncing and fetches every test block below it via devp2p.

For more context, the detailed documentation is intended to be read in this order:

Sync Payloads explains the design and mechanism.
Consumption in the EngineX fixture-format reference explains how these fixtures are consumed through the Engine API and devp2p full-sync paths.

Important

Companion PR: #3365 adds consume wirex and the devp2p peer that use these sync payloads. Ordinary Engine API consumers ignore syncPayloads; the EngineX consumption guide keeps the authored Engine API sequence separate.

Fixture shape

In abbreviated form, a fixture with one test block looks like this:

{
  "engineNewPayloads": [
    {
      "params": [
        {
          "blockNumber": "0x1",
          "blockHash": "0xabc...",
          "transactions": ["0x..."]
        }
      ]
    }
  ],
  "syncPayloads": [
    {
      "params": [
        {
          "parentHash": "0xabc...",
          "blockNumber": "0x2",
          "transactions": [],
          "extraData": "0x1234..."
        }
      ]
    }
  ],
  "lastblockhash": "0xabc..."
}

The matching parentHash shows that the sync payload is appended above the test block. lastblockhash continues to identify the test block, not the sync payload.

Existing sibling-chain examples

The EIP-7594 fork-transition test produces the two pre-Amsterdam cases: blob_count_10 has three expected-invalid siblings, while blob_count_7 has one valid block followed by two expected-invalid siblings.
The EIP-7954 fork-transition test produces an expected-invalid/valid sibling pair.
The EIP-7981 fork-transition test produces three chain heads across four test payloads.

PR-specific notes

This follows the same "empty block above the test chain" mechanism introduced for blockchain_test_sync by execution-spec-tests #2007. It adds a test-specific value to extraData and supports one sync payload per sibling-chain head, but does not replace the existing code that generates the singular syncPayload; refactor(test-fill): use the shared sync-payload builder for sync fixtures #3405 makes both fixture formats use the same builder.
No consensus test files, test blocks, assertions, or markers are changed.
The same node-ID normalization removes pytest-xdist's @bigmem scheduling suffix from published fixture IDs. This changes 1,299 IDs. Because packed pre-allocation groups are keyed by member IDs, 23,351 EngineX fixtures also receive new, internally consistent preHash pointers. These are release identity changes; they do not change the test payloads.

Related Issues or PRs

Consume-side counterpart:

feat(test-consume): add consume wirex, a devp2p full-sync simulator #3365.

Follow-up:

refactor(test-fill): use the shared sync-payload builder for sync fixtures #3405

Checklist

Ran fast static checks: just static
PR title has the form <type>(<area>): <title> matching C-*/A-* labels; the title matches the target squash commit message.

Cute Animal Picture

codecov · 2026-08-12T11:34:16Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.53%. Comparing base (43e3cd1) to head (707458c).
⚠️ Report is 3 commits behind head on forks/amsterdam.

Additional details and impacted files

@@               Coverage Diff                @@
##           forks/amsterdam    #3364   +/-   ##
================================================
  Coverage            93.53%   93.53%           
================================================
  Files                  624      624           
  Lines                37074    37074           
  Branches              3394     3394           
================================================
  Hits                 34679    34679           
  Misses                1645     1645           
  Partials               750      750

Flag	Coverage Δ
unittests	`93.53% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

danceratopz · 2026-08-13T20:35:40Z

@marioevz @spencer-tb I realized that I wasn't entirely sure whether invalid test blocks are guaranteed to be sent and rejected via devp2p, so I revisited this.

Prepending a new block is imo the most contentious change in this PR.

Appending doesn't change the chain or test expectation. It also doesn't incur much effort.

Prepending, however, does require effort to ensure that it doesn't change test expectations. As it turns out it's currently only worth it for 3/6 clients (which could also change against our favour in the future).

For an invalid singleton, WireX uses G → S → Tᵢ*: newPayload supplies the complete invalid head Tᵢ, while its unknown salted parent S gives the client a reason to sync. The six baseline clients currently split into two groups:

Announced `Tᵢ` body over devp2p?	Clients	Result
Yes	geth, Nethermind, Besu	The invalid test block crosses devp2p and is processed through the sync/backfill path. This is useful coverage beyond EngineX.
No	reth, ethrex, Erigon	The client syncs the missing ancestry, then judges `Tᵢ` using the payload already supplied by `newPayload`. The invalid body does not cross devp2p.

Both behaviors are correct. Only blocks below the announced head are required to travel over devp2p; refetching the head body is an implementation choice. Fresh checks reproduced both sides: geth requested Tᵢ and rejected it during beacon backfill, while ethrex requested only S and reused the Engine-delivered Tᵢ. Reth also reproduced the ancestors-only shape.

WireX asserts that test-block bodies were transported over devp2p, but only for valid tests, where every test block is placed below the announced sync head. It does not assert this for an invalid head because transport of the announced head's body is not enforceable across clients.

Currently, the prepend gives invalid-head devp2p ingestion coverage on three of six clients, but Geth, Nethermind or Besu could legitimately adopt the other group's optimization later, at which point the prepend would become futile for testing invalid-head body transport (although it would still exercise missing-ancestry sync followed by rejection).

We could ask clients to expose a test/debug flag that forces head-body download, but that adds per-client code, Hive configuration and ongoing maintenance, while testing a path different from the client's normal production behavior.

My conclusion is that the present three-client benefit makes the prepend worthwhile, but the PR should describe it as measured, client-dependent coverage rather than a guarantee. I also think it's worth reaching it to client teams, but wanted to get your take first.

danceratopz · 2026-08-14T11:25:22Z

I think the obvious and simple solution to this is to drop the prepend and append a second invalid synthetic sync trigger block. This should trigger the sync; both invalid blocks should be rejected! Will update.

When filling a blockchain_test_engine_x fixture, build one empty block above the chain's last built block - valid or not - and store it out of chain in the fixture's `syncPayload` field. A sync-based consumer announces that block instead of the chain's own head, which makes every block the test author wrote an ancestor a client must fetch and execute through its devp2p sync path. The test's own chain is untouched, byte for byte: a pytester suite fills whole modules both ways and asserts the fixtures are identical except for the `syncPayload` key itself. The announced block must be the block whose judgment the test asserts, so a chain asserting an `engine_api_error_code` keeps its own announcement and fills bare: a block appended above it would be announced instead, and the refusal the test verifies would never happen. This is part of the rule's meaning, not a special case - such a fill would succeed while silently destroying the assertion. The block's `extra_data` carries a per-test salt (the node id, any xdist group suffix stripped), because two tests of one pre-allocation group may declare byte-identical chains and a reused client only starts a sync for a head it has never seen. Benchmark chains opt out (`sync_block=False` at their conversion to a blockchain test): a framework block would distort what they measure. `--no-sync-block` turns the feature off for a fill. Not every chain can carry a block above its head; the follow-up commits teach the filler to decide those cases. Until then: - a head pinning a near-ceiling timestamp (the EIP-4788 beacon-root tests) silently emits a syncPayload whose timestamp overflows uint64, which no client could parse (verified on this commit); - a head pinning the maximum slot number, a sub-floor gas limit, or blob fields that overflow uint64 fails the fill loudly; - from Osaka on, a head pinning an excess blob gas near 2**64 hangs the fill in the blob price's Taylor series.

A block's timestamp and slot number must fit uint64, and the appended sync block takes its parent's plus a fixed step. A head pinned at or next to either ceiling therefore admits no child block at all - and nothing else on the fill side notices: Python integers do not wrap and t8n accepts the value, so before this commit such a chain silently emitted a syncPayload no client could parse (verified on the previous commit: the EIP-4788 beacon-root tests pinning timestamps at 2**64-1 and 2**64-2 produced sync timestamps of 2**64+11). Add sync_block_context_unavailable, the filler's own judgement of whether a block can be built above a head, with the two type-ceiling clauses as its first members. The filler declines with an INFO log and the chain fills as exactly the author's own - bare, never skipped, never marked: the condition is pure arithmetic on the head's header, so asking test authors to record it would only go stale. Timestamps and slot numbers are semantic (fork activation, TIMESTAMP and SLOTNUM expectations) and are never clamped or shifted. Verified by targeted fills: the beacon-root timestamps function at Cancun fills 24 engine_x fixtures with exactly the 12 ceiling parameters bare and their neighbours intact, and test_slotnum_value at Amsterdam fills 5 with exactly slot_max_u64 bare. The literal 12 gains its name, DEFAULT_TIMESTAMP_INCREMENT, because the guard's arithmetic must match Block.set_environment.

The appended block inherits its parent's gas limit, so a head pinning a gas limit below the fork's minimum admits no framework-built child: the fill failed loudly on the previous commits for test_block_gas_limit_below_minimum's zero, one and minimum_minus_one parameters at every fork. Add the gas-floor clause to sync_block_context_unavailable. The floor is fork arithmetic, not a constant a test author can see: from Amsterdam on, minimum_block_gas_limit is the budget an empty block's own access list needs (EIP-7928), and a fork raising it must not strand markers in test files - so the filler decides, silently, like the ceilings before it. The clause is judged under the fork the appended block *itself* would be built under, resolved at that block's own number and timestamp, not the fork of the head or of the chain's end. On a transition chain those disagree: a head at t=14,000 on a BPO2-to-Amsterdam-at-15k chain carries an appended block at t=14,012, which lands before Amsterdam and must be judged by BPO2's floor - judging by the chain's final fork would refuse a block on a chain that never reaches that fork. The type ceilings stay ahead of the fork resolution because they are fork-independent, and the fork of a block that cannot exist is not well-defined. A unit test pins the transition behaviour on both sides of the boundary. Verified by targeted fills of test_block_gas_limit_below_minimum at Paris and Osaka: at each fork exactly zero, one and minimum_minus_one fill bare and minimum keeps its block.

An intentionally invalid head can pin blob header fields from which no child block's fee context can be derived, in two ways: - excess blob gas plus blob gas used overflows uint64, so EELS's excess derivation raises before any child exists (the fill failed loudly on the previous commits: 35 fixtures across the Cancun excess-blob-gas file); - from Osaka on (EIP-7918) a child's excess blob gas needs its parent's blob gas price, whose Taylor series takes one step per update fraction of that excess on an integer that grows at every step. A head pinning an excess near 2**64 needs on the order of a trillion steps: not a failure but a hang, the one case where the appended block can do unbounded work. On the corpus this class is 216 fixtures at Osaka, each able to wedge a fill worker forever. Add the two blob clauses to sync_block_context_unavailable, bounded by MAX_SYNC_BLOCK_BLOB_PRICE_STEPS (1024): a uint256 cannot hold a price past about 177 steps, so anything between a few hundred and a few million separates every legitimate fee market (the most expensive test sits near 60 steps) from the pinned nonsense (about ten trillion). The guard is deliberately fork-agnostic: an absurd excess is refused everywhere, not only where the price series would diverge, at the cost of a little sync coverage on pre-Osaka forks in exchange for one rule instead of a fork-conditional one. Verified by targeted fills: the Cancun excess-blob-gas file fills 213 engine_x fixtures with exactly 35 bare, and Osaka's test_invalid_negative_excess_blob_gas fills all 216 bare in seconds - both figures matching the released corpus census exactly.

Some chains deliberately leave behind a state the appended empty block cannot execute on. The block is not a no-op: every block runs the fork's mandatory system operations, so a chain that sabotages a system contract, or ends a fork transition with one still undeployed, admits no framework-built child - and unlike the header conditions the filler decides itself, this is only discoverable by executing the block, which is exactly the work the fill cannot do twice. The test author, who created the condition on purpose, declares it instead: pass sync_block=False where the chain is built, the same opt-out benchmark tests already use. One mechanism, three users. The two system-contract test generators opt out their affected parametrizations - the out-of-gas, revert and throw error modes (their sabotaged contract sits in canonical state; the GAS_LIMIT variant's chain is valid and keeps its block) and the deploy-after-fork deployment case (the chain ends with the contract undeployed, so the appended block's mandatory call fails with SYSTEM_CONTRACT_EMPTY). No test file changes: both generators own their blockchain_test call. An undeclared chain still fails the fill loudly: build failures are wrapped with advice naming the opt-out and the real error chained, so nothing is ever silently dropped (verified by temporarily removing the generator's opt-out: the fill fails naming sync_block=False with SYSTEM_CONTRACT_CALL_FAILED chained). Verified by targeted fills: EIP-7002 plus EIP-7251 error modes at Prague (20 engine_x fixtures, exactly the 6 error modes bare), EIP-8282 at Amsterdam (20, same 6), and the deployment tests at the Cancun-to-Prague transition (12, exactly the 4 deploy_after_fork cases bare). Full-tree fills pass from this commit on.

A dedicated explanation page, The Sync Block, in the filling-tests section: the two blocks to keep apart, why announcing the sync block makes a client sync (valid chains first, invalid chains second, and the contract that nothing checkable in isolation may be wrong with it), why an empty block is not a no-op, the three kinds of chain that fill bare and who decides each, why the filler never fakes a block, and the decision flow. The command-line reference keeps the flag and a short summary that links to it. No marker documentation: there are no markers to document. Also corrects the engine_x format reference, which described a syncPayload field the model did not have and told Engine API consumers to execute it. The real contract is the opposite: consumers replaying payloads through the Engine API must ignore the field - above a rejected head it is a sync target only, never an executable continuation - and sync-based consumers announce it instead of the chain's own head.

Benchmark chains were the one deliberate opt-out, and its stated rationale does not hold: the benchmark's gas and opcode values are recorded in the build loop before the sync block is built, and no benchmark consumer reads syncPayload. Removing the opt-out makes the sync block canonical for engine_x - present everywhere physics and semantics permit. It also makes benchmark chains servable by a sync-based consumer at all: they are mostly single-block, so without the appended block they fall below a sync consumer's two-block minimum and skip. With it, every benchmark becomes a client sync-path stress test - distinct coverage from replaying the same payloads through engine_newPayload. Verified on tests/benchmark/compute/instruction/test_bitwise.py at Osaka (gas-benchmark-value 1M): all 20 engine_x fixtures carry the salted block above their single-block chains, at roughly 1.25x the fill time of the same family with --no-sync-block. Kept as its own commit so it can be reverted on its own. If problems surface after merge, benchmark releases can also opt out at fill time with --no-sync-block - the flag stays precisely as that kind of lever.

The twelve-second step a block takes above its parent when a test pins no timestamp was a module constant in the blockchain spec. It is fork arithmetic like the gas floor: mainnet's slot time since the Merge, and a value a future fork can change. Move it into the fork classes as Fork.block_time(), next to minimum_block_gas_limit(). Both consumers read it from the parent's fork, of necessity: a block's own fork cannot be resolved until its timestamp is known, and its timestamp is the parent's plus the step. Block.set_environment resolves the parent's fork from the environment it is given, and sync_block_context_unavailable resolves the head's fork before using the step for the timestamp ceiling and for the appended block's own fork - which is also why the type ceilings stay ahead of the child's fork resolution. The method is defined once, at Frontier, rather than as a slot time introduced at Paris. The consumers need a step at every fork - set_environment walks unpinned timestamps on pre-merge chains too, and published pre-merge fixtures have always stepped by twelve, so the value cannot change there - and a Paris-scoped method would force a pre-merge fallback constant, putting twelve in two places with a fork branch in every consumer. Pre-merge the step is a framework convention; from Paris on it equals the protocol's slot time, and a fork that changes the slot time overrides block_time() at itself.

@bigmem

The repo had three ways to strip an xdist group suffix from a node id, with three different contracts: _strip_xdist_group_suffix removed only the fill's own @t8n-cache-* groups and deliberately preserved author-set ones; the sync block's salt stripped every group; and pre_alloc's entropy derivation cut at the last @ unconditionally, which would silently truncate any test parameter containing @ (none exists today, checked by collection). The preserve behavior was not protecting anything - it was leaking distribution into published output. A test marked xdist_group("bigmem") and filled under -n --dist=loadgroup emits fixture ids ending in ]@bigmem: tests@v0.0.913 contains them (10 of 10 sampled push0 fixtures), while a serial fill of the same test does not, so a fixture's published identity depended on how the fill was distributed. The same asymmetry could desynchronize the phase-1 and phase-2 pre-alloc lookup keys whenever the two phases ran under different parallelism. Keep one function, _strip_any_xdist_group_suffix, in pre_alloc next to its lowest-level user, and point every consumer at it: fixture names and ids, the pre-alloc group keys and salts, the entropy derivation, and the sync block's salt. Everything derived from a node id is now independent of fill distribution. The visible consequence is deliberate: bigmem-marked fixtures lose the @bigmem suffix from their published ids in parallel fills, matching what a serial fill always produced.

Engine payload sequences can contain rejected attempts followed by valid siblings. A single appended payload can cover only one ancestry path. Emit an ordered syncPayloads list with one target above each expected-invalid leaf and, when valid, the final canonical leaf. Preserve authored payloads, head, and post state. Cover linear, terminal-invalid, and branching cases in unit and pytester tests.

Explain why Engine API directive sequences can branch, how fillers select leaves, and how consumers reconstruct each target path by hash. Rename the dedicated guide for sync payloads and update the CLI and fixture-format references.

danceratopz · 2026-08-18T14:18:07Z

Updated the PR with the new simpler approach; no test payloads are modified.

During self-review I learnt that we have tests today that include blocks that are siblings. This is typically used to test new fork behavior over a fork transition: The first block (before the transition) is invalid (feature unavailable) and the second block (after the transition) is valid. Both blocks build on the same valid block. Here's an example for eip7954_increase_max_contract_size/test_fork_transition.py: test_max_initcode_size_fork_transition. To test both paths via wirex, both paths need an appended sync block and this has been added to fill, at the cost of some complexity.

danceratopz added C-feat Category: an improvement or new feature A-test-fill Area: execution_testing.cli.pytest_commands.plugins.filler labels Aug 12, 2026

danceratopz mentioned this pull request Aug 12, 2026

feat(test-consume): add consume wirex, a devp2p full-sync simulator #3365

Draft

2 tasks

spencer-tb mentioned this pull request Aug 13, 2026

EL Specs Calls, Tue 11th August & Thu 13th August, 2026 ethspecs/pm#102

Closed

danceratopz added 11 commits August 17, 2026 17:23

docs(test-fill): document plural sync targets

230fddf

Explain why Engine API directive sequences can branch, how fillers select leaves, and how consumers reconstruct each target path by hash. Rename the dedicated guide for sync payloads and update the CLI and fixture-format references.

danceratopz force-pushed the experiments/wirex-fill branch from 06471b1 to 230fddf Compare August 18, 2026 12:43

danceratopz added 2 commits August 19, 2026 13:44

docs(test-fill): clarify sync payload documentation

7f31370

fix(test-fill): decline the sync block above all type-ceiling heads

9fb28fe

danceratopz force-pushed the experiments/wirex-fill branch from d921575 to 9fb28fe Compare August 19, 2026 12:26

refactor(test-spec): preserve sync payload leaf hash types

707458c

danceratopz mentioned this pull request Aug 20, 2026

refactor(test-fill): use the shared sync-payload builder for sync fixtures #3405

Draft

2 tasks

Navigation Menu

Uh oh!

feat(test-fill): make engine_x fixtures consumable via devp2p sync - #3364

feat(test-fill): make engine_x fixtures consumable via devp2p sync#3364
danceratopz wants to merge 14 commits into
forks/amsterdamfrom
experiments/wirex-fill

danceratopz commented Aug 12, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

danceratopz commented Aug 13, 2026

Uh oh!

danceratopz commented Aug 14, 2026

Uh oh!

danceratopz commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

danceratopz commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Fixture shape

PR-specific notes

Related Issues or PRs

Checklist

Cute Animal Picture

Uh oh!

codecov Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

danceratopz commented Aug 13, 2026

Uh oh!

danceratopz commented Aug 14, 2026

Uh oh!

danceratopz commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

danceratopz commented Aug 12, 2026 •

edited

Loading

codecov Bot commented Aug 12, 2026 •

edited

Loading