Feat/kurtosis testing - #1
Merged
Merged
Conversation
…nce correctly broken atm)
JasonVranek
added a commit
that referenced
this pull request
Aug 3, 2026
…acklog CHECKS.md: add feature.timing_games / .extra_validation / .skip_sigverify to the catalog table + a dedicated section; rewrite the "Law 3 gap" caveat as mostly-closed with skip_sigverify the residual (negative codepath, honest WARN). Backlog: mark #1/#9/#10/#11 done; file the bad-signature helix mock as the follow-up that would make skip_sigverify a real ON/OFF test.
JasonVranek
added a commit
that referenced
this pull request
Aug 11, 2026
* fix(helix): track latest helix-relay:main config schema + add setup
runbook
Reconcile the embedded helix config with current helix-relay:main:
- drop the removed `network_config: !Custom {...}` block (the relay now
fetches
chain spec + genesis from the beacon node at startup)
- migrate `cores` to the 10-field CoresConfig (drop sub_workers; add
decoder,
simulator, top_bid, data_gatherer, block_merging, housekeeper)
Bump the ethereum-package fork to the helix wait-for-genesis fix. Add
docs/local-kurtosis-e2e.md, a full runbook (build CB image -> deploy
devnet ->
run cb-basic -> verify) with every gotcha (kurtosis 1.18.1 pin, the 3
helix fixes,
pin-vs-latest coupling).
* feat(sim): P1 harness — preflight gate + triage (observability-first)
Introduce the `sim` Rust binary: the agent-drivable launch->triage->diagnose
loop the release flow never had. Library-first (extract src/lib.rs so cb-verify,
cb-orchestrator, and sim share one core, killing the duplication at the source).
- `sim preflight <args-file>`: render + have the REAL relay/CB image parse the
config in ~1s, before any ~10-min devnet spend. 3-valued verdict
(Pass | Fail{field} | Inconclusive{reason}) so config-schema drift is caught
as a labeled failure, never a masked runtime panic minutes into launch. Env /
pre-genesis / pull issues classify Inconclusive, not a false Fail.
- `sim triage <enclave>`: attach to a broken enclave, dump each non-RUNNING
service's ROOT panic via a docker-logs fallback that pierces kurtosis's
grpc/masking. Structured TriageReport JSON.
- Gate wiring in run-and-verify.sh: preflight blocks launch on Fail; triage
auto-fires on any launch failure and surfaces its JSON. Known digest-window
limitation documented at the site (deferred to `sim run` in P2).
Proven end-to-end vs ghcr.io/gattaca-com/helix-relay:main: valid cb-basic.yml ->
pass/exit0/~8.5s; renamed `hostname` field -> Fail{hostname}/exit1/~0.3s.
77 tests green (pure cores TDD'd on real + held-out fixtures; IO wirings are
Docker/kurtosis smoke checks per Law 4).
Also lands docs/NORTH-STAR.md (ratified direction: full-Rust sim, owned helix
mirror, dogfood one fork) and the P1 plan.
* style(clippy): clear pre-existing warnings so CI can enforce -D warnings
Mechanical clippy fixes across the pre-sim code (collapsible_if -> let-chains,
while_let_loop -> for/by_ref, manual_strip -> strip_prefix, unnecessary_unwrap,
useless_format/vec, needless_borrow, expect_fun_call). No behavior change; 77
tests still green.
Two justified #[allow]s (with comments): too_many_arguments on the enclave
pipeline launcher (an options struct would read worse), and dead_code on two
relay data-API Deserialize targets whose unread fields document the wire schema.
`cargo clippy --all-targets -- -D warnings` now passes clean, so every
subsequent commit builds under a real warning gate.
* docs(plan): P2 pivots from typed mirrors to Rust consolidation (post-grill)
Three adversarial reviews + a direct diff killed the typed-serde-mirror plan:
the "6 duplicated templates" premise is false (helix is byte-identical across
all 6 scenarios; CB varies <=7 lines), typed helix gains no guard (types aren't
importable; preflight stays the only check), and the serde mechanism is fragile
(sentinel collisions: 2000 vs late_in_slot_time_ms, db_name==user=="helix").
A code trace also refuted "just delete templating" — the ethereum-package fills
the {{ }} holes from runtime service-discovery values at launch.
New plan: port the (already-DRY) Python templates verbatim into Rust const
strings (holes stay literal — no serialization, no sentinel hazard); type only
the assembly (Scenario enum + one Images map). Byte-identity to golden is the
oracle. Fixes the live-wrong commit-boost/pbs image default along the way.
* test(sim): hermetic golden config fixtures + byte-diff harness (P2 Task 0)
The 6 Python-generated configs that produced the green e2e run, snapshotted with
the baked-default images (commit-boost/commit-boost, public helix) so the fixtures
are hermetic — not tied to the box's .env. `genmodel::assert_matches_golden` is the
acceptance oracle for the verbatim port to come: byte-identity, naming the first
differing line on mismatch. A guard test asserts the pbs image bug is absent from
the fixtures.
* feat(sim): sim generate — port config generation into Rust (P2 Task 1)
Ports scripts/generate_kurtosis_configs.py into the sim binary. The config
bodies are verbatim const/string templates (helix keeps its ~40 lines of
binary-verified drift comments; the {{ }} runtime holes stay literal — the
ethereum-package fills them at launch). Typing lives only at the assembly
layer: a Scenario enum + one Images map with the CORRECT commit-boost/commit-boost
default (fixing the live-wrong commit-boost/pbs Python default).
All 6 scenarios reproduce their golden fixtures byte-for-byte (the acceptance
oracle); cb-basic still passes sim preflight against the real helix image.
Review fixes folded in (adversarial review before commit):
- load_pubkeys returns Result; run assembles ALL bodies before writing any, so a
missing/malformed mux keys file fails cleanly with nothing written (matches the
Python's pre-write all-or-nothing), instead of panicking mid-generation with 5
stale files left behind.
- run_in(keys_dir, env_path) injectable core for hermetic negative-path tests.
- Scenario::ALL reordered to the true Python emission order; misleading
generate_all_matches_goldens renamed to reflect it checks IO faithfulness (the
hermetic golden match lives in scenario.rs on default images).
43 sim tests green (incl. missing-keys → clean-error + atomicity); clippy -D clean.
* refactor(sim): retire the Python config generator; sim generate is the one source
Deletes scripts/generate_kurtosis_configs.py (Law 1's root smell — it
reverse-engineered helix's serde layout from binary panics) and the stale
configs/example-kurtosis-config.yml hand-copy. `just generate-configs` now runs
`sim generate`; the tracked cb-basic.yml regenerates byte-identically.
Doc repointing: README image-default table fixed to the baked defaults
(commit-boost/commit-boost, public helix — no more commit-boost/pbs), the
config-gen section + repo-layout tree point at `sim generate`, and the runbook's
helix-drift fix instruction now targets src/bin/sim/genmodel/helix.rs's
HELIX_RELAY_CONFIG const instead of the deleted Python template.
* docs(sim): mark P2 landed (consolidation) + flag Law 1 tension for J
NORTH-STAR staged-plan P2 now reflects what actually shipped (verbatim port +
assembly typing, not cb_common mirrors) with a note that the grill outcome
tensions with Law 1's "built from cb_common structs" premise — J's call to
revise the Law. P2 plan doc stamped LANDED with the commit trail.
* docs(sim): record CB-preflight probe — partial coverage, defer trust call to J
Probed commit-boost/commit-boost:kurtosis: CB parses eagerly + errors structured
(chain-file-not-found at types.rs:442 is the clean post-deserialize PASS marker,
no spec mount needed). BUT [pbs] is #[serde(flatten)] + const-defaults, so a
renamed pbs field is silently defaulted and reaches PASS — a CB preflight would
false-pass the likeliest drift class. Not shipping an instrument that can lie
(pilot-breaks-the-instrument); deferred to J with the honest-Inconclusive vs
partial-preflight tradeoff written up.
* test(sim): guard the tracked cb-basic.yml against drifting from sim generate
configs/generated/cb-basic.yml is tracked (render.rs's fixture + what preflight
validates); a new test asserts sim generate reproduces it, so a hand-edit or a
stale regen fails CI instead of silently diverging — the staleness class that
rotted the old example config.
* test(checks): Law 4 guard-branch tests for chain_health + payload_matching
Adds decision-logic unit tests for the two Tier-1 checks that had zero:
- chain_health: inverted-range → Fail; single-slot window → Skip (not a false pass on zero data)
- payload_matching: empty relay set → Skip (not a false pass on zero comparisons)
Only the pre-fetch guard branches are unit-testable today — both checks inline
their real decision logic inside async fetch fns (no pure classifier seam, unlike
cb_metrics.rs). The deeper boundaries stay uncovered pending a classifier
extraction (flagged for P3). No production logic changed.
* docs(plan): P3 check-trustworthiness — 3 confirmed false-greens + fix approach
Consolidates the three code-verified false-greens (mux_routing pass-gate,
relay_pipeline + payload_matching first-wins-union-by-slot) with their shared
structural root cause: decision logic inlined in async fetch fns → no test seam →
bugs hide. Fix = extract pure classifiers (the healthy cb_metrics.rs pattern),
which serves Law 3 (no false greens) and Law 4 (testable verdict math) at once.
Lays out the per-check judgment calls (warn-vs-fail, tier, CB-debug-logging) for
J. Proposal only — verdict-changing, not started autonomously.
* feat(sim): sim generate --check — CI/agent drift gate for the configs
`sim generate --check` regenerates in-memory and verifies the on-disk configs
already match, exiting nonzero (naming the drifted files) without writing —
so CI or an agent can assert the checked-in configs are current in one command,
instead of only catching drift via `cargo test`. Shares the assembly path with
`run` so both fail identically on a bad mux keys file.
* fix(ci): repoint integration nightly off the deleted example config
Deleting configs/example-kurtosis-config.yml in the P2 retire commit left
integration.yml pointing at a nonexistent file (my Task 2 grep missed .github/).
Fix: generate cb-basic via `sim generate` and launch that, pinning the public
commit-boost image the old example used (the nightly builds no CB image). Also
align .env.example's documented defaults with Images::default and drop its
reference to the retired Python generator. Flagged in-workflow: confirm the
public pbs:latest tag is SSZ-current or switch to building the image.
* style: cargo fmt the tree (CI fmt-check was red)
This session's subagent edits (P1 sim bins, the clippy cleanup, P2 genmodel, the
Law 4 tests) weren't run through rustfmt, so `cargo fmt --check` (a CI job) was
failing. Pure formatting — rustfmt is AST-based and all 99 tests still pass;
byte-identity of `sim generate` output is unaffected (string literals untouched).
The branch now passes check + test + clippy + fmt-check.
* docs(plan): ratify J's P3 verdict-fix decisions + implementation findings
mux → WARN when unverifiable (debug logging already on in generated config);
payload → detect per-(relay,slot) conflict, verdict stays WARN; best-bid →
new arm comparing per-relay bid values from CB logs (keep delivered-count as
coverage); CB-preflight → keep honest Inconclusive.
* fix(checks): mux.routing WARNs when no routing decision is verifiable (P3)
The false-green: mux.routing PASSed whenever there were no misrouting violations,
even when ZERO routing decisions were actually checked (CB "using mux config"
DEBUG events absent) — reporting "All N verified" where N counted raw log lines.
Fix (J's call): extract a pure `classify_mux_routing` (the Law 4 test seam) and
count `routing_decisions_verified` = events with a known pubkey AND a mux_id. If
that's 0, WARN ("need CB debug logs") instead of PASS; PASS now counts verified
decisions, not log lines. Debug logging is already on in the generated cb-mux
config, so real runs stay green — only the unverifiable case flips to WARN.
4 decision tests added (pass / warn-unverifiable / warn-no-events / fail-misroute).
* fix(checks): payload_hash_match detects cross-relay conflicts (P3)
The false-green: `by_slot.entry(slot).or_insert(hash)` was first-wins union, so
when two relays reported DIFFERENT block_hash for one slot (relay equivocation —
the thing the cross-check exists to catch) all but the first were dropped before
the on-chain compare, and the verdict was order-dependent.
Fix (J's call): collect every (relay, hash) per slot; extract a pure generic
`classify_payload_matches` (Law 4 seam); WARN on a cross-relay conflict OR when
no relay hash matches chain. `missed` stays informational (doesn't downgrade),
matching prior behavior. 5 decision tests (u64 stands in for B256).
* feat(checks): relay.best_bid — verify aggregated bidding by comparing bid values (P3)
The false-green: check_payloads_delivered_multi unions by slot and counts distinct
delivered slots, so one delivering relay scored identically to genuine two-relay
aggregation — the "aggregated bidding" the multi-relay scenario exists to test was
never actually checked.
Fix (J's call): keep the delivered-count as a COVERAGE check and ADD a real
best-bid arm. check_best_bid_selection gathers each relay's best bid per slot (max
of its builder-blocks-received) + the delivered value, then classify_best_bid (a
pure Law 4 seam) asserts: WARN if no slot had >=2 relays competing (aggregation
never exercised — the anti-false-green), WARN if a competitive slot delivered LESS
than the best available bid, else PASS. 4 decision tests (u64 stands in for U256).
Data-semantics assumption to confirm in review: a relay's offered bid = max of its
builder_blocks_received for the slot; delivered value is comparable to bid value.
* fix(checks): payload conflict message no longer over-claims 'cross-relay'
A single relay reporting two hashes for one slot is also a conflict; the message
now says 'N distinct block hashes reported' rather than asserting cross-relay
disagreement. Verdict unchanged (WARN).
* revert: back out relay.best_bid — unsound data source (P3 review)
The adversarial review found check_best_bid_selection's data source wrong:
get_builder_blocks_received returns EVERY builder submission a relay received
(including bids that failed simulation and were never offered to the proposer),
so max(builder_blocks) overstates the relay's offered bid and would false-alarm
"value left on the table" on correct runs. The ~11-slot sampling also made the
"no competition" WARN the default even when aggregation works.
The pure classify_best_bid verdict logic was sound; only the IO source was wrong.
Correct design (P3 follow-up): source per-relay bids from the CB "received new
header" log events (relay_id + slot + value_eth, already parsed by
parse_cb_log_line, full-coverage not sampled), parse value_eth decimal->wei for
Ord, and compare the delivered value to the max OFFERED bid per slot. That needs
CB-log context (enclave + cb_service_names) plumbed into the check — a focused,
separately-reviewed piece rather than a rushed trust-core patch.
check_payloads_delivered_multi stays as the coverage check. mux.routing and
payload_hash_match fixes are unaffected.
* docs(plan): P3 status — mux+payload landed, best-bid backed out pending CB-log source
Records the adversarial-review outcome: WARN is non-fatal (won't break CI), the
best-bid data-source problem + correct follow-up design, and the lenient gaps
left within J's ratified scope.
* feat(checks): relay.best_bid v2 — bids from CB getHeader logs (P3 review fix)
Re-adds the aggregated-bidding check with the CORRECT data source the review
demanded. Per-relay OFFERED bids now come from CB's own "received new header"
getHeader log events (relay_id + slot + value_eth, full-coverage, already parsed
by parse_cb_log_line) — what CB actually compared — instead of the relay data
API's builder_blocks_received (which includes bids that failed sim and were never
offered, and was slot-sampled). Fixes both review findings: no false "value left
on the table" alarms, and no sampling-induced "no competition" default.
value_eth is parsed decimal->wei WITHOUT float (exact); bids and delivered are
compared in GWEI so sub-gwei cross-source rounding can't spuriously warn. Single-
relay runs SKIP. The pure classify_best_bid seam is unchanged. New module
src/checks/best_bid.rs; wired in main.rs alongside the mux check (needs CB-log
context). 4 tests incl. exact value_eth parsing.
* fix(checks): best_bid — verify only slots with delivered data; drop gwei (P3 review 2)
Second review of best_bid v2 found the crux sound ("received new header" IS the
per-relay offered bid) but caught a reintroduced Law-3 false-green: competitive
slots with NO delivered payload (out of window / missed) counted toward the PASS
"verified across N" claim. Now the verdict is based on VERIFIED slots (competitive
AND delivered present); if none are verified → WARN, never PASS. Also: window-filter
the bid harvest to [start,end]; drop the gwei rounding (both sources are exact wei,
so it only added a boundary artifact) and compare wei directly; soften the
suboptimal message (dropped the unfounded "misrouting" claim). +1 guard test.
* feat(just): human UX — build-cb-image, pull-images, one-command e2e
Closes the from-scratch gap: cb-testing had no recipe to build the CB image
(it lives in the sibling commit-boost-client repo) or pre-pull the public images.
- build-cb-image [tag] [cb_dir] — delegates to ../commit-boost-client `just build-all`
- pull-images — pre-pull helix/reth-rbuilder/lighthouse (public; helix isn't built)
- e2e [config] — generate-configs + pull-images + testnet, one command
README quickstart leads with the build-cb-image (once) -> e2e path.
* perf(run): pre-build cb-verify before launch + host-memory advisory
Two robustness fixes surfaced by the live-devnet validation:
- Pre-build cb-verify (release) BEFORE `kurtosis run` and invoke the built
binary, instead of `cargo run --release` mid-devnet — that compile is a
multi-GB spike while 10 services are live. Keeps it off the critical path.
- A host-memory advisory before launch (non-blocking; LOW_MEM_ABORT=1 to abort).
NOTE: this is a general host-resource guard, NOT the cause of the relay OOMs
seen on 2026-07-31 — those were per-container cgroup OOMs (CONSTRAINT_MEMCG) at
the relays' own RELAY_MAX_MEMORY cap, addressed separately in the fork.
* feat(sim): drop the flashbots relay — multi-relay = two helix instances
Helix is industry-predominant; the flashbots mev-boost-relay is a memory hog
(leaked ~825MB/min under spamoor). So the multi-relay scenarios (cb-multiple-
relays, cb-timing-games, cb-mux) now run TWO helix instances instead of
helix+flashbots. flashbots stays as the block BUILDER (reth-rbuilder).
- scenario.rs relays() -> ["helix","helix"] for the three multi-relay scenarios;
fixed the now-inaccurate comment prose baked into the configs.
- cb.rs mux labels node_1_to_flashbots/mux_flashbots -> node_1_to_helix/mux_helix_1
(cosmetic; routing is positional {{ index .Relays 0/1 }}).
- Regenerated the 3 multi-relay goldens (now the intended 2-helix output, not the
old Python baseline — mod.rs provenance updated).
- Bumps the ethereum-package submodule to the N-instance launcher (fbe3141).
Validated on a live devnet: helix-relay-2 + helix-relay-3 both survive, and every
multi-relay check PASSes on real data — best_bid 33 competitive/33 verified,
payload_hash 33 matched/0 conflicts, delivered across 2 relays. (Residual overall
FAIL is transient warmup 5xx: 1 get_header + 8 submit, down from 42+123 when the
flashbots relay was OOM-dying.)
* docs: sweep synthesis + new ARCH/CHECKS/fork-delta + fix flashbots-era staleness
Beefs up documentation after the P1/P2/P3 + 2-helix work, from a six-lens sweep:
NEW cross-cutting docs (the biggest gaps):
- docs/CHECKS.md — authoritative per-check catalog: tier, pass/warn/fail contract,
data source, and the load-bearing verdict rule (exit keys ONLY on tier-1 FAIL;
WARN/SKIP non-fatal, so a consumer must parse JSON result, not just exit code).
- docs/ARCH.md — how it all fits: sim generate → run-and-verify → cb-verify
(discovery→probes→checks→report) → fork; module map; key decisions.
- docs/fork-delta.md — the ethereum-package divergence from upstream (mev_resolver
component model, helix wait-for-genesis + N-instance + 8GB cap, rebase notes).
- docs/SWEEP-BACKLOG.md — the prioritized backlog (bugs/docs/tests/perf/refactor/
features); docs/plans/INDEX.md — landed-vs-live plan classification.
FIXES to now-stale docs:
- README + .env.example: multi-relay = two helix instances (flashbots relay dropped,
builder only); added relay.best_bid + sim verb contract + --skip-finalization-check
+ a pointer to CHECKS.md.
- Re-stamped P1/P2/P3 plan statuses (all shipped; read "uncommitted/backed-out").
* fix(orchestrator): cb-verify flag is --config, not --cb-config
cb-orchestrator (just test-all/test-one) invoked the cb-verify binary with
--cb-config, which cb-verify's clap doesn't define, so clap rejected it and the
batch runner failed at every check invocation. Surfaced by the doc sweep.
* fix(checks): C1 relay-death false-green + H3 mev-rate first-relay undercount
C1 (CRITICAL): when all relays are unreachable at check time (mid-run OOM death —
the exact scenario this repo exists to catch), run_relay_checks emitted SKIP for
the tier-1 relay.payloads_delivered_multi, and exit_code treats a tier-1 SKIP as
pass → the run exits 0/PASS with the MEV pipeline unverified. Extracted a pure
`all_relays_dead_results` that FAILs the tier-1 delivery check (tier-2/3 stay SKIP,
already gated). Unit-tested.
H3: check_mev_delivery_rate broke on the first relay that answered the data API,
so under mux (each relay holds only its half of deliveries) it undercounted and
spuriously WARNed. Now unions delivered payloads across ALL relays (like
check_payloads_delivered_multi); SKIP only if NO relay answered.
* fix(checks): H2 warmup-5xx false-red — rate-based tolerance, not any-5xx-fails
The cb_*_matrix checks read absolute CUMULATIVE relay status-code counters (incl.
the pre/early-window warmup phase) and FAILed on any 5xx>0, escalated to tier-1 —
so a handful of transient warmup 5xx flipped a genuinely-healthy run to red (the
df69c90 residual). classify_endpoint now classifies on the 5xx RATE: > MAX_5XX_RATE
(25%) or --strict → FAIL (materially broken relay/CB-to-relay link); a nonzero but
low rate → WARN (surfaced, non-fatal warmup noise). Test updated to the new
contract + a high-rate FAIL case.
* fix(checks): M4 finality gate + extract chain_health classifier seams
M4: FINALITY_POSSIBLE_AFTER_SLOT 96 → 160 (5*32). check_finality FAILs unless
finalized_epoch>=2, but epoch 2 first finalizes near the end of epoch 4 (~slot
160), not epoch 3 (slot 96) — justify(N+1)/finalize(N+2) lag. So any window
ending in ~[96,160) ran the check on a chain that had only finalized epoch 1 →
false tier-1 FAIL on a healthy chain. Default window (end_slot 288) unaffected.
Extracted pure Law-4 seams with tests: classify_finality, classify_missed_slots
(rate pass/warn), classify_cb_running (inspect-grep running/found-down/none).
* fix(relay): M6 pagination robust to helix ordering/cursor (not flashbots)
get_payloads_delivered assumed the flashbots data API's descending order + an
advancing cursor. Helix's contract is unverified, risking undercount (ascending
order → break after page 1) or 50× refetch (ignored cursor). Added a pure
page_made_progress seam: stop when the cursor doesn't advance (bounds an
ignored-cursor refetch to 2 pages) or when an in-range page adds no new slots
(order-agnostic); collect in-range rows into a HashSet to dedup; short-page break
keeps the normal single-page window fast. 5 tests; module doc updated to helix.
* fix(ci): nightly integration — enclave-name bug, matrix, reproducibility, runner note
- Pass --enclave "$ENCLAVE" to run-and-verify.sh so launch/verify/teardown all use
the same enclave (it was set but never passed → teardown targeted the wrong
enclave and the real one leaked).
- Matrix over [cb-basic, cb-mux] (fail-fast: false), each in its own enclave — the
mux/multi-relay/best_bid e2e paths had zero coverage before.
- CB image via a cb_image workflow_dispatch input (schedule falls back to the
default) instead of a bare inline :latest; REPRODUCIBILITY GAP + TODO to pin a
digest documented inline.
- Flagged the runner-memory requirement: a full devnet (esp. cb-mux's 2x8GB helix)
won't fit a 7GB GitHub-hosted runner — needs a large/self-hosted runner.
* fix(main): M5 drop the dead flashbots-Postgres post-mortem salvage
The preflight salvage queried mev-relay-postgres / db mev_boost_relay / table
mainnet_payload_delivered — all flashbots-mev-boost-relay specifics, dead after
the 2-helix migration (relays are helix-relay-postgres-N with a different schema).
So it always returned empty → the run failed with a MISLEADING "post-mortem found
no delivery records" error instead of the real cause. Now that C1 makes relay
death a clean tier-1 FAIL, dropped the salvage: relays-dead-at-preflight proceeds
to the window and the relay checks report the failure. The orphaned discovery
salvage fns are removed in the dead-code sweep (task 7).
* refactor: delete duplicate diagnostic bins + confirmed-dead code
Part A — deleted test_mux.rs (532 lines) + test_relay.rs (291 lines) and their
Cargo [[bin]] blocks: they re-implemented library code (better-tested in
mux_routing/discovery/relay), and test_mux's routing loop predated the P3
no-false-green fix — a REGRESSED copy that could print false greens. `just
test-mux` repointed to `cb-verify --config`.
Part B — dropped the crate-level #![allow(unused_imports/dead_code)] masks and
deleted zero-call-site dead code: metrics sum/has/values helpers; beacon
genesis/seconds-per-slot helpers; relay is_validator_registered; orchestrator's
write-only EnclaveState; discovery's write-only relay_identities field + the
orphaned flashbots post-mortem (PostMortemRecord/query_mev_relay_postgres/
parse_postmortem_output + tests — the M5 cleanup); diagnose CauseKind::Unknown.
No verdict/check logic changed. 77+5+45 tests green, clippy -D + fmt clean.
* docs: DEVELOPING.md (how to add checks/scenarios + dev loop) + README getting-started
New docs/DEVELOPING.md: the contributor guide — dev loop/prereqs, the Law-4
classify_* seam pattern for adding a check (with the anti-pattern named), the
Scenario/Images/golden flow for adding a scenario, the verdict contract, links to
ARCH/CHECKS. Fixes three real newcomer-path gaps in the README quickstart: the
Kurtosis 1.18.1 pin (not ">=0.90"), the submodule-init step missing from the
ordered flow, and the read-the-report/exit-code pointer. Also drops the deleted
test-relay refs + repoints test-mux to cb-verify.
* perf: thin-LTO + parallelize serial beacon loops + parsed-port precedence
No verdict/logic change — only how fast data is gathered:
- Cargo.toml: lto=true → "thin" (fat LTO buys nothing at runtime for an I/O-bound
binary but added 30-90s to every --release relink); tokio "full" → the used
feature set; futures added as a direct dep.
- The 3 serial per-slot beacon loops (check_mev_delivery_rate, check_missed_slots,
check_payload_hash_match) now fetch with buffer_unordered(16) — ~2s → ~0.2s each.
Folds are order-independent (commutative counters / slot-keyed maps), so the
CheckResult is byte-identical; classifiers untouched.
- discovery: try the already-parsed port before spawning a `kurtosis port print`
subprocess per port (fallback preserved) — removes ~20-40 startup subprocesses.
(check_cb_running's second inspect kept: its check-time freshness is load-bearing —
reusing startup liveness would false-green a CB that dies mid-run.)
* feat(report): provenance block + extract report::tier1_failed with exit-code tests
VerificationReport now carries an optional Provenance { config_path,
config_hash, images[] } gathered at run start, so a report is self-describing:
a consumer (or `sim diff`) can tell WHICH config + images produced it without
re-deriving from the environment. config_hash is a DefaultHasher over the
config bytes; images are the resolved {role, name, id} refs.
exit_code logic is extracted to report::tier1_failed (the tier-1 FAIL gate that
main's verdict keys on) and pinned by 7 unit tests covering the full matrix:
tier-1 FAIL -> 1, no tier-1 ran -> 2, WARN/SKIP/PASS-only -> 0. Previously this
verdict was inline and untested — the one number a CI consumer trusts.
* refactor(checks): CheckStatus worst-status Ord + classify seams for relay_pipeline
CheckStatus gets an explicit `Ord` (severity Fail>Warn>Pass>Skip, NOT derive
order) so worst-status aggregation is `.max()` over an iterator instead of a
hand-rolled fold. Two of the three folds in run_relay_checks are rewired to it;
the third is deliberately left (it's best-of-Pass-wins + Skip->Fail collapse,
the opposite of worst-status — a comment marks why).
Two pure verdict seams extracted (Law 4) with both-sides tests:
classify_mev_rate(mev_blocks, total, threshold) and
classify_registrations(registered, missing). discovery gains
test_parse_services_from_fixture pinning the kurtosis text-table parse.
13 new tests (lib 84 -> 97).
* feat(sim): `sim checks --list [--json]` catalog + `sim doctor` host preflight
Two agent-facing discovery verbs on the sim binary.
`sim checks --list` emits a machine-readable catalog of all 17 cb-verify
checks — {id, tier, title, data_source, feature_asserted, severity_note} —
so an agent can learn the harness contract (and the verdict rule: exit code
keys only on tier-1 FAIL; WARN/SKIP non-fatal) without reading source or
CHECKS.md. feature_asserted=true only for mux.routing and relay.best_bid.
Hand-maintained (no single check registry exists to derive from yet); a module
comment marks the sync obligation with src/checks + docs/CHECKS.md.
`sim doctor` is a host-prerequisite preflight: kurtosis installed + pinned
1.18.1 (WARN on newer, citing the config-version-9 clash), docker reachable,
>=18GB mem+swap headroom, CB image present, ethereum-package submodule
non-empty. Pure classify(&Probes)->Report split from the IO gather layer;
only kurtosis+docker are hard-fail, the rest advisory.
14 new tests (sim bin 45 -> 59).
* feat(sim): `sim diff` — verdict/provenance regression gate between two reports
Makes the report a round-trippable JSON interchange format (report + check
types now derive Deserialize, with #[serde(default)] on the optional/skipped
fields) and adds `sim diff <a.json> <b.json> [--json]`.
diff_reports (pure Law-4 seam) compares two reports: per-check verdict changes
classified by CheckStatus severity (Regressed = severity up, Improved = down),
added/removed checks, config_hash delta, and per-role image id/name deltas.
Exits nonzero iff a check got a strictly-worse verdict — the CI shape being
"bump an image, re-run, sim diff old new, fail on regression". A check dropping
to SKIP is surfaced but not a regression (the report's own tier-1 gate owns
pass/fail, not the diff).
9 unit tests incl. a serialize->deserialize->diff round-trip (the point of the
Deserialize derive). sim bin 59 -> 68.
* feat(checks): Law-3 feature-fired assertions for the toggle scenarios
skip-sigverify / extra-validation / timing-games were non-tests: they enable a
CB feature then assert only what cb-basic does. Add a config-driven check that
proves each enabled feature's codepath actually FIRED at runtime, via CB debug
logs (the scenarios already set [logs.stdout] level = debug):
- feature.timing_games -> PASS on >=1 "TG:" debug line (send_timed_get_header)
- feature.extra_validation -> PASS on >=1 "fetched parent block" line
- feature.skip_sigverify -> ALWAYS an honest WARN: it is a NEGATIVE codepath
(sigverify simply not called, no success log/metric) and is indistinguishable
from OFF on the happy path without a bad-signature relay. Never falsely green.
A marker feature enabled-but-unobserved WARNs (could be no getHeader in the
window), never FAILs — the same no-false-red discipline as the mux check. All
verdict logic is a pure classify_* seam (9 tests). Wired into the run pipeline
behind the same --config gate as mux.
mux_routing gains two shared primitives (read_cb_config_template,
fetch_filtered_logs) reused here; fetch_service_logs keeps its signature so the
diagnose caller is untouched. sim checks catalog + its feature-invariant test
updated to the five feature-asserting checks. lib 97 -> 106.
* docs: record Law-3 feature-fired checks + sim diff in CHECKS.md and backlog
CHECKS.md: add feature.timing_games / .extra_validation / .skip_sigverify to
the catalog table + a dedicated section; rewrite the "Law 3 gap" caveat as
mostly-closed with skip_sigverify the residual (negative codepath, honest WARN).
Backlog: mark #1/#9/#10/#11 done; file the bad-signature helix mock as the
follow-up that would make skip_sigverify a real ON/OFF test.
* docs: live-validation findings from the cb-timing-games devnet run
Records that this session's shipped work validated on real infra (provenance
populates with real image sha256s; feature.timing_games PASS on 1342 TG: log
lines; best_bid PASS on 37 competitive slots; H2 rate-classifier discriminates).
Files the timing-games get_header 47.5% 5xx tier-1 FAIL as a J design call:
real high 5xx rate (not warmup) from aggressive polling, but the delivery
pipeline was fully green — is it a false-red for the scenario? Not patched.
* docs: confirm timing-games get_header FAIL is scenario-specific (run 2 clean)
The second devnet run (cb-extra-validation, normal scenario) had
cb_get_header_matrix PASS with 0 get_header 5xx → overall PASS. Confirms the H2
rate-classifier does not false-fail a green run, and the timing-games 47.5% 5xx
is the aggressive config's own behavior. Also validated live: feature.extra_
validation PASS, best_bid SKIP on single-relay, and sim diff on two real reports.
* fix(cb_metrics): stop counting CB's synthetic timeout code 555 as relay 5xx
Root cause of the timing-games false-red: CB increments its status-code
counter with TIMEOUT_ERROR_CODE = 555 when IT cancels a request at its own
deadline (commit-boost constants.rs) — not a relay-served status. bucket_code
lumped any 5-prefixed code into 5xx, so timing-games (which cancels late polls
at the 400ms budget BY DESIGN) read as "47.5% relay 5xx" and tier-1-failed a
run whose relays served zero errors.
Live-confirmed with a dedicated capture run (raw counter before bucketing):
get_header = 200x48 / 204x6 / 400x4 / 555x42, ZERO real 5xx; helix logs 0
server errors; CB log 45 timeout mentions.
Fix, scenario-agnostic (no timing-games exemption):
- 555 buckets as `timeout`, its own category.
- timeouts > 25% of requests -> WARN, never FAIL, not even under --strict
(client-side deadline policy, not a pipeline error).
- real 5xx keeps the rate-based FAIL with timeouts EXCLUDED from the
denominator, so a genuine error storm still fails amid heavy timeout polling.
- get_header PASS detail notes sub-threshold timeout counts.
4 new both-sides tests (timing-games shape warns; strict doesn't promote;
sub-threshold passes with note; real 5xx still fails despite 90 timeouts).
lib 106 -> 110. CHECKS.md matrix section + stale pre-H2 caveat rewritten;
catalog severity_note updated.
* docs: live-confirm the 555 timeout fix — timing-games now overall PASS (exit 0) with annotative 44.4% timeout WARN
* feat(scenarios): divergent-bid 2-helix — cb-multiple-relays gets per-relay subsidies
Closes the degenerate-competition gap: one shared rbuilder submitted the
IDENTICAL bid to both helix instances, so relay.best_bid's "delivered >= best
offered" was a tie that proved nothing about CB's selection.
rbuilder upstream (which ethpandaops/reth-rbuilder:develop builds, no fork)
supports per-relay bid values from ONE instance via top-level
[[subsidy_overrides]] entries name-matched to [[relays]] blocks — each override
seals its own bid of true_block_value + subsidy. No second builder participant
needed; the previously-mapped 5-file two-builder surgery is obsolete.
- ethereum-package (submodule fc5e6a2): mev_builder_subsidy accepts a LIST
(positional per-relay values); list[0] = global subsidy, relays 1..n get
[[subsidy_overrides]]. Scalar shape renders byte-identically to before.
- cb-multiple-relays now emits mev_builder_subsidy: [1, 2] (helix-2 gets 1 ETH,
helix-3 gets 2 ETH on every slot). Other scenarios keep the scalar. Golden +
scenario comment updated.
- relay.best_bid data gains divergent_slots (competitive slots whose offered
values are NOT all equal) and the PASS detail states "real discrimination"
vs "degenerate tie" — the live proof of non-degeneracy. 2 both-sides tests.
lib 110 -> 112. NOTE: submodule commit fc5e6a2 is local-only (with 43fe436 +
fbe3141) — needs J's push to JasonVranek/ethereum-package.
* docs: live-validate divergent-bid 2-helix — 65/65 slots divergent, 0 suboptimal, overall PASS
* feat(cb_metrics): handle CB's WS transport code 556 + pin WS log-variant parsing
Readiness for commit-boost PR #483 (WS get_header streaming), which introduces
TRANSPORT_ERROR_CODE = 556 (connect refused / dns / tls / stream broke) and a
new bid log line "received new header from ws stream".
- bucket_code: 556 -> its own `transport` bucket (client-observed, not
relay-served — same misattribution class as the 555 fix; relay reachability's
FAIL owner stays the tier-1 relay_pipeline death checks).
- classify_endpoint: 555+556 aggregate into one client-side WARN above the 25%
rate (never FAIL, not under --strict); both excluded from the real-5xx
denominator so a genuine relay error storm still FAILs; PASS detail notes
sub-threshold counts of each.
- mux_routing: test pinning that the WS log variant (a superstring of the HTTP
message, latency/content_type fields absent) parses through parse_cb_log_line
with the fields best_bid needs (relay_id, slot, value_eth) and matches the
starts_with("received new header") prefix filter our consumers use.
5 new both-sides tests + extended bucket assertions (lib 112 -> 117).
CHECKS.md + sim checks catalog synced.
* feat(scenarios): cb-sigverify-diff — a REAL skip_sigverify ON/OFF differential
Turns feature.skip_sigverify from a permanent honest-WARN into a positive
test, with NO relay modification: CB validates bids against the pubkey in its
own [[relays]] url, so pointing CB at the real helix via a literal url whose
pubkey is a valid-but-WRONG BLS key (a mnemonic validator key, not helix's
DEFAULT_MEV_PUBKEY) makes validate_signature reject every bid (PubkeyMismatch)
— the exact function skip_sigverify skips. Bids winning the auction is then
positive proof the skip codepath fired.
- CbParams.literal_relay_url: replaces the {{ range }} relay loop with one
literal [[relays]] block (service DNS helix-relay-2:4040 resolves in-enclave;
confirmed by the live 2-helix runs' service names).
- Two new scenarios + goldens: cb-sigverify-diff (skip ON + poisoned url;
expect green with feature.skip_sigverify PASS) and cb-sigverify-diff-control
(same poison, skip OFF; EXPECTED to fail payload delivery — the control arm).
`sim diff control treatment` shows the flip that proves discrimination.
- feature_fired: has_poisoned_relay_pubkey (pure detector vs the known helix
signing pubkey) + classify_skip_sigverify(poisoned, auction_winners):
poisoned + >=1 "auction winner" (a post-validation log line) -> PASS;
poisoned + 0 -> WARN (no-false-green); unpoisoned -> the honest WARN as
before. 4 new tests incl. both-sides detection (lib 117 -> 120).
- Catalog + CHECKS.md synced.
Live differential run pending (2 devnets + sim diff).
* feat(scenarios): Law 7 — parametrize on EL/CL client pairs
"Coverage is a matrix, not a point": every scenario hardcoded geth+lighthouse,
so a CB regression specific to another client pair was invisible.
- New `ElCl { el, cl }` axis with DEFAULT (geth/lighthouse) and ALT
(nethermind/prysm); `Scenario::el_cl()` selects it. The participants block is
now derived from the pair instead of a const.
- The pair also drives every service name derived from it — notably
extra-validation's `rpc_url`, which was hardcoded to el-1-geth-lighthouse.
The ethereum-package names EL services `el-{index}-{el}-{cl}`
(el_launcher.star:177), so on any other pair that hardcode silently points at
a nonexistent service and extra validation no-ops (feature.extra_validation
would then WARN). Now derived — the exact coupling Law 7 exists to catch.
- New scenario cb-basic-nethermind-prysm (P3's "one alternate EL/CL pair"):
cb-basic's assertions on different clients. Full scenario x pair cross-product
is deliberately left for later — this proves the parametrization end to end
without exploding the fixture matrix.
Every pre-existing golden is byte-identical (zero regression); 2 new tests pin
both the participants flow and the rpc_url derivation. sim 68 -> 70.
Live devnet validation of the alt pair still pending (needs nethermind/prysm
image pulls).
* fix(sim diff): Skip transitions are coverage changes, not severity regressions
Found by dogfooding sim diff on the sigverify differential's two arms: the
report went FAIL -> PASS, yet the verdict read REGRESSION. Cause — Direction
was derived from CheckStatus's Ord (Fail > Warn > Pass > Skip), which is right
for worst-status AGGREGATION (a Skip must not win a fold over a Pass) but wrong
for TRANSITIONS: a Skip is not "better than a Pass", it is NO INFORMATION. So
SKIP -> PASS (a check that started running and passed) was classified as a
severity increase -> REGRESSED, and two of those dragged the whole run's
verdict to REGRESSION.
Direction now has CoverageGained (Skip -> real verdict) and CoverageLost (real
verdict -> Skip); neither counts as a regression, both are still printed
(cov-gain / cov-lost). Skip -> FAIL is deliberately NOT a regression either:
the failure existed before, it was just unmeasured. The same rule now backs
overall_regressed, so there is one classification in one place.
Verified against the two real reports that exposed it: the two false
REGRESSEDs became cov-gain, and the one genuine regression
(cb_relay_latency PASS -> WARN) is retained. 4 tests (sim 70 -> 72).
* docs: record the skip_sigverify differential result, the sim diff coverage fix, and the Law 7 slice
* fix(cb_metrics): distinguish "relay rejected the blinded blocks" from "proposer never chose one"
Found by the Law-7 alt-pair run (nethermind+prysm, 2026-08-04). The check
reported "0 deliveries (200+202=0); proposer never chose a builder block" — but
the data said the opposite: 26 blinded blocks were forwarded to the relay and
the relay rejected ALL 26 with 4xx (beacon side: 26 x 502 back to the CL). The
proposer very much chose builder blocks; the break was relay-side. A diagnosis
that names the wrong component is worse than none — it sends an operator to
debug the CL when the relay is refusing.
Now split:
- submissions present but zero delivered (r4xx > 0) -> FAIL naming the relay as
the rejecter (a relay refusing every blinded block is not a WARN).
- genuinely zero submissions -> the original "proposer never chose" WARN
(--strict FAILs), which is now only reachable when nothing was submitted.
3 tests incl. the live nethermind+prysm shape (lib 120 -> 121).
* docs: Law 7's first dividend — nethermind+prysm blinded blocks rejected by helix; plus the misdiagnosis fix
* fix(helix): enable the GetPayloadV2 route — prysm's MEV path was blocked by OUR config
Chasing the Law-7 alt-pair failure (nethermind+prysm delivered zero payloads)
ended at our own helix config, not at a client or CB defect.
Chain of evidence: prysm submits blinded blocks to /eth/v2/builder/blinded_blocks
at ~256ms into the slot (early, so the "late block" hypothesis is dead); helix
404s that route; CB deliberately refuses to downgrade to v1 (in v2 the relay
publishes the block after an empty 202, so forwarding a v1 payload would
silently drop it) and returns 502. Result: every builder block the proposer
chose was lost, and the 11 v2-unsupported events line up 1:1 with the 11 missed
slots. Lighthouse never hit it because it submits via v1.
The 404 was NOT a helix limitation: helix's Route enum has GetPayloadV2 (verified
in the shipped binary alongside GetPayload, GetTopBidV2, ProposerPayloadDeliveredV2)
and our generated router_config.enabled_routes listed only GetPayload. Added it.
All goldens regenerated (the helix block is shared by every scenario).
Also lands the check that would have named this in one line instead of a
multi-hour chase:
- NEW cb_relay_v2_unsupported reads pbs_submit_block_v2_unsupported_total, FAILs
when nonzero (escalating to tier 1 like the matrix checks — lost submissions
are at least as fatal as a 5xx) and points at the relay's route config first.
- cb_v2_fallback no longer claims "relays support v2" when the counter is zero.
That was a false reassurance: a relay that 404s v2 never reaches the fallback
path at all, so zero fallbacks says nothing about v2 support. It now reports
only what it measured ("No v2->v1 fallbacks recorded").
6 tests (lib 121 -> 125).
* docs: add CLAUDE.md (agent orientation) + sync the catalog drift it caught
CLAUDE.md is the router for the repo: what it is, USING it (real flags only,
verified against justfile/run-and-verify.sh/clap), the verdict model, a
DEBUGGING section built around the method that has repeatedly paid off here
(get the raw evidence - kurtosis logs with ANSI stripped, raw Prometheus
counters - before believing a check's summary), the Known traps (555/556
synthetic codes, helix enabled_routes/GetPayloadV2, wrong detail strings, the
kurtosis pin, the fork submodule push requirement, never pattern-kill), the
Law-4 classify_* seam for adding checks, and an explicit
keep-this-updated-in-the-same-commit contract.
Writing it immediately caught real drift, including mine: cb_relay_v2_unsupported
shipped in the previous commit without its catalog entry or CHECKS.md section -
the exact failure the discipline exists to prevent. Both synced here (catalog
now 24 entries; CHECKS.md gains the row + a section noting a FAIL usually means
the relay's route config, not a capability gap). README's stale "six scenarios"
corrected to nine.
* test(relay_pipeline): extract the 3 cross-relay aggregations as pure seams
relay_pipeline.rs held tier-1/2 verdict logic at 24% coverage: the aggregations
that decide relay.builder_blocks_received, relay.mev_delivery_rate and
relay.validator_registrations were embedded inside the async run_relay_checks,
so they could only be exercised by a live devnet. That is the Law-4 violation
the law exists to prevent - these decide whether a release is gated.
Extracted (behavior-identical, pure):
- aggregate_builder_blocks: any-Pass wins and sums counts (one silent relay is
not a pipeline failure since the builder only submits where configured), else
worst-status. Empty input now SKIPs instead of panicking on .max().unwrap().
- aggregate_mev_rate: BEST-of (Pass wins) and collapses Skip => Fail, both
deliberately opposite to CheckStatus::Ord - a Skip means no relay answered the
data API, i.e. a failure to MEASURE delivery. The comment explaining why it is
not unified with .max() now sits on the seam it describes.
- aggregate_registrations: worst-status, per-relay labelled detail, and the
documented "empty => omit the check entirely, not even SKIP" as an Option.
11 tests covering both sides of every boundary incl. the two panic guards
(empty vec, urls/results length mismatch). relay_pipeline 24.5% -> 59.3%
regions; lib 125 -> 136 tests; total coverage 62.3% -> 64.3%.
* docs(plan): bank the researched signer-on-Kurtosis runtime contract
Researched from commit-boost-client source: the container shape (same image,
CMD overridden to signer, uid 10001), the four required env vars incl. the
CB_SIGNER_ENDPOINT devnet trap (host defaults to 127.0.0.1), the minimum config
TOML, and four failure modes that would bite a naive launcher - the worst being
that a missing [[modules]] makes the service exit 0 SILENTLY, which Kurtosis
reads as a clean shutdown. Also the assertion ladder (status -> loaded_consensus
count -> JWT-authed get_pubkeys -> negative controls).
Gated behind the MEV scenario matrix: a signer test riding on an unreliable MEV
harness proves nothing.
* test(beacon): pin block-hash extraction against a REAL prysm response
beacon.rs sat at 6.6% coverage while owning the parse that payload_hash_match
and chain_health depend on. The risk is not a loud error: MinimalBlockBody
extracts execution_payload.block_hash through an Option, so a shape mismatch
silently yields None, which payload_hash_match reports as "missed" - a real
hash mismatch would be indistinguishable from a missing block.
Fixture captured LIVE from prysm v7.1.8 on the nethermind+prysm devnet rather
than invented, which matters now that Law 7 has us running more than one CL:
- parses block_hash from the real prysm response shape
- the extraction is insensitive to which sibling body fields are present (a
body carries ~12 more keys and grows every fork)
- a payload-less (phase0) block yields a clean None, not a parse error that
would abort the whole slot scan
- the {"data": ...} envelope unwraps
- trailing-slash base URLs normalize (string-concat URLs would 404)
beacon.rs 6.6% -> 41.8%; lib 136 -> 141 tests; total 64.3% -> 64.5%.
* fix(discovery): a relay's POSTGRES container was classified as a relay data API
is_relay_api_service excluded support services with ends_with("-postgres"), but
the N-relay-instance topology suffixes every service with its index, so the real
service name is `helix-relay-postgres-2` - which does not END with "-postgres"
and therefore passed the filter. Any 2-helix scenario could hand a postgres
container to the relay data-API checks. Switched to `contains`.
Found by extracting the service-SELECTION logic out of discover() into a pure
classify_services(services, port_fallback) seam (Law 4) - it decides WHAT gets
checked, so a misclassification silently invalidates every downstream check,
and it was previously unreachable without a live enclave.
The extraction also restored a port-precedence contract: try every
already-parsed port name BEFORE any `kurtosis port print` fallback. My first
version interleaved them per-name, which would shell out for "http" before
trying helix's parsed "endpoint" - one subprocess per relay per run, the same
regression an earlier perf pass removed. Now asserted by call-counting tests.
10 tests incl. client-pair agnosticism (Law 7: names carry the pair), all three
relay flavours vs their support services, pbs-over-http preference, optional CB
metrics, and the two precedence guards. discovery 58.4% -> 79.8%; lib 141 -> 151.
* test(report): pin the save->read interchange contract + render smoke
report.rs sat at 40% while owning the artifact every downstream consumer eats.
The load-bearing new test is the round trip: the lib WRITES the report and the
sim bin READS it, connected only by serde derives, so nothing but a save->parse
cycle proves the pipeline composes (incl. provenance survival and that a
check's status serializes as `result`, the key sim diff and CI gate on).
Plus: save-to-a-missing-dir must not panic (a finished verify must never die on
a bad --output-dir), short_hash determinism/width/empty-input, and a render
smoke over a report carrying every status + a FAIL with data.
lib 151 -> 156 tests.
* docs: Law 7 alt-pair layer 2 — GetPayloadV2 fix confirmed, block now rejected for a new reason
cb_relay_v2_unsupported PASSes (404-on-v2 gone, the route fix landed) but
submit_blinded_block still FAILs with all 25 blinded blocks rejected - the
failure moved from route-not-enabled to block-refused. Both new diagnostics
worked as designed. Parked as a known-fail scenario: it documents a real
interop gap, and the full MEV sweep is the priority.
* test(health,metrics): cover the two 0%-coverage modules
Both had zero tests while owning failure modes that are SILENT rather than loud:
health.rs - probe() counts any HTTP response as alive (only transport errors
mean death), so a typo in a probe path returns 404 and reads as ALIVE. That
would make the mid-run death detector permanently blind without any error. Now
pins all three liveness paths, the trailing-slash normalization (// 404s on some
servers, same blindness), and that a refused connection IS reported as death.
metrics.rs - a parse regression makes every matrix check SKIP, which is
indistinguishable from the normal devnet state (metrics usually absent) and is
non-fatal, so it would never surface. Now parses a real CB scrape shape incl.
label addressability (endpoint/http_status_code/relay_id are what every check
keys on) and histograms, plus empty/comment-only bodies parsing cleanly rather
than erroring, and a dead endpoint erroring rather than hanging.
lib 156 -> 167 tests.
* fix(test): health probe_all test used the wrong return type
probe_all returns Vec<(label, error)>, not Vec<label>. The previous commit
shipped a test that did not compile: I grepped for 'test result:' lines and
treated their ABSENCE as silence rather than as failure, so a broken build read
as green. Same class of mistake the harness itself keeps finding - a check that
cannot distinguish 'no signal' from 'bad signal'.
Test now asserts the labels AND that the death reason is carried rather than
discarded. Gates re-run with explicit exit codes (test/clippy/fmt all 0):
lib 160, sim 72, cb-verify 10.
* docs(CLAUDE.md): add the silence-is-not-success gate trap
* fix(cb_metrics): two checks read metric names that cannot exist — one produced a false PASS I built on
An independent audit of CB's metric surface caught both. They matter because
neither failed loudly; both reported green forever.
1. cb_relay_v2_unsupported never fired. CB's PBS registry is
Registry::new_custom(Some("cb_pbs")), and this counter is REGISTERED as
`pbs_submit_block_v2_unsupported_total`, so the EXPOSED name carries a
doubled prefix: `cb_pbs_pbs_submit_block_v2_unsupported_total`. Every sibling
is registered bare (`relay_status_code_total` -> `cb_pbs_relay_status_code_total`),
which is exactly why the odd one out slipped through. We matched the
registered name, so the check PASSed by construction.
RETRACTION: that false PASS was my only evidence that enabling helix's
GetPayloadV2 route fixed the nethermind+prysm failure. It was structural, not
evidence. submit_blinded_block failed identically before and after (26 vs 25
rejected), so the fix's EFFECT is now unverified - re-test before claiming it.
Adding the route is still correct on its own merits; its consequence is not
established.
2. cb_v2_fallback was permanently green. It read
`cb_pbs_submit_block_v2_fallback_to_v1_total`; commit-boost registers no
`*fallback*` metric anywhere. "Counter absent" was treated as "zero fallbacks
== PASS", so it could never fail - and it had also asserted "relays support
v2" on a run where the relay was 404ing v2. Now SKIPs, naming itself inert
and pointing at the check that owns v2 support.
Method rule: verify metric names against a REAL scrape, not against the
registration constant in CB source - the registry prefix is applied at gather
time. Catalog + backlog synced; tests re-pointed at the exposed names.
* docs(plan): signer grill verdict — build modified, two choices killed
An independent adversarial review killed the key layout (secrets/ is chmod
0600 -R, so uid 10001 cannot traverse it and the signer silently loads ZERO
keys; use teku-keys/teku-secrets, which web3signer already proves readable) and
killed launch Option A (the CB config artifact is not rendered until downstream
in main.star). It also corrected an error in my own research: a MISSING
[[modules]] bails loudly, not silently.
Plus: pbs.with_signer is dead code in the shipped binary, so 'PBS uses the
signer' is not an available escalation; the assertion ladder was mostly vanity
(/status is an unconditional 200 and there are two of them; loaded_consensus is
log-only and ANSI-mangled) and collapses to a JWT-authed get_pubkeys COUNT
assertion; and two afternoon-burning traps (negative controls rate-limit our own
IP for 300s; a commit-boost-* service name adds a 200k-line log fetch per check).
* feat(scenarios): cb-min-bid — prove the min_bid_eth floor drops bids (+ flatten-trap canary)
First scenario from the CB config-surface audit: 5 of 21 PbsConfig fields were
exercised by any scenario. min_bid_eth ranked highest because a wrong value
degrades SILENTLY (too high => every bid dropped, which reads as "no MEV") and
because it is observable.
The audit's own premise was stale, though, and the scenario only works once
corrected: it assumed ~0.05 ETH bids from old fixtures, but our scenarios set
mev_builder_subsidy: 1, so real bids land near 1.04 ETH (measured: 1.0439 /
2.0439 in the divergent run) - and CB validates min_bid_wei < 1 ETH, so NO legal
floor could ever reject a subsidized bid. cb-min-bid therefore runs with subsidy
0 (bids ~0.04 ETH of spamoor MEV) against a 0.5 ETH floor.
feature.min_bid FAILs iff an auction winner's value is BELOW the floor - the
definitive falsifier, since that is only possible if the floor was not applied.
That makes this the canary for CB's silent-flatten trap: [pbs] cannot carry
deny_unknown_fields (it must flatten PbsConfig), so a renamed/misspelled key
there is ignored rather than rejected. Nothing-rejected is a WARN, not a red:
"key ignored" and "every bid cleared the floor" look identical.
5 tests; the catalog's feature-invariant test caught the new id, as designed.
lib 157 -> 162; 10 scenarios; catalog + CHECKS.md synced.
* docs(plan): empirically confirm the signer keystore permissions KILL
Checked on a live enclave before writing any starlark: secrets/ is mode 600
root:root (no execute bit, so uid 10001 cannot traverse it) while teku-secrets
is 755 and teku-keys is 777. The originally-planned lighthouse layout would have
produced a healthy signer holding ZERO keys. Kurtosis does not chown on mount,
which is also why six other launchers force User(uid=0).
* docs: un-retract the GetPayloadV2 claim with delivery-counter evidence
Pre-fix runs (x3): submit_blinded_block 222 v1 (200), 0 v2 (202).
Post-fix cb-basic: 0 v1 (200), 222 v2 (202). A complete flip, on LIGHTHOUSE -
so enabling helix's GetPayloadV2 changed the relay-side path for every scenario,
not only the prysm one. This is counter evidence, not the structurally-broken
check that produced the earlier false PASS.
Also records a sweep caveat: run-and-verify.sh rebuilds cb-verify per scenario,
so a sweep spanning code changes mixes binary versions - cb-basic's v2 checks
came from the pre-fix build.
* docs: sharpen the prysm question — same v2 route, opposite outcome per CL
lighthouse over the relay's v2 route gets 202 Accepted (222 deliveries, PASS);
nethermind+prysm gets 4xx on all 25. So the variable is what CB forwards when
the request ORIGINATES from prysm (which calls CB's own v2 endpoint at 256ms
into the slot, ruling out timing), not the route or relay capability. Next: read
helix's rejection reason for a v2 submission and diff the content-type/body
between the lighthouse-origin and prysm-origin paths.
* feat(signer): config-gen half of the signer-on-Kurtosis North Star
Adds an OPT-IN CbParams.signer that appends [signer] + [signer.local.loader] +
[[modules]] to the CB config, plus a cb-signer scenario. Built to the adversarial
grill's verdict, not to my original plan - it killed two of my choices:
- **teku keystores, not lighthouse.** Verified live on an enclave: the package's
secrets/ dir is mode 600 root:root with NO execute bit, so CB's uid 10001
cannot traverse it and the signer would start healthy holding ZERO keys (the
loader is filter_map + warn!). teku-secrets is 755 and teku-keys 777 - the
same pair the package's own web3signer launcher already relies on.
- **Key paths stay placeholders.** They are per-participant
(node-<idx>-keystores/...) and the config template only carries
.Network/.Port/.Relays/.Timestamp, so they cannot be templated. CB's
CB_SIGNER_LOADER_{KEYS,SECRETS}_DIR env vars override the TOML at runtime.
- **Opt-in so the nine existing goldens stay byte-identical**, and appended
AFTER [logs.file] because interleaving top-level tables is invalid TOML once
[[relays]] has opened an array-of-tables (asserted by test).
- **A [[modules]] entry is mandatory**: with none CB bails loudly, with an empty
list it exits 0 SILENTLY. Every field is required, and signing_id must be
non-zero (asserted).
Generated TOML verified to parse and to match CB's schema shape. 4 tests;
sim 72 -> 76. Launcher/starlark half still to come.
* feat(scenarios): cb-signer emits commit_boost_signer + the [signer] config
Completes the config-gen half: the cb-signer scenario now also sets
mev_params.commit_boost_signer so the fork launches the signer container.
All 11 configs regenerate clean; the other 10 goldens are untouched.
* feat(checks): signer assertions — JWT-authed get_pubkeys with a key COUNT
The third piece of the signer North Star: what actually falsifies.
Built to the grill's collapsed ladder, deliberately NOT asserting the obvious:
GET /status is `Ok(StatusCode::OK)` with no logic (200 with zero keys loaded),
and the metrics server exposes a SECOND unconditional /status, so probing the
wrong port is an even emptier green. The startup log's loaded_consensus=N is
log-only (the signer registers exactly one metric, signer_status_code_total,
with no key-count gauge) and ANSI-colored, so the field is not even a contiguous
substring.
Instead: a JWT-authenticated GET /signer/v1/get_pubkeys with a COUNT assertion.
One HTTP call subsumes liveness, module registration, JWT auth AND key loading -
and it is by construction the assertion that fails if the keystore mount is
unreadable, which is this feature's most likely failure (CB's loader is
filter_map + warn!, so a permissions problem yields a healthy signer holding
nothing). Zero keys therefore FAILs and names the teku fix in the detail.
The JWT is hand-rolled HS256 rather than pulling `jsonwebtoken`, because the
claims are what must be exactly right: `route` binds to the exact request path,
and `payload_hash` must be NULL on a bodyless request (CB enforces both
directions). All of it is a pure, deterministic function with `now` injected -
5 tests decode the claims back and assert route binding, null payload_hash,
expiry and secret-sensitivity.
Also: the negative control classifies 401 PASS / 200 FAIL (auth not enforced) /
429 WARN naming the self-poisoning trap - CB rate-limits a source IP for 300s
after 3 failures, and every harness request shares one NAT address, so negatives
must run last.
Discovery gains signer_urls on a SEPARATE `cb-signer-*` pattern, asserted not to
leak into cb_service_names (three checks iterate that list shelling a 200k-line
log fetch each). 11 tests; lib 162 -> 173.
* feat(signer): wire the signer check into the run pipeline
The check module existed but nothing called it, so cb-signer would have run and
asserted nothing - a scenario that cannot fail is the exact defect class this
harness keeps finding. Now: for each discovered cb-signer-* service, mint a
module JWT and assert the get_pubkeys COUNT against the devnet's active
validator set. A non-200 or an unreachable signer FAILs at tier 1 rather than
being silently absent.
Catalog + CHECKS.md synced (the catalog's feature-invariant test caught the new
id again, as designed).
* docs: mark the signer BUILT-not-validated + add the uid-10001 keystore trap to CLAUDE.md
* docs: record the two-sweep overnight plan + cb-basic/cb-mux PASS
* test(cb_metrics): pin CB's exposed metric names after the dead-check audit
Audited every metric name the checks read against CB's registered names plus
its registry prefixes. Result: the two bad ones are already handled and the rest
are correct - a clean negative result worth banking rather than re-deriving.
The rule that produced the bug: Registry::new_custom(Some(..)) prefixes every
metric at gather time, and most PBS metrics are registered bare - but two carry
their own prefix and end up DOUBLED (cb_pbs_pbs_submit_block_v2_unsupported_total,
and cb_signer_signer_status_code_total, which we do not read yet but will if a
signer metrics check is added). The test pins them so an 'obvious tidy-up' of
the doubled prefix breaks loudly instead of silently disabling a check.
It deliberately cannot detect a CB-side rename - only a real scrape can, which
is the standing rule: verify metric names against a scrape, never against the
source constant.
* fix(cb_metrics): judge submit_blinded_block on the BEACON side — a false red my own feature caused
cb-multiple-relays FAILED the sweep on submit_blinded_block 186/626 5xx (29.7%)
while the pipeline was flawless: 65/65 payloads delivered across 2 relays, 100%
MEV rate, 65/65 payload hashes matched, best_bid verified over 65 competitive
slots, 0 missed slots.
The per-relay split shows why:
mev_relay_0 (subsidy 1, LOSES every auction): 202x1, 4xx x219, 5xx x185
mev_relay_1 (subsidy 2, WINS every auction): 202x219, 4xx x1, 5xx x1
beacon side: 202x220 <- CB served the CL every time
CB asks EVERY configured relay for the payl…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.