Repo-wide audit: rebuild the measurement spine, backfill the conventions

[AviAvni]

The claim this repo makes is that every performance figure comes from code you can run. An audit of all 44 packages, 230 reading guides, 45 crates and the book build found that claim only partly enforced. This makes it true.

42/42 verify.sh lanes pass · 45 crates clean under -D warnings · mdbook clean · no broken links across 292 files

---

1. A benchmark printed a physically impossible number

Topic 12's scan lane reported:

raw sum   800.0 MB   0.000 s   19047619.0 GB/s(raw-equiv)

That is roughly 20,000× this machine's memory bandwidth. time() took best-of-3 reps of a pure fold, so LLVM hoisted the whole computation out of its own repetition loop — reps 2 and 3 timed nothing and best-of-3 reported ~0.000 s. Fixed with black_box on the input inside each timed closure plus a MIN_CREDIBLE_SECS guard that prints n/a — below timer resolution rather than a figure.

Now measures 24.4 / 50.0 / 57.0 GB/s (150 GB/s peak on an M3 Pro, so one core gets about a third of the bus). The spread across the three column shapes is run-to-run noise on byte-identical work, and is recorded as a range rather than presented as a property of the data.

This is topic 0's first failure mode occurring in this repo's own code. It is written up in the topic and in the README rather than quietly deleted.

2. verify.sh covered 10 of 44 topics while CLAUDE.md claimed it ran every measured lane

The blocker was mechanical, not missing work: topics 34–43 caught the exercise-lane panic and printed [stub — ...]; topics 2–31 let a todo!() abort the process (rc=101) after lane 1 had already printed its numbers.

A quiet stub_lane helper now runs in every bench binary — suppress the panic hook, catch, print one marker line, exit 0. Topics 17–18 already had it; 19–22 got a STUBBED flag plus one summary line; 23–30 needed only hook suppression.

Result: 42 bin lanes across 41 topics, plus 3 criterion lanes for topic 0 behind --criterion. Added --list too.

The new coverage immediately caught one genuine failure — topic 8's txn_bench aborting the whole table on its MVCC stub — which is the argument for having it.

3. Two topics had no measurable lane, so two were written

Topic 7 had only the RESP server, which cannot run until the reader implements resp.rs. New loopback_bench measures the topic's actual thesis with no protocol parsing and no store:

P	ops/s	µs/request	syscalls/op	vs P=1
1	44,088	22.68	2.000	1.0×
64	2,919,728	0.34	0.031	66.2×
256	12,321,414	0.08	0.008	279.5×

Identical zero-work requests, 279× apart. Slightly super-linear against the 2/P syscall floor, because larger writes amortize per-byte costs too.

Topic 3's only bench interleaved the reader's DiskBTree with redb and died on the first stub. New btree_baseline prices the fanout arithmetic against redb measured warm — and its height ladder is a negative result, which is the best thing in the topic:

keys       ns/lookup   file MB   height
10000            367       1.6        2
100000           423       9.0        3
1000000          862      67.9        3
4000000         1101     270.0        3

"Height is the metric" predicts a step function. Cost keeps climbing while height stays pinned at 3, because height sets how many pages a lookup touches and cache residency sets what a touch costs. Two levers, and the second is why topic 6 exists. Reported as the finding rather than smoothed over.

4. CI never ran any Rust

The only workflow built the book, so the repo's central claim had zero automated protection. New verify.yml runs ./verify.sh --summary and a -D warnings build of all 45 crates.

Deliberately not cargo test — the stub tests are the specification and must fail on a fresh clone. Deliberately not clippy — style lints across 44 independent teaching crates would go red immediately and that is a different argument from this gate. Fixed the 8 warning-emitting crates so the gate passes: #[allow(..., reason = "...")] on stub scaffolding, plus three genuine fixes.

5. book/ was 7.2 GB, of which 6.7 GB was copied Rust build artifacts

src = "." makes mdbook copy every non-markdown file under the root, and it honours neither .gitignore nor .mdbookignore — both tested, hidden directories are copied too. All 44 target/ dirs were being duplicated into the rendered book on every build.

Fixed with a root .cargo/config.toml pointing target-dir outside the clone (verified that cargo resolves a relative target-dir against the config file's parent, and that nested crates inherit it).

16 GB reclaimed, book/ is now 49 MB, and the 44 crates share one dependency build instead of 44. Also un-ignored and committed 45 Cargo.lock files — for a repo whose pitch is that a seeded figure reproduces exactly, floating transitive versions were the one thing that could silently change a generator's output.

6. Conventions backfilled

convention	before	after
notes.md measured baseline	30 topics	44
numbers-first README opener	24 topics	42 (4 and 10 excluded on purpose)
## Done when in reading guides	116 of 230	230
pinned commit for file:line anchors	1 guide	84 clones, one table

The Done when gap was a clean band — topics 13–30 had zero. Each new checklist is specific to its guide's ### Step sections and questions, and ties back to the topic's measured number where there is one.

7. New pages

FINDINGS.md — the repo's differentiator was 4 rows in the README; it is now 42 rows, one measured headline per topic with the command that re-derives it. Plus a section on how to read it (which rows contradict their own topic's tidy story, which baselines refuse to be weak) and a dependency DAG with the three threads worth following deliberately:

• the fsync wall recurring at 337 / 341 / 14 ms across topics 5, 15 and 28
• skew in 13, 24, 36, 20
• the measurement itself lying in 0, 12, 34, 39–43

Presentation: PROGRESS.md showed 43 of 44 topics as todo — accurate for the study, but it reads as "unbuilt" to anyone arriving from the book. Split into Package (all 44 done) and Studied columns. SESSION-LOG.md was 132 KB under a single heading; added 50 ## date — topic NN — title headings for anchors, with the entry text verified byte-identical afterwards. capstone/README.md claimed milestones map to topics 0–31 and did not say M0 is the only one built. drafts/ committed with a README stating it is not part of the book.

---

Three places the audit over-stated the problem

Recording these because the first pass of the review was wrong about them:

• "notes.md measurement columns blank in topics 1–25" — those cells are the reader's prediction worksheet and are supposed to be empty. The real gap was 14 topics with no record of the provided lane at all. The worksheets were left alone.
• "32 topics lack a measured opener" — really 19 plus one borderline. Twelve already opened with numbers under other headings ("Our motivation numbers first", "The problem, priced").
• "18 of 44 topics record a pinned commit" — a regex artifact. Exactly one guide had a real SHA, against a few thousand file:line anchors.

Deliberately not done

• Topics 4 and 10 keep no verify lane. Their benchmarks measure only the reader's implementation, so there is nothing to report on a fresh clone. Their READMEs now open by saying so and giving the arithmetic (leveled vs tiered write amp) or the external oracle (DuckDB EXPLAIN) to predict against instead. Inventing a number to fill the slot would be the wrong fix.
• The 21 MB search index stays. It is inherent to a 500k-word book, mdbook has no per-page exclusion, and it is lazy-loaded on first search.

Test plan

./verify.sh --list        # 42 bin lanes + 3 criterion lanes
./verify.sh               # ~15 min; every lane must PASS and exit 0
./verify.sh --criterion 00
mdbook build              # check book/ stays ~49 MB, mermaid renders
python3 tools/pin-table.py --check

🤖 Generated with Claude Code