From 3794e75ae32a35cc3e714f125fbf5a2a0d0cf5fb Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 16:28:42 -0500 Subject: [PATCH] Proposal: entries are written once; keys are managed separately The store rewrites entries to manage keys, and the rewrites are what the commit path waits for: on the platform, nine stores on one NVMe put 518 of 519 seal goroutine samples in fsync by minute 4, behind 360 seal barriers a second and maintenance rewriting write-once data three to four times. The proposal: a heap with holes for the dynamic layer (a rewrite that fits reuses its slot, a hole is reused one seal late, one bounded move per pass), append-only data with merged index deltas for the permanent layer, and one commit point per store per block. Acceptance is the nine-store platform run flat. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- .../2026-09-16-entries-written-once.md | 161 ++++++++++++++++++ 1 file changed, 161 insertions(+) create mode 100644 docs/proposals/2026-09-16-entries-written-once.md diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md new file mode 100644 index 0000000..aadf39f --- /dev/null +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -0,0 +1,161 @@ +# Proposal: entries are written once; keys are managed separately + +*Status: proposal, 2026-09-16. On acceptance this becomes spec text +in 1.5 (dynamic layer), 1.8 (durability), 2.3 (seal) and 2.7 +(maintenance), and #94 leaves the register.* + +## The problem, measured + +The store rewrites entries in order to manage keys, and the rewrites +are what the commit path waits for. + +`cmd/bdbench -stores 9` (spec 2.11; PR #95) on the soak machine's +NVMe, nine stores of eight shards, each taking the soak's per-block +volume (11k dynamic rewrites, 8.3k permanent appends, 40k lookups) on +a 1 s block: + +| minute | seal p50 / p90 / max | block p90 / max | blocks of 540 | maintenance work per minute | write | +|---|---|---|---|---|---| +| 1 | 65 / 84 / 107 ms | 426 / 651 ms | 540 | 20 s | 117 MB/s | +| 3 | 116 / 731 / 1,244 ms | 1,218 / 2,889 ms | 494 | 229 s | 240 MB/s | +| 5 | 122 / 798 / 1,960 ms | 1,635 / 4,791 ms | 454 | 215 s | 233 MB/s | + +One store alone ran 30 minutes flat (seal p50 51 ms, block p50 +205 ms). Twenty goroutine snapshots of the nine-store run at minute +4: 518 of 519 seal shard goroutine samples in `fsync`; the maintenance +goroutines in `fsync` as well (55 of 56). The device is not short of +bandwidth. It is short of barriers, and two things fill its queue: + +1. **The seal's own barriers.** The permanent layer seals every shard + every block: four fsyncs per shard, three deep (data and index, + then the manifest's temp file, then the directory), plus the + dynamic tail's sync. Nine stores of eight shards ask the device for + about 360 fsyncs a second before any maintenance runs (#33). +2. **Maintenance rewriting entries.** Per minute across the nine + stores: dynamic compaction 121 s, permanent merge 127 s. The merge + costs as much as the compaction and rewrites data that never + changes: every block's segment is folded every 20 blocks, then + packed every 1,000, then packed again. Write amplification of + three to four on write-once data; and on the dynamic side a fold + reads a 220 MB segment and its 90 MB index back to drop the dead + records between them (soak 20260916T185711Z, bvn2-val4). + +## The rule + +> **An entry is written to the store once. Reclaiming space and +> bounding the walk are done on keys, never by rewriting entries.** + +Entries move only to make a bigger hole, one entry per step, bounded +(1.2). Everything else the store does to stay small and fast -- +compaction, merge, pack -- operates on indexes, which are an order of +magnitude smaller than what they index (a key and a location, ~40 B, +against a ~300 B value) and can be rebuilt from the entries. + +## The dynamic layer: a heap with holes + +The dynamic layer holds a bounded key set rewritten forever. Today +every rewrite appends and the old copy is garbage until a fold +rewrites everything around it. Instead: + +- **Entries live in a heap file per shard.** An entry is + `[len][key][value][checksum]`; its location is `(file, offset)`. +- **A rewrite that fits reuses the slot.** A value no larger than the + slot it replaces is written in place. The soak's dynamic rewrites + are chain heads, BPT nodes and account state whose size is stable, + so this is the common case and it creates no garbage at all. +- **A rewrite that does not fit appends and frees a hole.** The new + entry goes to the end of the heap (or into a hole that fits); the + old slot becomes a hole. +- **Holes are filled, not swept.** Free space is kept by size class + (`8 << n` bytes); a new entry takes the smallest hole that fits, or + the end of the file. Fragmentation is bounded by the size classes + the way a slab allocator's is, and the store reports the ratio of + hole bytes to live bytes. +- **Moving is the only rewrite, and it is bounded.** When the hole + ratio exceeds a threshold, one pass moves one entry -- the last + live entry of the file into the largest hole that fits -- and stops. + A pass costs one entry, never the file. The file shrinks from the + end when its tail is a hole. +- **The key map is in memory** for the live dynamic key set (the + soak's half million keys are ~24 MB per store; 1.2 allows memory + that scales with the working set). It is also what the seal makes + durable, below. + +The layer's size converges to O(live keys) by construction (1.5), +without the deeper fold 2.7 allows today. + +## The permanent layer: append-only data, merged indexes + +The permanent layer is write-once; it has no garbage. Its merges and +packs exist to bound the file count and the length of a history walk +(2.7). Both are properties of the *index*, so: + +- **A shard's data is one growing file** (or one per `PackEvery` + blocks). A block's records are appended; nothing is ever copied. +- **Each block seals an index delta**: the sorted `(key, offset)` pairs + of the records the block appended, with its filter (2.5). +- **Merge and pack fold index deltas** into one sorted index per shard + per level, on the adapter's cadence, off the protocol path, exactly + as today's tiers do -- but moving ~40 B per record instead of ~300. + The walk length and the file count are bounded by the index tiers; + the data file is never in the walk (a key resolves to an offset, + then one `pread`). + +## The seal: one commit point per store per block + +With indexes as the sealed object a block is one barrier round per +store rather than four per shard: + +1. Every shard's data appends (permanent) and slot writes (dynamic) + for the block are fsynced, in parallel: one round. +2. The block's index deltas -- permanent `(key, offset)` and dynamic + `(key, location)` for every key the block touched -- are written to + one per-store block file and fsynced: one round. +3. The store's manifest names the block file: the existing commit, + one round. + +Three rounds per store per block, none per shard beyond the parallel +data sync, in place of four rounds per shard. This is 1.8's "one +commit point" and closes #33. + +## Durability and crash consistency (1.8) + +- **A hole is reusable one seal late.** A slot freed in block N may + be reused only after N's seal is durable. Until then the durable + index still names the old slot, and a crash must find it intact. + Reuse is deferred the way 2.6 defers deletion. The adapter never + asks the store for an old version (its pre-images are memoized on + its side), so reuse waits on the seal and on nothing else. +- **A torn slot is detected, not misread.** Every entry carries its + length and a checksum; a slot whose checksum fails after a crash is + a hole, and the durable index never points at one, because an index + entry is durable only after the slot it names is. +- **The in-memory key map is rebuilt on open** from the last durable + index snapshot plus the block files after it, the same replay 2.8 + does for manifests today. A snapshot is taken on the pack cadence + so the replay is bounded. +- **Nothing durable is ever overwritten by a different entry** except + a slot the durable index no longer names. 1.7's identity rule + holds for files: a data file is appended, never republished. + +## What it is measured with + +`cmd/bdbench -stores 9`, the run above, is the acceptance test: + +- seal p90 under `-seal-budget` (100 ms) and flat from minute 1 to + minute 30; +- every block inside the interval; +- maintenance bytes per minute within a small factor of ingest, and + a pass never longer than a block; +- store size tracking the live key set plus the permanent appends; +- zero reads returning anything but the last written value. + +## Order of work + +1. The dynamic heap, behind the existing `KV2` dynamic surface (`Put`, + `Get`, `Seal`, `CompactHistory` becoming the bounded move, + `Stats`), so the sharding and the adapter do not change. The + platform measures it alone (`-stores 9 -perm 0`). +2. The permanent index deltas and the single block file, which also + brings the seal to one commit point. +3. Merge and pack over indexes.