From f2c10099cca8d76735cce3bbb8188ee6c0d4837f Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 16:46:06 -0500 Subject: [PATCH 01/58] The dynamic layer as a heap with holes HeapStore (heap.go) is step 1 of the entries-written-once proposal: entries in a heap file in size-class slots, the key map in memory, holes reused by class. A key rewritten within the block that took its slot is rewritten in place; rewritten in a later block it takes a hole or the end of the file, and the slot the durable index named becomes a hole only after the sync that stops naming it -- a crash leaves every durable entry intact and every torn slot unnamed, and a checksum catches a torn slot a stale index could name. The block sync fsyncs the heap and then appends and fsyncs the block's index delta; a snapshot on the maintenance cadence bounds the replay. KV2 asks its dynamic layer through a small interface (dynaLayer), so a shard opens with either sealed segments or the heap and everything above is unchanged; the heap is chosen at construction (NewKVShardHeapN / NewKV2Heap) and recognised on open by its directory. bdbench gains -dyna-heap, and its live page is organised into labelled groups (store, load, schedule, run; protocol path, maintenance, disk, store). Tests: in-place reuse within a block and a new slot across blocks with the hole reused one sync late; reopen replays the log and drops the unsynced block, cutting the file back; a torn log tail is dropped; a snapshot empties the log and the replay lands on it; a damaged slot is a checksum error; a sharded store round-trips through seal, compress, merge, close and reopen as a heap. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/live.html | 27 +- cmd/bdbench/main.go | 10 +- database/heap.go | 590 ++++++++++++++++++ database/heap_test.go | 235 +++++++ database/kv_2.go | 87 ++- database/kv_shard.go | 18 +- database/segstore.go | 12 + .../2026-09-16-entries-written-once.md | 23 +- 8 files changed, 966 insertions(+), 36 deletions(-) create mode 100644 database/heap.go create mode 100644 database/heap_test.go diff --git a/cmd/bdbench/live.html b/cmd/bdbench/live.html index a992fea..711f2e8 100644 --- a/cmd/bdbench/live.html +++ b/cmd/bdbench/live.html @@ -2,7 +2,12 @@

bdbench live

-
refreshes every 5 s from bdbench.csv
-
+
refreshes every 5 s from bdbench.csv and run.json
+
+

seal p50 / p90 / max (ms) — must stay flat with age

block p50 / p90 / max (ms) — interval is 1000

@@ -31,13 +37,22 @@

bdbench live

async function tick(){let t;try{t=await (await fetch("bdbench.csv?"+Date.now())).text()}catch(e){document.getElementById("state").textContent="unreachable";return} const rows=parse(t),last=rows[rows.length-1]||{};const st=document.getElementById("state"); st.textContent=rows.length?("minute "+last.minute):"waiting for the first minute"; - const cards=[["seal p90",last.seal_p90_ms+" ms","budget 100"],["block p90",last.block_p90_ms+" ms","interval 1000"],["blocks over interval",last.over_budget,"this minute"],["read p99",last.read_p99_us+" µs",""],["put p99 dyna/perm",last.dyna_put_p99_us+"/"+last.perm_put_p99_us+" µs",""],["maintenance",(+last.compress_s+ +last.merge_s+ +last.pack_s).toFixed(1)+" s/min",last.compress_passes+" passes, "+last.skipped+" skipped"],["disk write",last.write_MBps+" MB/s","read "+last.read_MBps],["store",(last.store_MB/1000).toFixed(2)+" GB",last.files+" files"],["history segments","perm "+last.perm_history+" dyna "+last.dyna_history,"bloom "+last.bloom_MB+" MB"],["wrong answers",last.mismatches,""]]; - document.getElementById("cards").innerHTML=cards.map(c=>`
${c[0]}
0?'class=bad':''}>${c[1]}
${c[2]}
`).join(""); + const tot=(+last.compress_s+ +last.merge_s+ +last.pack_s).toFixed(1); + document.getElementById("cards").innerHTML=[ + group("Protocol path, this minute",[["seal p90 (budget 100 ms)",last.seal_p90_ms+" ms",last.seal_p90_ms>100?"big warn":"big ok"],["block p90 (interval 1000 ms)",last.block_p90_ms+" ms",last.block_p90_ms>1000?"big bad":"big"],["blocks over the interval",last.over_budget,last.over_budget>0?"bad":"ok"],["blocks this minute",last.blocks],["read p99",last.read_p99_us+" µs"],["put p99 dyna / perm",last.dyna_put_p99_us+" / "+last.perm_put_p99_us+" µs"]]), + group("Maintenance, this minute",[["seconds of work",tot+" s","big"],["compress passes / s",last.compress_passes+" / "+last.compress_s],["merge passes / s",last.merge_passes+" / "+last.merge_s],["pack passes / s",last.pack_passes+" / "+last.pack_s],["skipped (one in flight)",last.skipped]]), + group("Disk",[["write",last.write_MBps+" MB/s","big"],["read",last.read_MBps+" MB/s"],["store size",(last.store_MB/1000).toFixed(2)+" GB"],["files",last.files]]), + group("Store",[["history segments perm / dyna",last.perm_history+" / "+last.dyna_history],["resident filters",last.bloom_MB+" MB"],["wrong answers",last.mismatches,last.mismatches>0?"big bad":"big ok"]])].join(""); line("c_seal",rows,["seal_p50_ms","seal_p90_ms","seal_max_ms"],["#3fb950","#d29922","#f85149"]); line("c_block",rows,["block_p50_ms","block_p90_ms","block_max_ms"],["#3fb950","#d29922","#f85149"],1000); line("c_maint",rows,["compress_s","merge_s","pack_s"],["#58a6ff","#bc8cff","#39c5cf"]); line("c_disk",rows.map(r=>({w:r.write_MBps,g:r.store_MB/1000*10})),["w","g"],["#d29922","#58a6ff"]); const T=document.getElementById("t");T.innerHTML=""+cols.map(c=>""+c.replace(/_/g," ")+"").join("")+""+rows.slice().reverse().map(r=>""+cols.map(c=>{let cls="";if(c=="seal_p90_ms"&&r[c]>100)cls="warn";if(c=="over_budget"&&r[c]>0)cls="bad";if(c=="mismatches"&&r[c]>0)cls="bad";return `${r[c]}`}).join("")+"").join("")} -fetch("run.json").then(r=>r.json()).then(c=>{document.getElementById("sub").textContent=`${c.stores} store(s) × ${c.shards} shards, seal limit ${c.sealLimit}, window ${c.window}, maintenance every ${c.compressEvery} blocks, pack every ${c.packEvery} · per ${c.interval} block: ${c.dynaPuts} dyna puts, ${c.permPuts} perm puts, ${c.reads} reads over ${c.hotKeys} hot keys, values ${c.valueMin}-${c.valueMax} B · ${c.dir} · started ${c.started} for ${c.duration} · refreshes every 5 s from bdbench.csv`}).catch(()=>{}); +function group(title,rows){return `

${title}

${rows.map(r=>``).join("")}
${r[0]}${r[1]}
`} +fetch("run.json").then(r=>r.json()).then(c=>{document.getElementById("config").innerHTML=[ + group("Store",[["dynamic layer",c.dynaHeap?"heap with holes":"sealed segments"],["stores on the disk",c.stores],["shards per store",c.shards],["seal limit (records/shard layer)",c.sealLimit],["window (blocks)",c.window]]), + group("Load per block per store",[["dynamic puts",c.dynaPuts],["permanent puts",c.permPuts],["reads",c.reads],["hot dynamic keys",c.hotKeys],["value bytes",c.valueMin+" to "+c.valueMax]]), + group("Schedule",[["block interval",c.interval],["duration",c.duration],["maintenance every (blocks)",c.compressEvery],["pack every (blocks)",c.packEvery],["started (UTC)",c.started]]), + group("Run",[["directory",c.dir.split("/").slice(-1)[0]],["seed",c.seed]])].join("")}).catch(()=>{}); tick();setInterval(tick,5000); diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index 643f6e0..4f47cd9 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -64,6 +64,7 @@ type config struct { seed uint64 pprof string http string + dynaHeap bool } //go:embed live.html @@ -91,6 +92,7 @@ func parseFlags() (config, error) { flag.Uint64Var(&c.seed, "seed", 1, "random seed") flag.StringVar(&c.pprof, "pprof", "", "serve net/http/pprof on this address (e.g. 127.0.0.1:6061)") flag.StringVar(&c.http, "http", "127.0.0.1:8098", "serve the live page and the run's files here; empty disables") + flag.BoolVar(&c.dynaHeap, "dyna-heap", false, "dynamic layer as a heap with holes (proposal 2026-09-16) instead of sealed segments") flag.Parse() if flag.NArg() != 0 { return c, fmt.Errorf("unexpected arguments: %q", flag.Args()) @@ -201,7 +203,11 @@ const ( func openStore(c config, id int) (*store, error) { dir := filepath.Join(c.dir, fmt.Sprintf("store-%d", id)) - kv, err := blockchainDB.NewKVShardN(dir, c.shards, c.sealLimit) + open := blockchainDB.NewKVShardN + if c.dynaHeap { + open = blockchainDB.NewKVShardHeapN + } + kv, err := open(dir, c.shards, c.sealLimit) if err != nil { return nil, fmt.Errorf("open store %d: %w", id, err) } @@ -406,7 +412,7 @@ func main() { "dir": c.dir, "stores": c.stores, "duration": c.duration.String(), "interval": c.interval.String(), "shards": c.shards, "sealLimit": c.sealLimit, "window": c.window, "compressEvery": c.compressEvery, "packEvery": c.packEvery, "dynaPuts": c.dynaPuts, "permPuts": c.permPuts, "reads": c.reads, "hotKeys": c.hotKeys, - "valueMin": c.valueMin, "valueMax": c.valueMax, "seed": c.seed, "started": time.Now().UTC().Format(time.RFC3339), + "valueMin": c.valueMin, "valueMax": c.valueMax, "seed": c.seed, "dynaHeap": c.dynaHeap, "started": time.Now().UTC().Format(time.RFC3339), }, "", " ") if err := os.WriteFile(filepath.Join(c.dir, "run.json"), runJSON, 0o644); err != nil { fail("run.json", err) diff --git a/database/heap.go b/database/heap.go new file mode 100644 index 0000000..84a537b --- /dev/null +++ b/database/heap.go @@ -0,0 +1,590 @@ +package blockchainDB + +import ( + "encoding/binary" + "errors" + "fmt" + "hash/crc32" + "io" + "os" + "path/filepath" + "sort" + "sync" + "sync/atomic" +) + +// HeapStore is the dynamic layer as a heap with holes (proposal +// docs/proposals/2026-09-16-entries-written-once.md): entries are +// written once, keys are managed separately, and space is reclaimed +// by reusing holes rather than by rewriting live data. +// +// Files, in one directory: +// +// heap.dat entries: [len u32][key 32][value][crc32 of key+value], +// each in a slot whose capacity is a size class (8 << c) +// index.log one delta per block sync: the (key, off, cap, len) of +// every key the block touched, checksummed, appended +// index.snap the whole key map, rewritten on the maintenance cadence +// +// The key map is in memory for the live key set (spec 1.2: memory that +// scales with the working set). Open loads the snapshot and replays +// the log; holes are the file's bytes no live slot covers. +// +// Durability (spec 1.8). The block sync fsyncs heap.dat and then +// appends and fsyncs the block's delta, so an index entry is durable +// only after the slot it names is. A slot the last durable index +// names is never overwritten: a key rewritten in a later block takes +// a new slot (a hole that fits, or the end of the file), and its old +// slot is freed only after the delta that stops naming it is durable +// -- one sync late, the way spec 2.6 defers deletion. A key rewritten +// again within the same block reuses the slot it took this block, +// since nothing durable names it yet. A crash therefore leaves every +// durable entry intact and every torn slot unnamed, and the checksum +// catches a torn slot that a stale index could name. +type HeapStore struct { + Directory string + + mu sync.RWMutex + file *os.File // heap.dat + log *os.File // index.log + size int64 // Append point: the end of heap.dat + index map[[32]byte]slot + free [heapClasses][]int64 // Reusable slots by size class + height uint64 // The block being written; slots taken in it may be rewritten in place + + // pendingFree holds the slots the current block stopped naming, + // reusable once the block's delta is durable. touched is the + // block's delta in the making. + pendingFree []slot + touched map[[32]byte]struct{} + + closed bool + holeBytes int64 // Capacity of every free slot + liveBytes int64 // Capacity of every named slot + + putTotal, putInPlace, putHole, putAppend atomic.Uint64 + lookups, hits atomic.Uint64 +} + +// slot is where an entry lives: its offset, the capacity of the slot +// (a size class) and the entry's value length. block is the height +// that took the slot, which decides whether a rewrite may reuse it. +type slot struct { + off int64 + cap uint32 + n uint32 + block uint64 +} + +const ( + heapHeader = 4 + 32 // len + key + heapTrailer = 4 // crc32 of key+value + heapClasses = 40 // 8 << 39 is far past any value + heapMinCap = 64 // Smallest slot + heapMagic = 0x48454150 // "HEAP", the delta record's marker +) + +// heapClass is the size class that holds need bytes: capacity +// 8 << c, the smallest not below need and not below heapMinCap. +func heapClass(need int) (c int, capacity uint32) { + capacity = heapMinCap + c = 3 // 8 << 3 + for int(capacity) < need { + capacity <<= 1 + c++ + } + return c, capacity +} + +// NewHeapStore creates an empty heap in directory, replacing anything +// there. +func NewHeapStore(directory string) (*HeapStore, error) { + os.RemoveAll(directory) + if err := os.MkdirAll(directory, 0o755); err != nil { + return nil, err + } + h := &HeapStore{Directory: directory} + return h, h.Open() +} + +// OpenHeapStore opens the heap in directory as it was left. +func OpenHeapStore(directory string) (*HeapStore, error) { + if _, err := os.Stat(filepath.Join(directory, "heap.dat")); err != nil { + return nil, fmt.Errorf("open heap at %s: %w", directory, err) + } + h := &HeapStore{Directory: directory} + return h, h.Open() +} + +// Open loads the key map from the snapshot and the log, and derives +// the holes. Idempotent. +func (h *HeapStore) Open() (err error) { + h.mu.Lock() + defer h.mu.Unlock() + if h.file != nil { + return nil + } + if h.file, err = os.OpenFile(filepath.Join(h.Directory, "heap.dat"), os.O_RDWR|os.O_CREATE, 0o644); err != nil { + return err + } + if h.log, err = os.OpenFile(filepath.Join(h.Directory, "index.log"), os.O_RDWR|os.O_CREATE|os.O_APPEND, 0o644); err != nil { + return err + } + h.index = map[[32]byte]slot{} + h.touched = map[[32]byte]struct{}{} + h.closed = false + if err = h.loadSnapshot(); err != nil { + return err + } + if err = h.replayLog(); err != nil { + return err + } + return h.deriveHoles() +} + +// Close syncs what is pending and closes the files. Reopen with Open. +func (h *HeapStore) Close() error { + p, err := h.beginBlockSync() + if err != nil { + if errors.Is(err, errStoreClosed) { + return nil + } + return err + } + if err := p.finish(); err != nil { + return err + } + h.mu.Lock() + defer h.mu.Unlock() + h.closed = true + err = h.file.Close() + if lerr := h.log.Close(); err == nil { + err = lerr + } + h.file, h.log = nil, nil + return err +} + +// Put writes value under key: in place if the key took its slot this +// block and the value fits, else into a hole that fits, else at the +// end of the file. The slot a durable index names is never rewritten. +func (h *HeapStore) Put(key [32]byte, value []byte) error { + need := heapHeader + len(value) + heapTrailer + buf := make([]byte, need) + binary.LittleEndian.PutUint32(buf, uint32(len(value))) + copy(buf[4:], key[:]) + copy(buf[heapHeader:], value) + binary.LittleEndian.PutUint32(buf[heapHeader+len(value):], crc32.ChecksumIEEE(buf[4:heapHeader+len(value)])) + + h.mu.Lock() + defer h.mu.Unlock() + if h.closed { + return errStoreClosed + } + h.putTotal.Add(1) + old, had := h.index[key] + var s slot + switch { + case had && old.block == h.height && int(old.cap) >= need: + // Taken this block, nothing durable names it: rewrite in place + s = old + s.n = uint32(len(value)) + h.putInPlace.Add(1) + default: + c, capacity := heapClass(need) + if n := len(h.free[c]); n > 0 { + s = slot{off: h.free[c][n-1], cap: capacity} + h.free[c] = h.free[c][:n-1] + h.holeBytes -= int64(capacity) + h.putHole.Add(1) + } else { + s = slot{off: h.size, cap: capacity} + h.size += int64(capacity) + h.putAppend.Add(1) + } + s.n = uint32(len(value)) + s.block = h.height + h.liveBytes += int64(capacity) + if had { + h.pendingFree = append(h.pendingFree, old) + h.liveBytes -= int64(old.cap) + } + } + if _, err := h.file.WriteAt(buf, s.off); err != nil { + return err + } + h.index[key] = s + h.touched[key] = struct{}{} + return nil +} + +// Get answers from the key map and one read of the slot. A slot +// whose checksum fails is reported as corrupt, never as a value. +func (h *HeapStore) Get(key [32]byte) ([]byte, error) { + h.mu.RLock() + if h.closed { + h.mu.RUnlock() + return nil, errStoreClosed + } + h.lookups.Add(1) + s, ok := h.index[key] + if !ok { + h.mu.RUnlock() + return nil, errNotFound + } + h.hits.Add(1) + buf := make([]byte, heapHeader+int(s.n)+heapTrailer) + _, err := h.file.ReadAt(buf, s.off) + h.mu.RUnlock() + if err != nil { + return nil, err + } + return heapEntryValue(buf, key) +} + +// heapEntryValue checks an entry read from a slot and returns its +// value. +func heapEntryValue(buf []byte, key [32]byte) ([]byte, error) { + n := int(binary.LittleEndian.Uint32(buf)) + if len(buf) != heapHeader+n+heapTrailer || [32]byte(buf[4:heapHeader]) != key { + return nil, fmt.Errorf("heap: slot does not hold the key it is named for") + } + if crc32.ChecksumIEEE(buf[4:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { + return nil, fmt.Errorf("heap: entry checksum failed") + } + return append([]byte(nil), buf[heapHeader:heapHeader+n]...), nil +} + +// GetDeep is Get: a heap has no history to reach into. +func (h *HeapStore) GetDeep(key [32]byte) ([]byte, error) { return h.Get(key) } + +// AdvanceBlock sets the block new writes belong to. +func (h *HeapStore) AdvanceBlock(height uint64) { + h.mu.Lock() + h.height = height + h.mu.Unlock() +} + +// heapSync is a block sync in flight: the delta to make durable and +// the slots to free once it is. +type heapSync struct { + h *HeapStore + delta []byte + freed []slot +} + +// beginBlockSync takes the block's delta and its freed slots under the +// lock; finish makes them durable outside it. +func (h *HeapStore) beginBlockSync() (blockSync, error) { + h.mu.Lock() + defer h.mu.Unlock() + if h.closed || h.file == nil { + return nil, errStoreClosed + } + p := &heapSync{h: h, freed: h.pendingFree} + h.pendingFree = nil + if len(h.touched) > 0 { + p.delta = h.encodeDelta() + h.touched = map[[32]byte]struct{}{} + } + return p, nil +} + +// encodeDelta is the block's index delta: marker, height, count, the +// touched keys' slots, and a checksum. The caller holds the lock. +func (h *HeapStore) encodeDelta() []byte { + const rec = 32 + 8 + 4 + 4 + buf := make([]byte, 4+8+4+len(h.touched)*rec+4) + binary.LittleEndian.PutUint32(buf, heapMagic) + binary.LittleEndian.PutUint64(buf[4:], h.height) + binary.LittleEndian.PutUint32(buf[12:], uint32(len(h.touched))) + at := 16 + for key := range h.touched { + s := h.index[key] + copy(buf[at:], key[:]) + binary.LittleEndian.PutUint64(buf[at+32:], uint64(s.off)) + binary.LittleEndian.PutUint32(buf[at+40:], s.cap) + binary.LittleEndian.PutUint32(buf[at+44:], s.n) + at += rec + } + binary.LittleEndian.PutUint32(buf[at:], crc32.ChecksumIEEE(buf[:at])) + return buf +} + +// finish: entries durable, then the delta durable, then the slots the +// block stopped naming become holes. +func (p *heapSync) finish() error { + h := p.h + if p.delta == nil && len(p.freed) == 0 { + return nil + } + if err := fsync(h.file); err != nil { + return err + } + if p.delta != nil { + if _, err := h.log.Write(p.delta); err != nil { + return err + } + if err := fsync(h.log); err != nil { + return err + } + } + h.mu.Lock() + for _, s := range p.freed { + h.addHole(s.off, s.cap) + } + h.mu.Unlock() + return nil +} + +// addHole puts a slot on its class's free list. The caller holds the +// lock. +func (h *HeapStore) addHole(off int64, capacity uint32) { + c, _ := heapClass(int(capacity)) + h.free[c] = append(h.free[c], off) + h.holeBytes += int64(capacity) +} + +// Snapshot writes the whole key map and truncates the log, so that +// the replay on open stays bounded. Off the protocol path, on the +// maintenance cadence. +func (h *HeapStore) Snapshot() error { + h.mu.RLock() + if h.closed { + h.mu.RUnlock() + return errStoreClosed + } + const rec = 32 + 8 + 4 + 4 + buf := make([]byte, 4+8+4+len(h.index)*rec+4) + binary.LittleEndian.PutUint32(buf, heapMagic) + binary.LittleEndian.PutUint64(buf[4:], h.height) + binary.LittleEndian.PutUint32(buf[12:], uint32(len(h.index))) + at := 16 + for key, s := range h.index { + copy(buf[at:], key[:]) + binary.LittleEndian.PutUint64(buf[at+32:], uint64(s.off)) + binary.LittleEndian.PutUint32(buf[at+40:], s.cap) + binary.LittleEndian.PutUint32(buf[at+44:], s.n) + at += rec + } + binary.LittleEndian.PutUint32(buf[at:], crc32.ChecksumIEEE(buf[:at])) + logSize, err := h.log.Seek(0, io.SeekCurrent) + h.mu.RUnlock() + if err != nil { + return err + } + // Written aside and renamed over the old snapshot, so a crash + // leaves one or the other whole + tmp := filepath.Join(h.Directory, "index.snap.tmp") + f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) + if err != nil { + return err + } + if _, err = f.Write(buf); err != nil { + f.Close() + return err + } + if err = fsync(f); err != nil { + f.Close() + return err + } + if err = f.Close(); err != nil { + return err + } + if err = os.Rename(tmp, filepath.Join(h.Directory, "index.snap")); err != nil { + return err + } + // The deltas the snapshot covers are what was in the log when it + // was taken; deltas appended since stay. The log is truncated + // from the front by rewriting what follows: rare, small, and it + // keeps one file per purpose. + h.mu.Lock() + defer h.mu.Unlock() + rest, err := readFrom(h.log, logSize) + if err != nil { + return err + } + if err = h.log.Truncate(0); err != nil { + return err + } + if _, err = h.log.Seek(0, io.SeekStart); err != nil { + return err + } + if len(rest) > 0 { + if _, err = h.log.Write(rest); err != nil { + return err + } + } + return fsync(h.log) +} + +func readFrom(f *os.File, off int64) ([]byte, error) { + end, err := f.Seek(0, io.SeekEnd) + if err != nil { + return nil, err + } + if end <= off { + return nil, nil + } + buf := make([]byte, end-off) + _, err = f.ReadAt(buf, off) + return buf, err +} + +// loadSnapshot reads index.snap into the key map, if there is one. +// The caller holds the lock. +func (h *HeapStore) loadSnapshot() error { + buf, err := os.ReadFile(filepath.Join(h.Directory, "index.snap")) + if errors.Is(err, os.ErrNotExist) { + return nil + } + if err != nil { + return err + } + n, err := h.applyDelta(buf) + if err != nil { + return fmt.Errorf("heap: snapshot: %w", err) + } + if n != len(buf) { + return fmt.Errorf("heap: snapshot has %d trailing bytes", len(buf)-n) + } + return nil +} + +// replayLog applies every whole delta in index.log; a torn tail is +// what a crash leaves and is dropped, its slots unnamed. The caller +// holds the lock. +func (h *HeapStore) replayLog() error { + buf, err := readFrom(h.log, 0) + if err != nil { + return err + } + at := 0 + for at < len(buf) { + n, err := h.applyDelta(buf[at:]) + if err != nil { + // Torn: keep what is whole, drop the rest + if err = h.log.Truncate(int64(at)); err != nil { + return err + } + break + } + at += n + } + if _, err = h.log.Seek(0, io.SeekEnd); err != nil { + return err + } + return nil +} + +var errHeapTorn = errors.New("heap: torn index record") + +// applyDelta applies one delta or snapshot record and returns its +// length. +func (h *HeapStore) applyDelta(buf []byte) (int, error) { + const rec = 32 + 8 + 4 + 4 + if len(buf) < 16 || binary.LittleEndian.Uint32(buf) != heapMagic { + return 0, errHeapTorn + } + count := int(binary.LittleEndian.Uint32(buf[12:])) + end := 16 + count*rec + if len(buf) < end+4 || crc32.ChecksumIEEE(buf[:end]) != binary.LittleEndian.Uint32(buf[end:]) { + return 0, errHeapTorn + } + if height := binary.LittleEndian.Uint64(buf[4:]); height > h.height { + h.height = height + } + for at := 16; at < end; at += rec { + var key [32]byte + copy(key[:], buf[at:]) + h.index[key] = slot{off: int64(binary.LittleEndian.Uint64(buf[at+32:])), + cap: binary.LittleEndian.Uint32(buf[at+40:]), n: binary.LittleEndian.Uint32(buf[at+44:])} + } + return end + 4, nil +} + +// deriveHoles rebuilds the free lists and the append point from the +// key map: the file's bytes no live slot covers are holes, split into +// size classes. The caller holds the lock. +func (h *HeapStore) deriveHoles() error { + slots := make([]slot, 0, len(h.index)) + for _, s := range h.index { + slots = append(slots, s) + } + sort.Slice(slots, func(i, j int) bool { return slots[i].off < slots[j].off }) + h.free = [heapClasses][]int64{} + h.holeBytes, h.liveBytes = 0, 0 + var at int64 + for _, s := range slots { + if s.off < at { + return fmt.Errorf("heap: slots overlap at %d", s.off) + } + h.gapToHoles(at, s.off) + h.liveBytes += int64(s.cap) + at = s.off + int64(s.cap) + } + h.size = at + // Anything past the last live slot is unnamed: a crash's torn + // writes, or entries of a block whose delta never became durable. + // The file is cut back to the append point so they are not holes + // that could be mistaken for anything. + return h.file.Truncate(at) +} + +// gapToHoles splits [from, to) into size-class slots, largest first. +func (h *HeapStore) gapToHoles(from, to int64) { + for from < to { + rest := to - from + capacity := uint32(heapMinCap) + for int64(capacity)*2 <= rest && capacity < 1<<30 { + capacity <<= 1 + } + if int64(capacity) > rest { + return // A remainder smaller than the smallest slot is lost until a move reclaims it + } + h.addHole(from, capacity) + from += int64(capacity) + } +} + +// Stats maps the heap's counters onto the store's report: every read +// is answered from the key map (LiveHit), there are no segments, and +// the resident memory is the key map. +func (h *HeapStore) Stats() StoreStats { + h.mu.RLock() + defer h.mu.RUnlock() + return StoreStats{ + PutTotal: h.putTotal.Load(), + PutNew: h.putAppend.Load() + h.putHole.Load(), + PutDuplicate: h.putInPlace.Load(), + LookupTotal: h.lookups.Load(), + LiveHit: h.hits.Load(), + ResidentBloomBytes: uint64(len(h.index)) * (32 + 24), + } +} + +// HoleRatio reports the heap's free capacity against its live +// capacity, the number a bounded move is scheduled on. +func (h *HeapStore) HoleRatio() (holes, live int64) { + h.mu.RLock() + defer h.mu.RUnlock() + return h.holeBytes, h.liveBytes +} + +// SetFilterBlocks and SetSealLimit are the segment layer's knobs; a +// heap has neither a window nor a tail. +func (h *HeapStore) SetFilterBlocks(uint64) error { return nil } +func (h *HeapStore) SetSealLimit(uint64) error { return nil } + +// compact is the heap's maintenance on the adapter's cadence: the key +// map snapshot that bounds the replay on open. (The bounded move +// that makes bigger holes is not written yet; holes cycle by size +// class meanwhile.) +func (h *HeapStore) compact() (bool, error) { return true, h.Snapshot() } + +// LiveRecords is the live key count; a heap is never sealed on it. +func (h *HeapStore) LiveRecords() uint64 { + h.mu.RLock() + defer h.mu.RUnlock() + return uint64(len(h.index)) +} diff --git a/database/heap_test.go b/database/heap_test.go new file mode 100644 index 0000000..c3c487f --- /dev/null +++ b/database/heap_test.go @@ -0,0 +1,235 @@ +package blockchainDB + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +func heapDir(t *testing.T) string { + t.Helper() + return filepath.Join(t.TempDir(), "heap") +} + +func key(b byte) (k [32]byte) { k[0] = b; return } + +// A key rewritten within the block reuses its slot; rewritten in a +// later block it takes a new slot, and the old one is a hole only +// after the sync that stops naming it. +func TestHeapRewriteReusesWithinTheBlockAndFreesOneSyncLate(t *testing.T) { + h, err := NewHeapStore(heapDir(t)) + require.NoError(t, err) + defer h.Close() + h.AdvanceBlock(1) + require.NoError(t, h.Put(key(1), []byte("one"))) + off := h.index[key(1)].off + require.NoError(t, h.Put(key(1), []byte("uno"))) + require.Equal(t, off, h.index[key(1)].off, "same block, fits: in place") + require.EqualValues(t, 1, h.putInPlace.Load()) + v, err := h.Get(key(1)) + require.NoError(t, err) + require.Equal(t, "uno", string(v)) + + // Block 1 durable; block 2 rewrites the key + p, err := h.beginBlockSync() + require.NoError(t, err) + require.NoError(t, p.finish()) + h.AdvanceBlock(2) + require.NoError(t, h.Put(key(1), []byte("two"))) + require.NotEqual(t, off, h.index[key(1)].off, "a durable slot is never rewritten") + holes, _ := h.HoleRatio() + require.Zero(t, holes, "the old slot is not a hole until block 2 is durable") + p, err = h.beginBlockSync() + require.NoError(t, err) + require.NoError(t, p.finish()) + holes, _ = h.HoleRatio() + require.EqualValues(t, heapMinCap, holes, "now it is") + + // Block 3 fills the hole + h.AdvanceBlock(3) + require.NoError(t, h.Put(key(2), []byte("three"))) + require.Equal(t, off, h.index[key(2)].off, "the hole is reused") + require.EqualValues(t, 1, h.putHole.Load()) + v, err = h.Get(key(1)) + require.NoError(t, err) + require.Equal(t, "two", string(v)) +} + +// Reopening replays the log: every synced value is back, the holes +// are derived, and a block that was never synced is gone -- its +// slots unnamed and its bytes cut from the file. +func TestHeapReopenKeepsTheDurableAndDropsTheRest(t *testing.T) { + dir := heapDir(t) + h, err := NewHeapStore(dir) + require.NoError(t, err) + h.AdvanceBlock(1) + for i := byte(1); i <= 50; i++ { + require.NoError(t, h.Put(key(i), []byte{i})) + } + p, err := h.beginBlockSync() + require.NoError(t, err) + require.NoError(t, p.finish()) + h.AdvanceBlock(2) + require.NoError(t, h.Put(key(1), []byte("rewritten in block 2"))) // New slot, old pending + p, err = h.beginBlockSync() + require.NoError(t, err) + require.NoError(t, p.finish()) + size := h.size + // Block 3: written, never synced -- the crash + h.AdvanceBlock(3) + require.NoError(t, h.Put(key(2), []byte("lost"))) + require.NoError(t, h.Put(key(99), []byte("lost too"))) + // Drop the store without Close: the OS has the bytes, the log has no delta + h.file.Close() + h.log.Close() + + r, err := OpenHeapStore(dir) + require.NoError(t, err) + defer r.Close() + v, err := r.Get(key(1)) + require.NoError(t, err) + require.Equal(t, "rewritten in block 2", string(v)) + v, err = r.Get(key(2)) + require.NoError(t, err) + require.Equal(t, []byte{2}, v, "block 3's rewrite was never durable") + _, err = r.Get(key(99)) + require.ErrorIs(t, err, errNotFound) + require.Equal(t, size, r.size, "the file is cut back to the durable append point") + holes, _ := r.HoleRatio() + require.EqualValues(t, heapMinCap, holes, "key 1's block-1 slot is a hole again") + require.EqualValues(t, 2, r.height, "the durable height: block 3 never synced") +} + +// A torn delta at the end of the log is dropped whole. +func TestHeapTornLogTailIsDropped(t *testing.T) { + dir := heapDir(t) + h, err := NewHeapStore(dir) + require.NoError(t, err) + h.AdvanceBlock(1) + require.NoError(t, h.Put(key(1), []byte("one"))) + require.NoError(t, h.Close()) + f, err := os.OpenFile(filepath.Join(dir, "index.log"), os.O_WRONLY|os.O_APPEND, 0o644) + require.NoError(t, err) + _, err = f.Write([]byte{0x50, 0x41, 0x45, 0x48, 9, 9}) // A marker and six bytes of nothing + require.NoError(t, err) + require.NoError(t, f.Close()) + + r, err := OpenHeapStore(dir) + require.NoError(t, err) + defer r.Close() + v, err := r.Get(key(1)) + require.NoError(t, err) + require.Equal(t, "one", string(v)) + st, err := os.Stat(filepath.Join(dir, "index.log")) + require.NoError(t, err) + require.EqualValues(t, 16+48+4, st.Size(), "one whole delta of one key remains") +} + +// A snapshot carries the map and empties the log; what comes after is +// replayed on top of it. +func TestHeapSnapshotBoundsTheReplay(t *testing.T) { + dir := heapDir(t) + h, err := NewHeapStore(dir) + require.NoError(t, err) + for b := uint64(1); b <= 30; b++ { + h.AdvanceBlock(b) + for i := byte(1); i <= 20; i++ { + require.NoError(t, h.Put(key(i), []byte{byte(b), i})) + } + p, err := h.beginBlockSync() + require.NoError(t, err) + require.NoError(t, p.finish()) + if b == 20 { + ok, err := h.compact() + require.NoError(t, err) + require.True(t, ok) + st, err := os.Stat(filepath.Join(dir, "index.log")) + require.NoError(t, err) + require.Zero(t, st.Size(), "the log is empty after the snapshot") + } + } + require.NoError(t, h.Close()) + r, err := OpenHeapStore(dir) + require.NoError(t, err) + defer r.Close() + for i := byte(1); i <= 20; i++ { + v, err := r.Get(key(i)) + require.NoError(t, err) + require.Equal(t, []byte{30, i}, v) + } + // A key rewritten every block cycles two slots: the one it holds + // and the one it held last block, a hole once this block is durable + // and the next block's slot. So after the last sync every key has + // one hole beside its live slot. + holes, live := r.HoleRatio() + require.EqualValues(t, 20*heapMinCap, live) + require.EqualValues(t, 20*heapMinCap, holes) +} + +// A slot whose bytes were damaged is an error, never a value. +func TestHeapChecksumCatchesADamagedSlot(t *testing.T) { + dir := heapDir(t) + h, err := NewHeapStore(dir) + require.NoError(t, err) + h.AdvanceBlock(1) + require.NoError(t, h.Put(key(1), []byte("intact"))) + off := h.index[key(1)].off + require.NoError(t, h.Close()) + f, err := os.OpenFile(filepath.Join(dir, "heap.dat"), os.O_WRONLY, 0o644) + require.NoError(t, err) + _, err = f.WriteAt([]byte("damaged"), off+heapHeader) + require.NoError(t, err) + require.NoError(t, f.Close()) + r, err := OpenHeapStore(dir) + require.NoError(t, err) + defer r.Close() + _, err = r.Get(key(1)) + require.ErrorContains(t, err, "checksum") +} + +// A shard built with the heap seals, compacts, closes and reopens as +// a heap, through the same KV2 and KVShard surface. +func TestHeapShardRoundTrip(t *testing.T) { + dir := filepath.Join(t.TempDir(), "shards") + kvs, err := NewKVShardHeapN(dir, 2, 1000) + require.NoError(t, err) + require.NoError(t, kvs.SetFilterBlocks(MinFilterBlocks)) + fr := NewFastRandom([]byte{7}) + hot := make([][32]byte, 200) + for i := range hot { + hot[i] = fr.NextHash() + } + for b := uint64(1); b <= 45; b++ { + for _, k := range hot { + require.NoError(t, kvs.PutDyna(k, append([]byte{byte(b)}, k[:8]...))) + } + require.NoError(t, kvs.PutPerm(fr.NextHash(), []byte("perm"))) + require.NoError(t, kvs.SealBlock(b)) + if b%20 == 0 { + require.NoError(t, kvs.Compress()) + _, err := kvs.MergeFinalized(b - MinFilterBlocks) + require.NoError(t, err) + } + } + for _, k := range hot { + v, err := kvs.GetDyna(k) + require.NoError(t, err) + require.Equal(t, byte(45), v[0]) + } + _, dyna := kvs.Stats() + require.EqualValues(t, 45*200, dyna.PutTotal) + require.NoError(t, kvs.Close()) + + re, err := OpenKVShard(dir) + require.NoError(t, err) + defer re.Close() + require.NotNil(t, re.Shards[0].Heap, "reopened as a heap") + require.Nil(t, re.Shards[0].DynaKV) + for _, k := range hot { + v, err := re.GetDyna(k) + require.NoError(t, err) + require.Equal(t, byte(45), v[0]) + } +} diff --git a/database/kv_2.go b/database/kv_2.go index f5008cd..e7ab8c4 100644 --- a/database/kv_2.go +++ b/database/kv_2.go @@ -13,6 +13,7 @@ import ( const PermDirName = "perm" const DynaDirName = "dyna" +const HeapDirName = "dyna-heap" // The dynamic layer as a heap with holes (heap.go) // KV2 // Maintains 2 layers of key value pairs with different immutability characteristics: @@ -59,7 +60,8 @@ type KV2 struct { Mutex sync.RWMutex Directory string // Directory where the PermKV and DynaKV directories are PermKV *SegmentStore // The Perm layer: sealed, immutable segments - DynaKV *SegmentStore // The Dyna layer: sealed, mutable segments + DynaKV *SegmentStore // The Dyna layer: sealed, mutable segments; nil when Heap is set + Heap *HeapStore // The Dyna layer as a heap with holes (heap.go); nil when DynaKV is set // dWrites and pWrites count writes to each layer since the last // Compress. Atomic, because the writes that bump them run // concurrently: KV2.Put takes the lock SHARED (see Put). @@ -153,7 +155,53 @@ func (k *KV2) SetFilterBlocks(n uint64) (err error) { if err = k.PermKV.SetFilterBlocks(n); err != nil { return err } - return k.DynaKV.SetFilterBlocks(n) + return k.dyna().SetFilterBlocks(n) +} + +// dynaLayer is what KV2 asks of its dynamic layer, whichever of the +// two it opened: sealed mutable segments (SegmentStore) or the heap +// with holes (HeapStore). +type dynaLayer interface { + Open() error + Close() error + Get(key [32]byte) ([]byte, error) + GetDeep(key [32]byte) ([]byte, error) + Put(key [32]byte, value []byte) error + AdvanceBlock(height uint64) + LiveRecords() uint64 + SetFilterBlocks(n uint64) error + SetSealLimit(limit uint64) error + beginBlockSync() (blockSync, error) + compact() (bool, error) + Stats() StoreStats +} + +// blockSync is the second half of a block sync, finished outside the +// shard's lock. +type blockSync interface{ finish() error } + +// dyna is the dynamic layer this shard opened. +func (k *KV2) dyna() dynaLayer { + if k.Heap != nil { + return k.Heap + } + return k.DynaKV +} + +// NewKV2Heap is NewKV2 with the dynamic layer as a heap with holes. +func NewKV2Heap(directory string, sealLimit uint64) (kv2 *KV2, err error) { + if kv2, err = NewKV2(directory, sealLimit); err != nil { + return nil, err + } + if err = kv2.DynaKV.Close(); err != nil { + return nil, err + } + os.RemoveAll(filepath.Join(directory, DynaDirName)) + kv2.DynaKV = nil + if kv2.Heap, err = NewHeapStore(filepath.Join(directory, HeapDirName)); err != nil { + return nil, err + } + return kv2, nil } func OpenKV2(directory string) (kv2 *KV2, err error) { @@ -164,6 +212,13 @@ func OpenKV2(directory string) (kv2 *KV2, err error) { if kv2.PermKV, err = OpenSegmentStore(permDirName); err != nil { return nil, err } + // The dynamic layer is whichever the store was built with + if _, statErr := os.Stat(filepath.Join(directory, HeapDirName)); statErr == nil { + if kv2.Heap, err = OpenHeapStore(filepath.Join(directory, HeapDirName)); err != nil { + return nil, err + } + return kv2, nil + } if kv2.DynaKV, err = OpenSegmentStore(dynaDirName); err != nil { return nil, err } @@ -191,7 +246,7 @@ func (k *KV2) Open() error { k.SealLimit = int(DefaultBloomCapacity) } } - if err := k.DynaKV.Open(); err != nil { + if err := k.dyna().Open(); err != nil { return err } k.opened.Store(true) @@ -213,7 +268,7 @@ func (k *KV2) Close() error { defer k.Mutex.Unlock() k.opened.Store(false) err := k.PermKV.Close() - if dynaErr := k.DynaKV.Close(); err == nil { + if dynaErr := k.dyna().Close(); err == nil { err = dynaErr } return err @@ -225,7 +280,7 @@ func (k *KV2) GetDyna(key [32]byte) (value []byte, err error) { k.Mutex.RLock() defer k.Mutex.RUnlock() - if value, err = k.DynaKV.Get(key); err != nil { // Not in DynaKV, then return whatever + if value, err = k.dyna().Get(key); err != nil { // Not in DynaKV, then return whatever return nil, err } return value, nil @@ -284,7 +339,7 @@ func (k *KV2) Get(key [32]byte) (value []byte, err error) { defer k.Mutex.RUnlock() // Check and see if this is a key that has been changed - value, err = k.DynaKV.Get(key) + value, err = k.dyna().Get(key) switch { case err == nil: return value, nil @@ -308,7 +363,7 @@ func (k *KV2) GetDeep(key [32]byte) (value []byte, err error) { k.Mutex.RLock() defer k.Mutex.RUnlock() - value, err = k.DynaKV.GetDeep(key) + value, err = k.dyna().GetDeep(key) switch { case err == nil: return value, nil @@ -326,7 +381,7 @@ func (k *KV2) PutDyna(key [32]byte, value []byte) (writes int, err error) { k.Mutex.RLock() // Shared: see Put (issue #66) defer k.Mutex.RUnlock() k.dWrites.Add(1) - if err = k.DynaKV.Put(key, value); err != nil { + if err = k.dyna().Put(key, value); err != nil { return int(k.dWrites.Load()), err } autoSeal, err = k.sealDynaIfFull() @@ -388,8 +443,8 @@ func (k *KV2) sealPermIfFull() (p *pendingSeal, err error) { // replayed in full on every open. The caller must hold the KV2 lock, // shared or exclusive; see sealPermIfFull. func (k *KV2) sealDynaIfFull() (p *pendingSeal, err error) { - if k.SealLimit <= 0 || k.DynaKV.LiveRecords() < uint64(k.SealLimit) { - return nil, nil + if k.Heap != nil || k.SealLimit <= 0 || k.DynaKV.LiveRecords() < uint64(k.SealLimit) { + return nil, nil // A heap is never sealed: it has no tail to fill } return k.DynaKV.beginSealNext() } @@ -437,8 +492,8 @@ func (k *KV2) sealDynaIfFull() (p *pendingSeal, err error) { func (k *KV2) Seal(height uint64) (meta SegmentMeta, err error) { k.Mutex.Lock() perm, err := k.PermKV.beginSeal(height) - dyna, dynaErr := k.DynaKV.beginSync() - k.DynaKV.AdvanceBlock(height + 1) + dyna, dynaErr := k.dyna().beginBlockSync() + k.dyna().AdvanceBlock(height + 1) k.Mutex.Unlock() // Both halves finish regardless of the other: a failure to sync @@ -517,12 +572,12 @@ func (k *KV2) Put(key [32]byte, value []byte) (writes int, err error) { k.Mutex.RLock() defer k.Mutex.RUnlock() - if value2, err2 := k.DynaKV.Get(key); err2 == nil { // Check. Is this a DynaKV key? + if value2, err2 := k.dyna().Get(key); err2 == nil { // Check. Is this a DynaKV key? if bytes.Equal(value, value2) { // If the key is in DynaKV, it stays there. return int(k.dWrites.Load()), nil // If the value is not changed, do nothing } k.dWrites.Add(1) - if err = k.DynaKV.Put(key, value); err != nil { // If the value DID change, update + if err = k.dyna().Put(key, value); err != nil { // If the value DID change, update return int(k.dWrites.Load()), err } autoSeal, err = k.sealDynaIfFull() @@ -553,7 +608,7 @@ func (k *KV2) Put(key [32]byte, value []byte) (writes int, err error) { return int(k.dWrites.Load()), nil } k.dWrites.Add(1) - if err = k.DynaKV.Put(key, value); err != nil { // If the perm value changed, it is now a DynaKV + if err = k.dyna().Put(key, value); err != nil { // If the perm value changed, it is now a DynaKV return int(k.dWrites.Load()), err } autoSeal, err = k.sealDynaIfFull() @@ -594,7 +649,7 @@ func (k *KV2) Put(key [32]byte, value []byte) (writes int, err error) { // // TODO: Cleanse PermKV of keys in DynaKV func (k *KV2) Compress() error { - if _, err := k.DynaKV.CompactHistory(); err != nil { + if _, err := k.dyna().compact(); err != nil { return err } k.Mutex.Lock() diff --git a/database/kv_shard.go b/database/kv_shard.go index 9a31c74..633e90c 100644 --- a/database/kv_shard.go +++ b/database/kv_shard.go @@ -227,7 +227,17 @@ func NewKVShard(directory string, sealLimit uint64) (kvs *KVShard, err error) { // // How many to ask for is a question about rate, not about size; see // DefaultNumShards. +// NewKVShardHeapN is NewKVShardN with every shard's dynamic layer a +// heap with holes (heap.go). +func NewKVShardHeapN(directory string, shards int, sealLimit uint64) (kvs *KVShard, err error) { + return newKVShardN(directory, shards, sealLimit, NewKV2Heap) +} + func NewKVShardN(directory string, shards int, sealLimit uint64) (kvs *KVShard, err error) { + return newKVShardN(directory, shards, sealLimit, NewKV2) +} + +func newKVShardN(directory string, shards int, sealLimit uint64, newShard func(string, uint64) (*KV2, error)) (kvs *KVShard, err error) { if shards < 1 { return nil, fmt.Errorf("a database needs at least one shard, asked for %d", shards) } @@ -241,7 +251,7 @@ func NewKVShardN(directory string, shards int, sealLimit uint64) (kvs *KVShard, kvs.Shards = make([]*KV2, shards) for i := range kvs.Shards { // Then create all the shards shardDir := kvs.ShardDir(i) - if kvs.Shards[i], err = NewKV2(shardDir, sealLimit); err != nil { // Create the KV2 for each shard + if kvs.Shards[i], err = newShard(shardDir, sealLimit); err != nil { // Create the KV2 for each shard return nil, err } } @@ -460,7 +470,7 @@ func (k *KVShard) adoptBlockHeight() error { shard.PermKV.AdvanceBlock(height) } if shard.DynaKV != nil { // So that its window is where the set's is - shard.DynaKV.AdvanceBlock(height) + shard.dyna().AdvanceBlock(height) } } return nil @@ -712,8 +722,8 @@ func (k *KVShard) Stats() (perm, dyna StoreStats) { if shard.PermKV != nil { add(&perm, shard.PermKV.Stats()) } - if shard.DynaKV != nil { - add(&dyna, shard.DynaKV.Stats()) + if shard.Heap != nil || shard.DynaKV != nil { + add(&dyna, shard.dyna().Stats()) } } return perm, dyna diff --git a/database/segstore.go b/database/segstore.go index 25c3175..8406705 100644 --- a/database/segstore.go +++ b/database/segstore.go @@ -2712,6 +2712,18 @@ func compactionRunWithin(history []*segment, ratio float64, budget uint64) (run // the merge's rule: an uncommitted output sits below the newest active // segment and recoverOrphans deletes it, while the inputs are still // named. +// beginBlockSync and compact are the dynaLayer surface (kv_2.go) +// over Sync and CompactHistory. +func (s *SegmentStore) beginBlockSync() (blockSync, error) { + p, err := s.beginSync() + if err != nil { + return nil, err + } + return p, nil +} + +func (s *SegmentStore) compact() (bool, error) { return s.CompactHistory() } + func (s *SegmentStore) CompactHistory() (compacted bool, err error) { s.maint.Lock() defer s.maint.Unlock() diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index aadf39f..5d6ff55 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -59,13 +59,17 @@ rewrites everything around it. Instead: - **Entries live in a heap file per shard.** An entry is `[len][key][value][checksum]`; its location is `(file, offset)`. -- **A rewrite that fits reuses the slot.** A value no larger than the - slot it replaces is written in place. The soak's dynamic rewrites - are chain heads, BPT nodes and account state whose size is stable, - so this is the common case and it creates no garbage at all. -- **A rewrite that does not fit appends and frees a hole.** The new - entry goes to the end of the heap (or into a hole that fits); the - old slot becomes a hole. +- **A rewrite within the block reuses the slot.** A key written + again in the block that took its slot is rewritten in place when the + value fits: nothing durable names the slot yet. +- **A rewrite in a later block takes a new slot.** The slot the last + durable index names is never overwritten, or a crash between the + write and the block's sync would expose an uncommitted value under + a committed name. The new entry goes into a hole that fits, or the + end of the heap; the old slot becomes a hole one sync later. A key + rewritten every block therefore cycles two slots, its own and last + block's; the heap's size for such a key is twice the entry, not a + history of it. - **Holes are filled, not swept.** Free space is kept by size class (`8 << n` bytes); a new entry takes the smallest hole that fits, or the end of the file. Fragmentation is bounded by the size classes @@ -155,7 +159,10 @@ commit point" and closes #33. 1. The dynamic heap, behind the existing `KV2` dynamic surface (`Put`, `Get`, `Seal`, `CompactHistory` becoming the bounded move, `Stats`), so the sharding and the adapter do not change. The - platform measures it alone (`-stores 9 -perm 0`). + platform measures it alone (`-stores 9 -perm 0`). *Written: + `database/heap.go`, opened with `NewKVShardHeapN` / `NewKV2Heap`, + detected on open by its directory; `bdbench -dyna-heap`. The + bounded move is not written yet; holes cycle by size class.* 2. The permanent index deltas and the single block file, which also brings the seal to one commit point. 3. Merge and pack over indexes. From 44d14210c9778429c2d6ea7a8d49b747af0b0c4e Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 17:00:22 -0500 Subject: [PATCH 02/58] The heap appends and cleans; it does not fill holes Filling holes wherever they lie was measured on the platform: nine stores' block syncs wrote 443 MB/s against the segment layer's 117 for the same ingest, because a block's 11k rewrites scattered over a 380 MB heap dirtied a page each and the barrier wrote them all (run 3). The commit path cannot afford scattered writes. Now a block's entries are appended contiguously and the sync is one sequential fsync per shard; a slot a key stops naming is dead where it lies; and a bounded cleaner on the maintenance cadence scans up to HeapCleanBytes from the head, re-appends the entries still live, and marks the region, which the sync after the delta naming the copies releases with a punched hole. The head is in every delta, so a reopen knows what is released. The cleaner reports bytes scanned against bytes moved: the heap's write amplification. The live page gets a two-second state (last ten seconds of seals and blocks, maintenance in flight, heap live and dead bytes) beside the per-minute rows. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/live.html | 12 +- cmd/bdbench/main.go | 89 +++- database/heap.go | 438 ++++++++++-------- database/heap_linux.go | 15 + database/heap_other.go | 9 + database/heap_test.go | 66 +-- .../2026-09-16-entries-written-once.md | 61 +-- 7 files changed, 449 insertions(+), 241 deletions(-) create mode 100644 database/heap_linux.go create mode 100644 database/heap_other.go diff --git a/cmd/bdbench/live.html b/cmd/bdbench/live.html index 711f2e8..1caa400 100644 --- a/cmd/bdbench/live.html +++ b/cmd/bdbench/live.html @@ -17,7 +17,8 @@ td:first-child,th:first-child{text-align:left}.warn{color:#d29922}.bad{color:#f85149}.ok{color:#3fb950}

bdbench live

-
refreshes every 5 s from bdbench.csv and run.json
+
live state every 2 s (live.json); a report row every minute (bdbench.csv)
+
@@ -34,9 +35,14 @@

bdbench live

function line(id,rows,keys,colors,ymax){const c=document.getElementById(id);const W=c.width=c.clientWidth*2,H=c.height=c.clientHeight*2;const x=c.getContext("2d");x.clearRect(0,0,W,H);if(!rows.length)return; let m=ymax||Math.max(1,...keys.flatMap(k=>rows.map(r=>r[k]||0)))*1.1;x.strokeStyle="#2a3240";x.lineWidth=1;for(let g=0;g<=4;g++){const y=H-6-(H-12)*g/4;x.beginPath();x.moveTo(40,y);x.lineTo(W,y);x.stroke();x.fillStyle="#8b949e";x.font="18px sans-serif";x.fillText((m*g/4).toFixed(0),0,y+6)} keys.forEach((k,i)=>{x.strokeStyle=colors[i];x.lineWidth=3;x.beginPath();rows.forEach((r,j)=>{const px=40+(W-44)*j/Math.max(1,rows.length-1),py=H-6-(H-12)*Math.min(1,(r[k]||0)/m);j?x.lineTo(px,py):x.moveTo(px,py)});x.stroke()})} +async function now(){try{const l=await (await fetch("live.json?"+Date.now())).json();const m=Math.floor(l.elapsedSec/60),s=l.elapsedSec%60;document.getElementById("state").textContent=`live · ${m}m ${String(s).padStart(2,"0")}s · block ${l.height} · ${l.blocks} blocks`;const L=l.last10s; + document.getElementById("now").innerHTML=[ + group("Now: seals, last 10 s",[["p50",L.sealP50ms.toFixed(0)+" ms","big"],["p90 (budget 100)",L.sealP90ms.toFixed(0)+" ms",L.sealP90ms>100?"warn":"ok"],["max",L.sealMaxMs.toFixed(0)+" ms"]]), + group("Now: blocks, last 10 s",[["p50",L.blockP50ms.toFixed(0)+" ms","big"],["p90 (interval 1000)",L.blockP90ms.toFixed(0)+" ms",L.blockP90ms>1000?"bad":"ok"],["max",L.blockMaxMs.toFixed(0)+" ms"],["over the interval",L.over+" of "+L.blocks,L.over>0?"bad":"ok"]]), + group("Now: maintenance and heap",[["passes in flight",l.maintenanceInFlight,"big"],["heap live / holes",l.heapLiveMB.toFixed(0)+" / "+l.heapHoleMB.toFixed(0)+" MB"],["wrong answers so far",l.mismatches,l.mismatches>0?"bad":"ok"]])].join("")}catch(e){}} async function tick(){let t;try{t=await (await fetch("bdbench.csv?"+Date.now())).text()}catch(e){document.getElementById("state").textContent="unreachable";return} const rows=parse(t),last=rows[rows.length-1]||{};const st=document.getElementById("state"); - st.textContent=rows.length?("minute "+last.minute):"waiting for the first minute"; + if(!rows.length)st.textContent="waiting for the first minute"; const tot=(+last.compress_s+ +last.merge_s+ +last.pack_s).toFixed(1); document.getElementById("cards").innerHTML=[ group("Protocol path, this minute",[["seal p90 (budget 100 ms)",last.seal_p90_ms+" ms",last.seal_p90_ms>100?"big warn":"big ok"],["block p90 (interval 1000 ms)",last.block_p90_ms+" ms",last.block_p90_ms>1000?"big bad":"big"],["blocks over the interval",last.over_budget,last.over_budget>0?"bad":"ok"],["blocks this minute",last.blocks],["read p99",last.read_p99_us+" µs"],["put p99 dyna / perm",last.dyna_put_p99_us+" / "+last.perm_put_p99_us+" µs"]]), @@ -54,5 +60,5 @@

bdbench live

group("Load per block per store",[["dynamic puts",c.dynaPuts],["permanent puts",c.permPuts],["reads",c.reads],["hot dynamic keys",c.hotKeys],["value bytes",c.valueMin+" to "+c.valueMax]]), group("Schedule",[["block interval",c.interval],["duration",c.duration],["maintenance every (blocks)",c.compressEvery],["pack every (blocks)",c.packEvery],["started (UTC)",c.started]]), group("Run",[["directory",c.dir.split("/").slice(-1)[0]],["seed",c.seed]])].join("")}).catch(()=>{}); -tick();setInterval(tick,5000); +now();tick();setInterval(now,2000);setInterval(tick,5000); diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index 4f47cd9..3b8003f 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -151,10 +151,22 @@ func pct(sorted []time.Duration, p float64) time.Duration { func ms(d time.Duration) string { return strconv.FormatFloat(float64(d)/1e6, 'f', 1, 64) } func us(d time.Duration) string { return strconv.FormatInt(int64(d/time.Microsecond), 10) } +// recent is one block as the live page sees it: when it ended and +// what it and its seal cost. +type recent struct { + at time.Time + block, seal time.Duration +} + // tallies is what every store adds to and the report takes from. type tallies struct { blockTimes, sealTimes, dynaPut, permPut, readT samples blocks, over, mismatches atomic.Uint64 + inFlight atomic.Int64 // Maintenance passes running now + + ringMu sync.Mutex + ring [1024]recent // The last blocks, for the live state + ringN uint64 mu sync.Mutex passes map[string]int @@ -287,10 +299,15 @@ func (s *store) block(c config, t *tallies) error { if err := s.kv.SealBlock(s.height); err != nil { return fmt.Errorf("store %d SealBlock: %w", s.id, err) } - t.sealTimes.add(time.Since(at)) + sealTook := time.Since(at) + t.sealTimes.add(sealTook) took := time.Since(start) t.blockTimes.add(took) t.blocks.Add(1) + t.ringMu.Lock() + t.ring[t.ringN%uint64(len(t.ring))] = recent{at: time.Now(), block: took, seal: sealTook} + t.ringN++ + t.ringMu.Unlock() if took > c.interval { t.over.Add(1) } else { @@ -313,9 +330,11 @@ func (s *store) maintain(c config, t *tallies) { } height := s.height s.maintWG.Add(1) + t.inFlight.Add(1) go func() { defer s.maintWG.Done() defer s.maintaining.Store(false) + defer t.inFlight.Add(-1) at := time.Now() err := s.kv.Compress() t.note("compress", time.Since(at)) @@ -373,6 +392,58 @@ func dirSize(dir string) (files int, bytes int64) { return } +// liveState is what the page shows between report rows: the last ten +// seconds of blocks and seals, and the run's running totals. Written +// to live.json every two seconds. +func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { + t.ringMu.Lock() + cut := time.Now().Add(-10 * time.Second) + var bt, st []time.Duration + n := t.ringN + if n > uint64(len(t.ring)) { + n = uint64(len(t.ring)) + } + for i := uint64(0); i < n; i++ { + r := t.ring[(t.ringN-1-i)%uint64(len(t.ring))] + if r.at.Before(cut) { + break + } + bt = append(bt, r.block) + st = append(st, r.seal) + } + t.ringMu.Unlock() + sort.Slice(bt, func(i, j int) bool { return bt[i] < bt[j] }) + sort.Slice(st, func(i, j int) bool { return st[i] < st[j] }) + over := 0 + for _, d := range bt { + if d > c.interval { + over++ + } + } + var height uint64 + var holes, live int64 + for _, s := range stores { + if s.height > height { + height = s.height + } + for _, sh := range s.kv.Shards { + if sh.Heap != nil { + h, l := sh.Heap.HoleRatio() + holes, live = holes+h, live+l + } + } + } + b, _ := json.Marshal(map[string]any{ + "elapsedSec": int(time.Since(start).Seconds()), "blocks": t.blocks.Load(), "height": height, + "last10s": map[string]any{"blocks": len(bt), "over": over, + "blockP50ms": float64(pct(bt, .5)) / 1e6, "blockP90ms": float64(pct(bt, .9)) / 1e6, "blockMaxMs": float64(pct(bt, 1)) / 1e6, + "sealP50ms": float64(pct(st, .5)) / 1e6, "sealP90ms": float64(pct(st, .9)) / 1e6, "sealMaxMs": float64(pct(st, 1)) / 1e6}, + "maintenanceInFlight": t.inFlight.Load(), "mismatches": t.mismatches.Load(), + "heapHoleMB": float64(holes) / 1e6, "heapLiveMB": float64(live) / 1e6, + }) + return b +} + func fail(what string, err error) { fmt.Fprintln(os.Stderr, "bdbench:", what+":", err) os.Exit(1) @@ -507,6 +578,22 @@ func main() { csvw.Flush() } + // The live state, every two seconds, beside the per-minute rows + liveStop := make(chan struct{}) + go func() { + tick := time.NewTicker(2 * time.Second) + defer tick.Stop() + for { + select { + case <-tick.C: + _ = os.WriteFile(filepath.Join(c.dir, "live.json"), t.liveState(c, stores, start), 0o644) + case <-liveStop: + return + } + } + }() + defer close(liveStop) + // Every store drives its own blocks; the first error stops the run. stop := make(chan struct{}) var stopOnce sync.Once diff --git a/database/heap.go b/database/heap.go index 84a537b..0138f8a 100644 --- a/database/heap.go +++ b/database/heap.go @@ -8,39 +8,45 @@ import ( "io" "os" "path/filepath" - "sort" "sync" "sync/atomic" ) -// HeapStore is the dynamic layer as a heap with holes (proposal +// HeapStore is the dynamic layer as an append-and-clean heap (proposal // docs/proposals/2026-09-16-entries-written-once.md): entries are -// written once, keys are managed separately, and space is reclaimed -// by reusing holes rather than by rewriting live data. +// managed by appending, keys are managed separately, and space is +// reclaimed by a bounded cleaner that moves the live entries out of +// the oldest region and releases it, never by rewriting an index or a +// filter. // // Files, in one directory: // -// heap.dat entries: [len u32][key 32][value][crc32 of key+value], -// each in a slot whose capacity is a size class (8 << c) -// index.log one delta per block sync: the (key, off, cap, len) of -// every key the block touched, checksummed, appended +// heap.dat entries: [cap u32][len u32][key 32][value][crc32 of +// key+value], each in a slot of cap bytes (a size class), +// appended in the order written; the head is the oldest +// byte still in use, and everything before it is released +// index.log one delta per block sync: the head, and the (key, off, +// cap, len) of every key the block touched, checksummed // index.snap the whole key map, rewritten on the maintenance cadence // -// The key map is in memory for the live key set (spec 1.2: memory that -// scales with the working set). Open loads the snapshot and replays -// the log; holes are the file's bytes no live slot covers. +// A block's writes are contiguous, so the block sync is one sequential +// fsync of the heap: filling holes wherever they lie was measured at +// 4x the ingest in page writes at every barrier, and is not done. A +// slot a key stops naming is dead where it lies until the cleaner +// reaches it. // // Durability (spec 1.8). The block sync fsyncs heap.dat and then // appends and fsyncs the block's delta, so an index entry is durable // only after the slot it names is. A slot the last durable index // names is never overwritten: a key rewritten in a later block takes -// a new slot (a hole that fits, or the end of the file), and its old -// slot is freed only after the delta that stops naming it is durable -// -- one sync late, the way spec 2.6 defers deletion. A key rewritten -// again within the same block reuses the slot it took this block, -// since nothing durable names it yet. A crash therefore leaves every -// durable entry intact and every torn slot unnamed, and the checksum -// catches a torn slot that a stale index could name. +// a new slot at the end, and its old slot is dead but intact until +// the head passes it -- and the head advances only after the delta +// naming the cleaner's copies is durable, one sync late, the way spec +// 2.6 defers deletion. A key rewritten again within the same block +// reuses the slot it took this block, since nothing durable names it +// yet. A crash therefore leaves every durable entry intact and every +// torn slot unnamed, and the checksum catches a torn slot that a +// stale index could name. type HeapStore struct { Directory string @@ -48,22 +54,23 @@ type HeapStore struct { file *os.File // heap.dat log *os.File // index.log size int64 // Append point: the end of heap.dat + head int64 // The oldest byte in use; everything before it is released index map[[32]byte]slot - free [heapClasses][]int64 // Reusable slots by size class - height uint64 // The block being written; slots taken in it may be rewritten in place + height uint64 // The block being written; slots taken in it may be rewritten in place - // pendingFree holds the slots the current block stopped naming, - // reusable once the block's delta is durable. touched is the - // block's delta in the making. - pendingFree []slot - touched map[[32]byte]struct{} + // touched is the block's delta in the making; cleanedTo is where + // the head moves once the delta naming the cleaner's copies is + // durable; snapshots counts compact calls between snapshots. + touched map[[32]byte]struct{} + cleanedTo int64 + snapshots int closed bool - holeBytes int64 // Capacity of every free slot liveBytes int64 // Capacity of every named slot - putTotal, putInPlace, putHole, putAppend atomic.Uint64 - lookups, hits atomic.Uint64 + putTotal, putInPlace, putAppend atomic.Uint64 + lookups, hits atomic.Uint64 + cleanedBytes, movedBytes atomic.Uint64 } // slot is where an entry lives: its offset, the capacity of the slot @@ -77,23 +84,32 @@ type slot struct { } const ( - heapHeader = 4 + 32 // len + key - heapTrailer = 4 // crc32 of key+value - heapClasses = 40 // 8 << 39 is far past any value - heapMinCap = 64 // Smallest slot - heapMagic = 0x48454150 // "HEAP", the delta record's marker + heapHeader = 4 + 4 + 32 // cap, len, key + heapTrailer = 4 // crc32 of key+value + heapMinCap = 64 // Smallest slot + heapMagic = 0x48454150 // "HEAP", the delta record's marker + heapDeltaHdr = 4 + 8 + 8 + 4 // magic, height, head, count + heapDeltaRec = 32 + 8 + 4 + 4 ) -// heapClass is the size class that holds need bytes: capacity -// 8 << c, the smallest not below need and not below heapMinCap. -func heapClass(need int) (c int, capacity uint32) { - capacity = heapMinCap - c = 3 // 8 << 3 +// HeapCleanBytes bounds one cleaning pass: the bytes of the oldest +// region scanned, of which only the live entries are copied. Sized so +// the cleaner keeps up with the adapter's cadence on the soak's +// volume (~10 MB appended per shard per 20 blocks) with room to spare. +var HeapCleanBytes int64 = 16 << 20 + +// HeapSnapshotEvery is how many compact calls pass between key-map +// snapshots; between them the log is what open replays. +var HeapSnapshotEvery = 5 + +// heapCap is the size class that holds need bytes: a power of two no +// smaller than heapMinCap. +func heapCap(need int) uint32 { + capacity := uint32(heapMinCap) for int(capacity) < need { capacity <<= 1 - c++ } - return c, capacity + return capacity } // NewHeapStore creates an empty heap in directory, replacing anything @@ -116,8 +132,8 @@ func OpenHeapStore(directory string) (*HeapStore, error) { return h, h.Open() } -// Open loads the key map from the snapshot and the log, and derives -// the holes. Idempotent. +// Open loads the key map from the snapshot and the log and derives +// the append point. Idempotent. func (h *HeapStore) Open() (err error) { h.mu.Lock() defer h.mu.Unlock() @@ -133,13 +149,14 @@ func (h *HeapStore) Open() (err error) { h.index = map[[32]byte]slot{} h.touched = map[[32]byte]struct{}{} h.closed = false + h.head, h.size, h.liveBytes, h.cleanedTo = 0, 0, 0, 0 if err = h.loadSnapshot(); err != nil { return err } if err = h.replayLog(); err != nil { return err } - return h.deriveHoles() + return h.deriveExtent() } // Close syncs what is pending and closes the files. Reopen with Open. @@ -165,17 +182,22 @@ func (h *HeapStore) Close() error { return err } +// encodeEntry lays out one entry for its slot. +func encodeEntry(capacity uint32, key [32]byte, value []byte) []byte { + buf := make([]byte, heapHeader+len(value)+heapTrailer) + binary.LittleEndian.PutUint32(buf, capacity) + binary.LittleEndian.PutUint32(buf[4:], uint32(len(value))) + copy(buf[8:], key[:]) + copy(buf[heapHeader:], value) + binary.LittleEndian.PutUint32(buf[heapHeader+len(value):], crc32.ChecksumIEEE(buf[8:heapHeader+len(value)])) + return buf +} + // Put writes value under key: in place if the key took its slot this -// block and the value fits, else into a hole that fits, else at the -// end of the file. The slot a durable index names is never rewritten. +// block and the value fits, else appended. The slot a durable index +// names is never rewritten. func (h *HeapStore) Put(key [32]byte, value []byte) error { need := heapHeader + len(value) + heapTrailer - buf := make([]byte, need) - binary.LittleEndian.PutUint32(buf, uint32(len(value))) - copy(buf[4:], key[:]) - copy(buf[heapHeader:], value) - binary.LittleEndian.PutUint32(buf[heapHeader+len(value):], crc32.ChecksumIEEE(buf[4:heapHeader+len(value)])) - h.mu.Lock() defer h.mu.Unlock() if h.closed { @@ -184,33 +206,21 @@ func (h *HeapStore) Put(key [32]byte, value []byte) error { h.putTotal.Add(1) old, had := h.index[key] var s slot - switch { - case had && old.block == h.height && int(old.cap) >= need: + if had && old.block == h.height && int(old.cap) >= need { // Taken this block, nothing durable names it: rewrite in place s = old s.n = uint32(len(value)) h.putInPlace.Add(1) - default: - c, capacity := heapClass(need) - if n := len(h.free[c]); n > 0 { - s = slot{off: h.free[c][n-1], cap: capacity} - h.free[c] = h.free[c][:n-1] - h.holeBytes -= int64(capacity) - h.putHole.Add(1) - } else { - s = slot{off: h.size, cap: capacity} - h.size += int64(capacity) - h.putAppend.Add(1) - } - s.n = uint32(len(value)) - s.block = h.height - h.liveBytes += int64(capacity) + } else { + s = slot{off: h.size, cap: heapCap(need), n: uint32(len(value)), block: h.height} + h.size += int64(s.cap) + h.liveBytes += int64(s.cap) if had { - h.pendingFree = append(h.pendingFree, old) - h.liveBytes -= int64(old.cap) + h.liveBytes -= int64(old.cap) // Dead where it lies } + h.putAppend.Add(1) } - if _, err := h.file.WriteAt(buf, s.off); err != nil { + if _, err := h.file.WriteAt(encodeEntry(s.cap, key, value), s.off); err != nil { return err } h.index[key] = s @@ -245,11 +255,11 @@ func (h *HeapStore) Get(key [32]byte) ([]byte, error) { // heapEntryValue checks an entry read from a slot and returns its // value. func heapEntryValue(buf []byte, key [32]byte) ([]byte, error) { - n := int(binary.LittleEndian.Uint32(buf)) - if len(buf) != heapHeader+n+heapTrailer || [32]byte(buf[4:heapHeader]) != key { + n := int(binary.LittleEndian.Uint32(buf[4:])) + if len(buf) != heapHeader+n+heapTrailer || [32]byte(buf[8:heapHeader]) != key { return nil, fmt.Errorf("heap: slot does not hold the key it is named for") } - if crc32.ChecksumIEEE(buf[4:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { + if crc32.ChecksumIEEE(buf[8:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { return nil, fmt.Errorf("heap: entry checksum failed") } return append([]byte(nil), buf[heapHeader:heapHeader+n]...), nil @@ -266,106 +276,188 @@ func (h *HeapStore) AdvanceBlock(height uint64) { } // heapSync is a block sync in flight: the delta to make durable and -// the slots to free once it is. +// the head to release once it is. type heapSync struct { - h *HeapStore - delta []byte - freed []slot + h *HeapStore + delta []byte + cleanedTo int64 } -// beginBlockSync takes the block's delta and its freed slots under the -// lock; finish makes them durable outside it. +// beginBlockSync takes the block's delta under the lock; finish makes +// it durable outside it. func (h *HeapStore) beginBlockSync() (blockSync, error) { h.mu.Lock() defer h.mu.Unlock() if h.closed || h.file == nil { return nil, errStoreClosed } - p := &heapSync{h: h, freed: h.pendingFree} - h.pendingFree = nil - if len(h.touched) > 0 { + p := &heapSync{h: h, cleanedTo: h.cleanedTo} + if len(h.touched) > 0 || h.cleanedTo > h.head { p.delta = h.encodeDelta() h.touched = map[[32]byte]struct{}{} } return p, nil } -// encodeDelta is the block's index delta: marker, height, count, the -// touched keys' slots, and a checksum. The caller holds the lock. +// encodeDelta is the block's index delta: marker, height, the head +// the block's copies let the heap release, the touched keys' slots, +// and a checksum. The caller holds the lock. func (h *HeapStore) encodeDelta() []byte { - const rec = 32 + 8 + 4 + 4 - buf := make([]byte, 4+8+4+len(h.touched)*rec+4) + buf := make([]byte, heapDeltaHdr+len(h.touched)*heapDeltaRec+4) binary.LittleEndian.PutUint32(buf, heapMagic) binary.LittleEndian.PutUint64(buf[4:], h.height) - binary.LittleEndian.PutUint32(buf[12:], uint32(len(h.touched))) - at := 16 + binary.LittleEndian.PutUint64(buf[12:], uint64(h.cleanedTo)) + binary.LittleEndian.PutUint32(buf[20:], uint32(len(h.touched))) + at := heapDeltaHdr for key := range h.touched { - s := h.index[key] - copy(buf[at:], key[:]) - binary.LittleEndian.PutUint64(buf[at+32:], uint64(s.off)) - binary.LittleEndian.PutUint32(buf[at+40:], s.cap) - binary.LittleEndian.PutUint32(buf[at+44:], s.n) - at += rec + at += putDeltaRec(buf[at:], key, h.index[key]) } binary.LittleEndian.PutUint32(buf[at:], crc32.ChecksumIEEE(buf[:at])) return buf } -// finish: entries durable, then the delta durable, then the slots the -// block stopped naming become holes. +func putDeltaRec(buf []byte, key [32]byte, s slot) int { + copy(buf, key[:]) + binary.LittleEndian.PutUint64(buf[32:], uint64(s.off)) + binary.LittleEndian.PutUint32(buf[40:], s.cap) + binary.LittleEndian.PutUint32(buf[44:], s.n) + return heapDeltaRec +} + +// finish: entries durable, then the delta durable, then the head the +// delta records is released. func (p *heapSync) finish() error { h := p.h - if p.delta == nil && len(p.freed) == 0 { + if p.delta == nil { return nil } if err := fsync(h.file); err != nil { return err } - if p.delta != nil { - if _, err := h.log.Write(p.delta); err != nil { - return err - } - if err := fsync(h.log); err != nil { + if _, err := h.log.Write(p.delta); err != nil { + return err + } + if err := fsync(h.log); err != nil { + return err + } + h.mu.Lock() + defer h.mu.Unlock() + if p.cleanedTo > h.head { + if err := punchHole(h.file, h.head, p.cleanedTo-h.head); err != nil { return err } + h.head = p.cleanedTo + } + return nil +} + +// compact is the heap's maintenance on the adapter's cadence: one +// bounded cleaning pass, and every HeapSnapshotEvery calls the key-map +// snapshot that bounds the replay on open. +func (h *HeapStore) compact() (bool, error) { + cleaned, err := h.clean(HeapCleanBytes) + if err != nil { + return cleaned, err } h.mu.Lock() - for _, s := range p.freed { - h.addHole(s.off, s.cap) + h.snapshots++ + due := h.snapshots >= HeapSnapshotEvery + if due { + h.snapshots = 0 } h.mu.Unlock() - return nil + if due { + err = h.Snapshot() + } + return cleaned, err } -// addHole puts a slot on its class's free list. The caller holds the -// lock. -func (h *HeapStore) addHole(off int64, capacity uint32) { - c, _ := heapClass(int(capacity)) - h.free[c] = append(h.free[c], off) - h.holeBytes += int64(capacity) +// clean scans up to budget bytes from the head, re-appends the entries +// still live, and marks the region for release at the next sync. The +// cost of a pass is the live fraction of the oldest region: for a hot +// key set rewritten every block it is small, and for a cold one it is +// the price of a bounded move (spec 1.2). Holds the lock for the +// pass: bounded, and the copies are ordinary appends. +func (h *HeapStore) clean(budget int64) (bool, error) { + h.mu.Lock() + defer h.mu.Unlock() + if h.closed { + return false, errStoreClosed + } + if h.cleanedTo > h.head { + return false, nil // The last pass's release is still waiting on a sync + } + from, to := h.head, h.head+budget + if to > h.size { + to = h.size + } + if from >= to || h.liveBytes == 0 { + return false, nil + } + // The pass never eats the block in progress: a slot taken this + // block may still be rewritten in place, and moving it would race + // that. Stop at the first slot of the current block. + region := make([]byte, to-from) + if _, err := h.file.ReadAt(region, from); err != nil && !errors.Is(err, io.EOF) { + return false, err + } + var off = from + var moved int64 + for off < to { + at := off - from + if at+heapHeader > int64(len(region)) { + break + } + capacity := binary.LittleEndian.Uint32(region[at:]) + n := binary.LittleEndian.Uint32(region[at+4:]) + if capacity == 0 || at+int64(capacity) > int64(len(region)) { + break // Unwritten, or a slot that straddles the budget: next pass + } + var key [32]byte + copy(key[:], region[at+8:]) + s, live := h.index[key] + if live && s.off == off { + if s.block == h.height { + break + } + entry := region[at : at+int64(heapHeader)+int64(n)+heapTrailer] + ns := slot{off: h.size, cap: capacity, n: n, block: h.height} + if _, err := h.file.WriteAt(entry, ns.off); err != nil { + return false, err + } + h.size += int64(capacity) + h.index[key] = ns + h.touched[key] = struct{}{} + moved += int64(capacity) + } + off += int64(capacity) + } + if off == from { + return false, nil + } + h.cleanedTo = off + h.cleanedBytes.Add(uint64(off - from)) + h.movedBytes.Add(uint64(moved)) + return true, nil } -// Snapshot writes the whole key map and truncates the log, so that -// the replay on open stays bounded. Off the protocol path, on the -// maintenance cadence. +// Snapshot writes the whole key map and drops the deltas it covers +// from the log, so that the replay on open stays bounded. Off the +// protocol path, on the maintenance cadence. func (h *HeapStore) Snapshot() error { h.mu.RLock() if h.closed { h.mu.RUnlock() return errStoreClosed } - const rec = 32 + 8 + 4 + 4 - buf := make([]byte, 4+8+4+len(h.index)*rec+4) + buf := make([]byte, heapDeltaHdr+len(h.index)*heapDeltaRec+4) binary.LittleEndian.PutUint32(buf, heapMagic) binary.LittleEndian.PutUint64(buf[4:], h.height) - binary.LittleEndian.PutUint32(buf[12:], uint32(len(h.index))) - at := 16 + binary.LittleEndian.PutUint64(buf[12:], uint64(h.head)) + binary.LittleEndian.PutUint32(buf[20:], uint32(len(h.index))) + at := heapDeltaHdr for key, s := range h.index { - copy(buf[at:], key[:]) - binary.LittleEndian.PutUint64(buf[at+32:], uint64(s.off)) - binary.LittleEndian.PutUint32(buf[at+40:], s.cap) - binary.LittleEndian.PutUint32(buf[at+44:], s.n) - at += rec + at += putDeltaRec(buf[at:], key, s) } binary.LittleEndian.PutUint32(buf[at:], crc32.ChecksumIEEE(buf[:at])) logSize, err := h.log.Seek(0, io.SeekCurrent) @@ -395,9 +487,7 @@ func (h *HeapStore) Snapshot() error { return err } // The deltas the snapshot covers are what was in the log when it - // was taken; deltas appended since stay. The log is truncated - // from the front by rewriting what follows: rare, small, and it - // keeps one file per purpose. + // was taken; deltas appended since stay h.mu.Lock() defer h.mu.Unlock() rest, err := readFrom(h.log, logSize) @@ -482,19 +572,21 @@ var errHeapTorn = errors.New("heap: torn index record") // applyDelta applies one delta or snapshot record and returns its // length. func (h *HeapStore) applyDelta(buf []byte) (int, error) { - const rec = 32 + 8 + 4 + 4 - if len(buf) < 16 || binary.LittleEndian.Uint32(buf) != heapMagic { + if len(buf) < heapDeltaHdr || binary.LittleEndian.Uint32(buf) != heapMagic { return 0, errHeapTorn } - count := int(binary.LittleEndian.Uint32(buf[12:])) - end := 16 + count*rec + count := int(binary.LittleEndian.Uint32(buf[20:])) + end := heapDeltaHdr + count*heapDeltaRec if len(buf) < end+4 || crc32.ChecksumIEEE(buf[:end]) != binary.LittleEndian.Uint32(buf[end:]) { return 0, errHeapTorn } if height := binary.LittleEndian.Uint64(buf[4:]); height > h.height { h.height = height } - for at := 16; at < end; at += rec { + if head := int64(binary.LittleEndian.Uint64(buf[12:])); head > h.head { + h.head = head + } + for at := heapDeltaHdr; at < end; at += heapDeltaRec { var key [32]byte copy(key[:], buf[at:]) h.index[key] = slot{off: int64(binary.LittleEndian.Uint64(buf[at+32:])), @@ -503,48 +595,32 @@ func (h *HeapStore) applyDelta(buf []byte) (int, error) { return end + 4, nil } -// deriveHoles rebuilds the free lists and the append point from the -// key map: the file's bytes no live slot covers are holes, split into -// size classes. The caller holds the lock. -func (h *HeapStore) deriveHoles() error { - slots := make([]slot, 0, len(h.index)) +// deriveExtent finds the append point from the key map and cuts the +// file back to it: anything past the last named slot is a crash's +// torn writes or the entries of a block whose delta never became +// durable, and must not be read as anything. The caller holds the +// lock. +func (h *HeapStore) deriveExtent() error { + var end int64 + h.liveBytes = 0 for _, s := range h.index { - slots = append(slots, s) - } - sort.Slice(slots, func(i, j int) bool { return slots[i].off < slots[j].off }) - h.free = [heapClasses][]int64{} - h.holeBytes, h.liveBytes = 0, 0 - var at int64 - for _, s := range slots { - if s.off < at { - return fmt.Errorf("heap: slots overlap at %d", s.off) + if s.off < h.head { + return fmt.Errorf("heap: a live slot at %d lies below the head %d", s.off, h.head) } - h.gapToHoles(at, s.off) - h.liveBytes += int64(s.cap) - at = s.off + int64(s.cap) - } - h.size = at - // Anything past the last live slot is unnamed: a crash's torn - // writes, or entries of a block whose delta never became durable. - // The file is cut back to the append point so they are not holes - // that could be mistaken for anything. - return h.file.Truncate(at) -} - -// gapToHoles splits [from, to) into size-class slots, largest first. -func (h *HeapStore) gapToHoles(from, to int64) { - for from < to { - rest := to - from - capacity := uint32(heapMinCap) - for int64(capacity)*2 <= rest && capacity < 1<<30 { - capacity <<= 1 - } - if int64(capacity) > rest { - return // A remainder smaller than the smallest slot is lost until a move reclaims it + if e := s.off + int64(s.cap); e > end { + end = e } - h.addHole(from, capacity) - from += int64(capacity) + h.liveBytes += int64(s.cap) } + h.size = end + h.cleanedTo = h.head + if err := h.file.Truncate(end); err != nil { + return err + } + if h.head > 0 { + return punchHole(h.file, 0, h.head) // Idempotent: the region is already released + } + return nil } // Stats maps the heap's counters onto the store's report: every read @@ -555,7 +631,7 @@ func (h *HeapStore) Stats() StoreStats { defer h.mu.RUnlock() return StoreStats{ PutTotal: h.putTotal.Load(), - PutNew: h.putAppend.Load() + h.putHole.Load(), + PutNew: h.putAppend.Load(), PutDuplicate: h.putInPlace.Load(), LookupTotal: h.lookups.Load(), LiveHit: h.hits.Load(), @@ -563,12 +639,18 @@ func (h *HeapStore) Stats() StoreStats { } } -// HoleRatio reports the heap's free capacity against its live -// capacity, the number a bounded move is scheduled on. -func (h *HeapStore) HoleRatio() (holes, live int64) { +// HoleRatio reports the dead bytes between the head and the append +// point against the live bytes: what the cleaner has yet to reclaim. +func (h *HeapStore) HoleRatio() (dead, live int64) { h.mu.RLock() defer h.mu.RUnlock() - return h.holeBytes, h.liveBytes + return h.size - h.head - h.liveBytes, h.liveBytes +} + +// Cleaned reports what the cleaner has scanned and what it had to +// copy: the ratio is the heap's write amplification. +func (h *HeapStore) Cleaned() (scanned, moved uint64) { + return h.cleanedBytes.Load(), h.movedBytes.Load() } // SetFilterBlocks and SetSealLimit are the segment layer's knobs; a @@ -576,12 +658,6 @@ func (h *HeapStore) HoleRatio() (holes, live int64) { func (h *HeapStore) SetFilterBlocks(uint64) error { return nil } func (h *HeapStore) SetSealLimit(uint64) error { return nil } -// compact is the heap's maintenance on the adapter's cadence: the key -// map snapshot that bounds the replay on open. (The bounded move -// that makes bigger holes is not written yet; holes cycle by size -// class meanwhile.) -func (h *HeapStore) compact() (bool, error) { return true, h.Snapshot() } - // LiveRecords is the live key count; a heap is never sealed on it. func (h *HeapStore) LiveRecords() uint64 { h.mu.RLock() diff --git a/database/heap_linux.go b/database/heap_linux.go new file mode 100644 index 0000000..876cabc --- /dev/null +++ b/database/heap_linux.go @@ -0,0 +1,15 @@ +//go:build linux + +package blockchainDB + +import ( + "os" + "syscall" +) + +// punchHole releases the bytes of [off, off+n) back to the filesystem +// without changing the file's size: the cleaned head of a heap. +func punchHole(f *os.File, off, n int64) error { + const punch = 0x02 | 0x01 // FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE + return syscall.Fallocate(int(f.Fd()), punch, off, n) +} diff --git a/database/heap_other.go b/database/heap_other.go new file mode 100644 index 0000000..1a20410 --- /dev/null +++ b/database/heap_other.go @@ -0,0 +1,9 @@ +//go:build !linux + +package blockchainDB + +import "os" + +// punchHole is a no-op where the filesystem cannot release a range; +// the cleaned head then costs disk until the file is rewritten. +func punchHole(*os.File, int64, int64) error { return nil } diff --git a/database/heap_test.go b/database/heap_test.go index c3c487f..d3ec213 100644 --- a/database/heap_test.go +++ b/database/heap_test.go @@ -16,9 +16,10 @@ func heapDir(t *testing.T) string { func key(b byte) (k [32]byte) { k[0] = b; return } // A key rewritten within the block reuses its slot; rewritten in a -// later block it takes a new slot, and the old one is a hole only -// after the sync that stops naming it. -func TestHeapRewriteReusesWithinTheBlockAndFreesOneSyncLate(t *testing.T) { +// later block it is appended, the old slot dead where it lies until a +// clean pass moves the live entries past it and the next sync +// releases the region. +func TestHeapRewriteReusesWithinTheBlockAndCleansOneSyncLate(t *testing.T) { h, err := NewHeapStore(heapDir(t)) require.NoError(t, err) defer h.Close() @@ -33,25 +34,35 @@ func TestHeapRewriteReusesWithinTheBlockAndFreesOneSyncLate(t *testing.T) { require.Equal(t, "uno", string(v)) // Block 1 durable; block 2 rewrites the key - p, err := h.beginBlockSync() - require.NoError(t, err) - require.NoError(t, p.finish()) + sync := func() { + p, err := h.beginBlockSync() + require.NoError(t, err) + require.NoError(t, p.finish()) + } + sync() h.AdvanceBlock(2) require.NoError(t, h.Put(key(1), []byte("two"))) require.NotEqual(t, off, h.index[key(1)].off, "a durable slot is never rewritten") - holes, _ := h.HoleRatio() - require.Zero(t, holes, "the old slot is not a hole until block 2 is durable") - p, err = h.beginBlockSync() - require.NoError(t, err) - require.NoError(t, p.finish()) - holes, _ = h.HoleRatio() - require.EqualValues(t, heapMinCap, holes, "now it is") + dead, live := h.HoleRatio() + require.EqualValues(t, heapMinCap, dead, "the old slot is dead where it lies") + require.EqualValues(t, heapMinCap, live) + sync() - // Block 3 fills the hole + // A clean pass in block 3 moves the live entry past the dead one; + // the region is released only by the sync after it h.AdvanceBlock(3) - require.NoError(t, h.Put(key(2), []byte("three"))) - require.Equal(t, off, h.index[key(2)].off, "the hole is reused") - require.EqualValues(t, 1, h.putHole.Load()) + cleaned, err := h.clean(1 << 20) + require.NoError(t, err) + require.True(t, cleaned) + require.EqualValues(t, 0, h.head, "not released yet: the copies are not durable") + scanned, moved := h.Cleaned() + require.EqualValues(t, 2*heapMinCap, scanned) + require.EqualValues(t, heapMinCap, moved, "one live entry copied, one dead skipped") + sync() + require.EqualValues(t, 2*heapMinCap, h.head, "released after the sync") + dead, live = h.HoleRatio() + require.Zero(t, dead) + require.EqualValues(t, heapMinCap, live) v, err = h.Get(key(1)) require.NoError(t, err) require.Equal(t, "two", string(v)) @@ -97,8 +108,8 @@ func TestHeapReopenKeepsTheDurableAndDropsTheRest(t *testing.T) { _, err = r.Get(key(99)) require.ErrorIs(t, err, errNotFound) require.Equal(t, size, r.size, "the file is cut back to the durable append point") - holes, _ := r.HoleRatio() - require.EqualValues(t, heapMinCap, holes, "key 1's block-1 slot is a hole again") + dead, _ := r.HoleRatio() + require.EqualValues(t, heapMinCap, dead, "key 1's block-1 slot is dead where it lies") require.EqualValues(t, 2, r.height, "the durable height: block 3 never synced") } @@ -124,7 +135,7 @@ func TestHeapTornLogTailIsDropped(t *testing.T) { require.Equal(t, "one", string(v)) st, err := os.Stat(filepath.Join(dir, "index.log")) require.NoError(t, err) - require.EqualValues(t, 16+48+4, st.Size(), "one whole delta of one key remains") + require.EqualValues(t, heapDeltaHdr+heapDeltaRec+4, st.Size(), "one whole delta of one key remains") } // A snapshot carries the map and empties the log; what comes after is @@ -133,6 +144,9 @@ func TestHeapSnapshotBoundsTheReplay(t *testing.T) { dir := heapDir(t) h, err := NewHeapStore(dir) require.NoError(t, err) + every := HeapSnapshotEvery + HeapSnapshotEvery = 1 + defer func() { HeapSnapshotEvery = every }() for b := uint64(1); b <= 30; b++ { h.AdvanceBlock(b) for i := byte(1); i <= 20; i++ { @@ -159,13 +173,13 @@ func TestHeapSnapshotBoundsTheReplay(t *testing.T) { require.NoError(t, err) require.Equal(t, []byte{30, i}, v) } - // A key rewritten every block cycles two slots: the one it holds - // and the one it held last block, a hole once this block is durable - // and the next block's slot. So after the last sync every key has - // one hole beside its live slot. - holes, live := r.HoleRatio() + // The one clean pass at block 20 found blocks 1-19 all dead and + // stopped at block 20's slots, which the sync at block 21 released; + // blocks 20-29 lie dead behind block 30's live slots. + dead, live := r.HoleRatio() require.EqualValues(t, 20*heapMinCap, live) - require.EqualValues(t, 20*heapMinCap, holes) + require.EqualValues(t, 10*20*heapMinCap, dead) + require.EqualValues(t, 19*20*heapMinCap, r.head, "released by the sync after the clean") } // A slot whose bytes were damaged is an error, never a value. diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index 5d6ff55..53415ec 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -51,42 +51,43 @@ compaction, merge, pack -- operates on indexes, which are an order of magnitude smaller than what they index (a key and a location, ~40 B, against a ~300 B value) and can be rebuilt from the entries. -## The dynamic layer: a heap with holes +## The dynamic layer: an append-and-clean heap The dynamic layer holds a bounded key set rewritten forever. Today every rewrite appends and the old copy is garbage until a fold -rewrites everything around it. Instead: - -- **Entries live in a heap file per shard.** An entry is - `[len][key][value][checksum]`; its location is `(file, offset)`. +rewrites everything around it, index and filter included. Instead: + +- **Entries are managed by appending.** An entry is + `[cap][len][key][value][checksum]` in a slot of a size class; a + block's entries are appended contiguously, so the block sync is one + sequential fsync of the heap per shard. (Filling holes wherever + they lie was built first and measured: a block's 11k rewrites + scattered over a 380 MB heap dirtied a page each, and the barrier + wrote 4x the ingest -- run 3, 443 MB/s against run 2's 117. The + commit path cannot afford scattered writes; only the cleaner can.) - **A rewrite within the block reuses the slot.** A key written again in the block that took its slot is rewritten in place when the value fits: nothing durable names the slot yet. -- **A rewrite in a later block takes a new slot.** The slot the last - durable index names is never overwritten, or a crash between the - write and the block's sync would expose an uncommitted value under - a committed name. The new entry goes into a hole that fits, or the - end of the heap; the old slot becomes a hole one sync later. A key - rewritten every block therefore cycles two slots, its own and last - block's; the heap's size for such a key is twice the entry, not a - history of it. -- **Holes are filled, not swept.** Free space is kept by size class - (`8 << n` bytes); a new entry takes the smallest hole that fits, or - the end of the file. Fragmentation is bounded by the size classes - the way a slab allocator's is, and the store reports the ratio of - hole bytes to live bytes. -- **Moving is the only rewrite, and it is bounded.** When the hole - ratio exceeds a threshold, one pass moves one entry -- the last - live entry of the file into the largest hole that fits -- and stops. - A pass costs one entry, never the file. The file shrinks from the - end when its tail is a hole. +- **A rewrite in a later block appends.** The slot the last durable + index names is never overwritten, or a crash between the write and + the block's sync would expose an uncommitted value under a committed + name. The old slot is dead where it lies. +- **A bounded cleaner makes the big hole.** On the maintenance + cadence, one pass scans up to `HeapCleanBytes` (16 MB) from the head + -- the oldest byte in use -- re-appends the entries still live, and + marks the region; the sync after the delta naming the copies + releases it (`fallocate` punch, size kept). A pass costs the live + fraction of the oldest region: small for a hot key set, and for a + cold one the price of a bounded move (1.2). The store reports bytes + scanned against bytes moved, which is the heap's write amplification. - **The key map is in memory** for the live dynamic key set (the soak's half million keys are ~24 MB per store; 1.2 allows memory that scales with the working set). It is also what the seal makes durable, below. The layer's size converges to O(live keys) by construction (1.5), -without the deeper fold 2.7 allows today. +without the deeper fold 2.7 allows today, and nothing but entries is +ever copied. ## The permanent layer: append-only data, merged indexes @@ -124,10 +125,11 @@ commit point" and closes #33. ## Durability and crash consistency (1.8) -- **A hole is reusable one seal late.** A slot freed in block N may - be reused only after N's seal is durable. Until then the durable - index still names the old slot, and a crash must find it intact. - Reuse is deferred the way 2.6 defers deletion. The adapter never +- **A region is released one seal late.** The head advances past a + cleaned region only after the delta naming the cleaner's copies is + durable. Until then the durable index still names the old slots, + and a crash must find them intact. Release is deferred the way 2.6 + defers deletion. The adapter never asks the store for an old version (its pre-images are memoized on its side), so reuse waits on the seal and on nothing else. - **A torn slot is detected, not misread.** Every entry carries its @@ -161,8 +163,7 @@ commit point" and closes #33. `Stats`), so the sharding and the adapter do not change. The platform measures it alone (`-stores 9 -perm 0`). *Written: `database/heap.go`, opened with `NewKVShardHeapN` / `NewKV2Heap`, - detected on open by its directory; `bdbench -dyna-heap`. The - bounded move is not written yet; holes cycle by size class.* + detected on open by its directory; `bdbench -dyna-heap`.* 2. The permanent index deltas and the single block file, which also brings the seal to one commit point. 3. Merge and pack over indexes. From e8d8bbd46b7218a41f17cc8c72669b47467f9100 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 17:05:51 -0500 Subject: [PATCH 03/58] The cleaner takes the deadest region, bounded by what it copies Cleaning the oldest region regardless was measured (run 4): with a skewed key set the oldest region is mostly live cold keys, so every pass copied most of its 16 MB into one block's sync, and the seal's p90 reached 1.2 s where the segment layer's was 0.2 s at the same age. The heap is now a sequence of HeapRegionBytes regions with live and dead accounting; a pass takes the region with the most dead bytes once HeapCleanRatio of it is dead, copies at most HeapCleanBytes of live entries (the rest next pass), and a region left with nothing live is punched after the sync that makes its copies durable. A byte released never costs more than a byte copied. The region a block is appending to is never taken. bdbench counts the bytes files occupy rather than their length, since a heap keeps its length and releases regions, and the live state shows the cleaner's bytes scanned against bytes copied. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/live.html | 2 +- cmd/bdbench/main.go | 12 +- database/heap.go | 252 ++++++++++++++++++++++++++---------------- database/heap_test.go | 24 ++-- 4 files changed, 181 insertions(+), 109 deletions(-) diff --git a/cmd/bdbench/live.html b/cmd/bdbench/live.html index 1caa400..8e569e8 100644 --- a/cmd/bdbench/live.html +++ b/cmd/bdbench/live.html @@ -39,7 +39,7 @@

bdbench live

document.getElementById("now").innerHTML=[ group("Now: seals, last 10 s",[["p50",L.sealP50ms.toFixed(0)+" ms","big"],["p90 (budget 100)",L.sealP90ms.toFixed(0)+" ms",L.sealP90ms>100?"warn":"ok"],["max",L.sealMaxMs.toFixed(0)+" ms"]]), group("Now: blocks, last 10 s",[["p50",L.blockP50ms.toFixed(0)+" ms","big"],["p90 (interval 1000)",L.blockP90ms.toFixed(0)+" ms",L.blockP90ms>1000?"bad":"ok"],["max",L.blockMaxMs.toFixed(0)+" ms"],["over the interval",L.over+" of "+L.blocks,L.over>0?"bad":"ok"]]), - group("Now: maintenance and heap",[["passes in flight",l.maintenanceInFlight,"big"],["heap live / holes",l.heapLiveMB.toFixed(0)+" / "+l.heapHoleMB.toFixed(0)+" MB"],["wrong answers so far",l.mismatches,l.mismatches>0?"bad":"ok"]])].join("")}catch(e){}} + group("Now: maintenance and heap",[["passes in flight",l.maintenanceInFlight,"big"],["heap live / dead",l.heapLiveMB.toFixed(0)+" / "+l.heapHoleMB.toFixed(0)+" MB"],["cleaner scanned / copied",l.heapScannedMB.toFixed(0)+" / "+l.heapMovedMB.toFixed(0)+" MB"],["wrong answers so far",l.mismatches,l.mismatches>0?"bad":"ok"]])].join("")}catch(e){}} async function tick(){let t;try{t=await (await fetch("bdbench.csv?"+Date.now())).text()}catch(e){document.getElementById("state").textContent="unreachable";return} const rows=parse(t),last=rows[rows.length-1]||{};const st=document.getElementById("state"); if(!rows.length)st.textContent="waiting for the first minute"; diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index 3b8003f..db26121 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -378,6 +378,8 @@ func procIO() (read, write uint64) { return } +// dirSize counts the bytes the files occupy, not their apparent size: +// a heap releases regions with punched holes and keeps its length. func dirSize(dir string) (files int, bytes int64) { _ = filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error { if err != nil || d.IsDir() { @@ -385,7 +387,11 @@ func dirSize(dir string) (files int, bytes int64) { } if info, err := d.Info(); err == nil { files++ - bytes += info.Size() + if st, ok := info.Sys().(*syscall.Stat_t); ok { + bytes += st.Blocks * 512 + } else { + bytes += info.Size() + } } return nil }) @@ -422,6 +428,7 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { } var height uint64 var holes, live int64 + var scanned, moved uint64 for _, s := range stores { if s.height > height { height = s.height @@ -430,6 +437,8 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { if sh.Heap != nil { h, l := sh.Heap.HoleRatio() holes, live = holes+h, live+l + sc, mv := sh.Heap.Cleaned() + scanned, moved = scanned+sc, moved+mv } } } @@ -440,6 +449,7 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { "sealP50ms": float64(pct(st, .5)) / 1e6, "sealP90ms": float64(pct(st, .9)) / 1e6, "sealMaxMs": float64(pct(st, 1)) / 1e6}, "maintenanceInFlight": t.inFlight.Load(), "mismatches": t.mismatches.Load(), "heapHoleMB": float64(holes) / 1e6, "heapLiveMB": float64(live) / 1e6, + "heapScannedMB": float64(scanned) / 1e6, "heapMovedMB": float64(moved) / 1e6, }) return b } diff --git a/database/heap.go b/database/heap.go index 0138f8a..ab89945 100644 --- a/database/heap.go +++ b/database/heap.go @@ -23,24 +23,29 @@ import ( // // heap.dat entries: [cap u32][len u32][key 32][value][crc32 of // key+value], each in a slot of cap bytes (a size class), -// appended in the order written; the head is the oldest -// byte still in use, and everything before it is released -// index.log one delta per block sync: the head, and the (key, off, -// cap, len) of every key the block touched, checksummed +// appended in the order written; the file is a sequence of +// regions of HeapRegionBytes, and a region whose entries +// are all dead is released (a punched hole, size kept) +// index.log one delta per block sync: the (key, off, cap, len) of +// every key the block touched, checksummed // index.snap the whole key map, rewritten on the maintenance cadence // // A block's writes are contiguous, so the block sync is one sequential // fsync of the heap: filling holes wherever they lie was measured at // 4x the ingest in page writes at every barrier, and is not done. A // slot a key stops naming is dead where it lies until the cleaner -// reaches it. +// takes its region: the region with the most dead bytes, once at +// least HeapCleanRatio of it is dead, so a pass copies little. +// Cleaning the oldest region regardless was measured too: with a +// skewed key set the oldest region is mostly live cold keys, and every +// pass copied most of it into one block's sync. // // Durability (spec 1.8). The block sync fsyncs heap.dat and then // appends and fsyncs the block's delta, so an index entry is durable // only after the slot it names is. A slot the last durable index // names is never overwritten: a key rewritten in a later block takes -// a new slot at the end, and its old slot is dead but intact until -// the head passes it -- and the head advances only after the delta +// a new slot at the end, and its old slot is dead but intact until its +// region is released -- and a region is released only after the delta // naming the cleaner's copies is durable, one sync late, the way spec // 2.6 defers deletion. A key rewritten again within the same block // reuses the slot it took this block, since nothing durable names it @@ -54,15 +59,18 @@ type HeapStore struct { file *os.File // heap.dat log *os.File // index.log size int64 // Append point: the end of heap.dat - head int64 // The oldest byte in use; everything before it is released index map[[32]byte]slot height uint64 // The block being written; slots taken in it may be rewritten in place - // touched is the block's delta in the making; cleanedTo is where - // the head moves once the delta naming the cleaner's copies is - // durable; snapshots counts compact calls between snapshots. + // regions is the live and dead capacity in each HeapRegionBytes of + // the file, what the cleaner chooses by; released marks the + // regions punched. touched is the block's delta in the making; + // release holds the regions emptied by the last pass, punched once + // the delta naming their copies is durable; snapshots counts + // compact calls between snapshots. + regions []region touched map[[32]byte]struct{} - cleanedTo int64 + release []int snapshots int closed bool @@ -73,6 +81,12 @@ type HeapStore struct { cleanedBytes, movedBytes atomic.Uint64 } +// region is the accounting for one HeapRegionBytes of the heap. +type region struct { + live, dead int64 + released bool +} + // slot is where an entry lives: its offset, the capacity of the slot // (a size class) and the entry's value length. block is the height // that took the slot, which decides whether a rewrite may reuse it. @@ -88,15 +102,22 @@ const ( heapTrailer = 4 // crc32 of key+value heapMinCap = 64 // Smallest slot heapMagic = 0x48454150 // "HEAP", the delta record's marker - heapDeltaHdr = 4 + 8 + 8 + 4 // magic, height, head, count + heapDeltaHdr = 4 + 8 + 8 + 4 // magic, height, reserved, count heapDeltaRec = 32 + 8 + 4 + 4 ) -// HeapCleanBytes bounds one cleaning pass: the bytes of the oldest -// region scanned, of which only the live entries are copied. Sized so -// the cleaner keeps up with the adapter's cadence on the soak's -// volume (~10 MB appended per shard per 20 blocks) with room to spare. -var HeapCleanBytes int64 = 16 << 20 +// HeapRegionBytes is the size of the regions the heap is cleaned and +// released by. +var HeapRegionBytes int64 = 4 << 20 + +// HeapCleanBytes bounds one cleaning pass by the bytes it COPIES: the +// most a pass can add to the next block's sync. +var HeapCleanBytes int64 = 2 << 20 + +// HeapCleanRatio is the dead fraction a region must reach before the +// cleaner takes it: what bounds the copying a pass does per byte it +// releases. +var HeapCleanRatio = 0.5 // HeapSnapshotEvery is how many compact calls pass between key-map // snapshots; between them the log is what open replays. @@ -149,7 +170,7 @@ func (h *HeapStore) Open() (err error) { h.index = map[[32]byte]slot{} h.touched = map[[32]byte]struct{}{} h.closed = false - h.head, h.size, h.liveBytes, h.cleanedTo = 0, 0, 0, 0 + h.size, h.liveBytes, h.regions, h.release = 0, 0, nil, nil if err = h.loadSnapshot(); err != nil { return err } @@ -213,10 +234,9 @@ func (h *HeapStore) Put(key [32]byte, value []byte) error { h.putInPlace.Add(1) } else { s = slot{off: h.size, cap: heapCap(need), n: uint32(len(value)), block: h.height} - h.size += int64(s.cap) - h.liveBytes += int64(s.cap) + h.append(s) if had { - h.liveBytes -= int64(old.cap) // Dead where it lies + h.kill(old) // Dead where it lies } h.putAppend.Add(1) } @@ -228,6 +248,27 @@ func (h *HeapStore) Put(key [32]byte, value []byte) error { return nil } +// append accounts a slot taken at the end of the file. The caller +// holds the lock and has set s.off to the append point. +func (h *HeapStore) append(s slot) { + h.size += int64(s.cap) + h.liveBytes += int64(s.cap) + r := int(s.off / HeapRegionBytes) + for len(h.regions) <= r { + h.regions = append(h.regions, region{}) + } + h.regions[r].live += int64(s.cap) +} + +// kill accounts a slot its key stopped naming. The caller holds the +// lock. +func (h *HeapStore) kill(s slot) { + h.liveBytes -= int64(s.cap) + r := &h.regions[int(s.off/HeapRegionBytes)] + r.live -= int64(s.cap) + r.dead += int64(s.cap) +} + // Get answers from the key map and one read of the slot. A slot // whose checksum fails is reported as corrupt, never as a value. func (h *HeapStore) Get(key [32]byte) ([]byte, error) { @@ -276,11 +317,11 @@ func (h *HeapStore) AdvanceBlock(height uint64) { } // heapSync is a block sync in flight: the delta to make durable and -// the head to release once it is. +// the regions to release once it is. type heapSync struct { - h *HeapStore - delta []byte - cleanedTo int64 + h *HeapStore + delta []byte + release []int } // beginBlockSync takes the block's delta under the lock; finish makes @@ -291,22 +332,22 @@ func (h *HeapStore) beginBlockSync() (blockSync, error) { if h.closed || h.file == nil { return nil, errStoreClosed } - p := &heapSync{h: h, cleanedTo: h.cleanedTo} - if len(h.touched) > 0 || h.cleanedTo > h.head { + p := &heapSync{h: h, release: h.release} + h.release = nil + if len(h.touched) > 0 || len(p.release) > 0 { p.delta = h.encodeDelta() h.touched = map[[32]byte]struct{}{} } return p, nil } -// encodeDelta is the block's index delta: marker, height, the head -// the block's copies let the heap release, the touched keys' slots, -// and a checksum. The caller holds the lock. +// encodeDelta is the block's index delta: marker, height, a reserved +// word, the touched keys' slots, and a checksum. The caller holds +// the lock. func (h *HeapStore) encodeDelta() []byte { buf := make([]byte, heapDeltaHdr+len(h.touched)*heapDeltaRec+4) binary.LittleEndian.PutUint32(buf, heapMagic) binary.LittleEndian.PutUint64(buf[4:], h.height) - binary.LittleEndian.PutUint64(buf[12:], uint64(h.cleanedTo)) binary.LittleEndian.PutUint32(buf[20:], uint32(len(h.touched))) at := heapDeltaHdr for key := range h.touched { @@ -324,8 +365,8 @@ func putDeltaRec(buf []byte, key [32]byte, s slot) int { return heapDeltaRec } -// finish: entries durable, then the delta durable, then the head the -// delta records is released. +// finish: entries durable, then the delta durable, then the regions +// the delta's copies emptied are released. func (p *heapSync) finish() error { h := p.h if p.delta == nil { @@ -342,11 +383,11 @@ func (p *heapSync) finish() error { } h.mu.Lock() defer h.mu.Unlock() - if p.cleanedTo > h.head { - if err := punchHole(h.file, h.head, p.cleanedTo-h.head); err != nil { + for _, r := range p.release { + if err := punchHole(h.file, int64(r)*HeapRegionBytes, HeapRegionBytes); err != nil { return err } - h.head = p.cleanedTo + h.regions[r] = region{released: true} } return nil } @@ -372,72 +413,80 @@ func (h *HeapStore) compact() (bool, error) { return cleaned, err } -// clean scans up to budget bytes from the head, re-appends the entries -// still live, and marks the region for release at the next sync. The -// cost of a pass is the live fraction of the oldest region: for a hot -// key set rewritten every block it is small, and for a cold one it is -// the price of a bounded move (spec 1.2). Holds the lock for the -// pass: bounded, and the copies are ordinary appends. +// clean takes the region with the most dead bytes, once at least +// HeapCleanRatio of it is dead, re-appends its live entries -- at most +// budget bytes of them, the rest next pass -- and, when none are left, +// marks it for release at the next sync. The cost of a pass is bounded +// by the copies it makes and by the ratio: a byte released never costs +// more than a byte copied (spec 1.2). Holds the lock for the pass: +// bounded, and the copies are ordinary appends. func (h *HeapStore) clean(budget int64) (bool, error) { h.mu.Lock() defer h.mu.Unlock() if h.closed { return false, errStoreClosed } - if h.cleanedTo > h.head { + if len(h.release) > 0 { return false, nil // The last pass's release is still waiting on a sync } - from, to := h.head, h.head+budget - if to > h.size { - to = h.size + // The region the block in progress is appending to is never taken: + // a slot taken this block may still be rewritten in place, and + // moving it would race that + current := int(h.size / HeapRegionBytes) + pick, best := -1, 0.0 + for i, r := range h.regions { + if i >= current || r.released || r.dead == 0 { + continue + } + if f := float64(r.dead) / float64(r.dead+r.live); f >= HeapCleanRatio && f > best { + pick, best = i, f + } } - if from >= to || h.liveBytes == 0 { + if pick < 0 { return false, nil } - // The pass never eats the block in progress: a slot taken this - // block may still be rewritten in place, and moving it would race - // that. Stop at the first slot of the current block. - region := make([]byte, to-from) - if _, err := h.file.ReadAt(region, from); err != nil && !errors.Is(err, io.EOF) { + from := int64(pick) * HeapRegionBytes + to := from + HeapRegionBytes + if to > h.size { + to = h.size + } + buf := make([]byte, to-from) + if _, err := h.file.ReadAt(buf, from); err != nil && !errors.Is(err, io.EOF) { return false, err } - var off = from - var moved int64 - for off < to { - at := off - from - if at+heapHeader > int64(len(region)) { - break - } - capacity := binary.LittleEndian.Uint32(region[at:]) - n := binary.LittleEndian.Uint32(region[at+4:]) - if capacity == 0 || at+int64(capacity) > int64(len(region)) { - break // Unwritten, or a slot that straddles the budget: next pass + var moved, at int64 + for at+heapHeader <= int64(len(buf)) { + capacity := binary.LittleEndian.Uint32(buf[at:]) + if capacity == 0 || at+int64(capacity) > int64(len(buf)) { + break // Unwritten, or a slot that straddles the region } var key [32]byte - copy(key[:], region[at+8:]) - s, live := h.index[key] - if live && s.off == off { - if s.block == h.height { - break + copy(key[:], buf[at+8:]) + if s, live := h.index[key]; live && s.off == from+at { + if moved >= budget { + h.cleanedBytes.Add(uint64(at)) + h.movedBytes.Add(uint64(moved)) + return true, nil // The rest next pass } - entry := region[at : at+int64(heapHeader)+int64(n)+heapTrailer] + n := binary.LittleEndian.Uint32(buf[at+4:]) + entry := buf[at : at+int64(heapHeader)+int64(n)+heapTrailer] ns := slot{off: h.size, cap: capacity, n: n, block: h.height} if _, err := h.file.WriteAt(entry, ns.off); err != nil { return false, err } - h.size += int64(capacity) + h.append(ns) + h.kill(s) h.index[key] = ns h.touched[key] = struct{}{} moved += int64(capacity) } - off += int64(capacity) + at += int64(capacity) } - if off == from { - return false, nil - } - h.cleanedTo = off - h.cleanedBytes.Add(uint64(off - from)) + h.cleanedBytes.Add(uint64(at)) h.movedBytes.Add(uint64(moved)) + if h.regions[pick].live == 0 { + h.release = append(h.release, pick) + } return true, nil } @@ -453,7 +502,6 @@ func (h *HeapStore) Snapshot() error { buf := make([]byte, heapDeltaHdr+len(h.index)*heapDeltaRec+4) binary.LittleEndian.PutUint32(buf, heapMagic) binary.LittleEndian.PutUint64(buf[4:], h.height) - binary.LittleEndian.PutUint64(buf[12:], uint64(h.head)) binary.LittleEndian.PutUint32(buf[20:], uint32(len(h.index))) at := heapDeltaHdr for key, s := range h.index { @@ -583,9 +631,6 @@ func (h *HeapStore) applyDelta(buf []byte) (int, error) { if height := binary.LittleEndian.Uint64(buf[4:]); height > h.height { h.height = height } - if head := int64(binary.LittleEndian.Uint64(buf[12:])); head > h.head { - h.head = head - } for at := heapDeltaHdr; at < end; at += heapDeltaRec { var key [32]byte copy(key[:], buf[at:]) @@ -595,30 +640,42 @@ func (h *HeapStore) applyDelta(buf []byte) (int, error) { return end + 4, nil } -// deriveExtent finds the append point from the key map and cuts the -// file back to it: anything past the last named slot is a crash's -// torn writes or the entries of a block whose delta never became -// durable, and must not be read as anything. The caller holds the -// lock. +// deriveExtent finds the append point and the regions' accounting +// from the key map and cuts the file back to it: anything past the +// last named slot is a crash's torn writes or the entries of a block +// whose delta never became durable, and must not be read as anything. +// A region with nothing live is released (idempotent for one already +// punched). The caller holds the lock. func (h *HeapStore) deriveExtent() error { var end int64 - h.liveBytes = 0 + h.liveBytes, h.regions, h.release = 0, nil, nil for _, s := range h.index { - if s.off < h.head { - return fmt.Errorf("heap: a live slot at %d lies below the head %d", s.off, h.head) - } if e := s.off + int64(s.cap); e > end { end = e } h.liveBytes += int64(s.cap) + r := int(s.off / HeapRegionBytes) + for len(h.regions) <= r { + h.regions = append(h.regions, region{}) + } + h.regions[r].live += int64(s.cap) } h.size = end - h.cleanedTo = h.head if err := h.file.Truncate(end); err != nil { return err } - if h.head > 0 { - return punchHole(h.file, 0, h.head) // Idempotent: the region is already released + for i := range h.regions { + extent := HeapRegionBytes + if e := end - int64(i)*HeapRegionBytes; e < extent { + extent = e + } + h.regions[i].dead = extent - h.regions[i].live + if h.regions[i].live == 0 && int64(i+1)*HeapRegionBytes <= end { + if err := punchHole(h.file, int64(i)*HeapRegionBytes, HeapRegionBytes); err != nil { + return err + } + h.regions[i] = region{released: true} + } } return nil } @@ -639,12 +696,15 @@ func (h *HeapStore) Stats() StoreStats { } } -// HoleRatio reports the dead bytes between the head and the append -// point against the live bytes: what the cleaner has yet to reclaim. +// HoleRatio reports the dead bytes in unreleased regions against the +// live bytes: what the cleaner has yet to reclaim. func (h *HeapStore) HoleRatio() (dead, live int64) { h.mu.RLock() defer h.mu.RUnlock() - return h.size - h.head - h.liveBytes, h.liveBytes + for _, r := range h.regions { + dead += r.dead + } + return dead, h.liveBytes } // Cleaned reports what the cleaner has scanned and what it had to diff --git a/database/heap_test.go b/database/heap_test.go index d3ec213..292025e 100644 --- a/database/heap_test.go +++ b/database/heap_test.go @@ -48,18 +48,22 @@ func TestHeapRewriteReusesWithinTheBlockAndCleansOneSyncLate(t *testing.T) { require.EqualValues(t, heapMinCap, live) sync() - // A clean pass in block 3 moves the live entry past the dead one; - // the region is released only by the sync after it + // A clean pass in block 3 takes the region (half dead) and moves + // the live entry out of it; the region is released only by the + // sync after it. The region must not be the one block 3 appends + // to, so the file is pushed into a second region first. h.AdvanceBlock(3) + h.size = HeapRegionBytes // Block 3 appends into region 1 cleaned, err := h.clean(1 << 20) require.NoError(t, err) require.True(t, cleaned) - require.EqualValues(t, 0, h.head, "not released yet: the copies are not durable") + require.Equal(t, []int{0}, h.release, "not released yet: the copies are not durable") + require.False(t, h.regions[0].released) scanned, moved := h.Cleaned() require.EqualValues(t, 2*heapMinCap, scanned) require.EqualValues(t, heapMinCap, moved, "one live entry copied, one dead skipped") sync() - require.EqualValues(t, 2*heapMinCap, h.head, "released after the sync") + require.True(t, h.regions[0].released, "released after the sync") dead, live = h.HoleRatio() require.Zero(t, dead) require.EqualValues(t, heapMinCap, live) @@ -156,9 +160,8 @@ func TestHeapSnapshotBoundsTheReplay(t *testing.T) { require.NoError(t, err) require.NoError(t, p.finish()) if b == 20 { - ok, err := h.compact() + _, err := h.compact() // Nothing to clean: one region, still being appended to require.NoError(t, err) - require.True(t, ok) st, err := os.Stat(filepath.Join(dir, "index.log")) require.NoError(t, err) require.Zero(t, st.Size(), "the log is empty after the snapshot") @@ -173,13 +176,12 @@ func TestHeapSnapshotBoundsTheReplay(t *testing.T) { require.NoError(t, err) require.Equal(t, []byte{30, i}, v) } - // The one clean pass at block 20 found blocks 1-19 all dead and - // stopped at block 20's slots, which the sync at block 21 released; - // blocks 20-29 lie dead behind block 30's live slots. + // Everything fits in the region the blocks append to, which the + // cleaner never takes: blocks 1-29 lie dead behind block 30's live + // slots, and the accounting survives the reopen. dead, live := r.HoleRatio() require.EqualValues(t, 20*heapMinCap, live) - require.EqualValues(t, 10*20*heapMinCap, dead) - require.EqualValues(t, 19*20*heapMinCap, r.head, "released by the sync after the clean") + require.EqualValues(t, 29*20*heapMinCap, dead) } // A slot whose bytes were damaged is an error, never a value. From a946758ffe680cf6211c901d2cdcf5ef8ac7a0c6 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 17:07:36 -0500 Subject: [PATCH 04/58] Time the heap's sync: the heap's fsync apart from the delta's The heap's seal is 3.5x the segment layer's at the same age with the cleaner idle (run 5, minute 1), so the sync's two barriers are timed separately and the bytes each fsync covered counted; the platform's live state shows the averages. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/live.html | 2 +- cmd/bdbench/main.go | 7 ++++++- database/heap.go | 26 +++++++++++++++++++++++--- 3 files changed, 30 insertions(+), 5 deletions(-) diff --git a/cmd/bdbench/live.html b/cmd/bdbench/live.html index 8e569e8..0123149 100644 --- a/cmd/bdbench/live.html +++ b/cmd/bdbench/live.html @@ -39,7 +39,7 @@

bdbench live

document.getElementById("now").innerHTML=[ group("Now: seals, last 10 s",[["p50",L.sealP50ms.toFixed(0)+" ms","big"],["p90 (budget 100)",L.sealP90ms.toFixed(0)+" ms",L.sealP90ms>100?"warn":"ok"],["max",L.sealMaxMs.toFixed(0)+" ms"]]), group("Now: blocks, last 10 s",[["p50",L.blockP50ms.toFixed(0)+" ms","big"],["p90 (interval 1000)",L.blockP90ms.toFixed(0)+" ms",L.blockP90ms>1000?"bad":"ok"],["max",L.blockMaxMs.toFixed(0)+" ms"],["over the interval",L.over+" of "+L.blocks,L.over>0?"bad":"ok"]]), - group("Now: maintenance and heap",[["passes in flight",l.maintenanceInFlight,"big"],["heap live / dead",l.heapLiveMB.toFixed(0)+" / "+l.heapHoleMB.toFixed(0)+" MB"],["cleaner scanned / copied",l.heapScannedMB.toFixed(0)+" / "+l.heapMovedMB.toFixed(0)+" MB"],["wrong answers so far",l.mismatches,l.mismatches>0?"bad":"ok"]])].join("")}catch(e){}} + group("Now: maintenance and heap",[["passes in flight",l.maintenanceInFlight,"big"],["heap live / dead",l.heapLiveMB.toFixed(0)+" / "+l.heapHoleMB.toFixed(0)+" MB"],["cleaner scanned / copied",l.heapScannedMB.toFixed(0)+" / "+l.heapMovedMB.toFixed(0)+" MB"],["shard sync: heap fsync / delta",l.heapFsyncMsAvg.toFixed(1)+" / "+l.heapDeltaMsAvg.toFixed(1)+" ms avg"],["shard sync covers",l.heapSyncKBAvg.toFixed(0)+" KB avg"],["wrong answers so far",l.mismatches,l.mismatches>0?"bad":"ok"]])].join("")}catch(e){}} async function tick(){let t;try{t=await (await fetch("bdbench.csv?"+Date.now())).text()}catch(e){document.getElementById("state").textContent="unreachable";return} const rows=parse(t),last=rows[rows.length-1]||{};const st=document.getElementById("state"); if(!rows.length)st.textContent="waiting for the first minute"; diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index db26121..9dc391b 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -428,7 +428,8 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { } var height uint64 var holes, live int64 - var scanned, moved uint64 + var scanned, moved, syncs, syncBytes uint64 + var heapFsync, deltaSync time.Duration for _, s := range stores { if s.height > height { height = s.height @@ -439,6 +440,8 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { holes, live = holes+h, live+l sc, mv := sh.Heap.Cleaned() scanned, moved = scanned+sc, moved+mv + n, b, hf, ds := sh.Heap.SyncCost() + syncs, syncBytes, heapFsync, deltaSync = syncs+n, syncBytes+b, heapFsync+hf, deltaSync+ds } } } @@ -450,6 +453,8 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { "maintenanceInFlight": t.inFlight.Load(), "mismatches": t.mismatches.Load(), "heapHoleMB": float64(holes) / 1e6, "heapLiveMB": float64(live) / 1e6, "heapScannedMB": float64(scanned) / 1e6, "heapMovedMB": float64(moved) / 1e6, + "heapSyncs": syncs, "heapSyncKBAvg": float64(syncBytes) / 1e3 / float64(max(syncs, 1)), + "heapFsyncMsAvg": float64(heapFsync) / 1e6 / float64(max(syncs, 1)), "heapDeltaMsAvg": float64(deltaSync) / 1e6 / float64(max(syncs, 1)), }) return b } diff --git a/database/heap.go b/database/heap.go index ab89945..4a8b2f7 100644 --- a/database/heap.go +++ b/database/heap.go @@ -10,6 +10,7 @@ import ( "path/filepath" "sync" "sync/atomic" + "time" ) // HeapStore is the dynamic layer as an append-and-clean heap (proposal @@ -72,6 +73,7 @@ type HeapStore struct { touched map[[32]byte]struct{} release []int snapshots int + syncedTo int64 // The append point at the last sync closed bool liveBytes int64 // Capacity of every named slot @@ -79,6 +81,9 @@ type HeapStore struct { putTotal, putInPlace, putAppend atomic.Uint64 lookups, hits atomic.Uint64 cleanedBytes, movedBytes atomic.Uint64 + // The sync's cost, split: nanoseconds in the heap's fsync and in + // the delta's write and fsync, and the syncs and bytes they covered + syncs, syncHeapNs, syncLogNs, syncBytes atomic.Uint64 } // region is the accounting for one HeapRegionBytes of the heap. @@ -170,7 +175,7 @@ func (h *HeapStore) Open() (err error) { h.index = map[[32]byte]slot{} h.touched = map[[32]byte]struct{}{} h.closed = false - h.size, h.liveBytes, h.regions, h.release = 0, 0, nil, nil + h.size, h.liveBytes, h.regions, h.release, h.syncedTo = 0, 0, nil, nil, 0 if err = h.loadSnapshot(); err != nil { return err } @@ -322,6 +327,7 @@ type heapSync struct { h *HeapStore delta []byte release []int + bytes int64 // Appended since the last sync: what the heap's fsync covers } // beginBlockSync takes the block's delta under the lock; finish makes @@ -332,7 +338,8 @@ func (h *HeapStore) beginBlockSync() (blockSync, error) { if h.closed || h.file == nil { return nil, errStoreClosed } - p := &heapSync{h: h, release: h.release} + p := &heapSync{h: h, release: h.release, bytes: h.size - h.syncedTo} + h.syncedTo = h.size h.release = nil if len(h.touched) > 0 || len(p.release) > 0 { p.delta = h.encodeDelta() @@ -372,15 +379,21 @@ func (p *heapSync) finish() error { if p.delta == nil { return nil } + t := time.Now() if err := fsync(h.file); err != nil { return err } + h.syncHeapNs.Add(uint64(time.Since(t))) + t = time.Now() if _, err := h.log.Write(p.delta); err != nil { return err } if err := fsync(h.log); err != nil { return err } + h.syncLogNs.Add(uint64(time.Since(t))) + h.syncs.Add(1) + h.syncBytes.Add(uint64(p.bytes)) h.mu.Lock() defer h.mu.Unlock() for _, r := range p.release { @@ -660,7 +673,7 @@ func (h *HeapStore) deriveExtent() error { } h.regions[r].live += int64(s.cap) } - h.size = end + h.size, h.syncedTo = end, end if err := h.file.Truncate(end); err != nil { return err } @@ -707,6 +720,13 @@ func (h *HeapStore) HoleRatio() (dead, live int64) { return dead, h.liveBytes } +// SyncCost reports the block syncs so far: how many, the bytes their +// heap fsyncs covered, and the time spent in the heap's fsync and in +// the delta's write and fsync. +func (h *HeapStore) SyncCost() (syncs, bytes uint64, heapFsync, delta time.Duration) { + return h.syncs.Load(), h.syncBytes.Load(), time.Duration(h.syncHeapNs.Load()), time.Duration(h.syncLogNs.Load()) +} + // Cleaned reports what the cleaner has scanned and what it had to // copy: the ratio is the heap's write amplification. func (h *HeapStore) Cleaned() (scanned, moved uint64) { From 82702eeed4dce02132219cb414f5b014fc371a5a Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 17:18:59 -0500 Subject: [PATCH 05/58] An entry carries its block; the heap repairs from the data alone The entry header gains the height of the block that wrote it, so that with the index files gone a sequential scan rebuilds the key map: the highest committed copy of a key wins, an entry above the committed height -- a block whose sync never finished -- is dropped, and a torn or damaged slot is skipped. RepairHeapStore takes the committed height from the store above, rebuilds, and snapshots; OpenHeapStore refuses a heap with data and no index rather than opening it empty. The cleaner's copies carry the height of the pass, so a repair prefers them to the originals. A pass now takes several mostly-dead regions within its copy budget, since one region per twenty blocks fell behind the append rate (run 5: 11 GB dead against 2 GB live). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 191 +++++++++++++++++++++++++++++++++++------- database/heap_test.go | 47 +++++++++++ 2 files changed, 209 insertions(+), 29 deletions(-) diff --git a/database/heap.go b/database/heap.go index 4a8b2f7..ca46e73 100644 --- a/database/heap.go +++ b/database/heap.go @@ -22,8 +22,10 @@ import ( // // Files, in one directory: // -// heap.dat entries: [cap u32][len u32][key 32][value][crc32 of -// key+value], each in a slot of cap bytes (a size class), +// heap.dat entries: [cap u32][len u32][height u64][key 32][value] +// [crc32 of height+key+value], each in a slot of cap bytes +// (a size class), self-describing so that Repair can +// rebuild the key map from the data alone, // appended in the order written; the file is a sequence of // regions of HeapRegionBytes, and a region whose entries // are all dead is released (a punched hole, size kept) @@ -90,6 +92,7 @@ type HeapStore struct { type region struct { live, dead int64 released bool + cleaning bool // Taken by the pass in progress } // slot is where an entry lives: its offset, the capacity of the slot @@ -103,11 +106,11 @@ type slot struct { } const ( - heapHeader = 4 + 4 + 32 // cap, len, key - heapTrailer = 4 // crc32 of key+value - heapMinCap = 64 // Smallest slot - heapMagic = 0x48454150 // "HEAP", the delta record's marker - heapDeltaHdr = 4 + 8 + 8 + 4 // magic, height, reserved, count + heapHeader = 4 + 4 + 8 + 32 // cap, len, height, key + heapTrailer = 4 // crc32 of key+value + heapMinCap = 64 // Smallest slot + heapMagic = 0x48454150 // "HEAP", the delta record's marker + heapDeltaHdr = 4 + 8 + 8 + 4 // magic, height, reserved, count heapDeltaRec = 32 + 8 + 4 + 4 ) @@ -116,9 +119,16 @@ const ( var HeapRegionBytes int64 = 4 << 20 // HeapCleanBytes bounds one cleaning pass by the bytes it COPIES: the -// most a pass can add to the next block's sync. +// most a pass can add to the next block's sync. A pass takes region +// after region until the budget is spent, so mostly-dead regions, +// which cost little to copy, are released several to a pass. var HeapCleanBytes int64 = 2 << 20 +// HeapCleanRegions bounds a pass by regions taken as well, so that a +// heap of wholly dead regions is not scanned end to end under the +// lock in one pass. +var HeapCleanRegions = 8 + // HeapCleanRatio is the dead fraction a region must reach before the // cleaner takes it: what bounds the copying a pass does per byte it // releases. @@ -149,15 +159,84 @@ func NewHeapStore(directory string) (*HeapStore, error) { return h, h.Open() } -// OpenHeapStore opens the heap in directory as it was left. +// OpenHeapStore opens the heap in directory as it was left. A heap +// whose data is there but whose index files are not is refused: that +// is what RepairHeapStore is for, and it needs the committed height, +// which only the store above knows. func OpenHeapStore(directory string) (*HeapStore, error) { - if _, err := os.Stat(filepath.Join(directory, "heap.dat")); err != nil { + st, err := os.Stat(filepath.Join(directory, "heap.dat")) + if err != nil { return nil, fmt.Errorf("open heap at %s: %w", directory, err) } + _, snapErr := os.Stat(filepath.Join(directory, "index.snap")) + _, logErr := os.Stat(filepath.Join(directory, "index.log")) + if st.Size() > 0 && snapErr != nil && logErr != nil { + return nil, fmt.Errorf("open heap at %s: %w", directory, ErrHeapNeedsRepair) + } h := &HeapStore{Directory: directory} return h, h.Open() } +// ErrHeapNeedsRepair says the heap's data is there and its index is +// not: RepairHeapStore rebuilds the index from the data. +var ErrHeapNeedsRepair = errors.New("heap has data but no index; repair it") + +// RepairHeapStore rebuilds the key map by reading the keys from the +// data: every entry carries its key, its height and a checksum, so a +// sequential scan recovers the map without any index. For each key +// the copy with the highest height wins, and an entry above the +// committed height -- a block whose sync never finished -- is +// dropped, as is any torn or damaged slot. The rebuilt map is +// snapshotted, so the next open is an ordinary one. +func RepairHeapStore(directory string, committed uint64) (*HeapStore, error) { + os.Remove(filepath.Join(directory, "index.snap")) + os.Remove(filepath.Join(directory, "index.log")) + h := &HeapStore{Directory: directory} + var err error + if h.file, err = os.OpenFile(filepath.Join(directory, "heap.dat"), os.O_RDWR, 0o644); err != nil { + return nil, err + } + if h.log, err = os.OpenFile(filepath.Join(directory, "index.log"), os.O_RDWR|os.O_CREATE|os.O_APPEND, 0o644); err != nil { + return nil, err + } + h.index = map[[32]byte]slot{} + h.touched = map[[32]byte]struct{}{} + data, err := readFrom(h.file, 0) + if err != nil { + return nil, err + } + heights := map[[32]byte]uint64{} + var off int64 + for off < int64(len(data)) { + capacity, height, key, value, ok := decodeEntry(data[off:]) + if !ok { + // A released region reads as zeros; skip to the next region + // boundary. Anything else torn ends the scan: nothing past + // a torn slot was named before it. + if binary.LittleEndian.Uint32(data[off:]) == 0 { + off = (off/HeapRegionBytes + 1) * HeapRegionBytes + continue + } + break + } + if height <= committed { + if prev, seen := heights[key]; !seen || height >= prev { + heights[key] = height + h.index[key] = slot{off: off, cap: capacity, n: uint32(len(value))} + } + } + off += int64(capacity) + } + h.height = committed + if err = h.deriveExtent(); err != nil { + return nil, err + } + if err = h.Snapshot(); err != nil { + return nil, err + } + return h, nil +} + // Open loads the key map from the snapshot and the log and derives // the append point. Idempotent. func (h *HeapStore) Open() (err error) { @@ -208,17 +287,40 @@ func (h *HeapStore) Close() error { return err } -// encodeEntry lays out one entry for its slot. -func encodeEntry(capacity uint32, key [32]byte, value []byte) []byte { +// encodeEntry lays out one entry for its slot: the block that wrote +// it is in the header so that, with the index gone, a scan can tell +// the current copy of a key (the highest height) from stale ones and +// a committed entry from one whose block never synced. +func encodeEntry(capacity uint32, height uint64, key [32]byte, value []byte) []byte { buf := make([]byte, heapHeader+len(value)+heapTrailer) binary.LittleEndian.PutUint32(buf, capacity) binary.LittleEndian.PutUint32(buf[4:], uint32(len(value))) - copy(buf[8:], key[:]) + binary.LittleEndian.PutUint64(buf[8:], height) + copy(buf[16:], key[:]) copy(buf[heapHeader:], value) binary.LittleEndian.PutUint32(buf[heapHeader+len(value):], crc32.ChecksumIEEE(buf[8:heapHeader+len(value)])) return buf } +// decodeEntry checks the entry at the start of buf and returns its +// fields; ok is false for an unwritten, torn or damaged slot. +func decodeEntry(buf []byte) (capacity uint32, height uint64, key [32]byte, value []byte, ok bool) { + if len(buf) < heapHeader { + return + } + capacity = binary.LittleEndian.Uint32(buf) + n := int(binary.LittleEndian.Uint32(buf[4:])) + if capacity == 0 || heapHeader+n+heapTrailer > int(capacity) || int(capacity) > len(buf) { + return + } + if crc32.ChecksumIEEE(buf[8:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { + return + } + height = binary.LittleEndian.Uint64(buf[8:]) + copy(key[:], buf[16:]) + return capacity, height, key, buf[heapHeader : heapHeader+n], true +} + // Put writes value under key: in place if the key took its slot this // block and the value fits, else appended. The slot a durable index // names is never rewritten. @@ -245,7 +347,7 @@ func (h *HeapStore) Put(key [32]byte, value []byte) error { } h.putAppend.Add(1) } - if _, err := h.file.WriteAt(encodeEntry(s.cap, key, value), s.off); err != nil { + if _, err := h.file.WriteAt(encodeEntry(s.cap, h.height, key, value), s.off); err != nil { return err } h.index[key] = s @@ -302,7 +404,7 @@ func (h *HeapStore) Get(key [32]byte) ([]byte, error) { // value. func heapEntryValue(buf []byte, key [32]byte) ([]byte, error) { n := int(binary.LittleEndian.Uint32(buf[4:])) - if len(buf) != heapHeader+n+heapTrailer || [32]byte(buf[8:heapHeader]) != key { + if len(buf) != heapHeader+n+heapTrailer || [32]byte(buf[16:heapHeader]) != key { return nil, fmt.Errorf("heap: slot does not hold the key it is named for") } if crc32.ChecksumIEEE(buf[8:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { @@ -442,22 +544,50 @@ func (h *HeapStore) clean(budget int64) (bool, error) { if len(h.release) > 0 { return false, nil // The last pass's release is still waiting on a sync } - // The region the block in progress is appending to is never taken: - // a slot taken this block may still be rewritten in place, and - // moving it would race that + var moved int64 + taken := 0 + for taken < HeapCleanRegions && moved < budget { + pick := h.pickRegion() + if pick < 0 { + break + } + m, err := h.cleanRegion(pick, budget-moved) + if err != nil { + return taken > 0, err + } + moved += m + taken++ + } + for i := range h.regions { + h.regions[i].cleaning = false + } + return taken > 0, nil +} + +// pickRegion is the region with the most dead bytes, once at least +// HeapCleanRatio of it is dead; -1 when none qualifies. The region +// the block in progress is appending to is never taken: a slot taken +// this block may still be rewritten in place, and moving it would +// race that. A region a pass has already partly cleaned keeps its +// dead bytes and is picked again. The caller holds the lock. +func (h *HeapStore) pickRegion() int { current := int(h.size / HeapRegionBytes) pick, best := -1, 0.0 for i, r := range h.regions { - if i >= current || r.released || r.dead == 0 { + if i >= current || r.released || r.dead == 0 || r.cleaning { continue } if f := float64(r.dead) / float64(r.dead+r.live); f >= HeapCleanRatio && f > best { pick, best = i, f } } - if pick < 0 { - return false, nil - } + return pick +} + +// cleanRegion re-appends the live entries of one region, at most +// budget bytes of them, and marks the region for release when none +// are left. Returns the bytes copied. The caller holds the lock. +func (h *HeapStore) cleanRegion(pick int, budget int64) (int64, error) { from := int64(pick) * HeapRegionBytes to := from + HeapRegionBytes if to > h.size { @@ -465,8 +595,9 @@ func (h *HeapStore) clean(budget int64) (bool, error) { } buf := make([]byte, to-from) if _, err := h.file.ReadAt(buf, from); err != nil && !errors.Is(err, io.EOF) { - return false, err + return 0, err } + h.regions[pick].cleaning = true // Not picked again this pass var moved, at int64 for at+heapHeader <= int64(len(buf)) { capacity := binary.LittleEndian.Uint32(buf[at:]) @@ -474,18 +605,20 @@ func (h *HeapStore) clean(budget int64) (bool, error) { break // Unwritten, or a slot that straddles the region } var key [32]byte - copy(key[:], buf[at+8:]) + copy(key[:], buf[at+16:]) if s, live := h.index[key]; live && s.off == from+at { if moved >= budget { h.cleanedBytes.Add(uint64(at)) h.movedBytes.Add(uint64(moved)) - return true, nil // The rest next pass + return moved, nil // The rest next pass } n := binary.LittleEndian.Uint32(buf[at+4:]) - entry := buf[at : at+int64(heapHeader)+int64(n)+heapTrailer] + // The copy is this block's write of the key: it carries + // this height, so a repair scan prefers it to the original + value := buf[at+int64(heapHeader) : at+int64(heapHeader)+int64(n)] ns := slot{off: h.size, cap: capacity, n: n, block: h.height} - if _, err := h.file.WriteAt(entry, ns.off); err != nil { - return false, err + if _, err := h.file.WriteAt(encodeEntry(capacity, h.height, key, value), ns.off); err != nil { + return moved, err } h.append(ns) h.kill(s) @@ -500,7 +633,7 @@ func (h *HeapStore) clean(budget int64) (bool, error) { if h.regions[pick].live == 0 { h.release = append(h.release, pick) } - return true, nil + return moved, nil } // Snapshot writes the whole key map and drops the deltas it covers diff --git a/database/heap_test.go b/database/heap_test.go index 292025e..473c339 100644 --- a/database/heap_test.go +++ b/database/heap_test.go @@ -249,3 +249,50 @@ func TestHeapShardRoundTrip(t *testing.T) { require.Equal(t, byte(45), v[0]) } } + +// With the index files gone, the map is rebuilt from the data: the +// highest committed copy of each key wins, an entry from the block +// that never synced is dropped, and a damaged slot is skipped. +func TestHeapRepairReadsTheKeysFromTheData(t *testing.T) { + dir := heapDir(t) + h, err := NewHeapStore(dir) + require.NoError(t, err) + sync := func() { + p, err := h.beginBlockSync() + require.NoError(t, err) + require.NoError(t, p.finish()) + } + h.AdvanceBlock(1) + for i := byte(1); i <= 10; i++ { + require.NoError(t, h.Put(key(i), []byte{1, i})) + } + sync() + h.AdvanceBlock(2) + require.NoError(t, h.Put(key(1), []byte{2, 1})) // Newer copy, higher offset + sync() + h.AdvanceBlock(3) + require.NoError(t, h.Put(key(2), []byte{3, 2})) // Never synced: not committed + h.file.Close() + h.log.Close() + require.NoError(t, os.Remove(filepath.Join(dir, "index.log"))) + + _, err = OpenHeapStore(dir) + require.ErrorIs(t, err, ErrHeapNeedsRepair) + r, err := RepairHeapStore(dir, 2) + require.NoError(t, err) + defer r.Close() + v, err := r.Get(key(1)) + require.NoError(t, err) + require.Equal(t, []byte{2, 1}, v, "the block-2 copy wins") + v, err = r.Get(key(2)) + require.NoError(t, err) + require.Equal(t, []byte{1, 2}, v, "block 3 never committed: its copy is dropped") + require.EqualValues(t, 10, r.LiveRecords()) + require.NoError(t, r.Close()) + re, err := OpenHeapStore(dir) + require.NoError(t, err, "the repair left a snapshot: an ordinary open") + defer re.Close() + v, err = re.Get(key(1)) + require.NoError(t, err) + require.Equal(t, []byte{2, 1}, v) +} From 6e28355241445cc4db454557dd58872a63732382 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 17:22:49 -0500 Subject: [PATCH 06/58] The mover syncs its own copies and holds the lock only to choose and name Alone on the disk with nine stores the heap's seal held at 55 ms p50 but the store grew 2.9 GB a minute: the cleaner's 2 MB copy budget per pass was a fraction of the append rate, and raising it would have made the copies bigger spikes in the next block's barrier, which is where they landed. The pass now writes and fsyncs its copies itself, outside the shard's lock, taking the lock only to choose the regions and reserve the copies' slots and again to name them -- a key rewritten meanwhile leaves its copy dead on arrival -- so its budget is about its own length (16 MB, 32 regions) and not the barrier's. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 151 +++++++++++++++++++++++++++++++---------------- 1 file changed, 99 insertions(+), 52 deletions(-) diff --git a/database/heap.go b/database/heap.go index ca46e73..ce49d96 100644 --- a/database/heap.go +++ b/database/heap.go @@ -118,16 +118,19 @@ const ( // released by. var HeapRegionBytes int64 = 4 << 20 -// HeapCleanBytes bounds one cleaning pass by the bytes it COPIES: the -// most a pass can add to the next block's sync. A pass takes region -// after region until the budget is spent, so mostly-dead regions, -// which cost little to copy, are released several to a pass. -var HeapCleanBytes int64 = 2 << 20 +// HeapCleanBytes bounds one cleaning pass by the bytes it COPIES. The +// pass syncs its own copies, so the bound is about the pass's own +// length, not a block's barrier; sized to keep up with the soak's +// append rate (~10 MB per shard per 20 blocks) at a live fraction of +// one half. A pass takes region after region until the budget is +// spent, so mostly-dead regions, which cost little, go several to a +// pass. +var HeapCleanBytes int64 = 16 << 20 // HeapCleanRegions bounds a pass by regions taken as well, so that a // heap of wholly dead regions is not scanned end to end under the // lock in one pass. -var HeapCleanRegions = 8 +var HeapCleanRegions = 32 // HeapCleanRatio is the dead fraction a region must reach before the // cleaner takes it: what bounds the copying a pass does per byte it @@ -528,48 +531,102 @@ func (h *HeapStore) compact() (bool, error) { return cleaned, err } -// clean takes the region with the most dead bytes, once at least -// HeapCleanRatio of it is dead, re-appends its live entries -- at most -// budget bytes of them, the rest next pass -- and, when none are left, -// marks it for release at the next sync. The cost of a pass is bounded -// by the copies it makes and by the ratio: a byte released never costs -// more than a byte copied (spec 1.2). Holds the lock for the pass: -// bounded, and the copies are ordinary appends. +// clean is the mover: it takes the regions with the most dead bytes, +// once at least HeapCleanRatio of each is dead, copies their live +// entries to the end -- at most budget bytes per pass -- and marks +// the regions for release once the delta naming the copies is +// durable. The pass syncs its own copies, so they never land in a +// block's barrier and the pass can be sized to the append rate; and +// it holds the shard's lock only to choose and to name, never across +// the copy or its fsync (spec 1.6). A byte released never costs more +// than a byte copied (spec 1.2). func (h *HeapStore) clean(budget int64) (bool, error) { + // 1. Under the lock: pick the regions, read them, decide what is + // live and where each copy will go h.mu.Lock() - defer h.mu.Unlock() if h.closed { + h.mu.Unlock() return false, errStoreClosed } if len(h.release) > 0 { + h.mu.Unlock() return false, nil // The last pass's release is still waiting on a sync } - var moved int64 - taken := 0 - for taken < HeapCleanRegions && moved < budget { + var moves []heapMove + var regions []int + var copied int64 + for len(regions) < HeapCleanRegions && copied < budget { pick := h.pickRegion() if pick < 0 { break } - m, err := h.cleanRegion(pick, budget-moved) + h.regions[pick].cleaning = true + regions = append(regions, pick) + m, n, err := h.planRegion(pick, budget-copied) if err != nil { - return taken > 0, err + h.mu.Unlock() + return false, err } - moved += m - taken++ + moves = append(moves, m...) + copied += n } - for i := range h.regions { - h.regions[i].cleaning = false + for _, r := range regions { + h.regions[r].cleaning = false + } + h.mu.Unlock() + if len(regions) == 0 { + return false, nil + } + // 2. Without the lock: write the copies and make them durable. + // Their slots were reserved at the end of the file under the lock, + // so puts landing meanwhile go past them. + for _, m := range moves { + if _, err := h.file.WriteAt(m.entry, m.to.off); err != nil { + return false, err + } } - return taken > 0, nil + if len(moves) > 0 { + if err := fsync(h.file); err != nil { + return false, err + } + } + // 3. Under the lock: name the copies, unless the key was rewritten + // meanwhile, in which case the copy is dead on arrival; and mark + // the regions the pass emptied + h.mu.Lock() + defer h.mu.Unlock() + for _, m := range moves { + s, live := h.index[m.key] + if live && s.off == m.from.off { + h.index[m.key] = m.to + h.touched[m.key] = struct{}{} + h.kill(m.from) + } else { + h.kill(m.to) // Reserved and written, but no longer wanted + } + } + for _, r := range regions { + if h.regions[r].live == 0 { + h.release = append(h.release, r) + } + } + h.movedBytes.Add(uint64(copied)) + return true, nil +} + +// heapMove is one live entry the mover copies: where it was, where +// it goes, and the bytes to write there. +type heapMove struct { + key [32]byte + from, to slot + entry []byte } // pickRegion is the region with the most dead bytes, once at least // HeapCleanRatio of it is dead; -1 when none qualifies. The region // the block in progress is appending to is never taken: a slot taken // this block may still be rewritten in place, and moving it would -// race that. A region a pass has already partly cleaned keeps its -// dead bytes and is picked again. The caller holds the lock. +// race that. The caller holds the lock. func (h *HeapStore) pickRegion() int { current := int(h.size / HeapRegionBytes) pick, best := -1, 0.0 @@ -584,21 +641,22 @@ func (h *HeapStore) pickRegion() int { return pick } -// cleanRegion re-appends the live entries of one region, at most -// budget bytes of them, and marks the region for release when none -// are left. Returns the bytes copied. The caller holds the lock. -func (h *HeapStore) cleanRegion(pick int, budget int64) (int64, error) { +// planRegion reads one region and reserves, at the end of the file, a +// slot for each live entry in it up to budget bytes; the rest wait for +// the next pass. The caller holds the lock. The reserved slots are +// counted live in their region at once, so the accounting is right +// whether or not the copy is wanted when it lands. +func (h *HeapStore) planRegion(pick int, budget int64) (moves []heapMove, copied int64, err error) { from := int64(pick) * HeapRegionBytes to := from + HeapRegionBytes if to > h.size { to = h.size } buf := make([]byte, to-from) - if _, err := h.file.ReadAt(buf, from); err != nil && !errors.Is(err, io.EOF) { - return 0, err + if _, err = h.file.ReadAt(buf, from); err != nil && !errors.Is(err, io.EOF) { + return nil, 0, err } - h.regions[pick].cleaning = true // Not picked again this pass - var moved, at int64 + var at int64 for at+heapHeader <= int64(len(buf)) { capacity := binary.LittleEndian.Uint32(buf[at:]) if capacity == 0 || at+int64(capacity) > int64(len(buf)) { @@ -607,33 +665,22 @@ func (h *HeapStore) cleanRegion(pick int, budget int64) (int64, error) { var key [32]byte copy(key[:], buf[at+16:]) if s, live := h.index[key]; live && s.off == from+at { - if moved >= budget { - h.cleanedBytes.Add(uint64(at)) - h.movedBytes.Add(uint64(moved)) - return moved, nil // The rest next pass + if copied >= budget { + break // The rest next pass } n := binary.LittleEndian.Uint32(buf[at+4:]) - // The copy is this block's write of the key: it carries - // this height, so a repair scan prefers it to the original value := buf[at+int64(heapHeader) : at+int64(heapHeader)+int64(n)] ns := slot{off: h.size, cap: capacity, n: n, block: h.height} - if _, err := h.file.WriteAt(encodeEntry(capacity, h.height, key, value), ns.off); err != nil { - return moved, err - } h.append(ns) - h.kill(s) - h.index[key] = ns - h.touched[key] = struct{}{} - moved += int64(capacity) + // The copy is this block's write of the key: it carries this + // height, so a repair scan prefers it to the original + moves = append(moves, heapMove{key: key, from: s, to: ns, entry: encodeEntry(capacity, h.height, key, value)}) + copied += int64(capacity) } at += int64(capacity) } h.cleanedBytes.Add(uint64(at)) - h.movedBytes.Add(uint64(moved)) - if h.regions[pick].live == 0 { - h.release = append(h.release, pick) - } - return moved, nil + return moves, copied, nil } // Snapshot writes the whole key map and drops the deltas it covers From 967743f92b69eb70428d3a628044b7e50598ad36 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 17:42:53 -0500 Subject: [PATCH 07/58] The heap is files of entries: the block's, the mover's, and deleted whole The dynamic layer is now fixed-size data files (HeapFileBytes) and a key is (file, offset, length). A block appends to the current file and the mover to a file of its own, so their barriers never share an inode -- the mover's fsync was flushing the block's pages and the block's the mover's. A file left with nothing live is deleted whole once the delta naming the mover's copies out of it is durable, which replaces hole punching and the region table, and the Linux-only build. Entries are laid out at their exact aligned length: the size classes wasted a third of the store and bought nothing without free lists; an in-place rewrite now requires the same aligned size, since a shorter one broke the scan a file's later entries depend on. The index is generations, index-G.log: a snapshot record starts a generation in a file of its own, written aside, fsynced, renamed into place and the directory fsynced, then the deltas append to it and the previous generation is removed. The old snapshot truncated the log and rewrote the deltas after it, and a crash in between lost committed blocks' names. Snapshots serialize with block syncs (syncMu) so no delta is in flight into a generation being retired. The mover's gate is the deadest file once half dead, or whatever its ratio while dead bytes exceed live, which bounds the heap at twice its live set. Get copies the slot under the lock and reads outside it, retrying once if the file was deleted meanwhile. Repair scans the files in order; later in the scan wins a tie in height. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 1190 +++++++++++++++++++++------------------- database/heap_linux.go | 15 - database/heap_other.go | 9 - database/heap_test.go | 285 +++++----- 4 files changed, 794 insertions(+), 705 deletions(-) delete mode 100644 database/heap_linux.go delete mode 100644 database/heap_other.go diff --git a/database/heap.go b/database/heap.go index ce49d96..741426c 100644 --- a/database/heap.go +++ b/database/heap.go @@ -5,152 +5,154 @@ import ( "errors" "fmt" "hash/crc32" - "io" "os" "path/filepath" + "sort" + "strconv" + "strings" "sync" "sync/atomic" "time" ) -// HeapStore is the dynamic layer as an append-and-clean heap (proposal -// docs/proposals/2026-09-16-entries-written-once.md): entries are -// managed by appending, keys are managed separately, and space is -// reclaimed by a bounded cleaner that moves the live entries out of -// the oldest region and releases it, never by rewriting an index or a -// filter. +// HeapStore is the dynamic layer as files of entries and a map of keys +// (proposal docs/proposals/2026-09-16-entries-written-once.md): an +// entry is written once at the end of a file, a key is (file, offset, +// length), and space comes back by deleting a file once nothing in it +// is live -- never by rewriting an index or a filter. // // Files, in one directory: // -// heap.dat entries: [cap u32][len u32][height u64][key 32][value] -// [crc32 of height+key+value], each in a slot of cap bytes -// (a size class), self-describing so that Repair can -// rebuild the key map from the data alone, -// appended in the order written; the file is a sequence of -// regions of HeapRegionBytes, and a region whose entries -// are all dead is released (a punched hole, size kept) -// index.log one delta per block sync: the (key, off, cap, len) of -// every key the block touched, checksummed -// index.snap the whole key map, rewritten on the maintenance cadence +// heap-N.dat entries, [len u32][height u64][key 32][value][crc32 +// of height+key+value], 8-byte aligned, appended in the +// order written. A block appends to the current data +// file; the mover appends to a file of its own, so the +// two never share a barrier. A file is rolled at +// HeapFileBytes and deleted once nothing in it is live. +// index-G.log generation G of the key map: a snapshot record of the +// whole map, then one delta per block sync naming the +// keys the block touched. A snapshot starts a new +// generation in a file of its own, switched to by +// rename, so no delta is ever truncated away. // -// A block's writes are contiguous, so the block sync is one sequential -// fsync of the heap: filling holes wherever they lie was measured at -// 4x the ingest in page writes at every barrier, and is not done. A -// slot a key stops naming is dead where it lies until the cleaner -// takes its region: the region with the most dead bytes, once at -// least HeapCleanRatio of it is dead, so a pass copies little. -// Cleaning the oldest region regardless was measured too: with a -// skewed key set the oldest region is mostly live cold keys, and every -// pass copied most of it into one block's sync. +// The key map is in memory for the live key set (spec 1.2: memory that +// scales with the working set). Open loads the newest whole +// generation, replays its deltas and derives every file's live and +// dead bytes; a data file nothing names is deleted then, and one that +// is missing while named is an error. // -// Durability (spec 1.8). The block sync fsyncs heap.dat and then -// appends and fsyncs the block's delta, so an index entry is durable -// only after the slot it names is. A slot the last durable index -// names is never overwritten: a key rewritten in a later block takes -// a new slot at the end, and its old slot is dead but intact until its -// region is released -- and a region is released only after the delta -// naming the cleaner's copies is durable, one sync late, the way spec -// 2.6 defers deletion. A key rewritten again within the same block -// reuses the slot it took this block, since nothing durable names it -// yet. A crash therefore leaves every durable entry intact and every -// torn slot unnamed, and the checksum catches a torn slot that a -// stale index could name. +// Durability (spec 1.8). The block sync fsyncs the data files the +// block wrote and then appends and fsyncs the block's delta, so an +// index entry is durable only after the entry it names is. A slot the +// last durable index names is never overwritten: a key rewritten in a +// later block takes a new slot, and its old slot is dead where it lies +// until its file is deleted -- and a file is deleted only after the +// delta naming the mover's copies out of it is durable, one sync late, +// the way spec 2.6 defers deletion (never unlink what a durable index +// names). A key rewritten again within the same block reuses the +// slot it took this block, since nothing durable names it yet. A +// crash therefore leaves every durable entry intact and every torn +// slot unnamed; the checksum catches a torn slot a stale index could +// name; and RepairHeapStore rebuilds the map from the data alone, +// since every entry carries its key and the height that wrote it. type HeapStore struct { Directory string mu sync.RWMutex - file *os.File // heap.dat - log *os.File // index.log - size int64 // Append point: the end of heap.dat + files map[uint32]*heapFile + cur *heapFile // The file the block appends to + mov *heapFile // The file the mover appends to + nextID uint32 index map[[32]byte]slot height uint64 // The block being written; slots taken in it may be rewritten in place - // regions is the live and dead capacity in each HeapRegionBytes of - // the file, what the cleaner chooses by; released marks the - // regions punched. touched is the block's delta in the making; - // release holds the regions emptied by the last pass, punched once - // the delta naming their copies is durable; snapshots counts - // compact calls between snapshots. - regions []region - touched map[[32]byte]struct{} - release []int + // touched is the block's delta in the making; dirty the files + // written since the last sync; release the files the mover + // emptied, deleted once the delta naming their copies is durable. + touched map[[32]byte]struct{} + dirty map[uint32]*heapFile + release []uint32 + + // syncMu serializes block syncs with each other and with a + // snapshot, so the map a snapshot writes is exactly the state of + // the last finished delta and no delta is in flight into a log + // about to be retired. + syncMu sync.Mutex + log *os.File + gen uint64 snapshots int - syncedTo int64 // The append point at the last sync closed bool - liveBytes int64 // Capacity of every named slot + liveBytes int64 + deadBytes int64 putTotal, putInPlace, putAppend atomic.Uint64 lookups, hits atomic.Uint64 cleanedBytes, movedBytes atomic.Uint64 - // The sync's cost, split: nanoseconds in the heap's fsync and in - // the delta's write and fsync, and the syncs and bytes they covered + // The sync's cost, split: nanoseconds in the data files' fsyncs + // and in the delta's write and fsync, and the syncs and bytes syncs, syncHeapNs, syncLogNs, syncBytes atomic.Uint64 } -// region is the accounting for one HeapRegionBytes of the heap. -type region struct { +// heapFile is one data file and its accounting. +type heapFile struct { + id uint32 + f *os.File + size int64 live, dead int64 - released bool cleaning bool // Taken by the pass in progress } -// slot is where an entry lives: its offset, the capacity of the slot -// (a size class) and the entry's value length. block is the height -// that took the slot, which decides whether a rewrite may reuse it. +// slot is where an entry lives: its file, its offset there, the value +// length, and the block that took it, which decides whether a rewrite +// may reuse it in place. type slot struct { - off int64 - cap uint32 + file uint32 + off uint32 n uint32 block uint64 } const ( - heapHeader = 4 + 4 + 8 + 32 // cap, len, height, key - heapTrailer = 4 // crc32 of key+value - heapMinCap = 64 // Smallest slot - heapMagic = 0x48454150 // "HEAP", the delta record's marker - heapDeltaHdr = 4 + 8 + 8 + 4 // magic, height, reserved, count - heapDeltaRec = 32 + 8 + 4 + 4 + heapHeader = 4 + 8 + 32 // len, height, key + heapTrailer = 4 // crc32 of height+key+value + heapAlign = 8 // Entries start on an 8-byte boundary + heapMagic = 0x48454150 // "HEAP", an index record's marker + heapSnapshot = 0x50414E53 // "SNAP", the record that starts a generation + heapIndexHdr = 4 + 8 + 4 // magic, height, count + heapIndexRec = 32 + 4 + 4 + 4 ) -// HeapRegionBytes is the size of the regions the heap is cleaned and -// released by. -var HeapRegionBytes int64 = 4 << 20 - -// HeapCleanBytes bounds one cleaning pass by the bytes it COPIES. The -// pass syncs its own copies, so the bound is about the pass's own -// length, not a block's barrier; sized to keep up with the soak's -// append rate (~10 MB per shard per 20 blocks) at a live fraction of -// one half. A pass takes region after region until the budget is -// spent, so mostly-dead regions, which cost little, go several to a -// pass. +// HeapFileBytes is the size a data file is rolled at. +var HeapFileBytes int64 = 16 << 20 + +// HeapCleanBytes bounds one mover pass by the bytes it copies; the +// pass syncs its own copies, so the bound is about the pass's length, +// not a block's barrier. var HeapCleanBytes int64 = 16 << 20 -// HeapCleanRegions bounds a pass by regions taken as well, so that a -// heap of wholly dead regions is not scanned end to end under the -// lock in one pass. -var HeapCleanRegions = 32 +// HeapCleanFiles bounds a pass by files taken as well. +var HeapCleanFiles = 8 -// HeapCleanRatio is the dead fraction a region must reach before the -// cleaner takes it: what bounds the copying a pass does per byte it -// releases. +// HeapCleanRatio is the dead fraction a file must reach before the +// mover takes it -- unless dead bytes exceed live bytes overall, when +// the deadest file is taken regardless, which bounds the heap at +// twice its live set. var HeapCleanRatio = 0.5 // HeapSnapshotEvery is how many compact calls pass between key-map -// snapshots; between them the log is what open replays. +// snapshots; between them the generation's deltas are what open +// replays. var HeapSnapshotEvery = 5 -// heapCap is the size class that holds need bytes: a power of two no -// smaller than heapMinCap. -func heapCap(need int) uint32 { - capacity := uint32(heapMinCap) - for int(capacity) < need { - capacity <<= 1 - } - return capacity +// entrySize is the bytes an entry of n value bytes takes, aligned. +func entrySize(n int) int64 { + return (int64(heapHeader+n+heapTrailer) + heapAlign - 1) &^ (heapAlign - 1) } +func dataName(id uint32) string { return fmt.Sprintf("heap-%06d.dat", id) } +func indexName(gen uint64) string { return fmt.Sprintf("index-%06d.log", gen) } + // NewHeapStore creates an empty heap in directory, replacing anything // there. func NewHeapStore(directory string) (*HeapStore, error) { @@ -163,17 +165,15 @@ func NewHeapStore(directory string) (*HeapStore, error) { } // OpenHeapStore opens the heap in directory as it was left. A heap -// whose data is there but whose index files are not is refused: that -// is what RepairHeapStore is for, and it needs the committed height, -// which only the store above knows. +// with data files but no index generation is refused: that is what +// RepairHeapStore is for, and it needs the committed height, which +// only the store above knows. func OpenHeapStore(directory string) (*HeapStore, error) { - st, err := os.Stat(filepath.Join(directory, "heap.dat")) + dataIDs, gens, err := listHeap(directory) if err != nil { return nil, fmt.Errorf("open heap at %s: %w", directory, err) } - _, snapErr := os.Stat(filepath.Join(directory, "index.snap")) - _, logErr := os.Stat(filepath.Join(directory, "index.log")) - if st.Size() > 0 && snapErr != nil && logErr != nil { + if len(dataIDs) > 0 && len(gens) == 0 { return nil, fmt.Errorf("open heap at %s: %w", directory, ErrHeapNeedsRepair) } h := &HeapStore{Directory: directory} @@ -184,87 +184,259 @@ func OpenHeapStore(directory string) (*HeapStore, error) { // not: RepairHeapStore rebuilds the index from the data. var ErrHeapNeedsRepair = errors.New("heap has data but no index; repair it") -// RepairHeapStore rebuilds the key map by reading the keys from the -// data: every entry carries its key, its height and a checksum, so a -// sequential scan recovers the map without any index. For each key -// the copy with the highest height wins, and an entry above the -// committed height -- a block whose sync never finished -- is -// dropped, as is any torn or damaged slot. The rebuilt map is -// snapshotted, so the next open is an ordinary one. -func RepairHeapStore(directory string, committed uint64) (*HeapStore, error) { - os.Remove(filepath.Join(directory, "index.snap")) - os.Remove(filepath.Join(directory, "index.log")) - h := &HeapStore{Directory: directory} - var err error - if h.file, err = os.OpenFile(filepath.Join(directory, "heap.dat"), os.O_RDWR, 0o644); err != nil { - return nil, err +// listHeap names the data files and the whole index generations in a +// directory, in order. A generation still being written (.tmp) is +// not whole and is removed. +func listHeap(directory string) (dataIDs []uint32, gens []uint64, err error) { + entries, err := os.ReadDir(directory) + if err != nil { + return nil, nil, err + } + for _, e := range entries { + name := e.Name() + switch { + case strings.HasPrefix(name, "heap-") && strings.HasSuffix(name, ".dat"): + id, err := strconv.ParseUint(strings.TrimSuffix(strings.TrimPrefix(name, "heap-"), ".dat"), 10, 32) + if err == nil { + dataIDs = append(dataIDs, uint32(id)) + } + case strings.HasPrefix(name, "index-") && strings.HasSuffix(name, ".log"): + g, err := strconv.ParseUint(strings.TrimSuffix(strings.TrimPrefix(name, "index-"), ".log"), 10, 64) + if err == nil { + gens = append(gens, g) + } + case strings.HasSuffix(name, segTmpSuffix): + os.Remove(filepath.Join(directory, name)) + } } - if h.log, err = os.OpenFile(filepath.Join(directory, "index.log"), os.O_RDWR|os.O_CREATE|os.O_APPEND, 0o644); err != nil { - return nil, err + sort.Slice(dataIDs, func(i, j int) bool { return dataIDs[i] < dataIDs[j] }) + sort.Slice(gens, func(i, j int) bool { return gens[i] < gens[j] }) + return dataIDs, gens, nil +} + +// Open loads the key map from the newest generation, opens the data +// files and derives their accounting. Idempotent. +func (h *HeapStore) Open() (err error) { + h.mu.Lock() + defer h.mu.Unlock() + if h.files != nil { + return nil + } + dataIDs, gens, err := listHeap(h.Directory) + if err != nil { + return err } h.index = map[[32]byte]slot{} h.touched = map[[32]byte]struct{}{} - data, err := readFrom(h.file, 0) + h.dirty = map[uint32]*heapFile{} + h.files = map[uint32]*heapFile{} + h.release = nil + h.closed = false + h.liveBytes, h.deadBytes = 0, 0 + // The newest whole generation is the index; older ones are what a + // snapshot left behind when it could not delete them + if len(gens) > 0 { + h.gen = gens[len(gens)-1] + for _, g := range gens[:len(gens)-1] { + os.Remove(filepath.Join(h.Directory, indexName(g))) + } + if err = h.replayGeneration(); err != nil { + return err + } + } else { + h.gen = 1 + if err = h.startGeneration(); err != nil { + return err + } + } + for _, id := range dataIDs { + if id >= h.nextID { + h.nextID = id + 1 + } + } + return h.deriveFiles(dataIDs) +} + +// replayGeneration applies the current generation: its snapshot, then +// every whole delta; a torn tail is what a crash leaves and is +// dropped, its slots unnamed. The caller holds the lock. +func (h *HeapStore) replayGeneration() (err error) { + path := filepath.Join(h.Directory, indexName(h.gen)) + if h.log, err = os.OpenFile(path, os.O_RDWR|os.O_APPEND, 0o644); err != nil { + return err + } + buf, err := os.ReadFile(path) if err != nil { - return nil, err + return err } - heights := map[[32]byte]uint64{} - var off int64 - for off < int64(len(data)) { - capacity, height, key, value, ok := decodeEntry(data[off:]) - if !ok { - // A released region reads as zeros; skip to the next region - // boundary. Anything else torn ends the scan: nothing past - // a torn slot was named before it. - if binary.LittleEndian.Uint32(data[off:]) == 0 { - off = (off/HeapRegionBytes + 1) * HeapRegionBytes - continue + at := 0 + for at < len(buf) { + n, err := h.applyRecord(buf[at:]) + if err != nil { + if err = h.log.Truncate(int64(at)); err != nil { + return err } break } - if height <= committed { - if prev, seen := heights[key]; !seen || height >= prev { - heights[key] = height - h.index[key] = slot{off: off, cap: capacity, n: uint32(len(value))} - } + at += n + } + return nil +} + +// startGeneration begins an index generation with a snapshot of the +// map, written aside and renamed into place, then fsynced; the +// previous generation's file is removed once the new one is durable. +// The caller holds syncMu and the lock (Open holds the lock alone, +// with nothing else running). +func (h *HeapStore) startGeneration() error { + path := filepath.Join(h.Directory, indexName(h.gen)) + tmp := path + segTmpSuffix + f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) + if err != nil { + return err + } + all := func(emit func(key [32]byte)) { + for key := range h.index { + emit(key) } - off += int64(capacity) } - h.height = committed - if err = h.deriveExtent(); err != nil { - return nil, err + if _, err = f.Write(h.encodeIndexOf(heapSnapshot, all, len(h.index))); err != nil { + f.Close() + return err } - if err = h.Snapshot(); err != nil { - return nil, err + if err = fsync(f); err != nil { + f.Close() + return err } - return h, nil -} - -// Open loads the key map from the snapshot and the log and derives -// the append point. Idempotent. -func (h *HeapStore) Open() (err error) { - h.mu.Lock() - defer h.mu.Unlock() - if h.file != nil { - return nil + if err = f.Close(); err != nil { + return err } - if h.file, err = os.OpenFile(filepath.Join(h.Directory, "heap.dat"), os.O_RDWR|os.O_CREATE, 0o644); err != nil { + if err = os.Rename(tmp, path); err != nil { return err } - if h.log, err = os.OpenFile(filepath.Join(h.Directory, "index.log"), os.O_RDWR|os.O_CREATE|os.O_APPEND, 0o644); err != nil { + if err = fsyncDir(h.Directory); err != nil { return err } - h.index = map[[32]byte]slot{} - h.touched = map[[32]byte]struct{}{} - h.closed = false - h.size, h.liveBytes, h.regions, h.release, h.syncedTo = 0, 0, nil, nil, 0 - if err = h.loadSnapshot(); err != nil { + old := h.log + if h.log, err = os.OpenFile(path, os.O_RDWR|os.O_APPEND, 0o644); err != nil { return err } - if err = h.replayLog(); err != nil { + if old != nil { + old.Close() + os.Remove(filepath.Join(h.Directory, indexName(h.gen-1))) + } + return nil +} + +// fsyncDir makes a directory's entries durable: a rename or an unlink +// is not on disk until the directory is. +func fsyncDir(directory string) error { + d, err := os.Open(directory) + if err != nil { return err } - return h.deriveExtent() + defer d.Close() + return fsync(d) +} + +// encodeIndex is one index record: a snapshot of the whole map or the +// block's delta of the keys it touched, checksummed. The caller holds +// the lock. +func (h *HeapStore) encodeIndex(magic uint32, keys map[[32]byte]struct{}) []byte { + return h.encodeIndexOf(magic, func(emit func(key [32]byte)) { + for key := range keys { + emit(key) + } + }, len(keys)) +} + +func (h *HeapStore) encodeIndexOf(magic uint32, each func(emit func(key [32]byte)), count int) []byte { + buf := make([]byte, heapIndexHdr+count*heapIndexRec+4) + binary.LittleEndian.PutUint32(buf, magic) + binary.LittleEndian.PutUint64(buf[4:], h.height) + binary.LittleEndian.PutUint32(buf[12:], uint32(count)) + at := heapIndexHdr + each(func(key [32]byte) { + s := h.index[key] + copy(buf[at:], key[:]) + binary.LittleEndian.PutUint32(buf[at+32:], s.file) + binary.LittleEndian.PutUint32(buf[at+36:], s.off) + binary.LittleEndian.PutUint32(buf[at+40:], s.n) + at += heapIndexRec + }) + binary.LittleEndian.PutUint32(buf[at:], crc32.ChecksumIEEE(buf[:at])) + return buf +} + +var errHeapTorn = errors.New("heap: torn index record") + +// applyRecord applies one index record and returns its length. +func (h *HeapStore) applyRecord(buf []byte) (int, error) { + if len(buf) < heapIndexHdr { + return 0, errHeapTorn + } + magic := binary.LittleEndian.Uint32(buf) + if magic != heapMagic && magic != heapSnapshot { + return 0, errHeapTorn + } + count := int(binary.LittleEndian.Uint32(buf[12:])) + end := heapIndexHdr + count*heapIndexRec + if len(buf) < end+4 || crc32.ChecksumIEEE(buf[:end]) != binary.LittleEndian.Uint32(buf[end:]) { + return 0, errHeapTorn + } + if height := binary.LittleEndian.Uint64(buf[4:]); height > h.height { + h.height = height + } + for at := heapIndexHdr; at < end; at += heapIndexRec { + var key [32]byte + copy(key[:], buf[at:]) + h.index[key] = slot{file: binary.LittleEndian.Uint32(buf[at+32:]), off: binary.LittleEndian.Uint32(buf[at+36:]), n: binary.LittleEndian.Uint32(buf[at+40:])} + } + return end + 4, nil +} + +// deriveFiles opens every data file the map names, derives its live +// and dead bytes, cuts the newest back to its last named entry, and +// deletes any file the map does not name at all: nothing durable +// names it, and its bytes are a crash's or the mover's leftovers. The +// caller holds the lock. +func (h *HeapStore) deriveFiles(dataIDs []uint32) error { + named := map[uint32]int64{} // file -> end of its last named slot + for _, s := range h.index { + if e := int64(s.off) + entrySize(int(s.n)); e > named[s.file] { + named[s.file] = e + } + h.liveBytes += entrySize(int(s.n)) + } + for id := range named { + if _, err := os.Stat(filepath.Join(h.Directory, dataName(id))); err != nil { + return fmt.Errorf("heap: the index names %s: %w", dataName(id), err) + } + } + for _, id := range dataIDs { + end, live := named[id] + if !live { + os.Remove(filepath.Join(h.Directory, dataName(id))) + continue + } + f, err := os.OpenFile(filepath.Join(h.Directory, dataName(id)), os.O_RDWR, 0o644) + if err != nil { + return err + } + if err = f.Truncate(end); err != nil { + f.Close() + return err + } + h.files[id] = &heapFile{id: id, f: f, size: end} + } + for _, s := range h.index { + h.files[s.file].live += entrySize(int(s.n)) + } + for _, hf := range h.files { + hf.dead = hf.size - hf.live + h.deadBytes += hf.dead + } + return nil } // Close syncs what is pending and closes the files. Reopen with Open. @@ -282,53 +454,102 @@ func (h *HeapStore) Close() error { h.mu.Lock() defer h.mu.Unlock() h.closed = true - err = h.file.Close() - if lerr := h.log.Close(); err == nil { - err = lerr + for _, hf := range h.files { + if cerr := hf.f.Close(); err == nil { + err = cerr + } + } + if cerr := h.log.Close(); err == nil { + err = cerr } - h.file, h.log = nil, nil + h.files, h.cur, h.mov, h.log = nil, nil, nil, nil return err } -// encodeEntry lays out one entry for its slot: the block that wrote -// it is in the header so that, with the index gone, a scan can tell -// the current copy of a key (the highest height) from stale ones and -// a committed entry from one whose block never synced. -func encodeEntry(capacity uint32, height uint64, key [32]byte, value []byte) []byte { - buf := make([]byte, heapHeader+len(value)+heapTrailer) - binary.LittleEndian.PutUint32(buf, capacity) - binary.LittleEndian.PutUint32(buf[4:], uint32(len(value))) - binary.LittleEndian.PutUint64(buf[8:], height) - copy(buf[16:], key[:]) +// encodeEntry lays out one entry: the block that wrote it is in the +// header so that, with the index gone, a scan can tell the current +// copy of a key (the highest height) from stale ones and a committed +// entry from one whose block never synced. +func encodeEntry(height uint64, key [32]byte, value []byte) []byte { + buf := make([]byte, entrySize(len(value))) + binary.LittleEndian.PutUint32(buf, uint32(len(value))) + binary.LittleEndian.PutUint64(buf[4:], height) + copy(buf[12:], key[:]) copy(buf[heapHeader:], value) - binary.LittleEndian.PutUint32(buf[heapHeader+len(value):], crc32.ChecksumIEEE(buf[8:heapHeader+len(value)])) + binary.LittleEndian.PutUint32(buf[heapHeader+len(value):], crc32.ChecksumIEEE(buf[4:heapHeader+len(value)])) return buf } // decodeEntry checks the entry at the start of buf and returns its -// fields; ok is false for an unwritten, torn or damaged slot. -func decodeEntry(buf []byte) (capacity uint32, height uint64, key [32]byte, value []byte, ok bool) { +// fields and its aligned size; ok is false for an unwritten, torn or +// damaged slot. +func decodeEntry(buf []byte) (size int64, height uint64, key [32]byte, value []byte, ok bool) { if len(buf) < heapHeader { return } - capacity = binary.LittleEndian.Uint32(buf) - n := int(binary.LittleEndian.Uint32(buf[4:])) - if capacity == 0 || heapHeader+n+heapTrailer > int(capacity) || int(capacity) > len(buf) { - return + n := int(binary.LittleEndian.Uint32(buf)) + size = entrySize(n) + if n == 0 && binary.LittleEndian.Uint64(buf[4:]) == 0 || size > int64(len(buf)) { + return 0, 0, key, nil, false } - if crc32.ChecksumIEEE(buf[8:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { - return + if crc32.ChecksumIEEE(buf[4:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { + return 0, 0, key, nil, false + } + height = binary.LittleEndian.Uint64(buf[4:]) + copy(key[:], buf[12:]) + return size, height, key, buf[heapHeader : heapHeader+n], true +} + +// newFile opens the next data file. The caller holds the lock. +func (h *HeapStore) newFile() (*heapFile, error) { + id := h.nextID + h.nextID++ + f, err := os.OpenFile(filepath.Join(h.Directory, dataName(id)), os.O_RDWR|os.O_CREATE|os.O_EXCL, 0o644) + if err != nil { + return nil, err + } + hf := &heapFile{id: id, f: f} + h.files[id] = hf + return hf, nil +} + +// reserve takes size bytes at the end of the block's or the mover's +// file, rolling to a new file at HeapFileBytes. The caller holds the +// lock. +func (h *HeapStore) reserve(mover bool, size int64) (hf *heapFile, off int64, err error) { + at := &h.cur + if mover { + at = &h.mov + } + if *at == nil || (*at).size+size > HeapFileBytes { + if *at, err = h.newFile(); err != nil { + return nil, 0, err + } } - height = binary.LittleEndian.Uint64(buf[8:]) - copy(key[:], buf[16:]) - return capacity, height, key, buf[heapHeader : heapHeader+n], true + hf = *at + off = hf.size + hf.size += size + hf.live += size + h.liveBytes += size + h.dirty[hf.id] = hf + return hf, off, nil +} + +// kill accounts a slot its key stopped naming. The caller holds the +// lock. +func (h *HeapStore) kill(s slot) { + size := entrySize(int(s.n)) + h.liveBytes -= size + h.deadBytes += size + hf := h.files[s.file] + hf.live -= size + hf.dead += size } // Put writes value under key: in place if the key took its slot this -// block and the value fits, else appended. The slot a durable index -// names is never rewritten. +// block and the entry keeps its size, else appended to the block's +// file. The slot a durable index names is never rewritten. func (h *HeapStore) Put(key [32]byte, value []byte) error { - need := heapHeader + len(value) + heapTrailer h.mu.Lock() defer h.mu.Unlock() if h.closed { @@ -337,20 +558,29 @@ func (h *HeapStore) Put(key [32]byte, value []byte) error { h.putTotal.Add(1) old, had := h.index[key] var s slot - if had && old.block == h.height && int(old.cap) >= need { - // Taken this block, nothing durable names it: rewrite in place + var hf *heapFile + if had && old.block == h.height && entrySize(len(value)) == entrySize(int(old.n)) { + // Taken this block, nothing durable names it, and the same + // aligned size, so the file stays a contiguous sequence of + // entries a scan can walk: rewrite in place s = old s.n = uint32(len(value)) + hf = h.files[s.file] + h.dirty[hf.id] = hf h.putInPlace.Add(1) } else { - s = slot{off: h.size, cap: heapCap(need), n: uint32(len(value)), block: h.height} - h.append(s) + var off int64 + var err error + if hf, off, err = h.reserve(false, entrySize(len(value))); err != nil { + return err + } + s = slot{file: hf.id, off: uint32(off), n: uint32(len(value)), block: h.height} if had { h.kill(old) // Dead where it lies } h.putAppend.Add(1) } - if _, err := h.file.WriteAt(encodeEntry(s.cap, h.height, key, value), s.off); err != nil { + if _, err := hf.f.WriteAt(encodeEntry(h.height, key, value), int64(s.off)); err != nil { return err } h.index[key] = s @@ -358,59 +588,48 @@ func (h *HeapStore) Put(key [32]byte, value []byte) error { return nil } -// append accounts a slot taken at the end of the file. The caller -// holds the lock and has set s.off to the append point. -func (h *HeapStore) append(s slot) { - h.size += int64(s.cap) - h.liveBytes += int64(s.cap) - r := int(s.off / HeapRegionBytes) - for len(h.regions) <= r { - h.regions = append(h.regions, region{}) - } - h.regions[r].live += int64(s.cap) -} - -// kill accounts a slot its key stopped naming. The caller holds the -// lock. -func (h *HeapStore) kill(s slot) { - h.liveBytes -= int64(s.cap) - r := &h.regions[int(s.off/HeapRegionBytes)] - r.live -= int64(s.cap) - r.dead += int64(s.cap) -} - -// Get answers from the key map and one read of the slot. A slot -// whose checksum fails is reported as corrupt, never as a value. +// Get answers from the key map and one read of the slot, taken +// outside the lock. A slot whose checksum fails is reported as +// corrupt, never as a value. A file deleted between the lookup and +// the read -- the mover moved the entry and a sync released the file +// meanwhile -- is looked up again. func (h *HeapStore) Get(key [32]byte) ([]byte, error) { - h.mu.RLock() - if h.closed { - h.mu.RUnlock() - return nil, errStoreClosed - } - h.lookups.Add(1) - s, ok := h.index[key] - if !ok { + for attempt := 0; ; attempt++ { + h.mu.RLock() + if h.closed { + h.mu.RUnlock() + return nil, errStoreClosed + } + h.lookups.Add(1) + s, ok := h.index[key] + var f *os.File + if ok { + f = h.files[s.file].f + } h.mu.RUnlock() - return nil, errNotFound - } - h.hits.Add(1) - buf := make([]byte, heapHeader+int(s.n)+heapTrailer) - _, err := h.file.ReadAt(buf, s.off) - h.mu.RUnlock() - if err != nil { - return nil, err + if !ok { + return nil, errNotFound + } + h.hits.Add(1) + buf := make([]byte, heapHeader+int(s.n)+heapTrailer) + if _, err := f.ReadAt(buf, int64(s.off)); err != nil { + if attempt == 0 && errors.Is(err, os.ErrClosed) { + continue + } + return nil, err + } + return heapEntryValue(buf, key) } - return heapEntryValue(buf, key) } // heapEntryValue checks an entry read from a slot and returns its // value. func heapEntryValue(buf []byte, key [32]byte) ([]byte, error) { - n := int(binary.LittleEndian.Uint32(buf[4:])) - if len(buf) != heapHeader+n+heapTrailer || [32]byte(buf[16:heapHeader]) != key { + n := int(binary.LittleEndian.Uint32(buf)) + if len(buf) != heapHeader+n+heapTrailer || [32]byte(buf[12:heapHeader]) != key { return nil, fmt.Errorf("heap: slot does not hold the key it is named for") } - if crc32.ChecksumIEEE(buf[8:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { + if crc32.ChecksumIEEE(buf[4:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { return nil, fmt.Errorf("heap: entry checksum failed") } return append([]byte(nil), buf[heapHeader:heapHeader+n]...), nil @@ -426,74 +645,61 @@ func (h *HeapStore) AdvanceBlock(height uint64) { h.mu.Unlock() } -// heapSync is a block sync in flight: the delta to make durable and -// the regions to release once it is. +// heapSync is a block sync in flight: the files to make durable, the +// delta, and the files to delete once it is. It holds syncMu until +// finished. type heapSync struct { h *HeapStore + dirty []*heapFile delta []byte - release []int - bytes int64 // Appended since the last sync: what the heap's fsync covers + release []uint32 + bytes int64 } // beginBlockSync takes the block's delta under the lock; finish makes // it durable outside it. func (h *HeapStore) beginBlockSync() (blockSync, error) { + h.syncMu.Lock() h.mu.Lock() defer h.mu.Unlock() - if h.closed || h.file == nil { + if h.closed || h.files == nil { + h.syncMu.Unlock() return nil, errStoreClosed } - p := &heapSync{h: h, release: h.release, bytes: h.size - h.syncedTo} - h.syncedTo = h.size + p := &heapSync{h: h, release: h.release} h.release = nil + for _, hf := range h.dirty { + p.dirty = append(p.dirty, hf) + p.bytes += hf.size + } + h.dirty = map[uint32]*heapFile{} if len(h.touched) > 0 || len(p.release) > 0 { - p.delta = h.encodeDelta() + p.delta = h.encodeIndex(heapMagic, h.touched) h.touched = map[[32]byte]struct{}{} } return p, nil } -// encodeDelta is the block's index delta: marker, height, a reserved -// word, the touched keys' slots, and a checksum. The caller holds -// the lock. -func (h *HeapStore) encodeDelta() []byte { - buf := make([]byte, heapDeltaHdr+len(h.touched)*heapDeltaRec+4) - binary.LittleEndian.PutUint32(buf, heapMagic) - binary.LittleEndian.PutUint64(buf[4:], h.height) - binary.LittleEndian.PutUint32(buf[20:], uint32(len(h.touched))) - at := heapDeltaHdr - for key := range h.touched { - at += putDeltaRec(buf[at:], key, h.index[key]) - } - binary.LittleEndian.PutUint32(buf[at:], crc32.ChecksumIEEE(buf[:at])) - return buf -} - -func putDeltaRec(buf []byte, key [32]byte, s slot) int { - copy(buf, key[:]) - binary.LittleEndian.PutUint64(buf[32:], uint64(s.off)) - binary.LittleEndian.PutUint32(buf[40:], s.cap) - binary.LittleEndian.PutUint32(buf[44:], s.n) - return heapDeltaRec -} - -// finish: entries durable, then the delta durable, then the regions -// the delta's copies emptied are released. -func (p *heapSync) finish() error { +// finish: entries durable, then the delta durable, then the files the +// delta's copies emptied are deleted. Releases syncMu. +func (p *heapSync) finish() (err error) { h := p.h + defer h.syncMu.Unlock() if p.delta == nil { return nil } t := time.Now() - if err := fsync(h.file); err != nil { - return err + for _, hf := range p.dirty { + if err = fsync(hf.f); err != nil { + return err + } } h.syncHeapNs.Add(uint64(time.Since(t))) t = time.Now() - if _, err := h.log.Write(p.delta); err != nil { + if _, err = h.log.Write(p.delta); err != nil { return err } - if err := fsync(h.log); err != nil { + if err = fsync(h.log); err != nil { return err } h.syncLogNs.Add(uint64(time.Since(t))) @@ -501,22 +707,25 @@ func (p *heapSync) finish() error { h.syncBytes.Add(uint64(p.bytes)) h.mu.Lock() defer h.mu.Unlock() - for _, r := range p.release { - if err := punchHole(h.file, int64(r)*HeapRegionBytes, HeapRegionBytes); err != nil { + for _, id := range p.release { + hf := h.files[id] + hf.f.Close() + delete(h.files, id) + h.deadBytes -= hf.dead + if err = os.Remove(filepath.Join(h.Directory, dataName(id))); err != nil { return err } - h.regions[r] = region{released: true} } return nil } // compact is the heap's maintenance on the adapter's cadence: one -// bounded cleaning pass, and every HeapSnapshotEvery calls the key-map -// snapshot that bounds the replay on open. +// bounded mover pass, and every HeapSnapshotEvery calls a new index +// generation, which bounds the replay on open. func (h *HeapStore) compact() (bool, error) { - cleaned, err := h.clean(HeapCleanBytes) + moved, err := h.clean(HeapCleanBytes) if err != nil { - return cleaned, err + return moved, err } h.mu.Lock() h.snapshots++ @@ -528,21 +737,45 @@ func (h *HeapStore) compact() (bool, error) { if due { err = h.Snapshot() } - return cleaned, err + return moved, err +} + +// Snapshot starts a new index generation: the whole map, then the +// deltas to come. Serialized with block syncs, so the map is exactly +// the state of the last finished delta and no delta lands in the +// generation being retired. Off the protocol path. +func (h *HeapStore) Snapshot() error { + h.syncMu.Lock() + defer h.syncMu.Unlock() + h.mu.Lock() + defer h.mu.Unlock() + if h.closed { + return errStoreClosed + } + h.gen++ + return h.startGeneration() +} + +// heapMove is one live entry the mover copies: where it was, where +// it goes, and the bytes to write there. +type heapMove struct { + key [32]byte + from, to slot + hf *heapFile + entry []byte } -// clean is the mover: it takes the regions with the most dead bytes, -// once at least HeapCleanRatio of each is dead, copies their live -// entries to the end -- at most budget bytes per pass -- and marks -// the regions for release once the delta naming the copies is -// durable. The pass syncs its own copies, so they never land in a -// block's barrier and the pass can be sized to the append rate; and -// it holds the shard's lock only to choose and to name, never across -// the copy or its fsync (spec 1.6). A byte released never costs more -// than a byte copied (spec 1.2). +// clean is the mover: it takes the files with the most dead bytes -- +// once HeapCleanRatio of each is dead, or the deadest whatever its +// ratio while dead bytes exceed live -- copies their live entries to +// the mover's file, at most budget bytes per pass, and marks a file +// left with nothing live for deletion once the delta naming the +// copies is durable. The pass syncs its own copies, so they never +// land in a block's barrier, and holds the shard's lock only to +// choose and reserve and again to name, never across the copy or its +// fsync (spec 1.6). A byte released never costs more than a byte +// copied unless the size bound forces it (spec 1.2). func (h *HeapStore) clean(budget int64) (bool, error) { - // 1. Under the lock: pick the regions, read them, decide what is - // live and where each copy will go h.mu.Lock() if h.closed { h.mu.Unlock() @@ -550,19 +783,19 @@ func (h *HeapStore) clean(budget int64) (bool, error) { } if len(h.release) > 0 { h.mu.Unlock() - return false, nil // The last pass's release is still waiting on a sync + return false, nil // The last pass's deletions are still waiting on a sync } var moves []heapMove - var regions []int + var taken []*heapFile var copied int64 - for len(regions) < HeapCleanRegions && copied < budget { - pick := h.pickRegion() - if pick < 0 { + for len(taken) < HeapCleanFiles && copied < budget { + hf := h.pickFile() + if hf == nil { break } - h.regions[pick].cleaning = true - regions = append(regions, pick) - m, n, err := h.planRegion(pick, budget-copied) + hf.cleaning = true + taken = append(taken, hf) + m, n, err := h.planFile(hf, budget-copied) if err != nil { h.mu.Unlock() return false, err @@ -570,34 +803,40 @@ func (h *HeapStore) clean(budget int64) (bool, error) { moves = append(moves, m...) copied += n } - for _, r := range regions { - h.regions[r].cleaning = false + for _, hf := range taken { + hf.cleaning = false + } + // The mover's file is dirty with the copies; the block's sync must + // not have to wait for them, so they are synced here and taken off + // the dirty set + var movFiles []*heapFile + for _, m := range moves { + if len(movFiles) == 0 || movFiles[len(movFiles)-1] != m.hf { + movFiles = append(movFiles, m.hf) + } + } + for _, hf := range movFiles { + delete(h.dirty, hf.id) } h.mu.Unlock() - if len(regions) == 0 { + if len(taken) == 0 { return false, nil } - // 2. Without the lock: write the copies and make them durable. - // Their slots were reserved at the end of the file under the lock, - // so puts landing meanwhile go past them. for _, m := range moves { - if _, err := h.file.WriteAt(m.entry, m.to.off); err != nil { + if _, err := m.hf.f.WriteAt(m.entry, int64(m.to.off)); err != nil { return false, err } } - if len(moves) > 0 { - if err := fsync(h.file); err != nil { + for _, hf := range movFiles { + if err := fsync(hf.f); err != nil { return false, err } } - // 3. Under the lock: name the copies, unless the key was rewritten - // meanwhile, in which case the copy is dead on arrival; and mark - // the regions the pass emptied h.mu.Lock() defer h.mu.Unlock() for _, m := range moves { s, live := h.index[m.key] - if live && s.off == m.from.off { + if live && s == m.from { h.index[m.key] = m.to h.touched[m.key] = struct{}{} h.kill(m.from) @@ -605,272 +844,119 @@ func (h *HeapStore) clean(budget int64) (bool, error) { h.kill(m.to) // Reserved and written, but no longer wanted } } - for _, r := range regions { - if h.regions[r].live == 0 { - h.release = append(h.release, r) + for _, hf := range taken { + if hf.live == 0 && hf != h.cur && hf != h.mov { + h.release = append(h.release, hf.id) } } h.movedBytes.Add(uint64(copied)) return true, nil } -// heapMove is one live entry the mover copies: where it was, where -// it goes, and the bytes to write there. -type heapMove struct { - key [32]byte - from, to slot - entry []byte -} - -// pickRegion is the region with the most dead bytes, once at least -// HeapCleanRatio of it is dead; -1 when none qualifies. The region -// the block in progress is appending to is never taken: a slot taken -// this block may still be rewritten in place, and moving it would -// race that. The caller holds the lock. -func (h *HeapStore) pickRegion() int { - current := int(h.size / HeapRegionBytes) - pick, best := -1, 0.0 - for i, r := range h.regions { - if i >= current || r.released || r.dead == 0 || r.cleaning { +// pickFile is the file the mover takes next: the deadest by fraction +// among those not open for append, if it is dead enough or the heap +// is over its size bound. The caller holds the lock. +func (h *HeapStore) pickFile() *heapFile { + var pick *heapFile + best := 0.0 + for _, hf := range h.files { + if hf == h.cur || hf == h.mov || hf.cleaning || hf.dead == 0 { continue } - if f := float64(r.dead) / float64(r.dead+r.live); f >= HeapCleanRatio && f > best { - pick, best = i, f + if f := float64(hf.dead) / float64(hf.size); f > best { + pick, best = hf, f } } + if pick == nil || (best < HeapCleanRatio && h.deadBytes <= h.liveBytes) { + return nil + } return pick } -// planRegion reads one region and reserves, at the end of the file, a -// slot for each live entry in it up to budget bytes; the rest wait for -// the next pass. The caller holds the lock. The reserved slots are -// counted live in their region at once, so the accounting is right -// whether or not the copy is wanted when it lands. -func (h *HeapStore) planRegion(pick int, budget int64) (moves []heapMove, copied int64, err error) { - from := int64(pick) * HeapRegionBytes - to := from + HeapRegionBytes - if to > h.size { - to = h.size - } - buf := make([]byte, to-from) - if _, err = h.file.ReadAt(buf, from); err != nil && !errors.Is(err, io.EOF) { +// planFile reads one file and reserves, in the mover's file, a slot +// for each live entry in it up to budget bytes; the rest wait for the +// next pass. The caller holds the lock. +func (h *HeapStore) planFile(hf *heapFile, budget int64) (moves []heapMove, copied int64, err error) { + buf := make([]byte, hf.size) + if _, err = hf.f.ReadAt(buf, 0); err != nil { return nil, 0, err } var at int64 - for at+heapHeader <= int64(len(buf)) { - capacity := binary.LittleEndian.Uint32(buf[at:]) - if capacity == 0 || at+int64(capacity) > int64(len(buf)) { - break // Unwritten, or a slot that straddles the region + for at < int64(len(buf)) { + size, _, key, value, ok := decodeEntry(buf[at:]) + if !ok { + break } - var key [32]byte - copy(key[:], buf[at+16:]) - if s, live := h.index[key]; live && s.off == from+at { + if s, live := h.index[key]; live && s.file == hf.id && int64(s.off) == at { if copied >= budget { break // The rest next pass } - n := binary.LittleEndian.Uint32(buf[at+4:]) - value := buf[at+int64(heapHeader) : at+int64(heapHeader)+int64(n)] - ns := slot{off: h.size, cap: capacity, n: n, block: h.height} - h.append(ns) + to, off, err := h.reserve(true, size) + if err != nil { + return nil, 0, err + } + ns := slot{file: to.id, off: uint32(off), n: uint32(len(value)), block: h.height} // The copy is this block's write of the key: it carries this // height, so a repair scan prefers it to the original - moves = append(moves, heapMove{key: key, from: s, to: ns, entry: encodeEntry(capacity, h.height, key, value)}) - copied += int64(capacity) + moves = append(moves, heapMove{key: key, from: s, to: ns, hf: to, entry: encodeEntry(h.height, key, value)}) + copied += size } - at += int64(capacity) + at += size } h.cleanedBytes.Add(uint64(at)) return moves, copied, nil } -// Snapshot writes the whole key map and drops the deltas it covers -// from the log, so that the replay on open stays bounded. Off the -// protocol path, on the maintenance cadence. -func (h *HeapStore) Snapshot() error { - h.mu.RLock() - if h.closed { - h.mu.RUnlock() - return errStoreClosed - } - buf := make([]byte, heapDeltaHdr+len(h.index)*heapDeltaRec+4) - binary.LittleEndian.PutUint32(buf, heapMagic) - binary.LittleEndian.PutUint64(buf[4:], h.height) - binary.LittleEndian.PutUint32(buf[20:], uint32(len(h.index))) - at := heapDeltaHdr - for key, s := range h.index { - at += putDeltaRec(buf[at:], key, s) - } - binary.LittleEndian.PutUint32(buf[at:], crc32.ChecksumIEEE(buf[:at])) - logSize, err := h.log.Seek(0, io.SeekCurrent) - h.mu.RUnlock() - if err != nil { - return err - } - // Written aside and renamed over the old snapshot, so a crash - // leaves one or the other whole - tmp := filepath.Join(h.Directory, "index.snap.tmp") - f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) - if err != nil { - return err - } - if _, err = f.Write(buf); err != nil { - f.Close() - return err - } - if err = fsync(f); err != nil { - f.Close() - return err - } - if err = f.Close(); err != nil { - return err - } - if err = os.Rename(tmp, filepath.Join(h.Directory, "index.snap")); err != nil { - return err - } - // The deltas the snapshot covers are what was in the log when it - // was taken; deltas appended since stay - h.mu.Lock() - defer h.mu.Unlock() - rest, err := readFrom(h.log, logSize) - if err != nil { - return err - } - if err = h.log.Truncate(0); err != nil { - return err - } - if _, err = h.log.Seek(0, io.SeekStart); err != nil { - return err - } - if len(rest) > 0 { - if _, err = h.log.Write(rest); err != nil { - return err - } - } - return fsync(h.log) -} - -func readFrom(f *os.File, off int64) ([]byte, error) { - end, err := f.Seek(0, io.SeekEnd) +// RepairHeapStore rebuilds the key map by reading the keys from the +// data: every entry carries its key, its height and a checksum, so a +// scan of the data files in order recovers the map without any index. +// For each key the copy with the highest height wins, later in the +// scan on a tie; an entry above the committed height -- a block whose +// sync never finished -- is dropped, as is any torn or damaged slot. +// The rebuilt map starts a new generation, so the next open is an +// ordinary one. +func RepairHeapStore(directory string, committed uint64) (*HeapStore, error) { + dataIDs, gens, err := listHeap(directory) if err != nil { return nil, err } - if end <= off { - return nil, nil - } - buf := make([]byte, end-off) - _, err = f.ReadAt(buf, off) - return buf, err -} - -// loadSnapshot reads index.snap into the key map, if there is one. -// The caller holds the lock. -func (h *HeapStore) loadSnapshot() error { - buf, err := os.ReadFile(filepath.Join(h.Directory, "index.snap")) - if errors.Is(err, os.ErrNotExist) { - return nil - } - if err != nil { - return err - } - n, err := h.applyDelta(buf) - if err != nil { - return fmt.Errorf("heap: snapshot: %w", err) - } - if n != len(buf) { - return fmt.Errorf("heap: snapshot has %d trailing bytes", len(buf)-n) - } - return nil -} - -// replayLog applies every whole delta in index.log; a torn tail is -// what a crash leaves and is dropped, its slots unnamed. The caller -// holds the lock. -func (h *HeapStore) replayLog() error { - buf, err := readFrom(h.log, 0) - if err != nil { - return err + for _, g := range gens { + os.Remove(filepath.Join(directory, indexName(g))) } - at := 0 - for at < len(buf) { - n, err := h.applyDelta(buf[at:]) + h := &HeapStore{Directory: directory, index: map[[32]byte]slot{}, touched: map[[32]byte]struct{}{}, + dirty: map[uint32]*heapFile{}, files: map[uint32]*heapFile{}, height: committed} + heights := map[[32]byte]uint64{} + for _, id := range dataIDs { + data, err := os.ReadFile(filepath.Join(directory, dataName(id))) if err != nil { - // Torn: keep what is whole, drop the rest - if err = h.log.Truncate(int64(at)); err != nil { - return err - } - break + return nil, err } - at += n - } - if _, err = h.log.Seek(0, io.SeekEnd); err != nil { - return err - } - return nil -} - -var errHeapTorn = errors.New("heap: torn index record") - -// applyDelta applies one delta or snapshot record and returns its -// length. -func (h *HeapStore) applyDelta(buf []byte) (int, error) { - if len(buf) < heapDeltaHdr || binary.LittleEndian.Uint32(buf) != heapMagic { - return 0, errHeapTorn - } - count := int(binary.LittleEndian.Uint32(buf[20:])) - end := heapDeltaHdr + count*heapDeltaRec - if len(buf) < end+4 || crc32.ChecksumIEEE(buf[:end]) != binary.LittleEndian.Uint32(buf[end:]) { - return 0, errHeapTorn - } - if height := binary.LittleEndian.Uint64(buf[4:]); height > h.height { - h.height = height - } - for at := heapDeltaHdr; at < end; at += heapDeltaRec { - var key [32]byte - copy(key[:], buf[at:]) - h.index[key] = slot{off: int64(binary.LittleEndian.Uint64(buf[at+32:])), - cap: binary.LittleEndian.Uint32(buf[at+40:]), n: binary.LittleEndian.Uint32(buf[at+44:])} - } - return end + 4, nil -} - -// deriveExtent finds the append point and the regions' accounting -// from the key map and cuts the file back to it: anything past the -// last named slot is a crash's torn writes or the entries of a block -// whose delta never became durable, and must not be read as anything. -// A region with nothing live is released (idempotent for one already -// punched). The caller holds the lock. -func (h *HeapStore) deriveExtent() error { - var end int64 - h.liveBytes, h.regions, h.release = 0, nil, nil - for _, s := range h.index { - if e := s.off + int64(s.cap); e > end { - end = e + var off int64 + for off < int64(len(data)) { + size, height, key, value, ok := decodeEntry(data[off:]) + if !ok { + break + } + if height <= committed { + if prev, seen := heights[key]; !seen || height >= prev { + heights[key] = height + h.index[key] = slot{file: id, off: uint32(off), n: uint32(len(value))} + } + } + off += size } - h.liveBytes += int64(s.cap) - r := int(s.off / HeapRegionBytes) - for len(h.regions) <= r { - h.regions = append(h.regions, region{}) + if id >= h.nextID { + h.nextID = id + 1 } - h.regions[r].live += int64(s.cap) } - h.size, h.syncedTo = end, end - if err := h.file.Truncate(end); err != nil { - return err + h.gen = 1 + if err = h.startGeneration(); err != nil { + return nil, err } - for i := range h.regions { - extent := HeapRegionBytes - if e := end - int64(i)*HeapRegionBytes; e < extent { - extent = e - } - h.regions[i].dead = extent - h.regions[i].live - if h.regions[i].live == 0 && int64(i+1)*HeapRegionBytes <= end { - if err := punchHole(h.file, int64(i)*HeapRegionBytes, HeapRegionBytes); err != nil { - return err - } - h.regions[i] = region{released: true} - } + if err = h.deriveFiles(dataIDs); err != nil { + return nil, err } - return nil + return h, nil } // Stats maps the heap's counters onto the store's report: every read @@ -885,29 +971,27 @@ func (h *HeapStore) Stats() StoreStats { PutDuplicate: h.putInPlace.Load(), LookupTotal: h.lookups.Load(), LiveHit: h.hits.Load(), + ActiveSegments: len(h.files), ResidentBloomBytes: uint64(len(h.index)) * (32 + 24), } } -// HoleRatio reports the dead bytes in unreleased regions against the -// live bytes: what the cleaner has yet to reclaim. +// HoleRatio reports the dead bytes in the files against the live +// bytes: what the mover has yet to reclaim. func (h *HeapStore) HoleRatio() (dead, live int64) { h.mu.RLock() defer h.mu.RUnlock() - for _, r := range h.regions { - dead += r.dead - } - return dead, h.liveBytes + return h.deadBytes, h.liveBytes } // SyncCost reports the block syncs so far: how many, the bytes their -// heap fsyncs covered, and the time spent in the heap's fsync and in -// the delta's write and fsync. +// data fsyncs covered, and the time spent in the data files' fsyncs +// and in the delta's write and fsync. func (h *HeapStore) SyncCost() (syncs, bytes uint64, heapFsync, delta time.Duration) { return h.syncs.Load(), h.syncBytes.Load(), time.Duration(h.syncHeapNs.Load()), time.Duration(h.syncLogNs.Load()) } -// Cleaned reports what the cleaner has scanned and what it had to +// Cleaned reports what the mover has scanned and what it had to // copy: the ratio is the heap's write amplification. func (h *HeapStore) Cleaned() (scanned, moved uint64) { return h.cleanedBytes.Load(), h.movedBytes.Load() diff --git a/database/heap_linux.go b/database/heap_linux.go deleted file mode 100644 index 876cabc..0000000 --- a/database/heap_linux.go +++ /dev/null @@ -1,15 +0,0 @@ -//go:build linux - -package blockchainDB - -import ( - "os" - "syscall" -) - -// punchHole releases the bytes of [off, off+n) back to the filesystem -// without changing the file's size: the cleaned head of a heap. -func punchHole(f *os.File, off, n int64) error { - const punch = 0x02 | 0x01 // FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE - return syscall.Fallocate(int(f.Fd()), punch, off, n) -} diff --git a/database/heap_other.go b/database/heap_other.go deleted file mode 100644 index 1a20410..0000000 --- a/database/heap_other.go +++ /dev/null @@ -1,9 +0,0 @@ -//go:build !linux - -package blockchainDB - -import "os" - -// punchHole is a no-op where the filesystem cannot release a range; -// the cleaned head then costs disk until the file is rewritten. -func punchHole(*os.File, int64, int64) error { return nil } diff --git a/database/heap_test.go b/database/heap_test.go index 473c339..4b06eb9 100644 --- a/database/heap_test.go +++ b/database/heap_test.go @@ -15,66 +15,89 @@ func heapDir(t *testing.T) string { func key(b byte) (k [32]byte) { k[0] = b; return } +func syncHeap(t *testing.T, h *HeapStore) { + t.Helper() + p, err := h.beginBlockSync() + require.NoError(t, err) + require.NoError(t, p.finish()) +} + +// Small files, so that rolling and deletion happen within a test. +func smallFiles(t *testing.T) { + t.Helper() + was := HeapFileBytes + HeapFileBytes = 1024 + t.Cleanup(func() { HeapFileBytes = was }) +} + // A key rewritten within the block reuses its slot; rewritten in a -// later block it is appended, the old slot dead where it lies until a -// clean pass moves the live entries past it and the next sync -// releases the region. -func TestHeapRewriteReusesWithinTheBlockAndCleansOneSyncLate(t *testing.T) { +// later block it is appended, the old slot dead where it lies until +// the mover copies the file's live entries out and the next sync +// deletes the file. +func TestHeapRewriteReusesWithinTheBlockAndMovesOneSyncLate(t *testing.T) { + smallFiles(t) h, err := NewHeapStore(heapDir(t)) require.NoError(t, err) defer h.Close() h.AdvanceBlock(1) require.NoError(t, h.Put(key(1), []byte("one"))) - off := h.index[key(1)].off + first := h.index[key(1)] require.NoError(t, h.Put(key(1), []byte("uno"))) - require.Equal(t, off, h.index[key(1)].off, "same block, fits: in place") + require.Equal(t, first, h.index[key(1)], "same block, fits: in place") require.EqualValues(t, 1, h.putInPlace.Load()) v, err := h.Get(key(1)) require.NoError(t, err) require.Equal(t, "uno", string(v)) - // Block 1 durable; block 2 rewrites the key - sync := func() { - p, err := h.beginBlockSync() - require.NoError(t, err) - require.NoError(t, p.finish()) - } - sync() + syncHeap(t, h) h.AdvanceBlock(2) require.NoError(t, h.Put(key(1), []byte("two"))) - require.NotEqual(t, off, h.index[key(1)].off, "a durable slot is never rewritten") + require.NotEqual(t, first, h.index[key(1)], "a durable slot is never rewritten") dead, live := h.HoleRatio() - require.EqualValues(t, heapMinCap, dead, "the old slot is dead where it lies") - require.EqualValues(t, heapMinCap, live) - sync() - - // A clean pass in block 3 takes the region (half dead) and moves - // the live entry out of it; the region is released only by the - // sync after it. The region must not be the one block 3 appends - // to, so the file is pushed into a second region first. + require.EqualValues(t, entrySize(3), dead, "the old slot is dead where it lies") + require.EqualValues(t, entrySize(3), live) + // Fill the block's file past its size so it rolls, then kill most + // of what the first file holds + for i := byte(10); i < 40; i++ { + require.NoError(t, h.Put(key(i), make([]byte, 64))) + } + syncHeap(t, h) + require.Greater(t, len(h.files), 1, "the block's file rolled") h.AdvanceBlock(3) - h.size = HeapRegionBytes // Block 3 appends into region 1 - cleaned, err := h.clean(1 << 20) + for i := byte(10); i < 39; i++ { + require.NoError(t, h.Put(key(i), make([]byte, 64))) + } + syncHeap(t, h) + + h.AdvanceBlock(4) + moved, err := h.clean(1 << 20) require.NoError(t, err) - require.True(t, cleaned) - require.Equal(t, []int{0}, h.release, "not released yet: the copies are not durable") - require.False(t, h.regions[0].released) - scanned, moved := h.Cleaned() - require.EqualValues(t, 2*heapMinCap, scanned) - require.EqualValues(t, heapMinCap, moved, "one live entry copied, one dead skipped") - sync() - require.True(t, h.regions[0].released, "released after the sync") - dead, live = h.HoleRatio() - require.Zero(t, dead) - require.EqualValues(t, heapMinCap, live) + require.True(t, moved, "a mostly dead file is taken") + require.NotEmpty(t, h.release, "a file emptied by the pass waits for the sync") + for _, id := range h.release { + _, err := os.Stat(filepath.Join(h.Directory, dataName(id))) + require.NoError(t, err, "not deleted yet: the copies are not durable") + } + released := append([]uint32(nil), h.release...) + syncHeap(t, h) + for _, id := range released { + _, err := os.Stat(filepath.Join(h.Directory, dataName(id))) + require.ErrorIs(t, err, os.ErrNotExist, "deleted after the sync") + } + _, copied := h.Cleaned() + require.Greater(t, copied, uint64(0), "the live entries left in it were copied out") v, err = h.Get(key(1)) require.NoError(t, err) require.Equal(t, "two", string(v)) + for i := byte(10); i < 40; i++ { + _, err := h.Get(key(i)) + require.NoError(t, err) + } } -// Reopening replays the log: every synced value is back, the holes -// are derived, and a block that was never synced is gone -- its -// slots unnamed and its bytes cut from the file. +// Reopening replays the generation: every synced value is back, the +// files' accounting is derived, and a block that was never synced is +// gone -- its slots unnamed and its bytes cut from the file. func TestHeapReopenKeepsTheDurableAndDropsTheRest(t *testing.T) { dir := heapDir(t) h, err := NewHeapStore(dir) @@ -83,21 +106,18 @@ func TestHeapReopenKeepsTheDurableAndDropsTheRest(t *testing.T) { for i := byte(1); i <= 50; i++ { require.NoError(t, h.Put(key(i), []byte{i})) } - p, err := h.beginBlockSync() - require.NoError(t, err) - require.NoError(t, p.finish()) + syncHeap(t, h) h.AdvanceBlock(2) - require.NoError(t, h.Put(key(1), []byte("rewritten in block 2"))) // New slot, old pending - p, err = h.beginBlockSync() - require.NoError(t, err) - require.NoError(t, p.finish()) - size := h.size + require.NoError(t, h.Put(key(1), []byte("rewritten in block 2"))) + syncHeap(t, h) + size := h.cur.size // Block 3: written, never synced -- the crash h.AdvanceBlock(3) require.NoError(t, h.Put(key(2), []byte("lost"))) require.NoError(t, h.Put(key(99), []byte("lost too"))) - // Drop the store without Close: the OS has the bytes, the log has no delta - h.file.Close() + for _, hf := range h.files { + hf.f.Close() + } h.log.Close() r, err := OpenHeapStore(dir) @@ -111,13 +131,16 @@ func TestHeapReopenKeepsTheDurableAndDropsTheRest(t *testing.T) { require.Equal(t, []byte{2}, v, "block 3's rewrite was never durable") _, err = r.Get(key(99)) require.ErrorIs(t, err, errNotFound) - require.Equal(t, size, r.size, "the file is cut back to the durable append point") + require.Len(t, r.files, 1) + for _, hf := range r.files { + require.Equal(t, size, hf.size, "the file is cut back to the durable append point") + } dead, _ := r.HoleRatio() - require.EqualValues(t, heapMinCap, dead, "key 1's block-1 slot is dead where it lies") + require.EqualValues(t, entrySize(1), dead, "key 1's block-1 slot is dead where it lies") require.EqualValues(t, 2, r.height, "the durable height: block 3 never synced") } -// A torn delta at the end of the log is dropped whole. +// A torn delta at the end of the generation is dropped whole. func TestHeapTornLogTailIsDropped(t *testing.T) { dir := heapDir(t) h, err := NewHeapStore(dir) @@ -125,7 +148,7 @@ func TestHeapTornLogTailIsDropped(t *testing.T) { h.AdvanceBlock(1) require.NoError(t, h.Put(key(1), []byte("one"))) require.NoError(t, h.Close()) - f, err := os.OpenFile(filepath.Join(dir, "index.log"), os.O_WRONLY|os.O_APPEND, 0o644) + f, err := os.OpenFile(filepath.Join(dir, indexName(1)), os.O_WRONLY|os.O_APPEND, 0o644) require.NoError(t, err) _, err = f.Write([]byte{0x50, 0x41, 0x45, 0x48, 9, 9}) // A marker and six bytes of nothing require.NoError(t, err) @@ -137,14 +160,15 @@ func TestHeapTornLogTailIsDropped(t *testing.T) { v, err := r.Get(key(1)) require.NoError(t, err) require.Equal(t, "one", string(v)) - st, err := os.Stat(filepath.Join(dir, "index.log")) + st, err := os.Stat(filepath.Join(dir, indexName(1))) require.NoError(t, err) - require.EqualValues(t, heapDeltaHdr+heapDeltaRec+4, st.Size(), "one whole delta of one key remains") + require.EqualValues(t, 2*(heapIndexHdr+4)+heapIndexRec, st.Size(), "the empty snapshot and one whole delta of one key remain") } -// A snapshot carries the map and empties the log; what comes after is -// replayed on top of it. -func TestHeapSnapshotBoundsTheReplay(t *testing.T) { +// A snapshot starts a new generation in its own file and retires the +// old one; the replay lands on the new one; and a generation whose +// write was interrupted is ignored. +func TestHeapSnapshotStartsAGenerationSafely(t *testing.T) { dir := heapDir(t) h, err := NewHeapStore(dir) require.NoError(t, err) @@ -156,32 +180,30 @@ func TestHeapSnapshotBoundsTheReplay(t *testing.T) { for i := byte(1); i <= 20; i++ { require.NoError(t, h.Put(key(i), []byte{byte(b), i})) } - p, err := h.beginBlockSync() - require.NoError(t, err) - require.NoError(t, p.finish()) + syncHeap(t, h) if b == 20 { - _, err := h.compact() // Nothing to clean: one region, still being appended to + _, err := h.compact() require.NoError(t, err) - st, err := os.Stat(filepath.Join(dir, "index.log")) + _, err = os.Stat(filepath.Join(dir, indexName(1))) + require.ErrorIs(t, err, os.ErrNotExist, "generation 1 retired") + _, err = os.Stat(filepath.Join(dir, indexName(2))) require.NoError(t, err) - require.Zero(t, st.Size(), "the log is empty after the snapshot") } } require.NoError(t, h.Close()) + // A crash mid-snapshot leaves a .tmp; it is not a generation + require.NoError(t, os.WriteFile(filepath.Join(dir, indexName(3)+".tmp"), []byte("half"), 0o644)) r, err := OpenHeapStore(dir) require.NoError(t, err) defer r.Close() + require.EqualValues(t, 2, r.gen) + _, err = os.Stat(filepath.Join(dir, indexName(3)+".tmp")) + require.ErrorIs(t, err, os.ErrNotExist) for i := byte(1); i <= 20; i++ { v, err := r.Get(key(i)) require.NoError(t, err) require.Equal(t, []byte{30, i}, v) } - // Everything fits in the region the blocks append to, which the - // cleaner never takes: blocks 1-29 lie dead behind block 30's live - // slots, and the accounting survives the reopen. - dead, live := r.HoleRatio() - require.EqualValues(t, 20*heapMinCap, live) - require.EqualValues(t, 29*20*heapMinCap, dead) } // A slot whose bytes were damaged is an error, never a value. @@ -191,11 +213,11 @@ func TestHeapChecksumCatchesADamagedSlot(t *testing.T) { require.NoError(t, err) h.AdvanceBlock(1) require.NoError(t, h.Put(key(1), []byte("intact"))) - off := h.index[key(1)].off + s := h.index[key(1)] require.NoError(t, h.Close()) - f, err := os.OpenFile(filepath.Join(dir, "heap.dat"), os.O_WRONLY, 0o644) + f, err := os.OpenFile(filepath.Join(dir, dataName(s.file)), os.O_WRONLY, 0o644) require.NoError(t, err) - _, err = f.WriteAt([]byte("damaged"), off+heapHeader) + _, err = f.WriteAt([]byte("damaged"), int64(s.off)+heapHeader) require.NoError(t, err) require.NoError(t, f.Close()) r, err := OpenHeapStore(dir) @@ -205,9 +227,58 @@ func TestHeapChecksumCatchesADamagedSlot(t *testing.T) { require.ErrorContains(t, err, "checksum") } +// With the index gone, the map is rebuilt from the data: the highest +// committed copy of each key wins, an entry from the block that never +// synced is dropped. +func TestHeapRepairReadsTheKeysFromTheData(t *testing.T) { + smallFiles(t) + dir := heapDir(t) + h, err := NewHeapStore(dir) + require.NoError(t, err) + h.AdvanceBlock(1) + for i := byte(1); i <= 10; i++ { + require.NoError(t, h.Put(key(i), make([]byte, 100))) + } + syncHeap(t, h) + h.AdvanceBlock(2) + require.NoError(t, h.Put(key(1), []byte{2, 1})) // Newer copy, in a later file + syncHeap(t, h) + h.AdvanceBlock(3) + require.NoError(t, h.Put(key(2), []byte{3, 2})) // Never synced: not committed + for _, hf := range h.files { + hf.f.Close() + } + h.log.Close() + require.NoError(t, os.Remove(filepath.Join(dir, indexName(1)))) + + _, err = OpenHeapStore(dir) + require.ErrorIs(t, err, ErrHeapNeedsRepair) + r, err := RepairHeapStore(dir, 2) + require.NoError(t, err) + defer r.Close() + v, err := r.Get(key(1)) + require.NoError(t, err) + require.Equal(t, []byte{2, 1}, v, "the block-2 copy wins") + v, err = r.Get(key(2)) + require.NoError(t, err) + require.Len(t, v, 100, "block 3 never committed: its copy is dropped") + require.EqualValues(t, 10, r.LiveRecords()) + require.NoError(t, r.Close()) + re, err := OpenHeapStore(dir) + require.NoError(t, err, "the repair left a generation: an ordinary open") + defer re.Close() + v, err = re.Get(key(1)) + require.NoError(t, err) + require.Equal(t, []byte{2, 1}, v) +} + // A shard built with the heap seals, compacts, closes and reopens as -// a heap, through the same KV2 and KVShard surface. +// a heap, through the same KV2 and KVShard surface, with files rolled +// and deleted along the way. func TestHeapShardRoundTrip(t *testing.T) { + was := HeapFileBytes + HeapFileBytes = 256 << 10 + defer func() { HeapFileBytes = was }() dir := filepath.Join(t.TempDir(), "shards") kvs, err := NewKVShardHeapN(dir, 2, 1000) require.NoError(t, err) @@ -217,25 +288,30 @@ func TestHeapShardRoundTrip(t *testing.T) { for i := range hot { hot[i] = fr.NextHash() } - for b := uint64(1); b <= 45; b++ { + for b := uint64(1); b <= 60; b++ { for _, k := range hot { - require.NoError(t, kvs.PutDyna(k, append([]byte{byte(b)}, k[:8]...))) + require.NoError(t, kvs.PutDyna(k, append([]byte{byte(b)}, fr.RandBuff(100, 300)...))) } require.NoError(t, kvs.PutPerm(fr.NextHash(), []byte("perm"))) require.NoError(t, kvs.SealBlock(b)) - if b%20 == 0 { + if b%10 == 0 { require.NoError(t, kvs.Compress()) - _, err := kvs.MergeFinalized(b - MinFilterBlocks) - require.NoError(t, err) + if b > MinFilterBlocks { + _, err := kvs.MergeFinalized(b - MinFilterBlocks) + require.NoError(t, err) + } } } for _, k := range hot { v, err := kvs.GetDyna(k) require.NoError(t, err) - require.Equal(t, byte(45), v[0]) + require.Equal(t, byte(60), v[0]) } _, dyna := kvs.Stats() - require.EqualValues(t, 45*200, dyna.PutTotal) + require.EqualValues(t, 60*200, dyna.PutTotal) + require.NoError(t, kvs.SealBlock(61)) // The sync that deletes what the last pass emptied + dead, live := kvs.Shards[0].Heap.HoleRatio() + require.Less(t, dead, 2*live, "the mover keeps dead bytes under twice the live set") require.NoError(t, kvs.Close()) re, err := OpenKVShard(dir) @@ -246,53 +322,6 @@ func TestHeapShardRoundTrip(t *testing.T) { for _, k := range hot { v, err := re.GetDyna(k) require.NoError(t, err) - require.Equal(t, byte(45), v[0]) - } -} - -// With the index files gone, the map is rebuilt from the data: the -// highest committed copy of each key wins, an entry from the block -// that never synced is dropped, and a damaged slot is skipped. -func TestHeapRepairReadsTheKeysFromTheData(t *testing.T) { - dir := heapDir(t) - h, err := NewHeapStore(dir) - require.NoError(t, err) - sync := func() { - p, err := h.beginBlockSync() - require.NoError(t, err) - require.NoError(t, p.finish()) - } - h.AdvanceBlock(1) - for i := byte(1); i <= 10; i++ { - require.NoError(t, h.Put(key(i), []byte{1, i})) + require.Equal(t, byte(60), v[0]) } - sync() - h.AdvanceBlock(2) - require.NoError(t, h.Put(key(1), []byte{2, 1})) // Newer copy, higher offset - sync() - h.AdvanceBlock(3) - require.NoError(t, h.Put(key(2), []byte{3, 2})) // Never synced: not committed - h.file.Close() - h.log.Close() - require.NoError(t, os.Remove(filepath.Join(dir, "index.log"))) - - _, err = OpenHeapStore(dir) - require.ErrorIs(t, err, ErrHeapNeedsRepair) - r, err := RepairHeapStore(dir, 2) - require.NoError(t, err) - defer r.Close() - v, err := r.Get(key(1)) - require.NoError(t, err) - require.Equal(t, []byte{2, 1}, v, "the block-2 copy wins") - v, err = r.Get(key(2)) - require.NoError(t, err) - require.Equal(t, []byte{1, 2}, v, "block 3 never committed: its copy is dropped") - require.EqualValues(t, 10, r.LiveRecords()) - require.NoError(t, r.Close()) - re, err := OpenHeapStore(dir) - require.NoError(t, err, "the repair left a snapshot: an ordinary open") - defer re.Close() - v, err = re.Get(key(1)) - require.NoError(t, err) - require.Equal(t, []byte{2, 1}, v) } From 1b078afecca83c6647dc9471ea38270c1aaff16f Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 17:50:42 -0500 Subject: [PATCH 08/58] The mover never picks a file it is writing to, and reads outside the lock Two faults from the first run on files. Within a pass the mover's file could roll, leaving the previous one with copies reserved but not yet written; if that file held any dead bytes the pass's next pick took it and read past its real length (EOF at store 7, shard 1). And a file a pass had emptied could be picked again while its deletion waited on the sync. A file with copies in flight or a deletion pending is now never picked, and a pass that fails after reserving accounts its reservations dead. The pass also read each picked file under the shard's exclusive lock -- 16 MB per file, and the seal's p50 went from 54 to 229 ms in the minute it worked hardest -- so it now picks under the lock, reads without it (a picked file's bytes do not change), and plans and names under it again (spec 1.6). StoreStats gains the heap's own figures (resident index bytes, files, live and dead bytes, bytes scanned and copied) instead of borrowing the segment layer's; the no-op knobs leave the dynamic-layer interface. Tests: a child process killed mid-block three times over, reopened and repaired to exactly the last durable block; a put between the mover's copy and its naming leaves the copy dead on arrival; all under the race detector. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 106 +++++++++++++++++++++++-------- database/heap_crash_test.go | 123 ++++++++++++++++++++++++++++++++++++ database/heap_test.go | 46 ++++++++++++++ database/kv_2.go | 7 +- database/kv_shard.go | 6 ++ database/segstore.go | 10 +++ 6 files changed, 267 insertions(+), 31 deletions(-) create mode 100644 database/heap_crash_test.go diff --git a/database/heap.go b/database/heap.go index 741426c..a003095 100644 --- a/database/heap.go +++ b/database/heap.go @@ -101,6 +101,8 @@ type heapFile struct { size int64 live, dead int64 cleaning bool // Taken by the pass in progress + inflight int // Copies reserved in it and not yet written: its size runs ahead of its bytes + releasing bool // Emptied by a pass; deleted by the next sync } // slot is where an entry lives: its file, its offset there, the value @@ -532,6 +534,9 @@ func (h *HeapStore) reserve(mover bool, size int64) (hf *heapFile, off int64, er hf.live += size h.liveBytes += size h.dirty[hf.id] = hf + if mover { + hf.inflight++ + } return hf, off, nil } @@ -756,6 +761,10 @@ func (h *HeapStore) Snapshot() error { return h.startGeneration() } +// moverHook, when set, runs between the mover's copy and its naming: +// the window in which a put can make a copy dead on arrival. +var moverHook func() + // heapMove is one live entry the mover copies: where it was, where // it goes, and the bytes to write there. type heapMove struct { @@ -771,11 +780,12 @@ type heapMove struct { // the mover's file, at most budget bytes per pass, and marks a file // left with nothing live for deletion once the delta naming the // copies is durable. The pass syncs its own copies, so they never -// land in a block's barrier, and holds the shard's lock only to -// choose and reserve and again to name, never across the copy or its +// land in a block's barrier, and it holds the shard's lock only to +// pick, to plan, and to name: never across a read, a write or an // fsync (spec 1.6). A byte released never costs more than a byte // copied unless the size bound forces it (spec 1.2). func (h *HeapStore) clean(budget int64) (bool, error) { + // 1. Under the lock: pick the files h.mu.Lock() if h.closed { h.mu.Unlock() @@ -785,30 +795,55 @@ func (h *HeapStore) clean(budget int64) (bool, error) { h.mu.Unlock() return false, nil // The last pass's deletions are still waiting on a sync } - var moves []heapMove var taken []*heapFile - var copied int64 - for len(taken) < HeapCleanFiles && copied < budget { + for len(taken) < HeapCleanFiles { hf := h.pickFile() if hf == nil { break } hf.cleaning = true taken = append(taken, hf) - m, n, err := h.planFile(hf, budget-copied) + } + h.mu.Unlock() + if len(taken) == 0 { + return false, nil + } + // 2. Without the lock: read them. A picked file is neither the + // block's nor the mover's, so its bytes do not change; only what + // the index says of them can, and that is checked under the lock + contents := make([][]byte, len(taken)) + for i, hf := range taken { + contents[i] = make([]byte, hf.size) + if _, err := hf.f.ReadAt(contents[i], 0); err != nil { + h.mu.Lock() + for _, hf := range taken { + hf.cleaning = false + } + h.mu.Unlock() + return false, err + } + } + // 3. Under the lock: decide what is live and reserve each copy's + // slot in the mover's file; the mover's files leave the dirty set, + // since the pass syncs them itself + h.mu.Lock() + var moves []heapMove + var copied int64 + for i, hf := range taken { + m, n, err := h.planFile(hf, contents[i], budget-copied) if err != nil { h.mu.Unlock() return false, err } moves = append(moves, m...) copied += n + if copied >= budget { + break + } } for _, hf := range taken { hf.cleaning = false } - // The mover's file is dirty with the copies; the block's sync must - // not have to wait for them, so they are synced here and taken off - // the dirty set var movFiles []*heapFile for _, m := range moves { if len(movFiles) == 0 || movFiles[len(movFiles)-1] != m.hf { @@ -819,22 +854,36 @@ func (h *HeapStore) clean(budget int64) (bool, error) { delete(h.dirty, hf.id) } h.mu.Unlock() - if len(taken) == 0 { - return false, nil + // 4. Without the lock: write the copies and make them durable + fail := func(err error) (bool, error) { + h.mu.Lock() + for _, m := range moves { + m.hf.inflight-- // Reserved, never written: dead, and read no further + h.kill(m.to) + } + h.mu.Unlock() + return false, err } for _, m := range moves { if _, err := m.hf.f.WriteAt(m.entry, int64(m.to.off)); err != nil { - return false, err + return fail(err) } } for _, hf := range movFiles { if err := fsync(hf.f); err != nil { - return false, err + return fail(err) } } + if moverHook != nil { + moverHook() // Tests: a put between the copy and its naming + } + // 5. Under the lock: name the copies, unless the key was rewritten + // meanwhile, in which case the copy is dead on arrival; and mark + // the files the pass emptied h.mu.Lock() defer h.mu.Unlock() for _, m := range moves { + m.hf.inflight-- s, live := h.index[m.key] if live && s == m.from { h.index[m.key] = m.to @@ -846,6 +895,7 @@ func (h *HeapStore) clean(budget int64) (bool, error) { } for _, hf := range taken { if hf.live == 0 && hf != h.cur && hf != h.mov { + hf.releasing = true h.release = append(h.release, hf.id) } } @@ -860,7 +910,7 @@ func (h *HeapStore) pickFile() *heapFile { var pick *heapFile best := 0.0 for _, hf := range h.files { - if hf == h.cur || hf == h.mov || hf.cleaning || hf.dead == 0 { + if hf == h.cur || hf == h.mov || hf.cleaning || hf.inflight > 0 || hf.releasing || hf.dead == 0 { continue } if f := float64(hf.dead) / float64(hf.size); f > best { @@ -873,14 +923,11 @@ func (h *HeapStore) pickFile() *heapFile { return pick } -// planFile reads one file and reserves, in the mover's file, a slot -// for each live entry in it up to budget bytes; the rest wait for the -// next pass. The caller holds the lock. -func (h *HeapStore) planFile(hf *heapFile, budget int64) (moves []heapMove, copied int64, err error) { - buf := make([]byte, hf.size) - if _, err = hf.f.ReadAt(buf, 0); err != nil { - return nil, 0, err - } +// planFile walks one file's bytes, read outside the lock, and +// reserves in the mover's file a slot for each entry the index still +// names, up to budget bytes; the rest wait for the next pass. The +// caller holds the lock. +func (h *HeapStore) planFile(hf *heapFile, buf []byte, budget int64) (moves []heapMove, copied int64, err error) { var at int64 for at < int64(len(buf)) { size, _, key, value, ok := decodeEntry(buf[at:]) @@ -959,20 +1006,23 @@ func RepairHeapStore(directory string, committed uint64) (*HeapStore, error) { return h, nil } -// Stats maps the heap's counters onto the store's report: every read -// is answered from the key map (LiveHit), there are no segments, and -// the resident memory is the key map. +// Stats is the heap's report in the store's terms: puts and lookups +// as the segment layer counts them (every hit is answered from the +// key map, so they are all LiveHit), and the heap's own figures. func (h *HeapStore) Stats() StoreStats { h.mu.RLock() defer h.mu.RUnlock() return StoreStats{ PutTotal: h.putTotal.Load(), PutNew: h.putAppend.Load(), - PutDuplicate: h.putInPlace.Load(), LookupTotal: h.lookups.Load(), LiveHit: h.hits.Load(), - ActiveSegments: len(h.files), - ResidentBloomBytes: uint64(len(h.index)) * (32 + 24), + ResidentIndexBytes: uint64(len(h.index)) * (32 + 24), + HeapFiles: len(h.files), + HeapLiveBytes: uint64(h.liveBytes), + HeapDeadBytes: uint64(h.deadBytes), + HeapScannedBytes: h.cleanedBytes.Load(), + HeapMovedBytes: h.movedBytes.Load(), } } diff --git a/database/heap_crash_test.go b/database/heap_crash_test.go new file mode 100644 index 0000000..01a4992 --- /dev/null +++ b/database/heap_crash_test.go @@ -0,0 +1,123 @@ +package blockchainDB + +import ( + "bufio" + "fmt" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +const heapCrashDirEnv = "BDB_HEAP_CRASH_DIR" + +// The child: blocks of the same keys, every block rewriting every key +// with a value that says which block, a sync per block and a mover +// pass every few, and a line on stdout at each durability point. +// Killed by the parent at a moment of its choosing. +func TestHeapCrashChild(t *testing.T) { + dir := os.Getenv(heapCrashDirEnv) + if dir == "" { + t.Skip("helper process for TestHeapCrashRecovery") + } + HeapFileBytes = 64 << 10 + h, err := NewHeapStore(dir) + require.NoError(t, err, "child: new") + for b := uint64(1); ; b++ { + h.AdvanceBlock(b) + for i := 0; i < 300; i++ { + require.NoError(t, h.Put(crashKey(i), crashValue(b, i)), "child: put") + } + p, err := h.beginBlockSync() + require.NoError(t, err, "child: sync") + require.NoError(t, p.finish(), "child: finish") + fmt.Printf("CHECKPOINT %d\n", b) + if b%4 == 0 { + _, err := h.compact() + require.NoError(t, err, "child: compact") + } + } +} + +func crashKey(i int) (k [32]byte) { + k[0], k[1] = byte(i>>8), byte(i) + return +} + +func crashValue(b uint64, i int) []byte { + v := make([]byte, 40+i%60) + v[0], v[1], v[2] = byte(b>>16), byte(b>>8), byte(b) + v[3] = byte(i) + return v +} + +// The parent: run the child, let it reach some durability points, +// kill it dead, and check that the heap opens to exactly the last +// durable block -- every key at that block's value, nothing torn, +// nothing from the block in flight -- and that a repair from the data +// alone agrees. +func TestHeapCrashRecovery(t *testing.T) { + if testing.Short() { + t.Skip("spawns processes") + } + for round := 0; round < 3; round++ { + dir := filepath.Join(t.TempDir(), fmt.Sprintf("crash-%d", round)) + cmd := exec.Command(os.Args[0], "-test.run", "TestHeapCrashChild$") + cmd.Env = append(os.Environ(), heapCrashDirEnv+"="+dir) + out, err := cmd.StdoutPipe() + require.NoError(t, err) + require.NoError(t, cmd.Start()) + var last uint64 + sc := bufio.NewScanner(out) + want := uint64(6 + round*5) + for sc.Scan() { + line := sc.Text() + if !strings.HasPrefix(line, "CHECKPOINT ") { + continue + } + n, err := strconv.ParseUint(strings.TrimPrefix(line, "CHECKPOINT "), 10, 64) + require.NoError(t, err) + last = n + if last >= want { + break + } + } + // Somewhere inside the next block: the child is mid-write or + // mid-sync when it dies + time.Sleep(time.Duration(round*7) * time.Millisecond) + require.NoError(t, cmd.Process.Kill()) + _ = cmd.Wait() + + check := func(h *HeapStore, what string) { + t.Helper() + for i := 0; i < 300; i++ { + v, err := h.Get(crashKey(i)) + require.NoError(t, err, "%s: key %d", what, i) + require.Equal(t, crashValue(h.height, i), v, "%s: key %d is at the durable block", what, i) + } + } + h, err := OpenHeapStore(dir) + require.NoError(t, err) + require.GreaterOrEqual(t, h.height, last, "the durable block is at least the last checkpoint the parent saw") + check(h, "open") + durable := h.height + require.NoError(t, h.Close()) + + // The index thrown away: the data alone must say the same + _, gens, err := listHeap(dir) + require.NoError(t, err) + for _, g := range gens { + require.NoError(t, os.Remove(filepath.Join(dir, indexName(g)))) + } + r, err := RepairHeapStore(dir, durable) + require.NoError(t, err) + require.Equal(t, durable, r.height) + check(r, "repair") + require.NoError(t, r.Close()) + } +} diff --git a/database/heap_test.go b/database/heap_test.go index 4b06eb9..065f216 100644 --- a/database/heap_test.go +++ b/database/heap_test.go @@ -325,3 +325,49 @@ func TestHeapShardRoundTrip(t *testing.T) { require.Equal(t, byte(60), v[0]) } } + +// A key rewritten between the mover's copy and its naming leaves the +// copy dead on arrival: the put's value stands, and the copy's bytes +// are accounted dead in the mover's file. +func TestHeapMoveIsDeadOnArrivalIfTheKeyWasRewritten(t *testing.T) { + smallFiles(t) // Nine 112-byte entries to a file + h, err := NewHeapStore(heapDir(t)) + require.NoError(t, err) + defer h.Close() + h.AdvanceBlock(1) + for i := byte(1); i <= 20; i++ { + require.NoError(t, h.Put(key(i), make([]byte, 64))) + } + syncHeap(t, h) + // Two more blocks rewriting every key but 20 leave key 20 the one + // live entry in a file otherwise dead: the mover's next pick + for b := uint64(2); b <= 3; b++ { + h.AdvanceBlock(b) + for i := byte(1); i <= 19; i++ { + require.NoError(t, h.Put(key(i), make([]byte, 64))) + } + syncHeap(t, h) + } + h.AdvanceBlock(4) + moverHook = func() { + require.NoError(t, h.Put(key(20), []byte("rewritten while moving"))) + } + defer func() { moverHook = nil }() + deadBefore, _ := h.HoleRatio() + moved, err := h.clean(1 << 20) + require.NoError(t, err) + require.True(t, moved) + _, copied := h.Cleaned() + require.GreaterOrEqual(t, copied, uint64(entrySize(64)), "key 20 was among the entries copied") + v, err := h.Get(key(20)) + require.NoError(t, err) + require.Equal(t, "rewritten while moving", string(v), "the put wins") + // Every copied entry left its old slot dead; key 20's copy is dead + // as well, since the put took the key elsewhere before it was named + dead, _ := h.HoleRatio() + require.EqualValues(t, deadBefore+int64(copied)+entrySize(64), dead, "the old slots and the unwanted copy are dead") + syncHeap(t, h) + v, err = h.Get(key(20)) + require.NoError(t, err) + require.Equal(t, "rewritten while moving", string(v)) +} diff --git a/database/kv_2.go b/database/kv_2.go index e7ab8c4..233f967 100644 --- a/database/kv_2.go +++ b/database/kv_2.go @@ -155,7 +155,10 @@ func (k *KV2) SetFilterBlocks(n uint64) (err error) { if err = k.PermKV.SetFilterBlocks(n); err != nil { return err } - return k.dyna().SetFilterBlocks(n) + if k.DynaKV == nil { + return nil // A heap has no window + } + return k.DynaKV.SetFilterBlocks(n) } // dynaLayer is what KV2 asks of its dynamic layer, whichever of the @@ -169,8 +172,6 @@ type dynaLayer interface { Put(key [32]byte, value []byte) error AdvanceBlock(height uint64) LiveRecords() uint64 - SetFilterBlocks(n uint64) error - SetSealLimit(limit uint64) error beginBlockSync() (blockSync, error) compact() (bool, error) Stats() StoreStats diff --git a/database/kv_shard.go b/database/kv_shard.go index 633e90c..d13d79c 100644 --- a/database/kv_shard.go +++ b/database/kv_shard.go @@ -714,6 +714,12 @@ func (k *KVShard) Stats() (perm, dyna StoreStats) { dst.HistorySegments += s.HistorySegments dst.ActiveSegments += s.ActiveSegments dst.ResidentBloomBytes += s.ResidentBloomBytes + dst.ResidentIndexBytes += s.ResidentIndexBytes + dst.HeapFiles += s.HeapFiles + dst.HeapLiveBytes += s.HeapLiveBytes + dst.HeapDeadBytes += s.HeapDeadBytes + dst.HeapScannedBytes += s.HeapScannedBytes + dst.HeapMovedBytes += s.HeapMovedBytes } for _, shard := range k.Shards { if shard == nil { diff --git a/database/segstore.go b/database/segstore.go index 8406705..93dac2e 100644 --- a/database/segstore.go +++ b/database/segstore.go @@ -814,6 +814,16 @@ type StoreStats struct { HistorySegments int // Segments in history now ActiveSegments int // Segments in the window now ResidentBloomBytes uint64 // History filter memory held, of BloomResidentBytes + + // The heap's own figures (heap.go), zero for a segment store: the + // key map's resident bytes, its data files, what is live and dead + // in them, and what the mover has scanned and copied so far. + ResidentIndexBytes uint64 + HeapFiles int + HeapLiveBytes uint64 + HeapDeadBytes uint64 + HeapScannedBytes uint64 + HeapMovedBytes uint64 } // storeCounters is StoreStats as the store keeps it: atomics, because From 5129bbe47cc9f9ffeaa0ebb76c0f0f353b60fa63 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 17:51:20 -0500 Subject: [PATCH 09/58] The proposal describes the dynamic layer as built Files of entries and a map of keys; the mover, its gate and its lock discipline with the measurements that set them; index generations; repair from the data; what the heap alone on the disk measures. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- .../2026-09-16-entries-written-once.md | 116 ++++++++++-------- 1 file changed, 68 insertions(+), 48 deletions(-) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index 53415ec..b8ee256 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -51,39 +51,52 @@ compaction, merge, pack -- operates on indexes, which are an order of magnitude smaller than what they index (a key and a location, ~40 B, against a ~300 B value) and can be rebuilt from the entries. -## The dynamic layer: an append-and-clean heap +## The dynamic layer: files of entries, a map of keys The dynamic layer holds a bounded key set rewritten forever. Today every rewrite appends and the old copy is garbage until a fold -rewrites everything around it, index and filter included. Instead: - -- **Entries are managed by appending.** An entry is - `[cap][len][key][value][checksum]` in a slot of a size class; a - block's entries are appended contiguously, so the block sync is one - sequential fsync of the heap per shard. (Filling holes wherever - they lie was built first and measured: a block's 11k rewrites - scattered over a 380 MB heap dirtied a page each, and the barrier - wrote 4x the ingest -- run 3, 443 MB/s against run 2's 117. The - commit path cannot afford scattered writes; only the cleaner can.) -- **A rewrite within the block reuses the slot.** A key written - again in the block that took its slot is rewritten in place when the - value fits: nothing durable names the slot yet. +rewrites everything around it, index and filter included. Instead +(`database/heap.go`, opened with `NewKVShardHeapN` / `NewKV2Heap`, +recognised on open by its directory; `bdbench -dyna-heap`): + +- **An entry is written once, at the end of a file.** An entry is + `[len][height][key][value][checksum]` at its exact aligned length; + a key is `(file, offset, length)`. Data files are fixed-size + (`HeapFileBytes`, 16 MB); a block appends to the current file, so + the block sync is one sequential fsync per file the block touched. + (Filling holes at put time was built first and measured: a block's + 11k rewrites scattered over a 380 MB heap dirtied a page each, and + the barrier wrote 4x the ingest -- run 3, 443 MB/s against run 2's + 117. Size classes were measured too: a third of the store wasted + for nothing, once nothing was allocated from free lists.) +- **A rewrite within the block reuses the slot** when the entry keeps + its aligned size: nothing durable names it yet, and the file stays a + contiguous sequence of entries a scan can walk. - **A rewrite in a later block appends.** The slot the last durable index names is never overwritten, or a crash between the write and the block's sync would expose an uncommitted value under a committed name. The old slot is dead where it lies. -- **A bounded cleaner makes the big hole.** On the maintenance - cadence, one pass scans up to `HeapCleanBytes` (16 MB) from the head - -- the oldest byte in use -- re-appends the entries still live, and - marks the region; the sync after the delta naming the copies - releases it (`fallocate` punch, size kept). A pass costs the live - fraction of the oldest region: small for a hot key set, and for a - cold one the price of a bounded move (1.2). The store reports bytes - scanned against bytes moved, which is the heap's write amplification. +- **The mover makes the space.** On the maintenance cadence a pass + takes the deadest files -- once half dead, or whatever their ratio + while dead bytes exceed live, which bounds the heap at twice its + live set -- copies their live entries into a file of its own, and + marks a file left with nothing live for deletion by the sync after + the delta naming the copies. The mover's file is not the block's, + so their barriers never share an inode (measured: the two fsyncs on + one file flushed each other's pages, seal p50 193 ms). The pass + holds the shard's lock only to pick, to plan and to name, never + across a read, a write or an fsync (measured: reading a 16 MB file + under the lock put the seal at 229 ms). A key rewritten while its + copy is in flight leaves the copy dead on arrival. The store + reports bytes scanned against bytes copied: the heap's write + amplification. - **The key map is in memory** for the live dynamic key set (the - soak's half million keys are ~24 MB per store; 1.2 allows memory - that scales with the working set). It is also what the seal makes - durable, below. + soak's half million keys are ~50 MB per store in Go; 1.2 allows + memory that scales with the working set). What the seal makes + durable is a delta of the keys the block touched; a snapshot starts + a new index generation in a file of its own on the maintenance + cadence, written aside, fsynced and renamed into place, so no delta + is ever truncated away. The layer's size converges to O(live keys) by construction (1.5), without the deeper fold 2.7 allows today, and nothing but entries is @@ -125,24 +138,31 @@ commit point" and closes #33. ## Durability and crash consistency (1.8) -- **A region is released one seal late.** The head advances past a - cleaned region only after the delta naming the cleaner's copies is - durable. Until then the durable index still names the old slots, - and a crash must find them intact. Release is deferred the way 2.6 - defers deletion. The adapter never - asks the store for an old version (its pre-images are memoized on - its side), so reuse waits on the seal and on nothing else. +- **A file is deleted one seal late.** A file emptied by the mover + is deleted only after the delta naming the mover's copies out of it + is durable. Until then the durable index still names its slots, + and a crash must find them intact: never unlink what a durable + index names. The adapter never asks the store for an old version + (its pre-images are memoized on its side), so deletion waits on the + seal and on nothing else. - **A torn slot is detected, not misread.** Every entry carries its - length and a checksum; a slot whose checksum fails after a crash is - a hole, and the durable index never points at one, because an index - entry is durable only after the slot it names is. -- **The in-memory key map is rebuilt on open** from the last durable - index snapshot plus the block files after it, the same replay 2.8 - does for manifests today. A snapshot is taken on the pack cadence - so the replay is bounded. -- **Nothing durable is ever overwritten by a different entry** except - a slot the durable index no longer names. 1.7's identity rule - holds for files: a data file is appended, never republished. + length and a checksum; an entry above the committed height is a + block that never synced. An index entry is durable only after the + slot it names is, so a torn slot is never named; open cuts the + files back to their last named slot and drops a torn delta whole. +- **The key map is rebuilt on open** from the newest whole index + generation: its snapshot, then its deltas. A generation whose + write was interrupted is not whole and is removed. +- **Repair reads the keys from the data.** With the index gone, a + scan of the data files rebuilds the map: every entry carries its + key and the height that wrote it, so the highest committed copy of + a key wins (later in the scan on a tie) and an entry above the + committed height, which only the store above knows, is dropped. A + child process killed mid-block, three times over, reopens and + repairs to exactly the last durable block (`heap_crash_test.go`). +- **Nothing durable is ever overwritten by a different entry.** 1.7's + identity rule holds for files: a data file is appended, never + republished. ## What it is measured with @@ -158,12 +178,12 @@ commit point" and closes #33. ## Order of work -1. The dynamic heap, behind the existing `KV2` dynamic surface (`Put`, - `Get`, `Seal`, `CompactHistory` becoming the bounded move, - `Stats`), so the sharding and the adapter do not change. The - platform measures it alone (`-stores 9 -perm 0`). *Written: - `database/heap.go`, opened with `NewKVShardHeapN` / `NewKV2Heap`, - detected on open by its directory; `bdbench -dyna-heap`.* +1. The dynamic heap, behind the existing `KV2` dynamic surface, so + the sharding and the adapter do not change. *Built.* Alone on + the disk with nine stores it holds the seal at ~54 ms p50 with no + compaction spikes, reads at 1-2 µs p99, and maintenance at a tenth + of the segment layer's; the segment layer alone had a compaction + storm in minute 4 (seal max 1.5 s, 58 blocks missed). 2. The permanent index deltas and the single block file, which also brings the seal to one commit point. 3. Merge and pack over indexes. From 4173fd47f9843104bccdc41459ed98e79631d102 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 17:52:40 -0500 Subject: [PATCH 10/58] The mover decodes picked files outside the lock Planning decoded every entry of each picked file under the shard's exclusive lock, checksums included: ~128 MB of CRC per pass, and the seal's p50 was 147 ms in the minute the mover worked. The read and the decode now happen together outside the lock; under it the pass only asks the index which entries are still named and reserves their copies. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 77 +++++++++++++++++++++++++++++++----------------- 1 file changed, 50 insertions(+), 27 deletions(-) diff --git a/database/heap.go b/database/heap.go index a003095..4aebfc6 100644 --- a/database/heap.go +++ b/database/heap.go @@ -808,13 +808,15 @@ func (h *HeapStore) clean(budget int64) (bool, error) { if len(taken) == 0 { return false, nil } - // 2. Without the lock: read them. A picked file is neither the - // block's nor the mover's, so its bytes do not change; only what - // the index says of them can, and that is checked under the lock - contents := make([][]byte, len(taken)) + // 2. Without the lock: read and decode them. A picked file is + // neither the block's nor the mover's, so its bytes do not change; + // only what the index says of them can, and that is checked under + // the lock. Decoding here, checksums included, keeps 16 MB of + // CRC per file off the lock + entries := make([][]heapEntry, len(taken)) for i, hf := range taken { - contents[i] = make([]byte, hf.size) - if _, err := hf.f.ReadAt(contents[i], 0); err != nil { + buf := make([]byte, hf.size) + if _, err := hf.f.ReadAt(buf, 0); err != nil { h.mu.Lock() for _, hf := range taken { hf.cleaning = false @@ -822,6 +824,8 @@ func (h *HeapStore) clean(budget int64) (bool, error) { h.mu.Unlock() return false, err } + entries[i] = decodeFile(buf) + h.cleanedBytes.Add(uint64(hf.size)) } // 3. Under the lock: decide what is live and reserve each copy's // slot in the mover's file; the mover's files leave the dirty set, @@ -830,7 +834,7 @@ func (h *HeapStore) clean(budget int64) (bool, error) { var moves []heapMove var copied int64 for i, hf := range taken { - m, n, err := h.planFile(hf, contents[i], budget-copied) + m, n, err := h.planFile(hf, entries[i], budget-copied) if err != nil { h.mu.Unlock() return false, err @@ -923,34 +927,53 @@ func (h *HeapStore) pickFile() *heapFile { return pick } -// planFile walks one file's bytes, read outside the lock, and -// reserves in the mover's file a slot for each entry the index still -// names, up to budget bytes; the rest wait for the next pass. The -// caller holds the lock. -func (h *HeapStore) planFile(hf *heapFile, buf []byte, budget int64) (moves []heapMove, copied int64, err error) { +// heapEntry is one entry of a picked file as decoded outside the +// lock: where it is, and its bytes ready to be copied. +type heapEntry struct { + off uint32 + size int64 + key [32]byte + value []byte +} + +// decodeFile walks a file's bytes into its entries, stopping at the +// first unwritten, torn or damaged slot. +func decodeFile(buf []byte) (entries []heapEntry) { var at int64 for at < int64(len(buf)) { size, _, key, value, ok := decodeEntry(buf[at:]) if !ok { break } - if s, live := h.index[key]; live && s.file == hf.id && int64(s.off) == at { - if copied >= budget { - break // The rest next pass - } - to, off, err := h.reserve(true, size) - if err != nil { - return nil, 0, err - } - ns := slot{file: to.id, off: uint32(off), n: uint32(len(value)), block: h.height} - // The copy is this block's write of the key: it carries this - // height, so a repair scan prefers it to the original - moves = append(moves, heapMove{key: key, from: s, to: ns, hf: to, entry: encodeEntry(h.height, key, value)}) - copied += size - } + entries = append(entries, heapEntry{off: uint32(at), size: size, key: key, value: value}) at += size } - h.cleanedBytes.Add(uint64(at)) + return entries +} + +// planFile reserves, in the mover's file, a slot for each entry of a +// picked file the index still names, up to budget bytes; the rest +// wait for the next pass. The caller holds the lock; the entries +// were decoded outside it. +func (h *HeapStore) planFile(hf *heapFile, entries []heapEntry, budget int64) (moves []heapMove, copied int64, err error) { + for _, e := range entries { + s, live := h.index[e.key] + if !live || s.file != hf.id || s.off != e.off { + continue + } + if copied >= budget { + break // The rest next pass + } + to, off, err := h.reserve(true, e.size) + if err != nil { + return nil, 0, err + } + ns := slot{file: to.id, off: uint32(off), n: uint32(len(e.value)), block: h.height} + // The copy is this block's write of the key: it carries this + // height, so a repair scan prefers it to the original + moves = append(moves, heapMove{key: e.key, from: s, to: ns, hf: to, entry: encodeEntry(h.height, e.key, e.value)}) + copied += e.size + } return moves, copied, nil } From 5519e5875265597ded3ef74a322be55781b10476 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:00:00 -0500 Subject: [PATCH 11/58] The mover plans a chunk of entries per lock hold The planning walk was the last hold: a map lookup per entry, up to a million entries per pass under the exclusive lock, ~100 ms at a time (seal p90 461 ms in the minute the size bound put the mover to work). The walk now takes the lock per 4,096 entries, and the copies' bytes are laid out outside it. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 47 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/database/heap.go b/database/heap.go index 4aebfc6..9207c3b 100644 --- a/database/heap.go +++ b/database/heap.go @@ -771,9 +771,14 @@ type heapMove struct { key [32]byte from, to slot hf *heapFile + height uint64 // The block the copy is written as + value []byte entry []byte } +// heapPlanChunk is how many entries a pass plans per lock hold. +const heapPlanChunk = 4096 + // clean is the mover: it takes the files with the most dead bytes -- // once HeapCleanRatio of each is dead, or the deadest whatever its // ratio while dead bytes exceed live -- copies their live entries to @@ -827,24 +832,35 @@ func (h *HeapStore) clean(budget int64) (bool, error) { entries[i] = decodeFile(buf) h.cleanedBytes.Add(uint64(hf.size)) } - // 3. Under the lock: decide what is live and reserve each copy's - // slot in the mover's file; the mover's files leave the dirty set, - // since the pass syncs them itself - h.mu.Lock() + // 3. Under the lock, a chunk of entries at a time: decide what is + // live and reserve each copy's slot in the mover's file. A pass + // walks up to a million entries; taking the lock per chunk keeps + // each hold to a millisecond or so. The mover's files leave the + // dirty set, since the pass syncs them itself var moves []heapMove var copied int64 for i, hf := range taken { - m, n, err := h.planFile(hf, entries[i], budget-copied) - if err != nil { + for at := 0; at < len(entries[i]) && copied < budget; at += heapPlanChunk { + end := at + heapPlanChunk + if end > len(entries[i]) { + end = len(entries[i]) + } + h.mu.Lock() + m, n, err := h.planFile(hf, entries[i][at:end], budget-copied) h.mu.Unlock() - return false, err - } - moves = append(moves, m...) - copied += n - if copied >= budget { - break + if err != nil { + h.mu.Lock() + for _, hf := range taken { + hf.cleaning = false + } + h.mu.Unlock() + return false, err + } + moves = append(moves, m...) + copied += n } } + h.mu.Lock() for _, hf := range taken { hf.cleaning = false } @@ -858,6 +874,11 @@ func (h *HeapStore) clean(budget int64) (bool, error) { delete(h.dirty, hf.id) } h.mu.Unlock() + // The copies' bytes are laid out outside the lock + for i := range moves { + m := &moves[i] + m.entry = encodeEntry(m.height, m.key, m.value) + } // 4. Without the lock: write the copies and make them durable fail := func(err error) (bool, error) { h.mu.Lock() @@ -971,7 +992,7 @@ func (h *HeapStore) planFile(hf *heapFile, entries []heapEntry, budget int64) (m ns := slot{file: to.id, off: uint32(off), n: uint32(len(e.value)), block: h.height} // The copy is this block's write of the key: it carries this // height, so a repair scan prefers it to the original - moves = append(moves, heapMove{key: e.key, from: s, to: ns, hf: to, entry: encodeEntry(h.height, e.key, e.value)}) + moves = append(moves, heapMove{key: e.key, from: s, to: ns, hf: to, height: h.height, value: e.value}) copied += e.size } return moves, copied, nil From 4d9d3d607af279bdb4eca8c8c6fec87b2bdd5d2d Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:04:02 -0500 Subject: [PATCH 12/58] The mover's file is never in a block sync's dirty set With planning unlocked between chunks a block sync could begin mid-pass and capture the mover's file as dirty, so the block fsynced the mover's 16 MB of copies: each shard sync covered 8.8 MB instead of 0.7, and the seal's p50 was 193 ms in the first minute (run 6). A mover reservation now never marks its file dirty; the pass syncs it itself. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/database/heap.go b/database/heap.go index 9207c3b..f173edf 100644 --- a/database/heap.go +++ b/database/heap.go @@ -533,9 +533,12 @@ func (h *HeapStore) reserve(mover bool, size int64) (hf *heapFile, off int64, er hf.size += size hf.live += size h.liveBytes += size - h.dirty[hf.id] = hf if mover { + // The pass syncs its own file; the block's sync must never + // capture it as dirty, or the block waits for the copies hf.inflight++ + } else { + h.dirty[hf.id] = hf } return hf, off, nil } @@ -835,8 +838,8 @@ func (h *HeapStore) clean(budget int64) (bool, error) { // 3. Under the lock, a chunk of entries at a time: decide what is // live and reserve each copy's slot in the mover's file. A pass // walks up to a million entries; taking the lock per chunk keeps - // each hold to a millisecond or so. The mover's files leave the - // dirty set, since the pass syncs them itself + // each hold to a millisecond or so. A block sync may begin + // between chunks; the mover's file is never in its dirty set var moves []heapMove var copied int64 for i, hf := range taken { @@ -864,16 +867,13 @@ func (h *HeapStore) clean(budget int64) (bool, error) { for _, hf := range taken { hf.cleaning = false } + h.mu.Unlock() var movFiles []*heapFile for _, m := range moves { if len(movFiles) == 0 || movFiles[len(movFiles)-1] != m.hf { movFiles = append(movFiles, m.hf) } } - for _, hf := range movFiles { - delete(h.dirty, hf.id) - } - h.mu.Unlock() // The copies' bytes are laid out outside the lock for i := range moves { m := &moves[i] From 4f83d61f038287b5fbdcb8583a6cf7a66513390e Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:09:38 -0500 Subject: [PATCH 13/58] Smaller mover passes: 4 MB copied, four files With every lock hold gone from the pass, the seal's tail follows the mover's own fsync volume in the device queue: 16 MB passes put p90 at 150-250 ms in the minutes they ran (run 7). 4 MB per pass releases more than the soak appends per shard per cadence. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/database/heap.go b/database/heap.go index f173edf..79be960 100644 --- a/database/heap.go +++ b/database/heap.go @@ -128,13 +128,16 @@ const ( // HeapFileBytes is the size a data file is rolled at. var HeapFileBytes int64 = 16 << 20 -// HeapCleanBytes bounds one mover pass by the bytes it copies; the -// pass syncs its own copies, so the bound is about the pass's length, -// not a block's barrier. -var HeapCleanBytes int64 = 16 << 20 +// HeapCleanBytes bounds one mover pass by the bytes it copies. The +// pass syncs its own copies, so the bound is not about a block's +// barrier but about the device queue the barrier shares: 16 MB +// passes put the seal's p90 at 150-250 ms in the minutes they ran +// (run 7), 4 MB releases more than the soak appends per shard per +// cadence (~4 MB, of which a quarter to a third is live). +var HeapCleanBytes int64 = 4 << 20 // HeapCleanFiles bounds a pass by files taken as well. -var HeapCleanFiles = 8 +var HeapCleanFiles = 4 // HeapCleanRatio is the dead fraction a file must reach before the // mover takes it -- unless dead bytes exceed live bytes overall, when From 82c33bba1894cc6b170106757ace938df26dceaa Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:10:44 -0500 Subject: [PATCH 14/58] The proposal carries run 7's numbers Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- docs/proposals/2026-09-16-entries-written-once.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index b8ee256..8bb8c81 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -179,11 +179,15 @@ commit point" and closes #33. ## Order of work 1. The dynamic heap, behind the existing `KV2` dynamic surface, so - the sharding and the adapter do not change. *Built.* Alone on - the disk with nine stores it holds the seal at ~54 ms p50 with no - compaction spikes, reads at 1-2 µs p99, and maintenance at a tenth - of the segment layer's; the segment layer alone had a compaction - storm in minute 4 (seal max 1.5 s, 58 blocks missed). + the sharding and the adapter do not change. *Built (PR #97).* + Alone on the disk with nine stores (run 7): seal p50 53-61 ms in + every minute, read p99 1-2 µs, maintenance ~10 s a minute, store + 3.8 GB at five minutes, zero wrong answers; the segment layer alone + seals at 33-38 ms p50 but reads at 11-13 µs, spends 22-110 s a + minute compacting, and had a compaction storm in minute 4 (seal + max 1.5 s, 58 blocks missed). The heap's remaining tail (p90 + 150-250 ms in the minutes the mover copies most) is the mover's + own fsync volume in the device queue, which the pass size paces. 2. The permanent index deltas and the single block file, which also brings the seal to one commit point. 3. Merge and pack over indexes. From 01b97fe0bfcab9a5f152f7e2b4ccf0bf24ebd7ad Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:15:29 -0500 Subject: [PATCH 15/58] The proposal specifies the permanent layer and the seal from the code as it is From a read of seal.go, segstore.go, blockset.go and indexmerge.go: what a block costs today (four barriers per shard, two per store), what a merge copies (bodies, byte-verbatim, because a 48-byte index record carries no file), and the replacement: files of records never moved, a 44-byte index record with the file in it, deltas per block behind the live filters, merge and pack over indexes only, and the block's deltas appended to the data files so a block is one barrier round per shard and one per store. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- .../2026-09-16-entries-written-once.md | 100 ++++++++++++------ 1 file changed, 70 insertions(+), 30 deletions(-) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index 8bb8c81..afd8ce6 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -105,36 +105,70 @@ ever copied. ## The permanent layer: append-only data, merged indexes The permanent layer is write-once; it has no garbage. Its merges and -packs exist to bound the file count and the length of a history walk -(2.7). Both are properties of the *index*, so: - -- **A shard's data is one growing file** (or one per `PackEvery` - blocks). A block's records are appended; nothing is ever copied. -- **Each block seals an index delta**: the sorted `(key, offset)` pairs - of the records the block appended, with its filter (2.5). -- **Merge and pack fold index deltas** into one sorted index per shard - per level, on the adapter's cadence, off the protocol path, exactly - as today's tiers do -- but moving ~40 B per record instead of ~300. - The walk length and the file count are bounded by the index tiers; - the data file is never in the walk (a key resolves to an offset, - then one `pread`). +packs exist to bound the file count and the length of a lookup's walk +(2.7), and both are properties of the *index*. Today (`segstore.go`, +`seal.go`, `blockset.go`) each block seals a segment -- a body file and +an index file with an embedded filter -- and `MergeBelow` folds +segments with `concatSegments`, which copies every body byte-verbatim +into a new body and rebuilds the index over it; `PackFinalized` copies +the bodies again into a cross-shard set file. An index record is 48 +bytes, `key, offset, length`, with no file identifier: several bodies +are only ever addressed through an out-of-band base, which is why the +bodies have to be concatenated to be merged. Instead: + +- **A shard's data is a sequence of fixed-size files, appended and + never moved**, the heap's file model (`HeapFileBytes`): a block's + records go to the end of the current file, and a record is + `(file, offset, length)`. Nothing is ever copied; a file is deleted + only by `DropBelow` when a pack no longer needs it -- and packs no + longer copy either, so a data file lives until the retention policy + above the store says otherwise. +- **An index record carries the file.** `key(32) file(4) offset(4) + length(4)`, 44 bytes, sorted by key as today. `mergeIndexes` and + `indexWriter` already merge indexes without touching a body; with + the file in the record they need no bases. +- **Each block seals an index delta** with its filter: the sorted + records of what the block appended, ~40 bytes a record. The delta + is what the window is made of; the live key filters (`keyfilter.go`) + are built over deltas exactly as they are built over segments now. +- **Merge and pack fold deltas into one sorted index per level**, on + the adapter's cadence, off the protocol path, with one filter over + the level: a k-way merge of 44-byte records instead of a copy of + ~300-byte bodies. The walk length and the file count are bounded by + the index levels; a lookup resolves a key to `(file, offset, + length)` and does one `pread` of the data file. A set is a merged + index over every shard's level, grouped by block range as today, + with no bodies in it. +- **Reads keep their protocol-path rule** (1.3): the window is the + last N blocks' deltas behind the live filters; an immutable key the + filters deny is absent; history and sets are reached only by + `GetDeep`. ## The seal: one commit point per store per block -With indexes as the sealed object a block is one barrier round per -store rather than four per shard: - -1. Every shard's data appends (permanent) and slot writes (dynamic) - for the block are fsynced, in parallel: one round. -2. The block's index deltas -- permanent `(key, offset)` and dynamic - `(key, location)` for every key the block touched -- are written to - one per-store block file and fsynced: one round. -3. The store's manifest names the block file: the existing commit, - one round. - -Three rounds per store per block, none per shard beyond the parallel -data sync, in place of four rounds per shard. This is 1.8's "one -commit point" and closes #33. +Today a non-empty block costs each shard four barriers -- the data +fsync, the index file's, the manifest's temp file, the directory -- +and the store two more for its block record (`seal.go`, +`commitJSON`). Nine stores of eight shards ask the device for about +360 barriers a second before any maintenance runs, and the platform +measured every seal goroutine waiting in `fsync` (#94). + +With deltas as the sealed object, a block's permanent index delta is +appended to the same data file right after the block's records, and +the dynamic layer's delta after its entries the same way, so that: + +1. Every shard fsyncs its data files once: entries and delta together, + in parallel across shards -- one round. +2. The store's block record names, for every shard, the file and + offset of the block's deltas: the existing `block.json` commit, + temp file and directory -- one round. + +Two rounds per store per block, in place of four per shard plus two. +Recovery trusts a delta only if its own checksum holds and every entry +it names checks, since one fsync does not order the delta's bytes +after the entries'; a delta that fails is the block that did not +commit, exactly as a torn delta is today. This is 1.8's "one commit +point" and closes #33. ## Durability and crash consistency (1.8) @@ -188,6 +222,12 @@ commit point" and closes #33. max 1.5 s, 58 blocks missed). The heap's remaining tail (p90 150-250 ms in the minutes the mover copies most) is the mover's own fsync volume in the device queue, which the pass size paces. -2. The permanent index deltas and the single block file, which also - brings the seal to one commit point. -3. Merge and pack over indexes. +2. The permanent layer as files of records with index deltas + (`PermStore`, behind `KV2`'s permanent surface: `PutIfAbsent`, + windowed `Get`, `GetDeep`, the two-half seal, `MergeBelow`, + `historyBelow`, `DropBelow`, `attachCold`, the filter knobs), the + 44-byte index record, and merge and pack over indexes. Measured + alone first (`-stores 9 -dyna 0`), then with the heap under the + full load, which is the acceptance run. +3. The block's deltas in the data files and the store-level commit: + one barrier round per shard, one per store. From f9188cc29bbe309443417c7f182b7dff428d9324 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:19:42 -0500 Subject: [PATCH 16/58] Step 2: the long search is bucketed and merged a bucket per block PermBuckets buckets of sorted runs with a filter each; every block the next bucket in rotation takes its records from the deltas since it was last merged as a new run, and runs fold by ratio, so the work a block does is a fixed slice proportional to what arrived and never a rewrite of the shard's index. A merge locks one bucket. The window's deltas behind the live filters stay the quick search. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- .../2026-09-16-entries-written-once.md | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index afd8ce6..f6dc2f6 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -131,18 +131,27 @@ bodies have to be concatenated to be merged. Instead: records of what the block appended, ~40 bytes a record. The delta is what the window is made of; the live key filters (`keyfilter.go`) are built over deltas exactly as they are built over segments now. -- **Merge and pack fold deltas into one sorted index per level**, on - the adapter's cadence, off the protocol path, with one filter over - the level: a k-way merge of 44-byte records instead of a copy of - ~300-byte bodies. The walk length and the file count are bounded by - the index levels; a lookup resolves a key to `(file, offset, - length)` and does one `pread` of the data file. A set is a merged - index over every shard's level, grouped by block range as today, - with no bodies in it. +- **The long search is bucketed, and merged a bucket at a time.** A + shard's index below the window is `PermBuckets` buckets (256 by + default; nothing about the number is special, and a modulus serves + as well as a mask), each holding a few sorted runs of 44-byte + records with a filter per run. Every block the maintenance step + takes the next bucket in rotation, gathers that bucket's records + from the deltas sealed since the bucket was last merged, writes them + as a new run, and folds the bucket's runs by ratio; a delta older + than every bucket's last merge is dropped. So the work every block + is a fixed slice, proportional to what arrived for one bucket, and + the big fold is rare and bounded per pass -- never a rewrite of the + shard's whole index, which would grow with the chain (1.2). A + merge swaps one bucket's runs under that bucket's lock; the other + buckets keep answering. A pack is the same fold across shards for + a block range, with no bodies in it. - **Reads keep their protocol-path rule** (1.3): the window is the last N blocks' deltas behind the live filters; an immutable key the - filters deny is absent; history and sets are reached only by - `GetDeep`. + filters deny is absent. Below the window a key's bucket is probed + newest run first, one filter and one binary search per run, and + that is `GetDeep`'s walk; a lookup resolves a key to `(file, + offset, length)` and does one `pread` of the data file. ## The seal: one commit point per store per block @@ -226,7 +235,8 @@ point" and closes #33. (`PermStore`, behind `KV2`'s permanent surface: `PutIfAbsent`, windowed `Get`, `GetDeep`, the two-half seal, `MergeBelow`, `historyBelow`, `DropBelow`, `attachCold`, the filter knobs), the - 44-byte index record, and merge and pack over indexes. Measured + 44-byte index record, the bucketed long search merged a bucket per + block, and packs over indexes. Measured alone first (`-stores 9 -dyna 0`), then with the heap under the full load, which is the acceptance run. 3. The block's deltas in the data files and the store-level commit: From fadc929ca5eabb62b0a11bbfb98d085200cd442e Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:26:02 -0500 Subject: [PATCH 17/58] Size the permanent index's buckets by simulation TestPermSizingSim models one shard at the soak's rate over a day, a week and a month against the store as it is. With any fixed bucket count the largest fold is 1/B of the index and grows with the chain; a bucket that splits at 32 MB bounds the fold at ~35 MB whatever the age, and the count follows the index (256 at a day, 1,024 at a week, 4,096 at a month). Maintenance writes fall four to six times and none is a body; the pack's 358 MB copy is gone. Ratio 0.25 stays; every bucket is merged every 256 blocks; filters live within a budget and older runs are probed cold. The proposal carries the table. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm_sizing_sim_test.go | 278 ++++++++++++++++++ .../2026-09-16-entries-written-once.md | 40 +++ 2 files changed, 318 insertions(+) create mode 100644 database/perm_sizing_sim_test.go diff --git a/database/perm_sizing_sim_test.go b/database/perm_sizing_sim_test.go new file mode 100644 index 0000000..b8d99b5 --- /dev/null +++ b/database/perm_sizing_sim_test.go @@ -0,0 +1,278 @@ +package blockchainDB + +import ( + "flag" + "fmt" + "sort" + "strings" + "testing" +) + +var simFlag = flag.Bool("sim", false, "run the permanent-index sizing simulation (TestPermSizingSim)") + +// A sizing simulation for the permanent layer's bucketed long search +// (docs/proposals/2026-09-16-entries-written-once.md, step 2). Not a +// test of the store: it models the index maintenance the proposal +// describes and prints, for a sweep of bucket counts, what a shard +// pays per block and what a lookup walks. Run with +// +// go test ./database/ -run TestPermSizingSim -v -sim +// +// The model, per shard: every block appends K records and seals a +// delta of K sorted 44-byte index records with a filter. A delta +// leaving the window (N blocks) feeds each record to its bucket's +// resident recent section. Every block the next bucket in rotation +// writes its recent section as a new sorted run and folds its runs by +// the store's ratio rule (compactionRunWithin: a suffix of runs is +// folded while each older run is no larger than 1/ratio of what has +// gathered behind it). Today's cost, for comparison: every +// MergeEvery blocks the shard folds its finished segments by copying +// bodies (~V bytes a record) and rebuilding the index, under the same +// ratio rule over merged segments; every PackEvery blocks the merged +// segments are copied again into a set. +func TestPermSizingSim(t *testing.T) { + if !*simFlag { + t.Skip("a sizing simulation; run with -sim") + } + const ( + K = 1040 // Permanent records per block per shard (soak: 8.3k per store, 8 shards) + V = 300 // Bytes a record's body costs to copy + rec = 44 // Bytes an index record costs + N = 20 // The window, blocks + ratio = 0.25 + mergeEvery = 20 + packEvery = 1000 + bloomBits = 12 // Bits per key in a filter + ) + horizons := []int{86_400, 7 * 86_400, 30 * 86_400} + buckets := []int{1, 64, 256, 1024, 4096} + splits := []int64{32 << 20, 64 << 20} // Bucket index bytes at which a bucket splits in two + const ( + M = 256 // Every bucket is merged every M blocks: B/M buckets a block + residentBudget = 64 << 20 // Filter bytes resident per shard, newest runs first; the rest are probed cold + runFile = 64 << 20 // Runs are appended to run files of this size; a run is (file, offset, length) + ) + + var out strings.Builder + fmt.Fprintf(&out, "\nPermanent index, one shard, K=%d records/block, ratio %.2f, window %d blocks\n", K, ratio, N) + for _, blocks := range horizons { + fmt.Fprintf(&out, "\n== %d blocks (%.0f days), %d M records ==\n", blocks, float64(blocks)/86_400, K*blocks/1_000_000) + // Today + today := simToday(blocks, K, V, rec, ratio, mergeEvery, packEvery) + fmt.Fprintf(&out, "today: index+body bytes written %.1f GB (%.1f MB/block avg), largest pass %.0f MB, merged segments at end %d, sets %d\n", + float64(today.written)/1e9, float64(today.written)/float64(blocks)/1e6, float64(today.largest)/1e6, today.segments, today.sets) + fmt.Fprintf(&out, "%-22s %12s %11s %11s %11s %9s %11s %11s %9s\n", "buckets", "idx KB/blk", "largest MB", "runs/bkt", "probes", "files", "resident MB", "cold probes", "recent MB") + for _, b := range buckets { + r := simBuckets(blocks, K, rec, N, ratio, b, 0, M, bloomBits, residentBudget, runFile) + fmt.Fprintf(&out, "%-22d %12.0f %11.1f %11.1f %11.1f %9d %11.1f %11.1f %9.1f\n", + b, float64(r.written)/float64(blocks)/1e3, float64(r.largest)/1e6, r.runsAvg, r.probes, r.files, float64(r.resident)/1e6, r.coldProbes, float64(r.recent)/1e6) + } + for _, sp := range splits { + for _, rt := range []float64{ratio, 0.1} { + r := simBuckets(blocks, K, rec, N, rt, 256, sp, M, bloomBits, residentBudget, runFile) + fmt.Fprintf(&out, "%-22s %12.0f %11.1f %11.1f %11.1f %9d %11.1f %11.1f %9.1f -> %d buckets\n", + fmt.Sprintf("256 split@%dMB r=%.2f", sp>>20, rt), float64(r.written)/float64(blocks)/1e3, float64(r.largest)/1e6, r.runsAvg, r.probes, r.files, float64(r.resident)/1e6, r.coldProbes, float64(r.recent)/1e6, r.buckets) + } + } + } + fmt.Fprintf(&out, "\nidx KB/blk: index bytes written per block, runs and folds; largest: the biggest single fold (or split);\n") + fmt.Fprintf(&out, "runs/bkt: sorted runs a lookup below the window probes (one filter each); probes: for a key not in the\n") + fmt.Fprintf(&out, "shard, window filter + recent section + runs, of which cold probes are on disk (K byte reads each);\n") + fmt.Fprintf(&out, "files: run files of %d MB; resident: filters within a %d MB budget, newest runs first; recent: the\n", runFile>>20, residentBudget>>20) + fmt.Fprintf(&out, "recent sections in memory (every bucket merged every %d blocks); split@S: a bucket whose runs exceed S\n", M) + fmt.Fprintf(&out, "splits in two; r: the fold ratio (a run folds into the older while the older is no larger than 1/r of it).\n") + t.Log(out.String()) +} + +type simResult struct { + written, largest int64 + runsAvg, probes float64 + coldProbes float64 + files, buckets int + resident, recent int64 +} + +// simBuckets models the bucketed long search. split > 0 splits a +// bucket in two once its runs hold more than split bytes (each half +// keeps half of every run: the keys are hashed, so a split is a +// rewrite of the bucket, counted as a fold). B/M buckets are merged +// each block so every bucket is merged every M blocks. +func simBuckets(blocks, K, rec, N int, ratio float64, B int, split int64, M, bloomBits int, residentBudget, runFile int64) simResult { + type bucket struct { + recent int64 // Records in the resident recent section + runs []int64 // Records per run, oldest first + } + bs := make([]bucket, B) + var r simResult + rotation := 0 + fold := func(b *bucket) { + if len(b.runs) < 2 { + return + } + var behind int64 + i := len(b.runs) - 1 + for ; i >= 0; i-- { + if i < len(b.runs)-1 && float64(b.runs[i])*ratio > float64(behind) { + break + } + behind += b.runs[i] + } + run := b.runs[i+1:] + if len(run) >= 2 { + var total int64 + for _, n := range run { + total += n + } + r.written += total * int64(rec) + if total*int64(rec) > r.largest { + r.largest = total * int64(rec) + } + b.runs = append(b.runs[:i+1], total) + } + } + for blk := 0; blk < blocks; blk++ { + if blk >= N { + perBucket := int64(K) / int64(len(bs)) + extra := int64(K) - perBucket*int64(len(bs)) + for i := range bs { + bs[i].recent += perBucket + } + for i := int64(0); i < extra; i++ { + bs[(blk*7+int(i))%len(bs)].recent++ + } + } + visits := (len(bs) + M - 1) / M + for v := 0; v < visits; v++ { + b := &bs[rotation%len(bs)] + rotation++ + if split > 0 { + var total int64 + for _, n := range b.runs { + total += n + } + if total*int64(rec) > split { + r.written += total * int64(rec) + if total*int64(rec) > r.largest { + r.largest = total * int64(rec) + } + half := bucket{recent: b.recent / 2} + b.recent -= half.recent + for i, n := range b.runs { + b.runs[i] = n / 2 + half.runs = append(half.runs, n-n/2) + } + bs = append(bs, half) + b = &bs[(rotation-1)%len(bs)] + } + } + if b.recent == 0 { + continue + } + r.written += b.recent * int64(rec) + b.runs = append(b.runs, b.recent) + b.recent = 0 + fold(b) + } + } + // Filters: newest runs first across the shard, within the budget + var runs, cold int + type runRef struct{ n int64 } + var newest, older []int64 // Per bucket: the newest run's records, then the rest + for _, b := range bs { + runs += len(b.runs) + r.recent += b.recent * int64(rec) + for i, n := range b.runs { + if i == len(b.runs)-1 { + newest = append(newest, n) + } else { + older = append(older, n) + } + } + } + budget := residentBudget + for _, n := range append(newest, older...) { + bytes := n * int64(bloomBits) / 8 + if budget >= bytes { + budget -= bytes + r.resident += bytes + } else { + cold++ + } + } + r.resident += r.recent + var indexBytes int64 + for _, b := range bs { + for _, n := range b.runs { + indexBytes += n * int64(rec) + } + } + r.files = int((indexBytes + runFile - 1) / runFile) + r.buckets = len(bs) + r.runsAvg = float64(runs) / float64(len(bs)) + r.coldProbes = float64(cold) / float64(len(bs)) + r.probes = 2 + r.runsAvg + _ = runRef{} + return r +} + +type simTodayResult struct { + written, largest int64 + segments, sets int +} + +// simToday models the store as it is: segments folded by copying +// bodies every mergeEvery blocks under the ratio rule, and packed +// every packEvery blocks by copying again. +func simToday(blocks, K, V, rec int, ratio float64, mergeEvery, packEvery int) simTodayResult { + perRecord := int64(V + rec) + var segs []int64 // Records per merged segment, oldest first + var r simTodayResult + pending := 0 // Finished blocks not yet merged + for blk := 1; blk <= blocks; blk++ { + pending++ + if blk%mergeEvery == 0 { + // The finished blocks become one merged segment, copied + n := int64(pending * K) + r.written += n * perRecord + segs = append(segs, n) + pending = 0 + var behind int64 + i := len(segs) - 1 + for ; i >= 0; i-- { + if i < len(segs)-1 && float64(segs[i])*ratio > float64(behind) { + break + } + behind += segs[i] + } + run := segs[i+1:] + if len(run) >= 2 { + var total int64 + for _, n := range run { + total += n + } + r.written += total * perRecord + if total*perRecord > r.largest { + r.largest = total * perRecord + } + segs = append(segs[:i+1], total) + } + } + if blk%packEvery == 0 { + // Everything merged so far below the watermark is copied + // into a set and dropped from the shard + var total int64 + for _, n := range segs { + total += n + } + r.written += total * perRecord + if total*perRecord > r.largest { + r.largest = total * perRecord + } + segs = segs[:0] + r.sets++ + } + } + r.segments = len(segs) + sort.Slice(segs, func(i, j int) bool { return segs[i] < segs[j] }) + return r +} diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index f6dc2f6..b8b6742 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -153,6 +153,46 @@ bodies have to be concatenated to be merged. Instead: that is `GetDeep`'s walk; a lookup resolves a key to `(file, offset, length)` and does one `pread` of the data file. +### Sizing the buckets (simulated) + +`TestPermSizingSim` (`go test ./database/ -run TestPermSizingSim -v +-args -sim`) models one shard at the soak's rate, 1,040 permanent +records a block, over a day, a week and a month, against the store as +it is (segments folded by copying bodies every 20 blocks under the +ratio rule, packed every 1,000). What it shows: + +| per shard | today | 256 buckets, fixed | 256 buckets, split at 32 MB | +|---|---|---|---| +| bytes written per block for maintenance | 2.5 MB (bodies and index) | 0.42-0.67 MB (index only) | 0.42-0.66 MB | +| largest single pass, day / week / month | 358 MB (a pack) | 14 / 102 / 460 MB | 14 / 35 / 35 MB | +| buckets, day / week / month | | 256 | 256 / 1,024 / 4,096 | +| runs a lookup below the window probes | | 3.4-3.9 | 3.4-4.3 | +| files (runs share 64 MB run files), month | | 1,768 | 1,768 | + +- **The number of buckets is not a constant.** With any fixed count + the largest fold is 1/B of the shard's index and grows with the + chain; a bucket that splits at a size bounds the fold at that size + whatever the age (1.2, bounded per pass), and the count follows the + index: about the index's bytes over the split size. Start at 256 a + shard; split at 32 MB, so a fold is never more than ~35 MB of + 44-byte records, off the protocol path. +- **Maintenance writes drop four to six times**, and every one of + them is an index byte: no body is ever copied. The pack's 358 MB + copy is gone. +- **The fold ratio stays 0.25.** A ratio of 0.1 halves the runs a + lookup probes but nearly doubles the bytes written. +- **Every bucket is merged every 256 blocks** (B/256 buckets a block), + which keeps the recent sections at ~6 MB a shard; one bucket a block + would leave records waiting hours once the count grows. +- **Filters are budgeted, not all resident.** Within a 64 MB budget + per shard the newest runs' filters are in memory and a miss below + the window costs 5-6 probes of which 2-3 are cold (K one-byte reads + each, as the store probes cold filters today); the window itself is + settled by the live filters in memory. The budget is the store's + to set (1.2: memory follows the working set). +- **A merge locks one bucket**, 1/B of the index; the rest keep + answering. + ## The seal: one commit point per store per block Today a non-empty block costs each shard four barriers -- the data From e1b6ca66a514210957b6913232094eb2aa20c27b Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:28:37 -0500 Subject: [PATCH 18/58] The buckets are bounded by the pack watermark, not the chain A bucket never drained holds 1/B of the whole chain, which is what the simulation's growing fold was. The store's pack watermark already drains history every 1,000 blocks; the buckets cover only what is above it, hashed keys make their shares even, so a bucket is ~4,000 records and nothing needs to split. About 180 KB of index a block per shard against today's 2.5 MB of bodies and index. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- .../2026-09-16-entries-written-once.md | 67 +++++++++---------- 1 file changed, 31 insertions(+), 36 deletions(-) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index b8b6742..dde323d 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -153,45 +153,40 @@ bodies have to be concatenated to be merged. Instead: that is `GetDeep`'s walk; a lookup resolves a key to `(file, offset, length)` and does one `pread` of the data file. -### Sizing the buckets (simulated) +### Sizing the buckets `TestPermSizingSim` (`go test ./database/ -run TestPermSizingSim -v -args -sim`) models one shard at the soak's rate, 1,040 permanent -records a block, over a day, a week and a month, against the store as -it is (segments folded by copying bodies every 20 blocks under the -ratio rule, packed every 1,000). What it shows: - -| per shard | today | 256 buckets, fixed | 256 buckets, split at 32 MB | -|---|---|---|---| -| bytes written per block for maintenance | 2.5 MB (bodies and index) | 0.42-0.67 MB (index only) | 0.42-0.66 MB | -| largest single pass, day / week / month | 358 MB (a pack) | 14 / 102 / 460 MB | 14 / 35 / 35 MB | -| buckets, day / week / month | | 256 | 256 / 1,024 / 4,096 | -| runs a lookup below the window probes | | 3.4-3.9 | 3.4-4.3 | -| files (runs share 64 MB run files), month | | 1,768 | 1,768 | - -- **The number of buckets is not a constant.** With any fixed count - the largest fold is 1/B of the shard's index and grows with the - chain; a bucket that splits at a size bounds the fold at that size - whatever the age (1.2, bounded per pass), and the count follows the - index: about the index's bytes over the split size. Start at 256 a - shard; split at 32 MB, so a fold is never more than ~35 MB of - 44-byte records, off the protocol path. -- **Maintenance writes drop four to six times**, and every one of - them is an index byte: no body is ever copied. The pack's 358 MB - copy is gone. -- **The fold ratio stays 0.25.** A ratio of 0.1 halves the runs a - lookup probes but nearly doubles the bytes written. -- **Every bucket is merged every 256 blocks** (B/256 buckets a block), - which keeps the recent sections at ~6 MB a shard; one bucket a block - would leave records waiting hours once the count grows. -- **Filters are budgeted, not all resident.** Within a 64 MB budget - per shard the newest runs' filters are in memory and a miss below - the window costs 5-6 probes of which 2-3 are cold (K one-byte reads - each, as the store probes cold filters today); the window itself is - settled by the live filters in memory. The budget is the store's - to set (1.2: memory follows the working set). -- **A merge locks one bucket**, 1/B of the index; the rest keep - answering. +records a block, against the store as it is (segments folded by +copying bodies every 20 blocks under the ratio rule, packed every +1,000 blocks). Its first result was a warning: a bucket that is +never drained holds 1/B of the whole chain, so with any fixed count +the largest fold grows with the age of the store (256 buckets: 14 MB +at a day, 102 at a week, 460 at a month), against 1.2. + +The bound is the pack watermark, which the store already has. Every +`PackEvery` (1,000) blocks the finished history is packed and dropped +from the shard; the buckets cover only the history above the +watermark, and since the keys are hashes their shares are even. A +bucket then holds at most 1,000 blocks of its share -- ~4,000 +records, ~180 KB, at 256 buckets -- so a fold is bounded by the pack +period, not the chain, and nothing needs to split. The count is a +free choice; 256 keeps runs small and a merge's lock small. + +Per shard per block that comes to about: the block's delta (46 KB), +the bucket runs (46 KB), their folds (~46 KB) and the pack amortized +(46 MB of index every 1,000 blocks, 46 KB): ~180 KB a block against +today's 2.5 MB, every byte an index byte. The pack is index-only, +46 MB a shard in place of 358 MB of bodies, and the chain's growth +lives where it lives today: in the sets below the watermark, grouped +by block range with one filter per finished group, reached only by +`GetDeep`. Filters for the buckets and the newest sets stay +resident within a budget; older groups are probed cold, as now. + +Ratio 0.25 stays (0.1 halves the runs a lookup probes but nearly +doubles the bytes written); every bucket is merged every 256 blocks +(B/256 buckets a block), which keeps the recent sections at a few MB +a shard. ## The seal: one commit point per store per block From c7f440c66a2a9708b988631b0b0959162d2c196f Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:33:08 -0500 Subject: [PATCH 19/58] bdbench: a per-store phase on the maintenance cadence Nine stores started together maintain in lockstep, every 20 blocks at the same block, and their movers' passes hit the device queue at once (the 30-minute heap run: seal p90 241 ms in the minute they coincide, 66 ms otherwise). Validators on one machine are in lockstep by consensus, so the soak has the same property. -maintenance-phase offsets each store's cadence by its share of the period; the adapter would derive the same offset from the node. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/main.go | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index 9dc391b..3244551 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -65,6 +65,7 @@ type config struct { pprof string http string dynaHeap bool + phase bool } //go:embed live.html @@ -93,6 +94,7 @@ func parseFlags() (config, error) { flag.StringVar(&c.pprof, "pprof", "", "serve net/http/pprof on this address (e.g. 127.0.0.1:6061)") flag.StringVar(&c.http, "http", "127.0.0.1:8098", "serve the live page and the run's files here; empty disables") flag.BoolVar(&c.dynaHeap, "dyna-heap", false, "dynamic layer as a heap with holes (proposal 2026-09-16) instead of sealed segments") + flag.BoolVar(&c.phase, "maintenance-phase", false, "offset each store's maintenance cadence by its share of the period, so stores in lockstep do not all maintain at once") flag.Parse() if flag.NArg() != 0 { return c, fmt.Errorf("unexpected arguments: %q", flag.Args()) @@ -204,6 +206,7 @@ type store struct { // answers cannot tell a fast wrong answer from a fast right one. last map[[32]byte][]byte height uint64 + phase uint64 // Blocks this store's maintenance cadence is offset by maintaining atomic.Bool maintWG sync.WaitGroup } @@ -233,6 +236,9 @@ func openStore(c config, id int) (*store, error) { } s := &store{id: id, kv: kv, rnd: blockchainDB.NewFastRandom(seed), hot: make([][32]byte, c.hotKeys), permKeys: make([][32]byte, 0, permSample), last: make(map[[32]byte][]byte, checked)} + if c.phase && c.compressEvery > 0 { + s.phase = uint64(id) * c.compressEvery / uint64(c.stores) + } // Hot dynamic keys are rewritten with a skew (index = hot * r^2, so // the low indexes take most writes); permanent keys are always new, // and a bounded sample of them, across all ages, is what the @@ -313,7 +319,7 @@ func (s *store) block(c config, t *tallies) error { } else { time.Sleep(c.interval - took) } - if c.compressEvery > 0 && s.height%c.compressEvery == 0 { + if c.compressEvery > 0 && (s.height+s.phase)%c.compressEvery == 0 { s.maintain(c, t) } return nil @@ -498,7 +504,7 @@ func main() { "dir": c.dir, "stores": c.stores, "duration": c.duration.String(), "interval": c.interval.String(), "shards": c.shards, "sealLimit": c.sealLimit, "window": c.window, "compressEvery": c.compressEvery, "packEvery": c.packEvery, "dynaPuts": c.dynaPuts, "permPuts": c.permPuts, "reads": c.reads, "hotKeys": c.hotKeys, - "valueMin": c.valueMin, "valueMax": c.valueMax, "seed": c.seed, "dynaHeap": c.dynaHeap, "started": time.Now().UTC().Format(time.RFC3339), + "valueMin": c.valueMin, "valueMax": c.valueMax, "seed": c.seed, "dynaHeap": c.dynaHeap, "maintenancePhase": c.phase, "started": time.Now().UTC().Format(time.RFC3339), }, "", " ") if err := os.WriteFile(filepath.Join(c.dir, "run.json"), runJSON, 0o644); err != nil { fail("run.json", err) From 21ee64da99610ce87a6cb4cb2a9eae508f282f54 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:35:30 -0500 Subject: [PATCH 20/58] The permanent index's primitives: a record that names an entry, runs, a merge Step 2's building blocks, with nothing that touches an entry's bytes: a 44-byte record (key, file, offset, length); a sorted run with a checksum and a filter, written into a run file and looked up resident or with the filter probed cold; a k-way merge over runs, newest wins. The reader rebuilds a run's filter from its stored byte count, since ByteMask indexes by NumBytes. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/permindex.go | 242 +++++++++++++++++++++++++++++++++++++ database/permindex_test.go | 94 ++++++++++++++ 2 files changed, 336 insertions(+) create mode 100644 database/permindex.go create mode 100644 database/permindex_test.go diff --git a/database/permindex.go b/database/permindex.go new file mode 100644 index 0000000..f18446b --- /dev/null +++ b/database/permindex.go @@ -0,0 +1,242 @@ +package blockchainDB + +import ( + "bytes" + "container/heap" + "encoding/binary" + "errors" + "fmt" + "hash/crc32" + "io" + "os" + "sort" +) + +// The permanent layer's index primitives (proposal +// docs/proposals/2026-09-16-entries-written-once.md, step 2): a +// record that names where an entry lives, sorted runs of records with +// a filter, a k-way merge over runs, and a lookup in a run. Nothing +// here touches an entry's bytes; entries are written once and only +// ever named. +// +// A run on disk: +// +// header magic "PRUN"(4) count(4) bloomBytes(4) bloomK(4) crc(4) +// records count sorted 44-byte records: key(32) file(4) off(4) len(4) +// bloom bloomBytes of filter over the keys +// +// Runs are immutable once written; a merge writes a new run from +// several and the inputs are dropped by whoever owns them. + +// permRecord names an entry: its key and where it is. +type permRecord struct { + key [32]byte + file uint32 + off uint32 + n uint32 +} + +const ( + permRecSize = 32 + 4 + 4 + 4 + permRunHdr = 4 + 4 + 4 + 4 + 4 + permRunMagic = 0x5052554E // "PRUN" + permBloomBits = 12 // Bits per key in a run's filter + permBloomK = 3 +) + +func (r permRecord) put(buf []byte) { + copy(buf, r.key[:]) + binary.LittleEndian.PutUint32(buf[32:], r.file) + binary.LittleEndian.PutUint32(buf[36:], r.off) + binary.LittleEndian.PutUint32(buf[40:], r.n) +} + +func getPermRecord(buf []byte) (r permRecord) { + copy(r.key[:], buf) + r.file = binary.LittleEndian.Uint32(buf[32:]) + r.off = binary.LittleEndian.Uint32(buf[36:]) + r.n = binary.LittleEndian.Uint32(buf[40:]) + return r +} + +// permRun is a sorted run of records with its filter, as held by a +// bucket: the file it lives in, where, and how many; the filter is +// resident when loaded and probed cold otherwise. +type permRun struct { + path string + off int64 // Where the run starts in its file + count uint32 + bloomAt int64 // Where the filter starts + bloom *Bloom + k int + bytes uint32 +} + +// writePermRun writes records, which must be sorted by key and free +// of duplicates, as a run appended to w, and returns it. The caller +// fsyncs the file. +func writePermRun(w io.WriterAt, at int64, path string, recs []permRecord) (*permRun, error) { + if !sort.SliceIsSorted(recs, func(i, j int) bool { return bytes.Compare(recs[i].key[:], recs[j].key[:]) < 0 }) { + return nil, errors.New("perm run: records are not sorted") + } + bloom := NewBloomSizedForKeys(uint64(len(recs)), permBloomK) + buf := make([]byte, permRunHdr+len(recs)*permRecSize+int(bloom.NumBytes)) + binary.LittleEndian.PutUint32(buf, permRunMagic) + binary.LittleEndian.PutUint32(buf[4:], uint32(len(recs))) + binary.LittleEndian.PutUint32(buf[8:], uint32(bloom.NumBytes)) + binary.LittleEndian.PutUint32(buf[12:], uint32(bloom.K)) + p := permRunHdr + for i, r := range recs { + if i > 0 && recs[i-1].key == r.key { + return nil, errors.New("perm run: duplicate key") + } + r.put(buf[p:]) + bloom.Set(r.key) + p += permRecSize + } + copy(buf[p:], bloom.Map) + binary.LittleEndian.PutUint32(buf[16:], crc32.ChecksumIEEE(buf[permRunHdr:])) + if _, err := w.WriteAt(buf, at); err != nil { + return nil, err + } + return &permRun{path: path, off: at, count: uint32(len(recs)), bloomAt: at + int64(p), bloom: bloom, k: bloom.K, bytes: uint32(len(buf))}, nil +} + +// openPermRun reads a run's header at off in f and verifies the run. +func openPermRun(f *os.File, path string, off int64, resident bool) (*permRun, error) { + hdr := make([]byte, permRunHdr) + if _, err := f.ReadAt(hdr, off); err != nil { + return nil, err + } + if binary.LittleEndian.Uint32(hdr) != permRunMagic { + return nil, fmt.Errorf("perm run at %s:%d: bad magic", path, off) + } + r := &permRun{path: path, off: off, count: binary.LittleEndian.Uint32(hdr[4:]), k: int(binary.LittleEndian.Uint32(hdr[12:]))} + bloomBytes := binary.LittleEndian.Uint32(hdr[8:]) + r.bloomAt = off + permRunHdr + int64(r.count)*permRecSize + r.bytes = permRunHdr + r.count*permRecSize + bloomBytes + body := make([]byte, r.bytes-permRunHdr) + if _, err := f.ReadAt(body, off+permRunHdr); err != nil { + return nil, err + } + if crc32.ChecksumIEEE(body) != binary.LittleEndian.Uint32(hdr[16:]) { + return nil, fmt.Errorf("perm run at %s:%d: checksum failed", path, off) + } + if resident { + // Rebuilt from the stored byte count: ByteMask indexes by NumBytes + r.bloom = &Bloom{NumBytes: uint64(bloomBytes), SizeOfMap: float64(bloomBytes) / (1 << 20), K: r.k, + Map: make([]byte, bloomBytes), Capacity: uint64(bloomBytes) * 8 / BloomBitsPerKey, Count: uint64(r.count)} + copy(r.bloom.Map, body[int64(r.count)*permRecSize:]) + } + return r, nil +} + +// lookup finds key in the run: the filter first (resident, or cold in +// the file), then a binary search over the records. +func (r *permRun) lookup(f *os.File, key [32]byte) (rec permRecord, found bool, err error) { + if r.bloom != nil { + if !r.bloom.Test(key) { + return rec, false, nil + } + } else if ok, err := r.bloomTestCold(f, key); err != nil || !ok { + return rec, false, err + } + lo, hi := int64(0), int64(r.count) + buf := make([]byte, permRecSize) + for lo < hi { + mid := (lo + hi) / 2 + if _, err := f.ReadAt(buf, r.off+permRunHdr+mid*permRecSize); err != nil { + return rec, false, err + } + switch c := bytes.Compare(buf[:32], key[:]); { + case c == 0: + return getPermRecord(buf), true, nil + case c < 0: + lo = mid + 1 + default: + hi = mid + } + } + return rec, false, nil +} + +// bloomTestCold probes the run's filter in the file: k one-byte +// reads, the way the segment store probes a cold filter. +func (r *permRun) bloomTestCold(f *os.File, key [32]byte) (bool, error) { + bloomBytes := r.bytes - permRunHdr - r.count*permRecSize + probe := &Bloom{NumBytes: uint64(bloomBytes), SizeOfMap: float64(bloomBytes) * 8, K: r.k} + one := make([]byte, 1) + for i := 0; i < r.k; i++ { + idx, mask := probe.ByteMask(key, i) + if _, err := f.ReadAt(one, r.bloomAt+int64(idx)); err != nil { + return false, err + } + if one[0]&mask == 0 { + return false, nil + } + } + return true, nil +} + +// records reads a run's records in order, for a merge. +func (r *permRun) records(f *os.File) ([]permRecord, error) { + buf := make([]byte, int64(r.count)*permRecSize) + if _, err := f.ReadAt(buf, r.off+permRunHdr); err != nil { + return nil, err + } + recs := make([]permRecord, r.count) + for i := range recs { + recs[i] = getPermRecord(buf[i*permRecSize:]) + } + return recs, nil +} + +// mergePermRuns merges sorted record lists, oldest first, into one +// sorted list; a key present in several takes the newest. Permanent +// keys are written once, so a duplicate is a replay or a fault, and +// newest-wins matches the store's rule everywhere else. +func mergePermRuns(inputs [][]permRecord) []permRecord { + h := &permMergeHeap{} + total := 0 + for src, in := range inputs { + total += len(in) + if len(in) > 0 { + heap.Push(h, permMergeCursor{src: src, recs: in}) + } + } + out := make([]permRecord, 0, total) + for h.Len() > 0 { + c := heap.Pop(h).(permMergeCursor) + r := c.recs[0] + if n := len(out); n == 0 || out[n-1].key != r.key { + out = append(out, r) + } + // The heap orders equal keys newest first, so the first copy + // out is the one kept and the rest are dropped here + if len(c.recs) > 1 { + heap.Push(h, permMergeCursor{src: c.src, recs: c.recs[1:]}) + } + } + return out +} + +type permMergeCursor struct { + src int + recs []permRecord +} + +type permMergeHeap []permMergeCursor + +func (h permMergeHeap) Len() int { return len(h) } +func (h permMergeHeap) Less(i, j int) bool { + if c := bytes.Compare(h[i].recs[0].key[:], h[j].recs[0].key[:]); c != 0 { + return c < 0 + } + return h[i].src > h[j].src // Newest first on a tie +} +func (h permMergeHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] } +func (h *permMergeHeap) Push(x any) { *h = append(*h, x.(permMergeCursor)) } +func (h *permMergeHeap) Pop() any { old := *h; x := old[len(old)-1]; *h = old[:len(old)-1]; return x } +func sortPermRecords(recs []permRecord) { + sort.Slice(recs, func(i, j int) bool { return bytes.Compare(recs[i].key[:], recs[j].key[:]) < 0 }) +} diff --git a/database/permindex_test.go b/database/permindex_test.go new file mode 100644 index 0000000..76a71d8 --- /dev/null +++ b/database/permindex_test.go @@ -0,0 +1,94 @@ +package blockchainDB + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +// A run round-trips: written sorted with its filter, reopened with the +// checksum verified, looked up resident and cold, merged newest-wins. +func TestPermRunWriteLookupMerge(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "runs.dat") + f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0o644) + require.NoError(t, err) + defer f.Close() + fr := NewFastRandom([]byte{11}) + older := make([]permRecord, 5000) + for i := range older { + older[i] = permRecord{key: fr.NextHash(), file: 1, off: uint32(i * 100), n: 64} + } + sortPermRecords(older) + // The newer run rewrites a tenth of the older keys and adds new ones + newer := make([]permRecord, 0, 1000) + for i := 0; i < 500; i++ { + r := older[i*10] + r.file, r.off = 2, uint32(i) + newer = append(newer, r) + } + for i := 0; i < 500; i++ { + newer = append(newer, permRecord{key: fr.NextHash(), file: 2, off: uint32(1000 + i), n: 32}) + } + sortPermRecords(newer) + + r1, err := writePermRun(f, 0, path, older) + require.NoError(t, err) + r2, err := writePermRun(f, int64(r1.bytes), path, newer) + require.NoError(t, err) + _, err = writePermRun(f, int64(r1.bytes+r2.bytes), path, []permRecord{older[3], older[2]}) + require.Error(t, err, "unsorted records are refused") + + // Reopen: resident and cold + rr, err := openPermRun(f, path, 0, true) + require.NoError(t, err) + require.EqualValues(t, 5000, rr.count) + cold, err := openPermRun(f, path, int64(r1.bytes), false) + require.NoError(t, err) + require.Nil(t, cold.bloom) + for _, rec := range older[:200] { + got, found, err := rr.lookup(f, rec.key) + require.NoError(t, err) + require.True(t, found) + require.Equal(t, rec, got) + } + for _, rec := range newer[:200] { + got, found, err := cold.lookup(f, rec.key) + require.NoError(t, err) + require.True(t, found) + require.Equal(t, rec, got) + } + misses := 0 + for i := 0; i < 2000; i++ { + _, found, err := rr.lookup(f, fr.NextHash()) + require.NoError(t, err) + if !found { + misses++ + } + } + require.Equal(t, 2000, misses, "an absent key is never found") + + // Merge: the newer copy of a rewritten key wins + recs1, err := rr.records(f) + require.NoError(t, err) + recs2, err := cold.records(f) + require.NoError(t, err) + merged := mergePermRuns([][]permRecord{recs1, recs2}) + require.Len(t, merged, 5500) + byKey := map[[32]byte]permRecord{} + for _, r := range merged { + byKey[r.key] = r + } + for _, r := range newer { + require.Equal(t, r, byKey[r.key]) + } + require.EqualValues(t, 1, byKey[older[1].key].file, "an unrewritten key keeps its older record") + + // A damaged run is refused + _, err = f.WriteAt([]byte{0xff, 0xff}, permRunHdr+40) + require.NoError(t, err) + _, err = openPermRun(f, path, 0, true) + require.ErrorContains(t, err, "checksum") +} From 2626aef9d4ce393bf4f4778dcd8ad30b6bc89200 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:39:34 -0500 Subject: [PATCH 21/58] The permanent layer as files of entries and buckets of keys, first cut PermStore (perm.go): entries appended to data files and never moved; a block's records sealed as a delta run with its filter; the window of the last FilterBlocks deltas is what the protocol path reads; a delta leaving the window feeds 256 buckets by the key's first byte, merged in rotation (PermBuckets/PermMergeEvery a step) and folded by ratio; a pack retires every bucket's runs and pending deltas into one sorted run, the deep history, probed cold. The manifest is written aside and renamed; deltas sealed after it replay on open and a torn one is cut. Runs carry a height and their file id. Not yet wired behind KV2; measured by its test only. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 1071 ++++++++++++++++++++++++++++++++++++ database/perm_test.go | 119 ++++ database/permindex.go | 20 +- database/permindex_test.go | 15 +- 4 files changed, 1210 insertions(+), 15 deletions(-) create mode 100644 database/perm.go create mode 100644 database/perm_test.go diff --git a/database/perm.go b/database/perm.go new file mode 100644 index 0000000..7afedc0 --- /dev/null +++ b/database/perm.go @@ -0,0 +1,1071 @@ +package blockchainDB + +import ( + "bytes" + "encoding/binary" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "sync" + "sync/atomic" +) + +// PermStore is the permanent layer as files of entries and buckets of +// keys (proposal docs/proposals/2026-09-16-entries-written-once.md, +// step 2). An entry is written once at the end of a data file and +// never moves; everything the store does to stay fast is done on +// 44-byte records that name entries (permindex.go). +// +// Files, in one directory: +// +// perm-N.dat entries, [len][height][key][value][crc] (heap.go's +// layout), appended in the order written, rolled at +// PermFileBytes; never rewritten +// runs-N.dat runs of records (permindex.go), appended: a block's +// delta at every seal, a bucket's run at every merge, +// a retired run at every pack; rolled at PermFileBytes +// and deleted once no run in it is referenced +// perm.json the manifest: which runs are the window, the buckets +// and the retired history, and where in the run files +// the deltas not yet in the manifest begin +// +// Tiers, newest to oldest: +// +// - the live map: this block's records, in memory until the seal; +// - the window: the last FilterBlocks deltas, each a run with its +// filter resident. The protocol path reads the live map and the +// window and nothing older (spec 1.3): an immutable key the window +// does not hold is absent; +// - the buckets: PermBuckets buckets by the key's first byte, each a +// few sorted runs. A delta that leaves the window feeds each +// bucket's recent records; every bucket is merged every +// PermMergeEvery blocks, in rotation, its recent records written +// as a run and its runs folded by the store's ratio. The buckets +// hold only the history above the pack watermark, so a bucket is +// bounded by the pack period and hashed keys keep it even; +// - retired runs: at every pack, every bucket's runs and every +// delta below the watermark fold into one sorted run, and the +// buckets start again. Retired runs are the deep history. +// +// GetDeep walks the tiers in that order; Get stops after the window. +// +// Durability (spec 1.8): a seal fsyncs the data files the block wrote +// and then appends and fsyncs the block's delta, so a delta names +// only durable entries. Open reads the manifest, then every whole +// delta after the offset it records; a torn delta is dropped whole. +// Maintenance writes new runs, fsyncs them, and commits the manifest +// (written aside and renamed) before any run file is deleted: never +// unlink what a durable manifest names. +type PermStore struct { + Directory string + + mu sync.RWMutex + files map[uint32]*heapFile // Data files, by id + cur *heapFile // The data file the block appends to + nextID uint32 + dirty map[uint32]*heapFile + + runs map[uint32]*runFile // Run files, by id + curRun *runFile + nextRun uint32 + live map[[32]byte]permRecord // This block's records + window []*permDelta // The last FilterBlocks deltas, oldest first + pending []*permDelta // Deltas below the window not yet in every bucket + buckets [PermBuckets]permBucket // The history above the watermark + retired []*permRun // The history below it, newest last + height uint64 + window_n uint64 // FilterBlocks + rotation int // The next bucket to merge + + // The manifest's view: the run file and offset after which deltas + // are replayed on open + deltasFrom struct { + file uint32 + off int64 + } + syncMu sync.Mutex // Serializes seals with each other and with manifest commits + closed bool + + putTotal, putDuplicate, lookups, windowHits, deepHits atomic.Uint64 + mergeRuns, foldRuns, packRuns atomic.Uint64 + indexBytes atomic.Uint64 +} + +// PermBuckets is how many buckets a shard's history above the +// watermark is kept in, by the key's first byte. +const PermBuckets = 256 + +// PermMergeEvery is how many blocks pass between merges of one +// bucket: PermBuckets/PermMergeEvery buckets are merged each block. +var PermMergeEvery uint64 = 256 + +// PermFileBytes is the size data files and run files are rolled at. +var PermFileBytes int64 = 64 << 20 + +// PermFoldRatio is the ratio a bucket's runs fold by: a suffix of runs +// folds while each older run is no larger than 1/ratio of what has +// gathered behind it. +var PermFoldRatio = 0.25 + +type permDelta struct { + height uint64 + run *permRun + rf *runFile +} + +type permBucket struct { + runs []*permRun // Oldest first + files []*runFile + merged uint64 // The height of the newest delta merged in +} + +// runFile is a file of runs and how many runs still reference it. +type runFile struct { + id uint32 + f *os.File + size int64 + refs int +} + +func permDataName(id uint32) string { return fmt.Sprintf("perm-%06d.dat", id) } +func permRunName(id uint32) string { return fmt.Sprintf("runs-%06d.dat", id) } + +// NewPermStore creates an empty store in directory, replacing anything +// there. +func NewPermStore(directory string, filterBlocks uint64) (*PermStore, error) { + os.RemoveAll(directory) + if err := os.MkdirAll(directory, 0o755); err != nil { + return nil, err + } + p := &PermStore{Directory: directory, window_n: filterBlocks} + return p, p.Open() +} + +// OpenPermStore opens the store in directory as it was left. +func OpenPermStore(directory string) (*PermStore, error) { + if _, err := os.Stat(filepath.Join(directory, "perm.json")); err != nil { + return nil, fmt.Errorf("open perm at %s: %w", directory, err) + } + p := &PermStore{Directory: directory} + return p, p.Open() +} + +// permManifest is perm.json. +type permManifest struct { + Version int `json:"version"` + Height uint64 `json:"height"` + FilterBlocks uint64 `json:"filterBlocks"` + Rotation int `json:"rotation"` + NextData uint32 `json:"nextData"` + NextRun uint32 `json:"nextRun"` + DeltasFile uint32 `json:"deltasFile"` // Deltas after this file:offset are replayed + DeltasOff int64 `json:"deltasOff"` + Window []permRunRef `json:"window"` + Pending []permRunRef `json:"pending"` + Buckets []permBucketM `json:"buckets"` + Retired []permRunRef `json:"retired"` + DataFiles []uint32 `json:"dataFiles"` + RunFiles []uint32 `json:"runFiles"` + Extra map[string]any `json:"-"` +} + +type permRunRef struct { + File uint32 `json:"file"` + Off int64 `json:"off"` + Height uint64 `json:"height,omitempty"` +} + +type permBucketM struct { + Runs []permRunRef `json:"runs"` + Merged uint64 `json:"merged"` +} + +// Open loads the manifest and replays the deltas after it. +func (p *PermStore) Open() (err error) { + p.mu.Lock() + defer p.mu.Unlock() + if p.files != nil { + return nil + } + p.files = map[uint32]*heapFile{} + p.dirty = map[uint32]*heapFile{} + p.runs = map[uint32]*runFile{} + p.live = map[[32]byte]permRecord{} + p.closed = false + var m permManifest + buf, err := os.ReadFile(filepath.Join(p.Directory, "perm.json")) + switch { + case errors.Is(err, os.ErrNotExist): + // A new store: the first data file and run file + if p.window_n == 0 { + p.window_n = MinFilterBlocks + } + if p.cur, err = p.newDataFile(); err != nil { + return err + } + if p.curRun, err = p.newRunFile(); err != nil { + return err + } + return p.commitManifest() + case err != nil: + return err + } + if err = json.Unmarshal(buf, &m); err != nil { + return fmt.Errorf("perm.json: %w", err) + } + if m.Version != 1 { + return fmt.Errorf("perm.json: version %d, want 1", m.Version) + } + p.height, p.window_n, p.rotation, p.nextID, p.nextRun = m.Height, m.FilterBlocks, m.Rotation, m.NextData, m.NextRun + for _, id := range m.DataFiles { + f, err := os.OpenFile(filepath.Join(p.Directory, permDataName(id)), os.O_RDWR, 0o644) + if err != nil { + return fmt.Errorf("perm: the manifest names %s: %w", permDataName(id), err) + } + st, _ := f.Stat() + p.files[id] = &heapFile{id: id, f: f, size: st.Size()} + } + for _, id := range m.RunFiles { + f, err := os.OpenFile(filepath.Join(p.Directory, permRunName(id)), os.O_RDWR, 0o644) + if err != nil { + return fmt.Errorf("perm: the manifest names %s: %w", permRunName(id), err) + } + st, _ := f.Stat() + p.runs[id] = &runFile{id: id, f: f, size: st.Size()} + } + load := func(ref permRunRef, resident bool) (*permRun, *runFile, error) { + rf := p.runs[ref.File] + if rf == nil { + return nil, nil, fmt.Errorf("perm: run file %d not open", ref.File) + } + r, err := openPermRun(rf.f, ref.File, ref.Off, resident) + if err != nil { + return nil, nil, err + } + rf.refs++ + return r, rf, nil + } + for _, ref := range m.Window { + r, rf, err := load(ref, true) + if err != nil { + return err + } + p.window = append(p.window, &permDelta{height: ref.Height, run: r, rf: rf}) + } + for _, ref := range m.Pending { + r, rf, err := load(ref, true) + if err != nil { + return err + } + p.pending = append(p.pending, &permDelta{height: ref.Height, run: r, rf: rf}) + } + for i, bm := range m.Buckets { + if i >= PermBuckets { + break + } + p.buckets[i].merged = bm.Merged + for _, ref := range bm.Runs { + r, rf, err := load(ref, true) + if err != nil { + return err + } + p.buckets[i].runs = append(p.buckets[i].runs, r) + p.buckets[i].files = append(p.buckets[i].files, rf) + } + } + for _, ref := range m.Retired { + r, _, err := load(ref, false) + if err != nil { + return err + } + p.retired = append(p.retired, r) + } + // The current files are the newest + for _, hf := range p.files { + if p.cur == nil || hf.id > p.cur.id { + p.cur = hf + } + } + for _, rf := range p.runs { + if p.curRun == nil || rf.id > p.curRun.id { + p.curRun = rf + } + } + p.deltasFrom.file, p.deltasFrom.off = m.DeltasFile, m.DeltasOff + return p.replayDeltas() +} + +// replayDeltas reads every whole delta after the manifest's offset +// into the window (and pending), cutting a torn tail. The caller +// holds the lock. +func (p *PermStore) replayDeltas() error { + ids := make([]uint32, 0, len(p.runs)) + for id := range p.runs { + if id >= p.deltasFrom.file { + ids = append(ids, id) + } + } + sort.Slice(ids, func(i, j int) bool { return ids[i] < ids[j] }) + for _, id := range ids { + rf := p.runs[id] + off := int64(0) + if id == p.deltasFrom.file { + off = p.deltasFrom.off + } + for off < rf.size { + r, err := openPermRun(rf.f, id, off, true) + if err != nil { + // Torn: cut the file here + if err := rf.f.Truncate(off); err != nil { + return err + } + rf.size = off + break + } + rf.refs++ + p.admit(&permDelta{height: r.height, run: r, rf: rf}) + if r.height > p.height { + p.height = r.height + } + off += int64(r.bytes) + } + } + // Entries past the last named one are a crash's leftovers + var end int64 + for _, d := range append(p.window, p.pending...) { + _ = d + } + _ = end + return nil +} + +// admit puts a delta in the window and moves what falls out of it to +// pending. The caller holds the lock. +func (p *PermStore) admit(d *permDelta) { + p.window = append(p.window, d) + for len(p.window) > int(p.window_n) { + p.pending = append(p.pending, p.window[0]) + p.window = p.window[1:] + } +} + +func (p *PermStore) newDataFile() (*heapFile, error) { + id := p.nextID + p.nextID++ + f, err := os.OpenFile(filepath.Join(p.Directory, permDataName(id)), os.O_RDWR|os.O_CREATE|os.O_EXCL, 0o644) + if err != nil { + return nil, err + } + hf := &heapFile{id: id, f: f} + p.files[id] = hf + return hf, nil +} + +func (p *PermStore) newRunFile() (*runFile, error) { + id := p.nextRun + p.nextRun++ + f, err := os.OpenFile(filepath.Join(p.Directory, permRunName(id)), os.O_RDWR|os.O_CREATE|os.O_EXCL, 0o644) + if err != nil { + return nil, err + } + rf := &runFile{id: id, f: f} + p.runs[id] = rf + return rf, nil +} + +// Close seals what is pending and closes the files. +func (p *PermStore) Close() error { + sync, err := p.beginSeal(p.height) + if err != nil { + return err + } + if err = sync.finish(); err != nil { + return err + } + p.mu.Lock() + defer p.mu.Unlock() + p.closed = true + for _, hf := range p.files { + if cerr := hf.f.Close(); err == nil { + err = cerr + } + } + for _, rf := range p.runs { + if cerr := rf.f.Close(); err == nil { + err = cerr + } + } + p.files, p.runs, p.cur, p.curRun = nil, nil, nil, nil + return err +} + +// PutIfAbsent writes value under key unless the window already holds +// the key, in which case the existing value is returned instead. The +// window is the store's immutability horizon (spec 1.3). +func (p *PermStore) PutIfAbsent(key [32]byte, value []byte) (existing []byte, existed bool, err error) { + p.mu.Lock() + defer p.mu.Unlock() + if p.closed { + return nil, false, errStoreClosed + } + p.putTotal.Add(1) + if rec, ok := p.live[key]; ok { + p.putDuplicate.Add(1) + v, err := p.readEntry(rec, key) + return v, true, err + } + for i := len(p.window) - 1; i >= 0; i-- { + d := p.window[i] + rec, found, err := d.run.lookup(d.rf.f, key) + if err != nil { + return nil, false, err + } + if found { + p.putDuplicate.Add(1) + v, err := p.readEntry(rec, key) + return v, true, err + } + } + size := entrySize(len(value)) + if p.cur.size+size > PermFileBytes { + if p.cur, err = p.newDataFile(); err != nil { + return nil, false, err + } + } + rec := permRecord{key: key, file: p.cur.id, off: uint32(p.cur.size), n: uint32(len(value))} + if _, err = p.cur.f.WriteAt(encodeEntry(p.height, key, value), p.cur.size); err != nil { + return nil, false, err + } + p.cur.size += size + p.dirty[p.cur.id] = p.cur + p.live[key] = rec + return nil, false, nil +} + +// Put is PutIfAbsent for callers that only need the error; a key +// already held is not an error, the value stands. +func (p *PermStore) Put(key [32]byte, value []byte) error { + _, _, err := p.PutIfAbsent(key, value) + return err +} + +func (p *PermStore) readEntry(rec permRecord, key [32]byte) ([]byte, error) { + hf := p.files[rec.file] + if hf == nil { + return nil, fmt.Errorf("perm: data file %d not open", rec.file) + } + buf := make([]byte, heapHeader+int(rec.n)+heapTrailer) + if _, err := hf.f.ReadAt(buf, int64(rec.off)); err != nil { + return nil, err + } + return heapEntryValue(buf, key) +} + +// Get answers from the live map and the window; a key older than the +// window is absent here (spec 1.3), GetDeep reaches it. +func (p *PermStore) Get(key [32]byte) ([]byte, error) { + p.mu.RLock() + defer p.mu.RUnlock() + if p.closed { + return nil, errStoreClosed + } + p.lookups.Add(1) + if rec, ok := p.live[key]; ok { + p.windowHits.Add(1) + return p.readEntry(rec, key) + } + for i := len(p.window) - 1; i >= 0; i-- { + d := p.window[i] + rec, found, err := d.run.lookup(d.rf.f, key) + if err != nil { + return nil, err + } + if found { + p.windowHits.Add(1) + return p.readEntry(rec, key) + } + } + return nil, errNotFound +} + +// GetDeep is Get, then the deltas not yet merged, the key's bucket +// newest run first, and the retired runs newest first. +func (p *PermStore) GetDeep(key [32]byte) ([]byte, error) { + v, err := p.Get(key) + if err == nil || !errors.Is(err, errNotFound) { + return v, err + } + p.mu.RLock() + defer p.mu.RUnlock() + for i := len(p.pending) - 1; i >= 0; i-- { + d := p.pending[i] + rec, found, err := d.run.lookup(d.rf.f, key) + if err != nil { + return nil, err + } + if found { + p.deepHits.Add(1) + return p.readEntry(rec, key) + } + } + b := &p.buckets[key[0]] + for i := len(b.runs) - 1; i >= 0; i-- { + rec, found, err := b.runs[i].lookup(b.files[i].f, key) + if err != nil { + return nil, err + } + if found { + p.deepHits.Add(1) + return p.readEntry(rec, key) + } + } + for i := len(p.retired) - 1; i >= 0; i-- { + r := p.retired[i] + rf := p.runs[r.file] + rec, found, err := r.lookup(rf.f, key) + if err != nil { + return nil, err + } + if found { + p.deepHits.Add(1) + return p.readEntry(rec, key) + } + } + return nil, errNotFound +} + +// permSeal is a seal in flight: the data files to fsync, the delta +// to append. Holds syncMu until finished. +type permSeal struct { + p *PermStore + dirty []*heapFile + recs []permRecord + height uint64 +} + +// beginSeal takes the block's records under the lock; finish makes +// them durable outside it. +func (p *PermStore) beginSeal(height uint64) (*permSeal, error) { + p.syncMu.Lock() + p.mu.Lock() + defer p.mu.Unlock() + if p.closed || p.files == nil { + p.syncMu.Unlock() + return nil, errStoreClosed + } + s := &permSeal{p: p, height: height} + for _, hf := range p.dirty { + s.dirty = append(s.dirty, hf) + } + p.dirty = map[uint32]*heapFile{} + s.recs = make([]permRecord, 0, len(p.live)) + for _, rec := range p.live { + s.recs = append(s.recs, rec) + } + sortPermRecords(s.recs) + return s, nil +} + +// finish: entries durable, then the delta appended and durable, then +// the delta admitted to the window and the live map cleared. +func (s *permSeal) finish() error { + p := s.p + defer p.syncMu.Unlock() + for _, hf := range s.dirty { + if err := fsync(hf.f); err != nil { + return err + } + } + if len(s.recs) == 0 { + p.mu.Lock() + if s.height >= p.height { + p.height = s.height + 1 + } + p.mu.Unlock() + return nil + } + p.mu.Lock() + rf := p.curRun + if rf.size > PermFileBytes { + var err error + if rf, err = p.newRunFile(); err != nil { + p.mu.Unlock() + return err + } + p.curRun = rf + } + at := rf.size + p.mu.Unlock() + run, err := writePermRun(rf.f, at, rf.id, s.recs, s.height) + if err != nil { + return err + } + if err = fsync(rf.f); err != nil { + return err + } + p.indexBytes.Add(uint64(run.bytes)) + p.mu.Lock() + defer p.mu.Unlock() + rf.size = at + int64(run.bytes) + rf.refs++ + p.admit(&permDelta{height: s.height, run: run, rf: rf}) + p.live = map[[32]byte]permRecord{} + if s.height >= p.height { + p.height = s.height + 1 + } + return nil +} + +// AdvanceBlock sets the block new writes belong to. +func (p *PermStore) AdvanceBlock(height uint64) { + p.mu.Lock() + p.height = height + p.mu.Unlock() +} + +// BlockHeight is the block being written. +func (p *PermStore) BlockHeight() uint64 { + p.mu.RLock() + defer p.mu.RUnlock() + return p.height +} + +// LiveCount is the records this block has written so far. +func (p *PermStore) LiveCount() int { + p.mu.RLock() + defer p.mu.RUnlock() + return len(p.live) +} + +// Merge is the maintenance step: the buckets due in rotation take +// their records from the pending deltas, fold, and the manifest is +// committed; deltas every bucket has absorbed are released. Runs +// are written and fsynced outside the lock; the lock is held to +// choose and to swap. +func (p *PermStore) Merge() error { + p.mu.Lock() + if p.closed { + p.mu.Unlock() + return errStoreClosed + } + if len(p.pending) == 0 { + p.mu.Unlock() + return nil + } + // Which buckets, and from which deltas + due := int(PermBuckets / PermMergeEvery) + if due < 1 { + due = 1 + } + newest := p.pending[len(p.pending)-1].height + type job struct { + b int + recs []permRecord + from []*permDelta + fold []*permRun + foldF []*runFile + foldI int + } + var jobs []job + for i := 0; i < due; i++ { + b := (p.rotation + i) % PermBuckets + bk := &p.buckets[b] + var j job + j.b = b + for _, d := range p.pending { + if d.height <= bk.merged { + continue + } + j.from = append(j.from, d) + } + jobs = append(jobs, j) + } + rotation := (p.rotation + due) % PermBuckets + // Read the deltas' records for these buckets outside the lock: + // runs are immutable + pendingCopy := append([]*permDelta(nil), p.pending...) + p.mu.Unlock() + byBucket := map[int][][]permRecord{} + for _, d := range pendingCopy { + recs, err := d.run.records(d.rf.f) + if err != nil { + return err + } + for _, j := range jobs { + if d.height <= p.buckets[j.b].merged { + continue + } + var mine []permRecord + for _, r := range recs { + if int(r.key[0]) == j.b { + mine = append(mine, r) + } + } + byBucket[j.b] = append(byBucket[j.b], mine) + } + } + // Write each due bucket's new run and its fold, in the current run file + p.mu.Lock() + rf := p.curRun + p.mu.Unlock() + var written []struct { + b int + run *permRun + fold bool + at int + } + for _, j := range jobs { + recs := mergePermRuns(byBucket[j.b]) + if len(recs) == 0 { + continue + } + p.mu.Lock() + if rf.size > PermFileBytes { + var err error + if rf, err = p.newRunFile(); err != nil { + p.mu.Unlock() + return err + } + p.curRun = rf + } + at := rf.size + rf.size += 0 + p.mu.Unlock() + run, err := writePermRun(rf.f, at, rf.id, recs, 0) + if err != nil { + return err + } + p.mu.Lock() + rf.size = at + int64(run.bytes) + p.mu.Unlock() + p.indexBytes.Add(uint64(run.bytes)) + p.mergeRuns.Add(1) + written = append(written, struct { + b int + run *permRun + fold bool + at int + }{b: j.b, run: run}) + } + if err := fsync(rf.f); err != nil { + return err + } + // Swap: the new runs join their buckets; then fold what the ratio + // says, one bucket at a time + p.mu.Lock() + for _, w := range written { + bk := &p.buckets[w.b] + bk.runs = append(bk.runs, w.run) + bk.files = append(bk.files, rf) + rf.refs++ + } + for _, j := range jobs { + p.buckets[j.b].merged = newest + } + p.rotation = rotation + // Deltas every bucket has absorbed are released + minMerged := ^uint64(0) + for i := range p.buckets { + if p.buckets[i].merged < minMerged { + minMerged = p.buckets[i].merged + } + } + var keep []*permDelta + for _, d := range p.pending { + if d.height <= minMerged { + d.rf.refs-- + } else { + keep = append(keep, d) + } + } + p.pending = keep + folds := p.planFolds() + p.mu.Unlock() + for _, f := range folds { + if err := p.fold(f); err != nil { + return err + } + } + p.mu.Lock() + defer p.mu.Unlock() + if err := p.commitManifest(); err != nil { + return err + } + return p.dropUnreferencedRunFiles() +} + +type permFold struct { + b int + at int // The runs from this index on fold into one + count int +} + +// planFolds chooses, per bucket, the suffix of runs the ratio says to +// fold. The caller holds the lock. +func (p *PermStore) planFolds() (folds []permFold) { + for b := range p.buckets { + runs := p.buckets[b].runs + if len(runs) < 2 { + continue + } + var behind uint32 + i := len(runs) - 1 + for ; i >= 0; i-- { + if i < len(runs)-1 && float64(runs[i].count)*PermFoldRatio > float64(behind) { + break + } + behind += runs[i].count + } + if n := len(runs) - (i + 1); n >= 2 { + folds = append(folds, permFold{b: b, at: i + 1, count: n}) + } + } + return folds +} + +// fold merges a bucket's chosen runs into one, written to the current +// run file, and swaps it in under the lock. +func (p *PermStore) fold(f permFold) error { + p.mu.RLock() + bk := &p.buckets[f.b] + if f.at+f.count > len(bk.runs) { + p.mu.RUnlock() + return nil // The bucket changed under us; next time + } + runs := append([]*permRun(nil), bk.runs[f.at:f.at+f.count]...) + files := append([]*runFile(nil), bk.files[f.at:f.at+f.count]...) + p.mu.RUnlock() + inputs := make([][]permRecord, len(runs)) + for i, r := range runs { + recs, err := r.records(files[i].f) + if err != nil { + return err + } + inputs[i] = recs + } + merged := mergePermRuns(inputs) + p.mu.Lock() + rf := p.curRun + if rf.size > PermFileBytes { + var err error + if rf, err = p.newRunFile(); err != nil { + p.mu.Unlock() + return err + } + p.curRun = rf + } + at := rf.size + p.mu.Unlock() + run, err := writePermRun(rf.f, at, rf.id, merged, 0) + if err != nil { + return err + } + if err = fsync(rf.f); err != nil { + return err + } + p.indexBytes.Add(uint64(run.bytes)) + p.foldRuns.Add(1) + p.mu.Lock() + defer p.mu.Unlock() + rf.size = at + int64(run.bytes) + bk = &p.buckets[f.b] + if f.at+f.count > len(bk.runs) { + return nil + } + for _, old := range files { + old.refs-- + } + rest := append([]*permRun(nil), bk.runs[f.at+f.count:]...) + restF := append([]*runFile(nil), bk.files[f.at+f.count:]...) + bk.runs = append(append(bk.runs[:f.at], run), rest...) + bk.files = append(append(bk.files[:f.at], rf), restF...) + rf.refs++ + return nil +} + +// Pack retires the history above the watermark: every bucket's runs +// and every pending delta fold into one sorted run, and the buckets +// start again. The retired run is the deep history's newest. +func (p *PermStore) Pack() error { + p.mu.RLock() + var inputs [][]permRecord + var release []*runFile + for _, d := range p.pending { + recs, err := d.run.records(d.rf.f) + if err != nil { + p.mu.RUnlock() + return err + } + inputs = append(inputs, recs) + release = append(release, d.rf) + } + for b := range p.buckets { + for i, r := range p.buckets[b].runs { + recs, err := r.records(p.buckets[b].files[i].f) + if err != nil { + p.mu.RUnlock() + return err + } + inputs = append(inputs, recs) + release = append(release, p.buckets[b].files[i]) + } + } + height := p.height + p.mu.RUnlock() + if len(inputs) == 0 { + return nil + } + merged := mergePermRuns(inputs) + p.mu.Lock() + rf := p.curRun + if rf.size > PermFileBytes { + var err error + if rf, err = p.newRunFile(); err != nil { + p.mu.Unlock() + return err + } + p.curRun = rf + } + at := rf.size + p.mu.Unlock() + run, err := writePermRun(rf.f, at, rf.id, merged, height) + if err != nil { + return err + } + if err = fsync(rf.f); err != nil { + return err + } + p.indexBytes.Add(uint64(run.bytes)) + p.packRuns.Add(1) + p.mu.Lock() + defer p.mu.Unlock() + rf.size = at + int64(run.bytes) + run.bloom = nil // Retired runs are probed cold + p.retired = append(p.retired, run) + rf.refs++ + for _, r := range release { + r.refs-- + } + p.pending = nil + for b := range p.buckets { + p.buckets[b] = permBucket{merged: height} + } + if err := p.commitManifest(); err != nil { + return err + } + return p.dropUnreferencedRunFiles() +} + +// commitManifest writes perm.json aside and renames it into place. +// The caller holds the lock. +func (p *PermStore) commitManifest() error { + m := permManifest{Version: 1, Height: p.height, FilterBlocks: p.window_n, Rotation: p.rotation, NextData: p.nextID, NextRun: p.nextRun} + ref := func(r *permRun, rf *runFile, height uint64) permRunRef { + return permRunRef{File: rf.id, Off: r.off, Height: height} + } + for _, d := range p.window { + m.Window = append(m.Window, ref(d.run, d.rf, d.height)) + } + for _, d := range p.pending { + m.Pending = append(m.Pending, ref(d.run, d.rf, d.height)) + } + for b := range p.buckets { + bm := permBucketM{Merged: p.buckets[b].merged} + for i, r := range p.buckets[b].runs { + bm.Runs = append(bm.Runs, ref(r, p.buckets[b].files[i], 0)) + } + m.Buckets = append(m.Buckets, bm) + } + for _, r := range p.retired { + m.Retired = append(m.Retired, permRunRef{File: r.file, Off: r.off, Height: r.height}) + } + for id := range p.files { + m.DataFiles = append(m.DataFiles, id) + } + for id := range p.runs { + m.RunFiles = append(m.RunFiles, id) + } + sort.Slice(m.DataFiles, func(i, j int) bool { return m.DataFiles[i] < m.DataFiles[j] }) + sort.Slice(m.RunFiles, func(i, j int) bool { return m.RunFiles[i] < m.RunFiles[j] }) + // Deltas sealed after this commit append to the current run file + // from its end; open replays from there + m.DeltasFile, m.DeltasOff = p.curRun.id, p.curRun.size + p.deltasFrom.file, p.deltasFrom.off = m.DeltasFile, m.DeltasOff + buf, err := json.MarshalIndent(m, "", " ") + if err != nil { + return err + } + path := filepath.Join(p.Directory, "perm.json") + tmp := path + segTmpSuffix + f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) + if err != nil { + return err + } + if _, err = f.Write(buf); err != nil { + f.Close() + return err + } + if err = fsync(f); err != nil { + f.Close() + return err + } + if err = f.Close(); err != nil { + return err + } + if err = os.Rename(tmp, path); err != nil { + return err + } + return fsyncDir(p.Directory) +} + +// dropUnreferencedRunFiles deletes run files no run references, after +// the manifest that no longer names them is durable. The caller +// holds the lock. +func (p *PermStore) dropUnreferencedRunFiles() error { + for id, rf := range p.runs { + if rf.refs > 0 || rf == p.curRun { + continue + } + rf.f.Close() + delete(p.runs, id) + if err := os.Remove(filepath.Join(p.Directory, permRunName(id))); err != nil { + return err + } + } + return nil +} + +// Stats is the store's report. +func (p *PermStore) Stats() StoreStats { + p.mu.RLock() + defer p.mu.RUnlock() + var runs int + for b := range p.buckets { + runs += len(p.buckets[b].runs) + } + return StoreStats{ + PutTotal: p.putTotal.Load(), + PutNew: p.putTotal.Load() - p.putDuplicate.Load(), + PutDuplicate: p.putDuplicate.Load(), + LookupTotal: p.lookups.Load(), + LiveHit: p.windowHits.Load(), + ActiveSegments: len(p.window), + HistorySegments: runs + len(p.retired), + HeapFiles: len(p.files) + len(p.runs), + HeapMovedBytes: p.indexBytes.Load(), + } +} + +// keyPrefixBucket is the bucket a key belongs to. +func keyPrefixBucket(key [32]byte) int { return int(key[0]) } + +var _ = strings.TrimSpace +var _ = strconv.Itoa +var _ = binary.LittleEndian +var _ = bytes.Compare +var _ = keyPrefixBucket diff --git a/database/perm_test.go b/database/perm_test.go new file mode 100644 index 0000000..94ce46d --- /dev/null +++ b/database/perm_test.go @@ -0,0 +1,119 @@ +package blockchainDB + +import ( + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +func sealPerm(t *testing.T, p *PermStore, height uint64) { + t.Helper() + s, err := p.beginSeal(height) + require.NoError(t, err) + require.NoError(t, s.finish()) +} + +// Records are written once and found in the window; past the window +// Get says absent and GetDeep finds them through the buckets and, +// after a pack, the retired run; the manifest and the deltas replay +// on reopen; a torn delta is dropped whole. +func TestPermStoreTiersAndReopen(t *testing.T) { + every := PermMergeEvery + PermMergeEvery = 4 // 64 buckets a merge + defer func() { PermMergeEvery = every }() + dir := filepath.Join(t.TempDir(), "perm") + p, err := NewPermStore(dir, MinFilterBlocks) + require.NoError(t, err) + fr := NewFastRandom([]byte{21}) + const perBlock = 300 + var keys [][32]byte + value := func(b uint64, i int) []byte { return []byte{byte(b >> 8), byte(b), byte(i >> 8), byte(i), 'v'} } + for b := uint64(1); b <= 3*MinFilterBlocks; b++ { + p.AdvanceBlock(b) + for i := 0; i < perBlock; i++ { + k := fr.NextHash() + keys = append(keys, k) + existing, existed, err := p.PutIfAbsent(k, value(b, i)) + require.NoError(t, err) + require.False(t, existed) + require.Nil(t, existing) + } + // A key written again in its block is refused with its value + existing, existed, err := p.PutIfAbsent(keys[len(keys)-1], []byte("other")) + require.NoError(t, err) + require.True(t, existed) + require.Equal(t, value(b, perBlock-1), existing) + sealPerm(t, p, b) + if b%5 == 0 { + require.NoError(t, p.Merge()) + } + } + // The window holds the last MinFilterBlocks blocks; older keys are + // absent to Get and present to GetDeep + inWindow := keys[2*MinFilterBlocks*perBlock:] + older := keys[:2*MinFilterBlocks*perBlock] + for i, k := range inWindow[:perBlock] { + v, err := p.Get(k) + require.NoError(t, err) + require.Equal(t, value(2*MinFilterBlocks+1, i), v) + } + for _, k := range older[:100] { + _, err := p.Get(k) + require.ErrorIs(t, err, errNotFound, "past the window, absent on the protocol path") + _, err = p.GetDeep(k) + require.NoError(t, err, "and found deep") + } + _, err = p.Get(fr.NextHash()) + require.ErrorIs(t, err, errNotFound) + st := p.Stats() + require.Greater(t, st.HistorySegments, 0, "buckets hold runs") + + // A pack retires the buckets and the pending deltas into one run + require.NoError(t, p.Pack()) + require.Len(t, p.retired, 1) + for b := range p.buckets { + require.Empty(t, p.buckets[b].runs) + } + for _, k := range older[:200] { + _, err = p.GetDeep(k) + require.NoError(t, err, "found in the retired run") + } + + // More blocks, sealed but never in a manifest, then reopen + for b := uint64(3*MinFilterBlocks + 1); b <= 3*MinFilterBlocks+5; b++ { + p.AdvanceBlock(b) + for i := 0; i < perBlock; i++ { + k := fr.NextHash() + keys = append(keys, k) + require.NoError(t, p.Put(k, value(b, i))) + } + sealPerm(t, p, b) + } + // A torn delta after the last seal: the crash + rf := p.curRun + _, err = rf.f.WriteAt([]byte("PRUNgarbage"), rf.size) + require.NoError(t, err) + last := keys[len(keys)-perBlock:] + for _, hf := range p.files { + hf.f.Close() + } + for _, rf := range p.runs { + rf.f.Close() + } + r, err := OpenPermStore(dir) + require.NoError(t, err) + defer r.Close() + require.EqualValues(t, 3*MinFilterBlocks+5, r.height-1+1, "the replayed deltas set the height") + for i, k := range last { + v, err := r.Get(k) + require.NoError(t, err, "a delta replayed after the manifest") + require.Equal(t, value(3*MinFilterBlocks+5, i), v) + } + for _, k := range older[:100] { + _, err = r.GetDeep(k) + require.NoError(t, err, "the retired run came back through the manifest") + } + _, err = r.Get(fr.NextHash()) + require.ErrorIs(t, err, errNotFound) +} diff --git a/database/permindex.go b/database/permindex.go index f18446b..6c09ac6 100644 --- a/database/permindex.go +++ b/database/permindex.go @@ -21,7 +21,7 @@ import ( // // A run on disk: // -// header magic "PRUN"(4) count(4) bloomBytes(4) bloomK(4) crc(4) +// header magic "PRUN"(4) count(4) bloomBytes(4) bloomK(4) crc(4) height(8) // records count sorted 44-byte records: key(32) file(4) off(4) len(4) // bloom bloomBytes of filter over the keys // @@ -38,7 +38,7 @@ type permRecord struct { const ( permRecSize = 32 + 4 + 4 + 4 - permRunHdr = 4 + 4 + 4 + 4 + 4 + permRunHdr = 4 + 4 + 4 + 4 + 4 + 8 permRunMagic = 0x5052554E // "PRUN" permBloomBits = 12 // Bits per key in a run's filter permBloomK = 3 @@ -64,9 +64,11 @@ func getPermRecord(buf []byte) (r permRecord) { // resident when loaded and probed cold otherwise. type permRun struct { path string - off int64 // Where the run starts in its file + file uint32 // The run file's id + off int64 // Where the run starts in its file count uint32 - bloomAt int64 // Where the filter starts + height uint64 // The block a delta is; the watermark a retired run is; 0 for a bucket's run + bloomAt int64 // Where the filter starts bloom *Bloom k int bytes uint32 @@ -75,7 +77,7 @@ type permRun struct { // writePermRun writes records, which must be sorted by key and free // of duplicates, as a run appended to w, and returns it. The caller // fsyncs the file. -func writePermRun(w io.WriterAt, at int64, path string, recs []permRecord) (*permRun, error) { +func writePermRun(w io.WriterAt, at int64, file uint32, recs []permRecord, height uint64) (*permRun, error) { if !sort.SliceIsSorted(recs, func(i, j int) bool { return bytes.Compare(recs[i].key[:], recs[j].key[:]) < 0 }) { return nil, errors.New("perm run: records are not sorted") } @@ -85,6 +87,7 @@ func writePermRun(w io.WriterAt, at int64, path string, recs []permRecord) (*per binary.LittleEndian.PutUint32(buf[4:], uint32(len(recs))) binary.LittleEndian.PutUint32(buf[8:], uint32(bloom.NumBytes)) binary.LittleEndian.PutUint32(buf[12:], uint32(bloom.K)) + binary.LittleEndian.PutUint64(buf[20:], height) p := permRunHdr for i, r := range recs { if i > 0 && recs[i-1].key == r.key { @@ -99,11 +102,12 @@ func writePermRun(w io.WriterAt, at int64, path string, recs []permRecord) (*per if _, err := w.WriteAt(buf, at); err != nil { return nil, err } - return &permRun{path: path, off: at, count: uint32(len(recs)), bloomAt: at + int64(p), bloom: bloom, k: bloom.K, bytes: uint32(len(buf))}, nil + return &permRun{path: permRunName(file), file: file, off: at, count: uint32(len(recs)), height: height, bloomAt: at + int64(p), bloom: bloom, k: bloom.K, bytes: uint32(len(buf))}, nil } // openPermRun reads a run's header at off in f and verifies the run. -func openPermRun(f *os.File, path string, off int64, resident bool) (*permRun, error) { +func openPermRun(f *os.File, file uint32, off int64, resident bool) (*permRun, error) { + path := permRunName(file) hdr := make([]byte, permRunHdr) if _, err := f.ReadAt(hdr, off); err != nil { return nil, err @@ -111,7 +115,7 @@ func openPermRun(f *os.File, path string, off int64, resident bool) (*permRun, e if binary.LittleEndian.Uint32(hdr) != permRunMagic { return nil, fmt.Errorf("perm run at %s:%d: bad magic", path, off) } - r := &permRun{path: path, off: off, count: binary.LittleEndian.Uint32(hdr[4:]), k: int(binary.LittleEndian.Uint32(hdr[12:]))} + r := &permRun{path: path, file: file, off: off, count: binary.LittleEndian.Uint32(hdr[4:]), k: int(binary.LittleEndian.Uint32(hdr[12:])), height: binary.LittleEndian.Uint64(hdr[20:])} bloomBytes := binary.LittleEndian.Uint32(hdr[8:]) r.bloomAt = off + permRunHdr + int64(r.count)*permRecSize r.bytes = permRunHdr + r.count*permRecSize + bloomBytes diff --git a/database/permindex_test.go b/database/permindex_test.go index 76a71d8..056f268 100644 --- a/database/permindex_test.go +++ b/database/permindex_test.go @@ -12,7 +12,7 @@ import ( // checksum verified, looked up resident and cold, merged newest-wins. func TestPermRunWriteLookupMerge(t *testing.T) { dir := t.TempDir() - path := filepath.Join(dir, "runs.dat") + path := filepath.Join(dir, permRunName(7)) f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0o644) require.NoError(t, err) defer f.Close() @@ -34,18 +34,19 @@ func TestPermRunWriteLookupMerge(t *testing.T) { } sortPermRecords(newer) - r1, err := writePermRun(f, 0, path, older) + r1, err := writePermRun(f, 0, 7, older, 1) require.NoError(t, err) - r2, err := writePermRun(f, int64(r1.bytes), path, newer) + r2, err := writePermRun(f, int64(r1.bytes), 7, newer, 2) require.NoError(t, err) - _, err = writePermRun(f, int64(r1.bytes+r2.bytes), path, []permRecord{older[3], older[2]}) + _, err = writePermRun(f, int64(r1.bytes+r2.bytes), 7, []permRecord{older[3], older[2]}, 3) require.Error(t, err, "unsorted records are refused") // Reopen: resident and cold - rr, err := openPermRun(f, path, 0, true) + rr, err := openPermRun(f, 7, 0, true) require.NoError(t, err) require.EqualValues(t, 5000, rr.count) - cold, err := openPermRun(f, path, int64(r1.bytes), false) + require.EqualValues(t, 1, rr.height) + cold, err := openPermRun(f, 7, int64(r1.bytes), false) require.NoError(t, err) require.Nil(t, cold.bloom) for _, rec := range older[:200] { @@ -89,6 +90,6 @@ func TestPermRunWriteLookupMerge(t *testing.T) { // A damaged run is refused _, err = f.WriteAt([]byte{0xff, 0xff}, permRunHdr+40) require.NoError(t, err) - _, err = openPermRun(f, path, 0, true) + _, err = openPermRun(f, 7, 0, true) require.ErrorContains(t, err, "checksum") } From 53f22b7a8bbc40d1575b1acc0fc05051d3478ae2 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:41:57 -0500 Subject: [PATCH 22/58] The permanent store behind KV2; a store with both layers as files KV2 asks its permanent layer through a small interface (permLayer): open, close, get, deep get, put, put-if-absent, live count, advance, the window, the seal's two halves and the merge. NewKV2Files opens the heap and the PermStore; OpenKV2 recognises the store by its perm.json; NewKVShardFilesN builds a sharded one. KVShard's pack calls a file-backed shard's own Pack (buckets retire into one run of keys, no set file), skips attaching sets and dropping history for it, and its stats and block advance go through the interface. bdbench -perm-files opens the files store. A shard round trip through seal, merge, pack, close and reopen finds permanent keys in the window, then deep, and dynamic keys at their last value. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/main.go | 7 +++- database/kv_2.go | 89 +++++++++++++++++++++++++++++++++++-------- database/kv_shard.go | 33 ++++++++++++++-- database/perm.go | 22 +++++++++++ database/perm_test.go | 74 +++++++++++++++++++++++++++++++++++ database/segstore.go | 22 +++++++++++ 6 files changed, 227 insertions(+), 20 deletions(-) diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index 3244551..508c638 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -65,6 +65,7 @@ type config struct { pprof string http string dynaHeap bool + permFiles bool phase bool } @@ -94,6 +95,7 @@ func parseFlags() (config, error) { flag.StringVar(&c.pprof, "pprof", "", "serve net/http/pprof on this address (e.g. 127.0.0.1:6061)") flag.StringVar(&c.http, "http", "127.0.0.1:8098", "serve the live page and the run's files here; empty disables") flag.BoolVar(&c.dynaHeap, "dyna-heap", false, "dynamic layer as a heap with holes (proposal 2026-09-16) instead of sealed segments") + flag.BoolVar(&c.permFiles, "perm-files", false, "both layers as files of entries: the heap and the permanent layer with buckets of keys (implies -dyna-heap)") flag.BoolVar(&c.phase, "maintenance-phase", false, "offset each store's maintenance cadence by its share of the period, so stores in lockstep do not all maintain at once") flag.Parse() if flag.NArg() != 0 { @@ -222,6 +224,9 @@ func openStore(c config, id int) (*store, error) { if c.dynaHeap { open = blockchainDB.NewKVShardHeapN } + if c.permFiles { + open = blockchainDB.NewKVShardFilesN + } kv, err := open(dir, c.shards, c.sealLimit) if err != nil { return nil, fmt.Errorf("open store %d: %w", id, err) @@ -504,7 +509,7 @@ func main() { "dir": c.dir, "stores": c.stores, "duration": c.duration.String(), "interval": c.interval.String(), "shards": c.shards, "sealLimit": c.sealLimit, "window": c.window, "compressEvery": c.compressEvery, "packEvery": c.packEvery, "dynaPuts": c.dynaPuts, "permPuts": c.permPuts, "reads": c.reads, "hotKeys": c.hotKeys, - "valueMin": c.valueMin, "valueMax": c.valueMax, "seed": c.seed, "dynaHeap": c.dynaHeap, "maintenancePhase": c.phase, "started": time.Now().UTC().Format(time.RFC3339), + "valueMin": c.valueMin, "valueMax": c.valueMax, "seed": c.seed, "dynaHeap": c.dynaHeap || c.permFiles, "permFiles": c.permFiles, "maintenancePhase": c.phase, "started": time.Now().UTC().Format(time.RFC3339), }, "", " ") if err := os.WriteFile(filepath.Join(c.dir, "run.json"), runJSON, 0o644); err != nil { fail("run.json", err) diff --git a/database/kv_2.go b/database/kv_2.go index 233f967..390d1e4 100644 --- a/database/kv_2.go +++ b/database/kv_2.go @@ -13,7 +13,8 @@ import ( const PermDirName = "perm" const DynaDirName = "dyna" -const HeapDirName = "dyna-heap" // The dynamic layer as a heap with holes (heap.go) +const HeapDirName = "dyna-heap" // The dynamic layer as a heap with holes (heap.go) +const PermFilesDirName = "perm-files" // The permanent layer as files of entries and buckets of keys (perm.go) // KV2 // Maintains 2 layers of key value pairs with different immutability characteristics: @@ -59,7 +60,8 @@ type KV2 struct { // every commit and read of the shard for the whole copy (issue #57). Mutex sync.RWMutex Directory string // Directory where the PermKV and DynaKV directories are - PermKV *SegmentStore // The Perm layer: sealed, immutable segments + PermKV *SegmentStore // The Perm layer: sealed, immutable segments; nil when Perm is set + Perm *PermStore // The Perm layer as files of entries and buckets of keys (perm.go); nil when PermKV is set DynaKV *SegmentStore // The Dyna layer: sealed, mutable segments; nil when Heap is set Heap *HeapStore // The Dyna layer as a heap with holes (heap.go); nil when DynaKV is set // dWrites and pWrites count writes to each layer since the last @@ -152,7 +154,7 @@ func (k *KV2) Writes() (dyna, perm int) { func (k *KV2) SetFilterBlocks(n uint64) (err error) { k.Mutex.Lock() defer k.Mutex.Unlock() - if err = k.PermKV.SetFilterBlocks(n); err != nil { + if err = k.perm().SetFilterBlocks(n); err != nil { return err } if k.DynaKV == nil { @@ -181,6 +183,50 @@ type dynaLayer interface { // shard's lock. type blockSync interface{ finish() error } +// permLayer is what KV2 asks of its permanent layer, whichever of the +// two it opened: sealed immutable segments (SegmentStore) or files of +// entries and buckets of keys (PermStore). +type permLayer interface { + Open() error + Close() error + Get(key [32]byte) ([]byte, error) + GetDeep(key [32]byte) ([]byte, error) + Put(key [32]byte, value []byte) error + PutIfAbsent(key [32]byte, value []byte) (existing []byte, existed bool, err error) + LiveCount() int + AdvanceBlock(height uint64) + SetFilterBlocks(n uint64) error + beginPermSeal(height uint64) (blockSync, error) + mergeBelow(height uint64) (bool, error) + Stats() StoreStats +} + +// perm is the permanent layer this shard opened. +func (k *KV2) perm() permLayer { + if k.Perm != nil { + return k.Perm + } + return k.PermKV +} + +// NewKV2Files is NewKV2 with both layers as files of entries: the +// dynamic layer a heap (heap.go), the permanent layer files with +// buckets of keys (perm.go). +func NewKV2Files(directory string, sealLimit uint64) (kv2 *KV2, err error) { + if kv2, err = NewKV2Heap(directory, sealLimit); err != nil { + return nil, err + } + if err = kv2.PermKV.Close(); err != nil { + return nil, err + } + os.RemoveAll(filepath.Join(directory, PermDirName)) + kv2.PermKV = nil + if kv2.Perm, err = NewPermStore(filepath.Join(directory, PermFilesDirName), MinFilterBlocks); err != nil { + return nil, err + } + return kv2, nil +} + // dyna is the dynamic layer this shard opened. func (k *KV2) dyna() dynaLayer { if k.Heap != nil { @@ -210,7 +256,12 @@ func OpenKV2(directory string) (kv2 *KV2, err error) { kv2.Directory = directory permDirName := filepath.Join(directory, PermDirName) // Add directory names dynaDirName := filepath.Join(directory, DynaDirName) // Add directory names - if kv2.PermKV, err = OpenSegmentStore(permDirName); err != nil { + // The permanent layer is whichever the store was built with + if _, statErr := os.Stat(filepath.Join(directory, PermFilesDirName, "perm.json")); statErr == nil { + if kv2.Perm, err = OpenPermStore(filepath.Join(directory, PermFilesDirName)); err != nil { + return nil, err + } + } else if kv2.PermKV, err = OpenSegmentStore(permDirName); err != nil { return nil, err } // The dynamic layer is whichever the store was built with @@ -235,13 +286,15 @@ func (k *KV2) Open() error { } k.Mutex.Lock() defer k.Mutex.Unlock() - if err := k.PermKV.Open(); err != nil { + if err := k.perm().Open(); err != nil { return err } if k.SealLimit == 0 { // Restore what the database was built with; only a store // predating the persisted field falls back to a default - if limit := k.PermKV.SealLimit; limit > 0 { + if k.PermKV == nil { + k.SealLimit = int(DefaultBloomCapacity) + } else if limit := k.PermKV.SealLimit; limit > 0 { k.SealLimit = int(limit) } else { k.SealLimit = int(DefaultBloomCapacity) @@ -268,7 +321,7 @@ func (k *KV2) Close() error { k.Mutex.Lock() defer k.Mutex.Unlock() k.opened.Store(false) - err := k.PermKV.Close() + err := k.perm().Close() if dynaErr := k.dyna().Close(); err == nil { err = dynaErr } @@ -300,7 +353,7 @@ func (k *KV2) GetPerm(key [32]byte) (value []byte, err error) { k.Mutex.RLock() defer k.Mutex.RUnlock() - if value, err = k.PermKV.Get(key); err != nil { // Not in PermKV, then return whatever + if value, err = k.perm().Get(key); err != nil { // Not in PermKV, then return whatever return nil, err } return value, nil @@ -314,7 +367,7 @@ func (k *KV2) GetPerm(key [32]byte) (value []byte, err error) { func (k *KV2) GetPermDeep(key [32]byte) (value []byte, err error) { k.Mutex.RLock() defer k.Mutex.RUnlock() - return k.PermKV.GetDeep(key) + return k.perm().GetDeep(key) } // Get @@ -353,7 +406,7 @@ func (k *KV2) Get(key [32]byte) (value []byte, err error) { // is how it says so. return nil, err } - return k.PermKV.Get(key) // Not in DynaKV; return whatever PermKV has + return k.perm().Get(key) // Not in DynaKV; return whatever PermKV has } // GetDeep @@ -371,7 +424,7 @@ func (k *KV2) GetDeep(key [32]byte) (value []byte, err error) { case !errors.Is(err, errNotFound): return nil, err // See Get: a failure is not an absence } - return k.PermKV.GetDeep(key) + return k.perm().GetDeep(key) } // PutDyna @@ -397,7 +450,7 @@ func (k *KV2) PutPerm(key [32]byte, value []byte) (writes int, err error) { k.Mutex.RLock() // Shared: see Put (issue #66) defer k.Mutex.RUnlock() k.pWrites.Add(1) - if err = k.PermKV.Put(key, value); err != nil { + if err = k.perm().Put(key, value); err != nil { return int(k.pWrites.Load()), err } autoSeal, err = k.sealPermIfFull() @@ -430,7 +483,7 @@ func finishAutoSeal(p *pendingSeal, err error) error { // published and under the exclusive lock in Open, and the layer // serializes its own seal. func (k *KV2) sealPermIfFull() (p *pendingSeal, err error) { - if k.SealLimit <= 0 || k.PermKV.LiveCount() < k.SealLimit { + if k.Perm != nil || k.SealLimit <= 0 || k.PermKV.LiveCount() < k.SealLimit { return nil, nil } return k.PermKV.beginSealNext() @@ -492,7 +545,7 @@ func (k *KV2) sealDynaIfFull() (p *pendingSeal, err error) { // out (issue #84; the two halves are seal.go's). func (k *KV2) Seal(height uint64) (meta SegmentMeta, err error) { k.Mutex.Lock() - perm, err := k.PermKV.beginSeal(height) + perm, err := k.perm().beginPermSeal(height) dyna, dynaErr := k.dyna().beginBlockSync() k.dyna().AdvanceBlock(height + 1) k.Mutex.Unlock() @@ -508,7 +561,7 @@ func (k *KV2) Seal(height uint64) (meta SegmentMeta, err error) { }() } if perm != nil { - meta, err = perm.finish() + err = perm.finish() } wg.Wait() if err == nil { @@ -530,6 +583,10 @@ func (k *KV2) Seal(height uint64) (meta SegmentMeta, err error) { // KV2 lock here held every Put, Get and Seal of the shard for the // whole copy, which is the pause issue #57 measures. func (k *KV2) MergeBelow(height uint64) (meta SegmentMeta, merged bool, err error) { + if k.Perm != nil { + merged, err = k.Perm.mergeBelow(height) + return meta, merged, err + } return k.PermKV.MergeBelow(height) } @@ -590,7 +647,7 @@ func (k *KV2) Put(key [32]byte, value []byte) (writes int, err error) { // whether the key was there and then calling Put, which asked again // to enforce immutability, made a new key -- the common case, and a // miss by definition -- pay for the answer twice. - existing, existed, err := k.PermKV.PutIfAbsent(key, value) + existing, existed, err := k.perm().PutIfAbsent(key, value) if err != nil { return int(k.dWrites.Load()), err } diff --git a/database/kv_shard.go b/database/kv_shard.go index d13d79c..e4d0061 100644 --- a/database/kv_shard.go +++ b/database/kv_shard.go @@ -103,6 +103,9 @@ func (k *KVShard) attachSets() (err error) { if shard == nil || shard.PermKV == nil { continue } + if shard.PermKV == nil { + continue // A file-backed permanent layer keeps its own deep history + } if err = shard.PermKV.attachCold(shardSets{k.Sets, i}); err != nil { return fmt.Errorf("shard %d: %w", i, err) } @@ -195,6 +198,9 @@ func OpenKVShard(directory string) (kVShard *KVShard, err error) { // again, which commits and retires them if newest, ok := kVShard.Sets.Newest(); ok { for i, shard := range kVShard.Shards { + if shard.PermKV == nil { + continue + } if _, err = shard.PermKV.DropBelow(newest.Last + 1); err != nil { return nil, fmt.Errorf("shard %d: %w", i, err) } @@ -233,6 +239,12 @@ func NewKVShardHeapN(directory string, shards int, sealLimit uint64) (kvs *KVSha return newKVShardN(directory, shards, sealLimit, NewKV2Heap) } +// NewKVShardFilesN is NewKVShardN with both layers as files of entries +// (heap.go, perm.go). +func NewKVShardFilesN(directory string, shards int, sealLimit uint64) (kvs *KVShard, err error) { + return newKVShardN(directory, shards, sealLimit, NewKV2Files) +} + func NewKVShardN(directory string, shards int, sealLimit uint64) (kvs *KVShard, err error) { return newKVShardN(directory, shards, sealLimit, NewKV2) } @@ -467,7 +479,7 @@ func (k *KVShard) adoptBlockHeight() error { continue } if shard.PermKV != nil { - shard.PermKV.AdvanceBlock(height) + shard.perm().AdvanceBlock(height) } if shard.DynaKV != nil { // So that its window is where the set's is shard.dyna().AdvanceBlock(height) @@ -481,7 +493,9 @@ func (k *KVShard) adoptBlockHeight() error { func (k *KVShard) useSharedBlockRecord() { for _, shard := range k.Shards { if shard != nil && shard.PermKV != nil { - shard.PermKV.ExternalBlockRecord = true + if shard.PermKV != nil { + shard.PermKV.ExternalBlockRecord = true + } } } } @@ -602,6 +616,19 @@ func (k *KVShard) MergeFinalized(height uint64) (mergedShards int, err error) { // pack is done, and whichever of the two lands first, the drop that // follows removes what the set holds. func (k *KVShard) PackFinalized(height uint64) (meta SetMeta, packed bool, err error) { + // A file-backed permanent layer packs within the shard: its + // buckets retire into one run of keys, no bodies, no set file + if len(k.Shards) > 0 && k.Shards[0].Perm != nil { + for i, shard := range k.Shards { + if err = shard.Open(); err != nil { + return meta, false, fmt.Errorf("shard %d: %w", i, err) + } + if err = shard.Perm.Pack(); err != nil { + return meta, false, fmt.Errorf("shard %d: %w", i, err) + } + } + return meta, true, nil + } // One pack at a time. Every guard in here and below is a // check-then-act on state another pack can change: two calls read // the same watermark and pack the same segments into overlapping @@ -726,7 +753,7 @@ func (k *KVShard) Stats() (perm, dyna StoreStats) { continue } if shard.PermKV != nil { - add(&perm, shard.PermKV.Stats()) + add(&perm, shard.perm().Stats()) } if shard.Heap != nil || shard.DynaKV != nil { add(&dyna, shard.dyna().Stats()) diff --git a/database/perm.go b/database/perm.go index 7afedc0..cc6cff6 100644 --- a/database/perm.go +++ b/database/perm.go @@ -1061,6 +1061,28 @@ func (p *PermStore) Stats() StoreStats { } } +// beginPermSeal and mergeBelow are the permLayer surface (kv_2.go). +func (p *PermStore) beginPermSeal(height uint64) (blockSync, error) { + s, err := p.beginSeal(height) + if err != nil { + return nil, err + } + return s, nil +} + +func (p *PermStore) mergeBelow(uint64) (bool, error) { return true, p.Merge() } + +// SetFilterBlocks sets the window. +func (p *PermStore) SetFilterBlocks(n uint64) error { + if n < MinFilterBlocks { + return fmt.Errorf("perm: filter blocks %d below the minimum %d", n, MinFilterBlocks) + } + p.mu.Lock() + p.window_n = n + p.mu.Unlock() + return nil +} + // keyPrefixBucket is the bucket a key belongs to. func keyPrefixBucket(key [32]byte) int { return int(key[0]) } diff --git a/database/perm_test.go b/database/perm_test.go index 94ce46d..a7a6958 100644 --- a/database/perm_test.go +++ b/database/perm_test.go @@ -117,3 +117,77 @@ func TestPermStoreTiersAndReopen(t *testing.T) { _, err = r.Get(fr.NextHash()) require.ErrorIs(t, err, errNotFound) } + +// A shard built with both layers as files seals, merges, packs, +// closes and reopens as files, through the same KV2 and KVShard +// surface; permanent keys are found in the window and, once past it, +// by GetDeep; dynamic keys keep their last value. +func TestFilesShardRoundTrip(t *testing.T) { + every := PermMergeEvery + PermMergeEvery = 8 + defer func() { PermMergeEvery = every }() + dir := filepath.Join(t.TempDir(), "shards") + kvs, err := NewKVShardFilesN(dir, 2, 1000) + require.NoError(t, err) + require.NoError(t, kvs.SetFilterBlocks(MinFilterBlocks)) + fr := NewFastRandom([]byte{9}) + hot := make([][32]byte, 100) + for i := range hot { + hot[i] = fr.NextHash() + } + var perm [][32]byte + for b := uint64(1); b <= 3*MinFilterBlocks; b++ { + for _, k := range hot { + require.NoError(t, kvs.PutDyna(k, append([]byte{byte(b)}, k[:4]...))) + } + for i := 0; i < 50; i++ { + k := fr.NextHash() + perm = append(perm, k) + require.NoError(t, kvs.PutPerm(k, append([]byte{byte(b)}, k[:4]...))) + } + require.NoError(t, kvs.SealBlock(b)) + if b%10 == 0 { + require.NoError(t, kvs.Compress()) + _, err := kvs.MergeFinalized(b) + require.NoError(t, err) + } + } + for _, k := range hot { + v, err := kvs.GetDyna(k) + require.NoError(t, err) + require.Equal(t, byte(3*MinFilterBlocks), v[0]) + } + recent := perm[len(perm)-50:] + old := perm[:50] + for _, k := range recent { + _, err := kvs.GetPerm(k) + require.NoError(t, err, "in the window") + } + for _, k := range old { + _, err := kvs.GetPerm(k) + require.ErrorIs(t, err, errNotFound, "past the window") + v, err := kvs.Shards[kvs.ShardIndex(k[:])].GetPermDeep(k) + require.NoError(t, err, "found deep") + require.Equal(t, byte(1), v[0]) + } + _, packed, err := kvs.PackFinalized(3 * MinFilterBlocks) + require.NoError(t, err) + require.True(t, packed) + require.NoError(t, kvs.Close()) + + re, err := OpenKVShard(dir) + require.NoError(t, err) + defer re.Close() + require.NotNil(t, re.Shards[0].Perm, "reopened as files") + require.NotNil(t, re.Shards[0].Heap) + for _, k := range hot { + v, err := re.GetDyna(k) + require.NoError(t, err) + require.Equal(t, byte(3*MinFilterBlocks), v[0]) + } + for _, k := range old { + v, err := re.Shards[re.ShardIndex(k[:])].GetPermDeep(k) + require.NoError(t, err, "the retired run came back") + require.Equal(t, byte(1), v[0]) + } +} diff --git a/database/segstore.go b/database/segstore.go index 93dac2e..0543e5d 100644 --- a/database/segstore.go +++ b/database/segstore.go @@ -2734,6 +2734,28 @@ func (s *SegmentStore) beginBlockSync() (blockSync, error) { func (s *SegmentStore) compact() (bool, error) { return s.CompactHistory() } +// beginPermSeal and mergeBelow are the permLayer surface (kv_2.go) +// over beginSeal and MergeBelow. +func (s *SegmentStore) beginPermSeal(height uint64) (blockSync, error) { + p, err := s.beginSeal(height) + if err != nil { + return nil, err + } + if p == nil { + return nil, nil + } + return segSeal{p}, nil +} + +type segSeal struct{ p *pendingSeal } + +func (s segSeal) finish() error { _, err := s.p.finish(); return err } + +func (s *SegmentStore) mergeBelow(height uint64) (bool, error) { + _, merged, err := s.MergeBelow(height) + return merged, err +} + func (s *SegmentStore) CompactHistory() (compacted bool, err error) { s.maint.Lock() defer s.maint.Unlock() From 0a7392c1338be229638545cd59c9b9d77890c0c2 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:42:43 -0500 Subject: [PATCH 23/58] perm.go: the draft's placeholders removed Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 82 ++++++++++++------------------------------------ 1 file changed, 20 insertions(+), 62 deletions(-) diff --git a/database/perm.go b/database/perm.go index cc6cff6..f41dc0f 100644 --- a/database/perm.go +++ b/database/perm.go @@ -1,16 +1,12 @@ package blockchainDB import ( - "bytes" - "encoding/binary" "encoding/json" "errors" "fmt" "os" "path/filepath" "sort" - "strconv" - "strings" "sync" "sync/atomic" ) @@ -157,21 +153,20 @@ func OpenPermStore(directory string) (*PermStore, error) { // permManifest is perm.json. type permManifest struct { - Version int `json:"version"` - Height uint64 `json:"height"` - FilterBlocks uint64 `json:"filterBlocks"` - Rotation int `json:"rotation"` - NextData uint32 `json:"nextData"` - NextRun uint32 `json:"nextRun"` - DeltasFile uint32 `json:"deltasFile"` // Deltas after this file:offset are replayed - DeltasOff int64 `json:"deltasOff"` - Window []permRunRef `json:"window"` - Pending []permRunRef `json:"pending"` - Buckets []permBucketM `json:"buckets"` - Retired []permRunRef `json:"retired"` - DataFiles []uint32 `json:"dataFiles"` - RunFiles []uint32 `json:"runFiles"` - Extra map[string]any `json:"-"` + Version int `json:"version"` + Height uint64 `json:"height"` + FilterBlocks uint64 `json:"filterBlocks"` + Rotation int `json:"rotation"` + NextData uint32 `json:"nextData"` + NextRun uint32 `json:"nextRun"` + DeltasFile uint32 `json:"deltasFile"` // Deltas after this file:offset are replayed + DeltasOff int64 `json:"deltasOff"` + Window []permRunRef `json:"window"` + Pending []permRunRef `json:"pending"` + Buckets []permBucketM `json:"buckets"` + Retired []permRunRef `json:"retired"` + DataFiles []uint32 `json:"dataFiles"` + RunFiles []uint32 `json:"runFiles"` } type permRunRef struct { @@ -335,12 +330,6 @@ func (p *PermStore) replayDeltas() error { off += int64(r.bytes) } } - // Entries past the last named one are a crash's leftovers - var end int64 - for _, d := range append(p.window, p.pending...) { - _ = d - } - _ = end return nil } @@ -663,27 +652,10 @@ func (p *PermStore) Merge() error { due = 1 } newest := p.pending[len(p.pending)-1].height - type job struct { - b int - recs []permRecord - from []*permDelta - fold []*permRun - foldF []*runFile - foldI int - } + type job struct{ b int } var jobs []job for i := 0; i < due; i++ { - b := (p.rotation + i) % PermBuckets - bk := &p.buckets[b] - var j job - j.b = b - for _, d := range p.pending { - if d.height <= bk.merged { - continue - } - j.from = append(j.from, d) - } - jobs = append(jobs, j) + jobs = append(jobs, job{b: (p.rotation + i) % PermBuckets}) } rotation := (p.rotation + due) % PermBuckets // Read the deltas' records for these buckets outside the lock: @@ -714,10 +686,8 @@ func (p *PermStore) Merge() error { rf := p.curRun p.mu.Unlock() var written []struct { - b int - run *permRun - fold bool - at int + b int + run *permRun } for _, j := range jobs { recs := mergePermRuns(byBucket[j.b]) @@ -734,7 +704,6 @@ func (p *PermStore) Merge() error { p.curRun = rf } at := rf.size - rf.size += 0 p.mu.Unlock() run, err := writePermRun(rf.f, at, rf.id, recs, 0) if err != nil { @@ -746,10 +715,8 @@ func (p *PermStore) Merge() error { p.indexBytes.Add(uint64(run.bytes)) p.mergeRuns.Add(1) written = append(written, struct { - b int - run *permRun - fold bool - at int + b int + run *permRun }{b: j.b, run: run}) } if err := fsync(rf.f); err != nil { @@ -1082,12 +1049,3 @@ func (p *PermStore) SetFilterBlocks(n uint64) error { p.mu.Unlock() return nil } - -// keyPrefixBucket is the bucket a key belongs to. -func keyPrefixBucket(key [32]byte) int { return int(key[0]) } - -var _ = strings.TrimSpace -var _ = strconv.Itoa -var _ = binary.LittleEndian -var _ = bytes.Compare -var _ = keyPrefixBucket From 48cdf726161ff88b0f8bbb5f38fb798affa73291 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 18:55:48 -0500 Subject: [PATCH 24/58] The heap's snapshot is written outside the sync mutex Every hundredth block each shard wrote its whole key map -- 3 MB -- while holding syncMu, so the block's sync waited behind the snapshot's write and fsync on all nine stores at the same block (seal p90 276 ms in those minutes with the mover nearly idle). The snapshot is now encoded under the map's read lock and written and fsynced with no lock held; syncMu is taken only to copy the deltas appended meanwhile after it, rename it into place and switch. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 98 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 79 insertions(+), 19 deletions(-) diff --git a/database/heap.go b/database/heap.go index 79be960..0defbb4 100644 --- a/database/heap.go +++ b/database/heap.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "hash/crc32" + "io" "os" "path/filepath" "sort" @@ -250,7 +251,12 @@ func (h *HeapStore) Open() (err error) { } } else { h.gen = 1 - if err = h.startGeneration(); err != nil { + // A fresh store: the first generation, written with the locks + // released since startGeneration takes them itself + h.mu.Unlock() + err = h.startGeneration() + h.mu.Lock() + if err != nil { return err } } @@ -288,24 +294,42 @@ func (h *HeapStore) replayGeneration() (err error) { return nil } -// startGeneration begins an index generation with a snapshot of the -// map, written aside and renamed into place, then fsynced; the -// previous generation's file is removed once the new one is durable. -// The caller holds syncMu and the lock (Open holds the lock alone, -// with nothing else running). +// startGeneration begins an index generation: a snapshot of the map, +// written aside and fsynced, then the deltas appended to the old +// generation meanwhile copied after it, renamed into place, the +// directory fsynced, and the previous generation's file removed once +// the new one is durable. The snapshot itself is written with no +// lock held but the map's read lock; only the tail copy and the +// switch hold syncMu, so a block's sync waits milliseconds for a +// snapshot, not for 3 MB of map (measured: seal p90 276 ms at every +// hundredth block with the whole write under syncMu). Open calls it +// with nothing else running. func (h *HeapStore) startGeneration() error { - path := filepath.Join(h.Directory, indexName(h.gen)) - tmp := path + segTmpSuffix - f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) - if err != nil { - return err + next := h.gen + if h.log != nil { + next = h.gen + 1 // A snapshot starts the generation after the current one } + path := filepath.Join(h.Directory, indexName(next)) + tmp := path + segTmpSuffix + // 1. The map as of now, and where the old generation's log ends: + // deltas after that point are copied over below + h.mu.RLock() all := func(emit func(key [32]byte)) { for key := range h.index { emit(key) } } - if _, err = f.Write(h.encodeIndexOf(heapSnapshot, all, len(h.index))); err != nil { + snap := h.encodeIndexOf(heapSnapshot, all, len(h.index)) + var copiedTo int64 + if h.log != nil { + copiedTo, _ = h.log.Seek(0, io.SeekEnd) + } + h.mu.RUnlock() + f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) + if err != nil { + return err + } + if _, err = f.Write(snap); err != nil { f.Close() return err } @@ -313,6 +337,29 @@ func (h *HeapStore) startGeneration() error { f.Close() return err } + // 2. Under syncMu: no delta is in flight, so the old log's tail + // past copiedTo is exactly the deltas since the snapshot + h.syncMu.Lock() + defer h.syncMu.Unlock() + h.mu.Lock() + defer h.mu.Unlock() + if h.log != nil { + rest, err := readFrom(h.log, copiedTo) + if err != nil { + f.Close() + return err + } + if len(rest) > 0 { + if _, err = f.Write(rest); err != nil { + f.Close() + return err + } + if err = fsync(f); err != nil { + f.Close() + return err + } + } + } if err = f.Close(); err != nil { return err } @@ -328,11 +375,26 @@ func (h *HeapStore) startGeneration() error { } if old != nil { old.Close() - os.Remove(filepath.Join(h.Directory, indexName(h.gen-1))) + os.Remove(filepath.Join(h.Directory, indexName(h.gen))) } + h.gen = next return nil } +// readFrom reads a file from off to its end. +func readFrom(f *os.File, off int64) ([]byte, error) { + end, err := f.Seek(0, io.SeekEnd) + if err != nil { + return nil, err + } + if end <= off { + return nil, nil + } + buf := make([]byte, end-off) + _, err = f.ReadAt(buf, off) + return buf, err +} + // fsyncDir makes a directory's entries durable: a rename or an unlink // is not on disk until the directory is. func fsyncDir(directory string) error { @@ -756,14 +818,12 @@ func (h *HeapStore) compact() (bool, error) { // the state of the last finished delta and no delta lands in the // generation being retired. Off the protocol path. func (h *HeapStore) Snapshot() error { - h.syncMu.Lock() - defer h.syncMu.Unlock() - h.mu.Lock() - defer h.mu.Unlock() - if h.closed { + h.mu.RLock() + closed := h.closed + h.mu.RUnlock() + if closed { return errStoreClosed } - h.gen++ return h.startGeneration() } From 1b28c413391b82484fe7ee403edba4f6219de691 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 19:01:45 -0500 Subject: [PATCH 25/58] A data file rolls by age as well as by size At 64 shards a shard appends ~34 KB a block, its 16 MB file takes eight minutes to roll, and the current file is never the mover's: no dead byte was reclaimed and the store grew 2.3 GB a minute. A file that has served HeapFileBlocks (64) blocks rolls whatever its size. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/database/heap.go b/database/heap.go index 0defbb4..cf3e4f7 100644 --- a/database/heap.go +++ b/database/heap.go @@ -101,9 +101,10 @@ type heapFile struct { f *os.File size int64 live, dead int64 - cleaning bool // Taken by the pass in progress - inflight int // Copies reserved in it and not yet written: its size runs ahead of its bytes - releasing bool // Emptied by a pass; deleted by the next sync + firstBlock uint64 // The block that first appended to it + cleaning bool // Taken by the pass in progress + inflight int // Copies reserved in it and not yet written: its size runs ahead of its bytes + releasing bool // Emptied by a pass; deleted by the next sync } // slot is where an entry lives: its file, its offset there, the value @@ -126,9 +127,17 @@ const ( heapIndexRec = 32 + 4 + 4 + 4 ) -// HeapFileBytes is the size a data file is rolled at. +// HeapFileBytes is the size a data file is rolled at, and +// HeapFileBlocks the age: a shard that appends little per block (many +// shards, or a quiet one) would otherwise keep one file current for +// long stretches, and the current file is never the mover's, so its +// dead bytes could not be reclaimed (measured at 64 shards: 34 KB a +// block, no file rolled in three minutes, the store growing 2.3 GB a +// minute with the mover idle). var HeapFileBytes int64 = 16 << 20 +var HeapFileBlocks uint64 = 64 + // HeapCleanBytes bounds one mover pass by the bytes it copies. The // pass syncs its own copies, so the bound is not about a block's // barrier but about the device queue the barrier shares: 16 MB @@ -588,10 +597,11 @@ func (h *HeapStore) reserve(mover bool, size int64) (hf *heapFile, off int64, er if mover { at = &h.mov } - if *at == nil || (*at).size+size > HeapFileBytes { + if *at == nil || (*at).size+size > HeapFileBytes || (!mover && h.height >= (*at).firstBlock+HeapFileBlocks) { if *at, err = h.newFile(); err != nil { return nil, 0, err } + (*at).firstBlock = h.height } hf = *at off = hf.size From 737f783e36c507ea82a2e3f076e258495b98e33b Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 19:24:36 -0500 Subject: [PATCH 26/58] The permanent store's maintenance writes to its own file, synced once a merge Under the full load (files5) the seal climbed from 59 to 209 ms p50 in five minutes: merges and folds appended their runs to the same run file the seal appends deltas to, so the block's fsync flushed the merge's writes, and every fold took a barrier of its own -- up to 256 a merge per shard. Maintenance now has a run file of its own (runs-N.dat), the seal's deltas theirs (deltas-N.dat), a merge syncs its runs and folds once before the manifest names them, and open replays deltas from the seal's files alone. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 191 +++++++++++++++++++++++++++++++---------------- 1 file changed, 127 insertions(+), 64 deletions(-) diff --git a/database/perm.go b/database/perm.go index f41dc0f..19ba4ba 100644 --- a/database/perm.go +++ b/database/perm.go @@ -22,10 +22,14 @@ import ( // perm-N.dat entries, [len][height][key][value][crc] (heap.go's // layout), appended in the order written, rolled at // PermFileBytes; never rewritten -// runs-N.dat runs of records (permindex.go), appended: a block's -// delta at every seal, a bucket's run at every merge, -// a retired run at every pack; rolled at PermFileBytes -// and deleted once no run in it is referenced +// deltas-N.dat the seal's run file: a block's delta appended at every +// seal; open replays the deltas after the manifest from +// these files alone +// runs-N.dat maintenance's run file: a bucket's run at every merge, +// a retired run at every pack; never the seal's, so their +// barriers never share an inode; both kinds are rolled +// at PermFileBytes and deleted once no run in them is +// referenced // perm.json the manifest: which runs are the window, the buckets // and the retired history, and where in the run files // the deltas not yet in the manifest begin @@ -67,7 +71,8 @@ type PermStore struct { dirty map[uint32]*heapFile runs map[uint32]*runFile // Run files, by id - curRun *runFile + curRun *runFile // The run file the seal appends deltas to + maintRun *runFile // The run file maintenance appends to: never the seal's, so their barriers never share an inode nextRun uint32 live map[[32]byte]permRecord // This block's records window []*permDelta // The last FilterBlocks deltas, oldest first @@ -122,14 +127,23 @@ type permBucket struct { // runFile is a file of runs and how many runs still reference it. type runFile struct { - id uint32 - f *os.File - size int64 - refs int + id uint32 + f *os.File + size int64 + refs int + deltas bool // The seal's, replayed on open; else maintenance's } -func permDataName(id uint32) string { return fmt.Sprintf("perm-%06d.dat", id) } -func permRunName(id uint32) string { return fmt.Sprintf("runs-%06d.dat", id) } +func permDataName(id uint32) string { return fmt.Sprintf("perm-%06d.dat", id) } +func permRunName(id uint32) string { return fmt.Sprintf("runs-%06d.dat", id) } +func permDeltaName(id uint32) string { return fmt.Sprintf("deltas-%06d.dat", id) } + +func (rf *runFile) name() string { + if rf.deltas { + return permDeltaName(rf.id) + } + return permRunName(rf.id) +} // NewPermStore creates an empty store in directory, replacing anything // there. @@ -167,6 +181,7 @@ type permManifest struct { Retired []permRunRef `json:"retired"` DataFiles []uint32 `json:"dataFiles"` RunFiles []uint32 `json:"runFiles"` + DeltaFiles []uint32 `json:"deltaFiles"` } type permRunRef struct { @@ -203,7 +218,7 @@ func (p *PermStore) Open() (err error) { if p.cur, err = p.newDataFile(); err != nil { return err } - if p.curRun, err = p.newRunFile(); err != nil { + if p.curRun, err = p.newRunFile(true); err != nil { return err } return p.commitManifest() @@ -233,6 +248,30 @@ func (p *PermStore) Open() (err error) { st, _ := f.Stat() p.runs[id] = &runFile{id: id, f: f, size: st.Size()} } + // Delta files: the manifest's, and any the seal created after it + deltaIDs := append([]uint32(nil), m.DeltaFiles...) + if entries, err := os.ReadDir(p.Directory); err == nil { + for _, e := range entries { + var id uint32 + if n, _ := fmt.Sscanf(e.Name(), "deltas-%06d.dat", &id); n == 1 && id > m.NextRun-1 { + deltaIDs = append(deltaIDs, id) + } + } + } + for _, id := range deltaIDs { + if p.runs[id] != nil { + continue + } + f, err := os.OpenFile(filepath.Join(p.Directory, permDeltaName(id)), os.O_RDWR, 0o644) + if err != nil { + return fmt.Errorf("perm: delta file %s: %w", permDeltaName(id), err) + } + st, _ := f.Stat() + p.runs[id] = &runFile{id: id, f: f, size: st.Size(), deltas: true} + if id >= p.nextRun { + p.nextRun = id + 1 + } + } load := func(ref permRunRef, resident bool) (*permRun, *runFile, error) { rf := p.runs[ref.File] if rf == nil { @@ -280,15 +319,17 @@ func (p *PermStore) Open() (err error) { } p.retired = append(p.retired, r) } - // The current files are the newest + // The current data file is the newest; the seal's run file is the + // manifest's; maintenance opens a file of its own when it runs for _, hf := range p.files { if p.cur == nil || hf.id > p.cur.id { p.cur = hf } } - for _, rf := range p.runs { - if p.curRun == nil || rf.id > p.curRun.id { - p.curRun = rf + p.curRun = p.runs[m.DeltasFile] + if p.curRun == nil { + if p.curRun, err = p.newRunFile(true); err != nil { + return err } } p.deltasFrom.file, p.deltasFrom.off = m.DeltasFile, m.DeltasOff @@ -300,8 +341,8 @@ func (p *PermStore) Open() (err error) { // holds the lock. func (p *PermStore) replayDeltas() error { ids := make([]uint32, 0, len(p.runs)) - for id := range p.runs { - if id >= p.deltasFrom.file { + for id, rf := range p.runs { + if rf.deltas && id >= p.deltasFrom.file { ids = append(ids, id) } } @@ -329,6 +370,10 @@ func (p *PermStore) replayDeltas() error { } off += int64(r.bytes) } + // The newest delta file is the seal's current one + if p.curRun == nil || rf.id > p.curRun.id { + p.curRun = rf + } } return nil } @@ -355,14 +400,15 @@ func (p *PermStore) newDataFile() (*heapFile, error) { return hf, nil } -func (p *PermStore) newRunFile() (*runFile, error) { +func (p *PermStore) newRunFile(deltas bool) (*runFile, error) { id := p.nextRun p.nextRun++ - f, err := os.OpenFile(filepath.Join(p.Directory, permRunName(id)), os.O_RDWR|os.O_CREATE|os.O_EXCL, 0o644) + rf := &runFile{id: id, deltas: deltas} + f, err := os.OpenFile(filepath.Join(p.Directory, rf.name()), os.O_RDWR|os.O_CREATE|os.O_EXCL, 0o644) if err != nil { return nil, err } - rf := &runFile{id: id, f: f} + rf.f = f p.runs[id] = rf return rf, nil } @@ -582,7 +628,7 @@ func (s *permSeal) finish() error { rf := p.curRun if rf.size > PermFileBytes { var err error - if rf, err = p.newRunFile(); err != nil { + if rf, err = p.newRunFile(true); err != nil { p.mu.Unlock() return err } @@ -631,6 +677,19 @@ func (p *PermStore) LiveCount() int { return len(p.live) } +// maintFile is the run file maintenance appends to, rolled at +// PermFileBytes. The caller holds the lock. +func (p *PermStore) maintFile() (*runFile, error) { + if p.maintRun == nil || p.maintRun.size > PermFileBytes { + rf, err := p.newRunFile(false) + if err != nil { + return nil, err + } + p.maintRun = rf + } + return p.maintRun, nil +} + // Merge is the maintenance step: the buckets due in rotation take // their records from the pending deltas, fold, and the manifest is // committed; deltas every bucket has absorbed are released. Runs @@ -681,10 +740,13 @@ func (p *PermStore) Merge() error { byBucket[j.b] = append(byBucket[j.b], mine) } } - // Write each due bucket's new run and its fold, in the current run file + // Write each due bucket's new run to the maintenance run file p.mu.Lock() - rf := p.curRun + rf, err := p.maintFile() p.mu.Unlock() + if err != nil { + return err + } var written []struct { b int run *permRun @@ -695,15 +757,8 @@ func (p *PermStore) Merge() error { continue } p.mu.Lock() - if rf.size > PermFileBytes { - var err error - if rf, err = p.newRunFile(); err != nil { - p.mu.Unlock() - return err - } - p.curRun = rf - } at := rf.size + rf.size += 0 p.mu.Unlock() run, err := writePermRun(rf.f, at, rf.id, recs, 0) if err != nil { @@ -753,8 +808,20 @@ func (p *PermStore) Merge() error { p.pending = keep folds := p.planFolds() p.mu.Unlock() + // The folds' runs are written without a barrier each and synced + // once, before the manifest that names them + var synced []*runFile for _, f := range folds { - if err := p.fold(f); err != nil { + rf, err := p.fold(f) + if err != nil { + return err + } + if rf != nil && (len(synced) == 0 || synced[len(synced)-1] != rf) { + synced = append(synced, rf) + } + } + for _, rf := range synced { + if err := fsync(rf.f); err != nil { return err } } @@ -795,14 +862,15 @@ func (p *PermStore) planFolds() (folds []permFold) { return folds } -// fold merges a bucket's chosen runs into one, written to the current -// run file, and swaps it in under the lock. -func (p *PermStore) fold(f permFold) error { +// fold merges a bucket's chosen runs into one, written to the +// maintenance run file (returned, for the caller to sync), and swaps +// it in under the lock. +func (p *PermStore) fold(f permFold) (*runFile, error) { p.mu.RLock() bk := &p.buckets[f.b] if f.at+f.count > len(bk.runs) { p.mu.RUnlock() - return nil // The bucket changed under us; next time + return nil, nil // The bucket changed under us; next time } runs := append([]*permRun(nil), bk.runs[f.at:f.at+f.count]...) files := append([]*runFile(nil), bk.files[f.at:f.at+f.count]...) @@ -811,29 +879,23 @@ func (p *PermStore) fold(f permFold) error { for i, r := range runs { recs, err := r.records(files[i].f) if err != nil { - return err + return nil, err } inputs[i] = recs } merged := mergePermRuns(inputs) p.mu.Lock() - rf := p.curRun - if rf.size > PermFileBytes { - var err error - if rf, err = p.newRunFile(); err != nil { - p.mu.Unlock() - return err - } - p.curRun = rf + rf, err := p.maintFile() + if err != nil { + p.mu.Unlock() + return nil, err } at := rf.size + rf.size += 0 p.mu.Unlock() run, err := writePermRun(rf.f, at, rf.id, merged, 0) if err != nil { - return err - } - if err = fsync(rf.f); err != nil { - return err + return nil, err } p.indexBytes.Add(uint64(run.bytes)) p.foldRuns.Add(1) @@ -842,7 +904,7 @@ func (p *PermStore) fold(f permFold) error { rf.size = at + int64(run.bytes) bk = &p.buckets[f.b] if f.at+f.count > len(bk.runs) { - return nil + return rf, nil } for _, old := range files { old.refs-- @@ -852,7 +914,7 @@ func (p *PermStore) fold(f permFold) error { bk.runs = append(append(bk.runs[:f.at], run), rest...) bk.files = append(append(bk.files[:f.at], rf), restF...) rf.refs++ - return nil + return rf, nil } // Pack retires the history above the watermark: every bucket's runs @@ -889,14 +951,10 @@ func (p *PermStore) Pack() error { } merged := mergePermRuns(inputs) p.mu.Lock() - rf := p.curRun - if rf.size > PermFileBytes { - var err error - if rf, err = p.newRunFile(); err != nil { - p.mu.Unlock() - return err - } - p.curRun = rf + rf, err := p.maintFile() + if err != nil { + p.mu.Unlock() + return err } at := rf.size p.mu.Unlock() @@ -954,11 +1012,16 @@ func (p *PermStore) commitManifest() error { for id := range p.files { m.DataFiles = append(m.DataFiles, id) } - for id := range p.runs { - m.RunFiles = append(m.RunFiles, id) + for id, rf := range p.runs { + if rf.deltas { + m.DeltaFiles = append(m.DeltaFiles, id) + } else { + m.RunFiles = append(m.RunFiles, id) + } } sort.Slice(m.DataFiles, func(i, j int) bool { return m.DataFiles[i] < m.DataFiles[j] }) sort.Slice(m.RunFiles, func(i, j int) bool { return m.RunFiles[i] < m.RunFiles[j] }) + sort.Slice(m.DeltaFiles, func(i, j int) bool { return m.DeltaFiles[i] < m.DeltaFiles[j] }) // Deltas sealed after this commit append to the current run file // from its end; open replays from there m.DeltasFile, m.DeltasOff = p.curRun.id, p.curRun.size @@ -995,12 +1058,12 @@ func (p *PermStore) commitManifest() error { // holds the lock. func (p *PermStore) dropUnreferencedRunFiles() error { for id, rf := range p.runs { - if rf.refs > 0 || rf == p.curRun { + if rf.refs > 0 || rf == p.curRun || rf == p.maintRun { continue } rf.f.Close() delete(p.runs, id) - if err := os.Remove(filepath.Join(p.Directory, permRunName(id))); err != nil { + if err := os.Remove(filepath.Join(p.Directory, rf.name())); err != nil { return err } } From 16b1599135e2fdcf928ae591887f2a6316279eb3 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 19:25:16 -0500 Subject: [PATCH 27/58] The permanent store's counters on the live page; the shard-count finding in the proposal Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/live.html | 2 +- cmd/bdbench/main.go | 6 +++++ database/perm.go | 6 +++++ .../2026-09-16-entries-written-once.md | 25 +++++++++++++++---- 4 files changed, 33 insertions(+), 6 deletions(-) diff --git a/cmd/bdbench/live.html b/cmd/bdbench/live.html index 0123149..f789bd6 100644 --- a/cmd/bdbench/live.html +++ b/cmd/bdbench/live.html @@ -39,7 +39,7 @@

bdbench live

document.getElementById("now").innerHTML=[ group("Now: seals, last 10 s",[["p50",L.sealP50ms.toFixed(0)+" ms","big"],["p90 (budget 100)",L.sealP90ms.toFixed(0)+" ms",L.sealP90ms>100?"warn":"ok"],["max",L.sealMaxMs.toFixed(0)+" ms"]]), group("Now: blocks, last 10 s",[["p50",L.blockP50ms.toFixed(0)+" ms","big"],["p90 (interval 1000)",L.blockP90ms.toFixed(0)+" ms",L.blockP90ms>1000?"bad":"ok"],["max",L.blockMaxMs.toFixed(0)+" ms"],["over the interval",L.over+" of "+L.blocks,L.over>0?"bad":"ok"]]), - group("Now: maintenance and heap",[["passes in flight",l.maintenanceInFlight,"big"],["heap live / dead",l.heapLiveMB.toFixed(0)+" / "+l.heapHoleMB.toFixed(0)+" MB"],["cleaner scanned / copied",l.heapScannedMB.toFixed(0)+" / "+l.heapMovedMB.toFixed(0)+" MB"],["shard sync: heap fsync / delta",l.heapFsyncMsAvg.toFixed(1)+" / "+l.heapDeltaMsAvg.toFixed(1)+" ms avg"],["shard sync covers",l.heapSyncKBAvg.toFixed(0)+" KB avg"],["wrong answers so far",l.mismatches,l.mismatches>0?"bad":"ok"]])].join("")}catch(e){}} + group("Now: maintenance and heap",[["passes in flight",l.maintenanceInFlight,"big"],["heap live / dead",l.heapLiveMB.toFixed(0)+" / "+l.heapHoleMB.toFixed(0)+" MB"],["cleaner scanned / copied",l.heapScannedMB.toFixed(0)+" / "+l.heapMovedMB.toFixed(0)+" MB"],["shard sync: heap fsync / delta",l.heapFsyncMsAvg.toFixed(1)+" / "+l.heapDeltaMsAvg.toFixed(1)+" ms avg"],["shard sync covers",l.heapSyncKBAvg.toFixed(0)+" KB avg"],["perm merges / folds / packs",l.permMerges+" / "+l.permFolds+" / "+l.permPacks],["perm index written",(l.permIndexMB||0).toFixed(0)+" MB"],["wrong answers so far",l.mismatches,l.mismatches>0?"bad":"ok"]])].join("")}catch(e){}} async function tick(){let t;try{t=await (await fetch("bdbench.csv?"+Date.now())).text()}catch(e){document.getElementById("state").textContent="unreachable";return} const rows=parse(t),last=rows[rows.length-1]||{};const st=document.getElementById("state"); if(!rows.length)st.textContent="waiting for the first minute"; diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index 508c638..90c4d1c 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -441,6 +441,7 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { var holes, live int64 var scanned, moved, syncs, syncBytes uint64 var heapFsync, deltaSync time.Duration + var permMerges, permFolds, permPacks, permIndexBytes uint64 for _, s := range stores { if s.height > height { height = s.height @@ -454,6 +455,10 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { n, b, hf, ds := sh.Heap.SyncCost() syncs, syncBytes, heapFsync, deltaSync = syncs+n, syncBytes+b, heapFsync+hf, deltaSync+ds } + if sh.Perm != nil { + m, f, pk, ib := sh.Perm.Counters() + permMerges, permFolds, permPacks, permIndexBytes = permMerges+m, permFolds+f, permPacks+pk, permIndexBytes+ib + } } } b, _ := json.Marshal(map[string]any{ @@ -465,6 +470,7 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { "heapHoleMB": float64(holes) / 1e6, "heapLiveMB": float64(live) / 1e6, "heapScannedMB": float64(scanned) / 1e6, "heapMovedMB": float64(moved) / 1e6, "heapSyncs": syncs, "heapSyncKBAvg": float64(syncBytes) / 1e3 / float64(max(syncs, 1)), + "permMerges": permMerges, "permFolds": permFolds, "permPacks": permPacks, "permIndexMB": float64(permIndexBytes) / 1e6, "heapFsyncMsAvg": float64(heapFsync) / 1e6 / float64(max(syncs, 1)), "heapDeltaMsAvg": float64(deltaSync) / 1e6 / float64(max(syncs, 1)), }) return b diff --git a/database/perm.go b/database/perm.go index 19ba4ba..72a93d7 100644 --- a/database/perm.go +++ b/database/perm.go @@ -1070,6 +1070,12 @@ func (p *PermStore) dropUnreferencedRunFiles() error { return nil } +// Counters reports the maintenance so far: merge runs, folds, packs, +// and the index bytes written. +func (p *PermStore) Counters() (merges, folds, packs, indexBytes uint64) { + return p.mergeRuns.Load(), p.foldRuns.Load(), p.packRuns.Load(), p.indexBytes.Load() +} + // Stats is the store's report. func (p *PermStore) Stats() StoreStats { p.mu.RLock() diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index dde323d..62a7c11 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -188,6 +188,19 @@ doubles the bytes written); every bucket is merged every 256 blocks (B/256 buckets a block), which keeps the recent sections at a few MB a shard. +### Shards partition keys, not files (measured) + +The dynamic layer alone, nine stores, five minutes: seal p50 54-65 ms +at 8 storage shards, 67-79 ms at 64, 75-86 ms at 128, with a burst of +lost blocks every other minute at 64 and 128 and 8x and 16x the +files. Every shard's files sync on their own at every block, so the +barrier count rises with the shard count; a hundred-plus shards are +right for spreading the retiring work and shrinking every lock, and +they must not each bring a barrier. A block's records for every +shard go into one per-store append file, shard-tagged, and each +shard's index points into it: one data fsync per store however many +shards. + ## The seal: one commit point per store per block Today a non-empty block costs each shard four barriers -- the data @@ -267,11 +280,13 @@ point" and closes #33. 150-250 ms in the minutes the mover copies most) is the mover's own fsync volume in the device queue, which the pass size paces. 2. The permanent layer as files of records with index deltas - (`PermStore`, behind `KV2`'s permanent surface: `PutIfAbsent`, - windowed `Get`, `GetDeep`, the two-half seal, `MergeBelow`, - `historyBelow`, `DropBelow`, `attachCold`, the filter knobs), the - 44-byte index record, the bucketed long search merged a bucket per - block, and packs over indexes. Measured + (`PermStore`, behind `KV2`'s permanent surface), the 44-byte index + record, the bucketed long search merged in rotation, and packs over + indexes. *First cut built and wired (`perm.go`, + `NewKVShardFilesN`, `bdbench -perm-files`); its first full-load run + put the seal at 59 ms p50 in minute 1, then climbing to 209 by + minute 5 because maintenance shared the seal's run file; fixed, + remeasured next.* Measured alone first (`-stores 9 -dyna 0`), then with the heap under the full load, which is the acceptance run. 3. The block's deltas in the data files and the store-level commit: From 6e767dc4be5ffd8a7f89b998c28329cf78e77c88 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 19:39:37 -0500 Subject: [PATCH 28/58] The mover's size bound has hysteresis and a floor The bound engaged the moment dead bytes passed live and released the moment they fell back, so nine stores in lockstep all took files at once and the seal's p90 went to 461-704 ms in those minutes and 70-90 otherwise. It now engages at dead > 1.5x live, releases below live, and never takes a file less than a quarter dead. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/database/heap.go b/database/heap.go index cf3e4f7..43cfa93 100644 --- a/database/heap.go +++ b/database/heap.go @@ -86,6 +86,7 @@ type HeapStore struct { closed bool liveBytes int64 deadBytes int64 + bound bool // The size bound engaged: the mover takes files below the ratio putTotal, putInPlace, putAppend atomic.Uint64 lookups, hits atomic.Uint64 @@ -150,10 +151,18 @@ var HeapCleanBytes int64 = 4 << 20 var HeapCleanFiles = 4 // HeapCleanRatio is the dead fraction a file must reach before the -// mover takes it -- unless dead bytes exceed live bytes overall, when -// the deadest file is taken regardless, which bounds the heap at -// twice its live set. -var HeapCleanRatio = 0.5 +// mover takes it -- unless the heap is over its size bound, when the +// deadest file is taken at HeapCleanFloor or more. The bound has +// hysteresis: it engages when dead bytes exceed live by +// HeapBoundOn and releases when they fall below live again, so the +// movers of nine stores in lockstep do not all engage on the same +// block and then all disengage (measured: p90 461-704 ms in the +// minutes the bound flipped, 70-90 ms otherwise). +var ( + HeapCleanRatio = 0.5 + HeapCleanFloor = 0.25 + HeapBoundOn = 1.5 +) // HeapSnapshotEvery is how many compact calls pass between key-map // snapshots; between them the generation's deltas are what open @@ -1015,7 +1024,13 @@ func (h *HeapStore) pickFile() *heapFile { pick, best = hf, f } } - if pick == nil || (best < HeapCleanRatio && h.deadBytes <= h.liveBytes) { + switch { + case !h.bound && float64(h.deadBytes) > HeapBoundOn*float64(h.liveBytes): + h.bound = true + case h.bound && h.deadBytes < h.liveBytes: + h.bound = false + } + if pick == nil || best < HeapCleanRatio && (!h.bound || best < HeapCleanFloor) { return nil } return pick From c66fa35c503b101cfd663b654745b8233d17b3d8 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 19:42:37 -0500 Subject: [PATCH 29/58] The permanent store's manifest is written outside the lock Every merge ended with the manifest's write, fsync, rename and directory fsync under the shard's exclusive lock, every twenty blocks per shard, and the seal waited behind it (files rerun, minute 2: seal p90 185 ms). The manifest is now encoded under the lock and written with it released; a delta sealed meanwhile lies past the offset the manifest records and is replayed on open. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/database/perm.go b/database/perm.go index 72a93d7..746f4c9 100644 --- a/database/perm.go +++ b/database/perm.go @@ -986,9 +986,26 @@ func (p *PermStore) Pack() error { return p.dropUnreferencedRunFiles() } -// commitManifest writes perm.json aside and renames it into place. -// The caller holds the lock. +// commitManifest writes perm.json: encoded under the lock, written, +// fsynced and renamed with the lock released, so that a merge's +// manifest commit -- every twenty blocks, per shard -- is not a +// barrier the shard's puts and seal wait behind (spec 1.6). A delta +// the seal appends meanwhile lies after the offset the manifest +// records, and open replays it. The caller holds the lock and gets +// it back. func (p *PermStore) commitManifest() error { + buf, err := p.encodeManifest() + if err != nil { + return err + } + p.mu.Unlock() + err = p.writeManifest(buf) + p.mu.Lock() + return err +} + +// encodeManifest is the manifest as of now. The caller holds the lock. +func (p *PermStore) encodeManifest() ([]byte, error) { m := permManifest{Version: 1, Height: p.height, FilterBlocks: p.window_n, Rotation: p.rotation, NextData: p.nextID, NextRun: p.nextRun} ref := func(r *permRun, rf *runFile, height uint64) permRunRef { return permRunRef{File: rf.id, Off: r.off, Height: height} @@ -1022,14 +1039,16 @@ func (p *PermStore) commitManifest() error { sort.Slice(m.DataFiles, func(i, j int) bool { return m.DataFiles[i] < m.DataFiles[j] }) sort.Slice(m.RunFiles, func(i, j int) bool { return m.RunFiles[i] < m.RunFiles[j] }) sort.Slice(m.DeltaFiles, func(i, j int) bool { return m.DeltaFiles[i] < m.DeltaFiles[j] }) - // Deltas sealed after this commit append to the current run file + // Deltas sealed after this point append to the current delta file // from its end; open replays from there m.DeltasFile, m.DeltasOff = p.curRun.id, p.curRun.size p.deltasFrom.file, p.deltasFrom.off = m.DeltasFile, m.DeltasOff - buf, err := json.MarshalIndent(m, "", " ") - if err != nil { - return err - } + return json.MarshalIndent(m, "", " ") +} + +// writeManifest writes the encoded manifest aside, fsyncs it, renames +// it into place and fsyncs the directory. No lock is held. +func (p *PermStore) writeManifest(buf []byte) error { path := filepath.Join(p.Directory, "perm.json") tmp := path + segTmpSuffix f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) From 7556ddde9d8ca987b3b2cb4cc4f825f1981471f2 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 19:44:48 -0500 Subject: [PATCH 30/58] A merge takes as many buckets as the blocks since the last one call for Merge took PermBuckets/PermMergeEvery buckets per call -- one -- but the adapter calls it every twenty blocks, so a bucket was merged every 5,120 blocks, pending deltas never drained, and each merge read a growing pile of them (files rerun: seal p50 223 ms by minute 4). The buckets due now follow the blocks elapsed since the last call, so every bucket is merged every PermMergeEvery blocks whatever the caller's cadence. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/database/perm.go b/database/perm.go index 746f4c9..72b5e53 100644 --- a/database/perm.go +++ b/database/perm.go @@ -70,18 +70,19 @@ type PermStore struct { nextID uint32 dirty map[uint32]*heapFile - runs map[uint32]*runFile // Run files, by id - curRun *runFile // The run file the seal appends deltas to - maintRun *runFile // The run file maintenance appends to: never the seal's, so their barriers never share an inode - nextRun uint32 - live map[[32]byte]permRecord // This block's records - window []*permDelta // The last FilterBlocks deltas, oldest first - pending []*permDelta // Deltas below the window not yet in every bucket - buckets [PermBuckets]permBucket // The history above the watermark - retired []*permRun // The history below it, newest last - height uint64 - window_n uint64 // FilterBlocks - rotation int // The next bucket to merge + runs map[uint32]*runFile // Run files, by id + curRun *runFile // The run file the seal appends deltas to + maintRun *runFile // The run file maintenance appends to: never the seal's, so their barriers never share an inode + nextRun uint32 + live map[[32]byte]permRecord // This block's records + window []*permDelta // The last FilterBlocks deltas, oldest first + pending []*permDelta // Deltas below the window not yet in every bucket + buckets [PermBuckets]permBucket // The history above the watermark + retired []*permRun // The history below it, newest last + height uint64 + window_n uint64 // FilterBlocks + rotation int // The next bucket to merge + lastMerge uint64 // The height Merge last ran at: what decides how many buckets are due // The manifest's view: the run file and offset after which deltas // are replayed on open @@ -705,11 +706,21 @@ func (p *PermStore) Merge() error { p.mu.Unlock() return nil } - // Which buckets, and from which deltas - due := int(PermBuckets / PermMergeEvery) + // Which buckets: every bucket is due once per PermMergeEvery + // blocks, so a call that comes after n blocks takes n/PermMergeEvery + // of them, whatever the caller's cadence + elapsed := p.height - p.lastMerge + if p.lastMerge == 0 || elapsed > PermMergeEvery { + elapsed = PermMergeEvery + } + due := int(uint64(PermBuckets) * elapsed / PermMergeEvery) if due < 1 { due = 1 } + if due > PermBuckets { + due = PermBuckets + } + p.lastMerge = p.height newest := p.pending[len(p.pending)-1].height type job struct{ b int } var jobs []job From fda3e9b810c0d020ede94a9523d32f79a13cb7a5 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 20:15:39 -0500 Subject: [PATCH 31/58] The heap's block delta lives in its data file: one barrier per block The delta -- the keys the block touched and where they are -- is appended to the block's data file right behind its entries, as an entry under a reserved key, so a block is one fsync per shard for the dynamic layer instead of two; the barrier count is what the device queue charges for, and the second barrier was a third of the sync. The index generation file keeps only the snapshot and the replay point; open scans the data files from that point for delta entries, verifies the last delta's entries by their checksums before trusting it (one fsync does not order the delta's bytes after the entries'), and cuts the newest file back to the last delta or named slot. A delta's bytes are live until a snapshot supersedes them and dead after; a scan steps over a damaged entry rather than stopping at it; a repair records the end of the data as its replay point so no old delta is replayed over it. The crash test passes as before. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 368 +++++++++++++++++++++++++++++------------- database/heap_test.go | 29 ++-- 2 files changed, 276 insertions(+), 121 deletions(-) diff --git a/database/heap.go b/database/heap.go index 43cfa93..7f2df67 100644 --- a/database/heap.go +++ b/database/heap.go @@ -31,16 +31,28 @@ import ( // two never share a barrier. A file is rolled at // HeapFileBytes and deleted once nothing in it is live. // index-G.log generation G of the key map: a snapshot record of the -// whole map, then one delta per block sync naming the -// keys the block touched. A snapshot starts a new -// generation in a file of its own, switched to by -// rename, so no delta is ever truncated away. +// whole map and the point in the data files from which +// the deltas after it are replayed. A snapshot starts a +// new generation in a file of its own, switched to by +// rename. +// +// A block's delta -- the keys it touched and where they are -- is +// appended to the block's data file right behind the block's +// entries, as an entry under a reserved key, so a block is ONE fsync +// per shard (entries and delta together) rather than two (measured: +// the second barrier was a third of the sync's cost, and the barrier +// count is what the device queue charges for). One fsync does not +// order the delta's bytes after the entries', so open verifies the +// last delta's entries by their checksums before trusting it; an +// earlier delta was followed by a later block's fsync, which covers it. +// A delta's bytes are dead once a snapshot supersedes them. // // The key map is in memory for the live key set (spec 1.2: memory that // scales with the working set). Open loads the newest whole -// generation, replays its deltas and derives every file's live and -// dead bytes; a data file nothing names is deleted then, and one that -// is missing while named is an error. +// generation's snapshot, replays the deltas in the data files after +// its replay point, and derives every file's live and dead bytes; a +// data file nothing names is deleted then, and one that is missing +// while named is an error. // // Durability (spec 1.8). The block sync fsyncs the data files the // block wrote and then appends and fsyncs the block's delta, so an @@ -73,6 +85,12 @@ type HeapStore struct { touched map[[32]byte]struct{} dirty map[uint32]*heapFile release []uint32 + // deltaAt is where the last delta ended: the replay point a + // snapshot records, and what a file is cut back to on open + deltaAt struct { + file uint32 + off int64 + } // syncMu serializes block syncs with each other and with a // snapshot, so the map a snapshot writes is exactly the state of @@ -102,6 +120,7 @@ type heapFile struct { f *os.File size int64 live, dead int64 + deltas int64 // Bytes of delta entries not yet superseded by a snapshot: live until then firstBlock uint64 // The block that first appended to it cleaning bool // Taken by the pass in progress inflight int // Copies reserved in it and not yet written: its size runs ahead of its bytes @@ -119,15 +138,21 @@ type slot struct { } const ( - heapHeader = 4 + 8 + 32 // len, height, key - heapTrailer = 4 // crc32 of height+key+value - heapAlign = 8 // Entries start on an 8-byte boundary - heapMagic = 0x48454150 // "HEAP", an index record's marker - heapSnapshot = 0x50414E53 // "SNAP", the record that starts a generation - heapIndexHdr = 4 + 8 + 4 // magic, height, count + heapHeader = 4 + 8 + 32 // len, height, key + heapTrailer = 4 // crc32 of height+key+value + heapAlign = 8 // Entries start on an 8-byte boundary + heapMagic = 0x48454150 // "HEAP", an index record's marker + heapSnapshot = 0x50414E53 // "SNAP", the record that starts a generation + heapIndexHdr = 4 + 8 + 4 + 4 + 4 // magic, height, count, replay file, replay offset heapIndexRec = 32 + 4 + 4 + 4 ) +// heapDeltaKey is the reserved key a delta entry is written under in +// a data file. A real key is a hash; the odds of one being all ones +// are those of a hash collision. +var heapDeltaKey = [32]byte{0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff} + // HeapFileBytes is the size a data file is rolled at, and // HeapFileBlocks the age: a shard that appends little per block (many // shards, or a quiet one) would otherwise keep one file current for @@ -269,8 +294,8 @@ func (h *HeapStore) Open() (err error) { } } else { h.gen = 1 - // A fresh store: the first generation, written with the locks - // released since startGeneration takes them itself + // A fresh store: the first generation, written with the lock + // released since startGeneration takes it itself h.mu.Unlock() err = h.startGeneration() h.mu.Lock() @@ -283,12 +308,35 @@ func (h *HeapStore) Open() (err error) { h.nextID = id + 1 } } + if err = h.openFiles(dataIDs); err != nil { + return err + } + if err = h.replayDeltas(dataIDs); err != nil { + return err + } return h.deriveFiles(dataIDs) } -// replayGeneration applies the current generation: its snapshot, then -// every whole delta; a torn tail is what a crash leaves and is -// dropped, its slots unnamed. The caller holds the lock. +// openFiles opens every data file present. The caller holds the lock. +func (h *HeapStore) openFiles(dataIDs []uint32) error { + for _, id := range dataIDs { + f, err := os.OpenFile(filepath.Join(h.Directory, dataName(id)), os.O_RDWR, 0o644) + if err != nil { + return err + } + st, err := f.Stat() + if err != nil { + f.Close() + return err + } + h.files[id] = &heapFile{id: id, f: f, size: st.Size()} + } + return nil +} + +// replayGeneration loads the current generation's snapshot; the +// deltas after it are in the data files and replayDeltas applies +// them. The caller holds the lock. func (h *HeapStore) replayGeneration() (err error) { path := filepath.Join(h.Directory, indexName(h.gen)) if h.log, err = os.OpenFile(path, os.O_RDWR|os.O_APPEND, 0o644); err != nil { @@ -298,30 +346,112 @@ func (h *HeapStore) replayGeneration() (err error) { if err != nil { return err } - at := 0 - for at < len(buf) { - n, err := h.applyRecord(buf[at:]) - if err != nil { - if err = h.log.Truncate(int64(at)); err != nil { - return err + n, err := h.applyRecord(buf) + if err != nil { + return fmt.Errorf("heap: generation %d: %w", h.gen, err) + } + if n != len(buf) { + return fmt.Errorf("heap: generation %d has %d trailing bytes", h.gen, len(buf)-n) + } + return nil +} + +// replayDeltas applies every delta entry in the data files after the +// snapshot's replay point, in file and offset order. The last delta +// is trusted only if every entry it names checks: one fsync covered +// it and its entries together, and a crash inside that fsync can +// leave the delta durable and an entry torn. An earlier delta was +// followed by a later block's fsync. The caller holds the lock; the +// files are open. +func (h *HeapStore) replayDeltas(dataIDs []uint32) error { + type found struct { + file uint32 + off int64 + end int64 + recs []byte + } + var deltas []found + for _, id := range dataIDs { + if id < h.deltaAt.file { + continue + } + hf := h.files[id] + if hf == nil { + continue + } + buf := make([]byte, hf.size) + if _, err := hf.f.ReadAt(buf, 0); err != nil && !errors.Is(err, io.EOF) { + return err + } + at := int64(0) + if id == h.deltaAt.file { + at = h.deltaAt.off + } + for at < int64(len(buf)) { + size, _, key, value, ok := decodeEntry(buf[at:]) + if size == 0 { + break } + if ok && key == heapDeltaKey { + deltas = append(deltas, found{file: id, off: at, end: at + size, recs: append([]byte(nil), value...)}) + } + at += size // A damaged entry is stepped over; if it is named, Get reports it + } + } + for i, d := range deltas { + if i == len(deltas)-1 && !h.deltaEntriesCheck(d.recs) { + break // The block did not commit + } + if _, err := h.applyRecord(d.recs); err != nil { break } - at += n + h.deltaAt.file, h.deltaAt.off = d.file, d.end + h.files[d.file].deltas += d.end - d.off } return nil } -// startGeneration begins an index generation: a snapshot of the map, -// written aside and fsynced, then the deltas appended to the old -// generation meanwhile copied after it, renamed into place, the -// directory fsynced, and the previous generation's file removed once -// the new one is durable. The snapshot itself is written with no -// lock held but the map's read lock; only the tail copy and the -// switch hold syncMu, so a block's sync waits milliseconds for a -// snapshot, not for 3 MB of map (measured: seal p90 276 ms at every -// hundredth block with the whole write under syncMu). Open calls it -// with nothing else running. +// deltaEntriesCheck reads every entry a delta names and checks it. +func (h *HeapStore) deltaEntriesCheck(recs []byte) bool { + if len(recs) < heapIndexHdr { + return false + } + count := int(binary.LittleEndian.Uint32(recs[12:])) + end := heapIndexHdr + count*heapIndexRec + if len(recs) < end+4 || crc32.ChecksumIEEE(recs[:end]) != binary.LittleEndian.Uint32(recs[end:]) { + return false + } + for at := heapIndexHdr; at < end; at += heapIndexRec { + var key [32]byte + copy(key[:], recs[at:]) + file := binary.LittleEndian.Uint32(recs[at+32:]) + off := binary.LittleEndian.Uint32(recs[at+36:]) + n := binary.LittleEndian.Uint32(recs[at+40:]) + hf := h.files[file] + if hf == nil { + return false + } + buf := make([]byte, heapHeader+int(n)+heapTrailer) + if _, err := hf.f.ReadAt(buf, int64(off)); err != nil { + return false + } + if _, err := heapEntryValue(buf, key); err != nil { + return false + } + } + return true +} + +// startGeneration begins an index generation: a snapshot of the map +// with the replay point -- where the last delta ended -- written +// aside and fsynced, renamed into place, the directory fsynced, and +// the previous generation's file removed once the new one is durable. +// The map and the point are taken together under the lock; a delta +// the seal appends after that lies past the point and is replayed +// over the snapshot on open, which is idempotent. The snapshot's +// write holds no lock (measured: a 3 MB snapshot under syncMu put +// the seal's p90 at 276 ms every hundredth block). A delta's bytes +// before the point are dead once the generation is durable. func (h *HeapStore) startGeneration() error { next := h.gen if h.log != nil { @@ -329,20 +459,18 @@ func (h *HeapStore) startGeneration() error { } path := filepath.Join(h.Directory, indexName(next)) tmp := path + segTmpSuffix - // 1. The map as of now, and where the old generation's log ends: - // deltas after that point are copied over below - h.mu.RLock() + h.mu.Lock() all := func(emit func(key [32]byte)) { for key := range h.index { emit(key) } } snap := h.encodeIndexOf(heapSnapshot, all, len(h.index)) - var copiedTo int64 - if h.log != nil { - copiedTo, _ = h.log.Seek(0, io.SeekEnd) + superseded := map[uint32]int64{} + for id, hf := range h.files { + superseded[id] = hf.deltas } - h.mu.RUnlock() + h.mu.Unlock() f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) if err != nil { return err @@ -355,29 +483,6 @@ func (h *HeapStore) startGeneration() error { f.Close() return err } - // 2. Under syncMu: no delta is in flight, so the old log's tail - // past copiedTo is exactly the deltas since the snapshot - h.syncMu.Lock() - defer h.syncMu.Unlock() - h.mu.Lock() - defer h.mu.Unlock() - if h.log != nil { - rest, err := readFrom(h.log, copiedTo) - if err != nil { - f.Close() - return err - } - if len(rest) > 0 { - if _, err = f.Write(rest); err != nil { - f.Close() - return err - } - if err = fsync(f); err != nil { - f.Close() - return err - } - } - } if err = f.Close(); err != nil { return err } @@ -387,6 +492,8 @@ func (h *HeapStore) startGeneration() error { if err = fsyncDir(h.Directory); err != nil { return err } + h.mu.Lock() + defer h.mu.Unlock() old := h.log if h.log, err = os.OpenFile(path, os.O_RDWR|os.O_APPEND, 0o644); err != nil { return err @@ -396,21 +503,17 @@ func (h *HeapStore) startGeneration() error { os.Remove(filepath.Join(h.Directory, indexName(h.gen))) } h.gen = next - return nil -} - -// readFrom reads a file from off to its end. -func readFrom(f *os.File, off int64) ([]byte, error) { - end, err := f.Seek(0, io.SeekEnd) - if err != nil { - return nil, err - } - if end <= off { - return nil, nil + // The deltas the snapshot covers are dead where they lie + for id, n := range superseded { + if hf := h.files[id]; hf != nil && n > 0 { + hf.deltas -= n + hf.live -= n + hf.dead += n + h.liveBytes -= n + h.deadBytes += n + } } - buf := make([]byte, end-off) - _, err = f.ReadAt(buf, off) - return buf, err + return nil } // fsyncDir makes a directory's entries durable: a rename or an unlink @@ -440,6 +543,8 @@ func (h *HeapStore) encodeIndexOf(magic uint32, each func(emit func(key [32]byte binary.LittleEndian.PutUint32(buf, magic) binary.LittleEndian.PutUint64(buf[4:], h.height) binary.LittleEndian.PutUint32(buf[12:], uint32(count)) + binary.LittleEndian.PutUint32(buf[16:], h.deltaAt.file) + binary.LittleEndian.PutUint32(buf[20:], uint32(h.deltaAt.off)) at := heapIndexHdr each(func(key [32]byte) { s := h.index[key] @@ -472,6 +577,10 @@ func (h *HeapStore) applyRecord(buf []byte) (int, error) { if height := binary.LittleEndian.Uint64(buf[4:]); height > h.height { h.height = height } + if magic == heapSnapshot { + h.deltaAt.file = binary.LittleEndian.Uint32(buf[16:]) + h.deltaAt.off = int64(binary.LittleEndian.Uint32(buf[20:])) + } for at := heapIndexHdr; at < end; at += heapIndexRec { var key [32]byte copy(key[:], buf[at:]) @@ -480,11 +589,11 @@ func (h *HeapStore) applyRecord(buf []byte) (int, error) { return end + 4, nil } -// deriveFiles opens every data file the map names, derives its live -// and dead bytes, cuts the newest back to its last named entry, and -// deletes any file the map does not name at all: nothing durable -// names it, and its bytes are a crash's or the mover's leftovers. The -// caller holds the lock. +// deriveFiles derives every open file's live and dead bytes, cuts +// the newest back to its last named slot or its last delta, and +// deletes any file the map does not name and no delta needed: +// nothing durable names it, and its bytes are a crash's or the +// mover's leftovers. The caller holds the lock. func (h *HeapStore) deriveFiles(dataIDs []uint32) error { named := map[uint32]int64{} // file -> end of its last named slot for _, s := range h.index { @@ -494,30 +603,33 @@ func (h *HeapStore) deriveFiles(dataIDs []uint32) error { h.liveBytes += entrySize(int(s.n)) } for id := range named { - if _, err := os.Stat(filepath.Join(h.Directory, dataName(id))); err != nil { - return fmt.Errorf("heap: the index names %s: %w", dataName(id), err) + if h.files[id] == nil { + return fmt.Errorf("heap: the index names %s: missing", dataName(id)) } } for _, id := range dataIDs { + hf := h.files[id] end, live := named[id] - if !live { + if id == h.deltaAt.file && h.deltaAt.off > end { + end, live = h.deltaAt.off, true + } + if !live && hf.deltas == 0 { + hf.f.Close() + delete(h.files, id) os.Remove(filepath.Join(h.Directory, dataName(id))) continue } - f, err := os.OpenFile(filepath.Join(h.Directory, dataName(id)), os.O_RDWR, 0o644) - if err != nil { + if err := hf.f.Truncate(end); err != nil { return err } - if err = f.Truncate(end); err != nil { - f.Close() - return err - } - h.files[id] = &heapFile{id: id, f: f, size: end} + hf.size = end } for _, s := range h.index { h.files[s.file].live += entrySize(int(s.n)) } for _, hf := range h.files { + hf.live += hf.deltas + h.liveBytes += hf.deltas hf.dead = hf.size - hf.live h.deadBytes += hf.dead } @@ -578,7 +690,7 @@ func decodeEntry(buf []byte) (size int64, height uint64, key [32]byte, value []b return 0, 0, key, nil, false } if crc32.ChecksumIEEE(buf[4:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { - return 0, 0, key, nil, false + return size, 0, key, nil, false // Damaged: its size lets a scan step over it } height = binary.LittleEndian.Uint64(buf[4:]) copy(key[:], buf[12:]) @@ -760,15 +872,28 @@ func (h *HeapStore) beginBlockSync() (blockSync, error) { } p := &heapSync{h: h, release: h.release} h.release = nil + if len(h.touched) > 0 || len(p.release) > 0 { + // The delta goes into the block's data file behind its entries, + // under the reserved key, so the block's one fsync covers both + p.delta = h.encodeIndex(heapMagic, h.touched) + h.touched = map[[32]byte]struct{}{} + hf, off, err := h.reserve(false, entrySize(len(p.delta))) + if err != nil { + h.syncMu.Unlock() + return nil, err + } + if _, err := hf.f.WriteAt(encodeEntry(h.height, heapDeltaKey, p.delta), off); err != nil { + h.syncMu.Unlock() + return nil, err + } + hf.deltas += entrySize(len(p.delta)) + h.deltaAt.file, h.deltaAt.off = hf.id, off+entrySize(len(p.delta)) + } for _, hf := range h.dirty { p.dirty = append(p.dirty, hf) p.bytes += hf.size } h.dirty = map[uint32]*heapFile{} - if len(h.touched) > 0 || len(p.release) > 0 { - p.delta = h.encodeIndex(heapMagic, h.touched) - h.touched = map[[32]byte]struct{}{} - } return p, nil } @@ -787,14 +912,6 @@ func (p *heapSync) finish() (err error) { } } h.syncHeapNs.Add(uint64(time.Since(t))) - t = time.Now() - if _, err = h.log.Write(p.delta); err != nil { - return err - } - if err = fsync(h.log); err != nil { - return err - } - h.syncLogNs.Add(uint64(time.Since(t))) h.syncs.Add(1) h.syncBytes.Add(uint64(p.bytes)) h.mu.Lock() @@ -1051,11 +1168,13 @@ func decodeFile(buf []byte) (entries []heapEntry) { var at int64 for at < int64(len(buf)) { size, _, key, value, ok := decodeEntry(buf[at:]) - if !ok { + if size == 0 { break } - entries = append(entries, heapEntry{off: uint32(at), size: size, key: key, value: value}) - at += size + if ok { + entries = append(entries, heapEntry{off: uint32(at), size: size, key: key, value: value}) + } + at += size // A damaged entry stays where it is: not moved, not named by the mover } return entries } @@ -1113,9 +1232,13 @@ func RepairHeapStore(directory string, committed uint64) (*HeapStore, error) { var off int64 for off < int64(len(data)) { size, height, key, value, ok := decodeEntry(data[off:]) - if !ok { + if size == 0 { break } + if !ok || key == heapDeltaKey { + off += size + continue + } if height <= committed { if prev, seen := heights[key]; !seen || height >= prev { heights[key] = height @@ -1128,10 +1251,20 @@ func RepairHeapStore(directory string, committed uint64) (*HeapStore, error) { h.nextID = id + 1 } } + // A repair trusts the data alone: the snapshot's replay point is + // the end of the newest file, so no old delta is replayed over it + if n := len(dataIDs); n > 0 { + if st, err := os.Stat(filepath.Join(directory, dataName(dataIDs[n-1]))); err == nil { + h.deltaAt.file, h.deltaAt.off = dataIDs[n-1], st.Size() + } + } h.gen = 1 if err = h.startGeneration(); err != nil { return nil, err } + if err = h.openFiles(dataIDs); err != nil { + return nil, err + } if err = h.deriveFiles(dataIDs); err != nil { return nil, err } @@ -1159,13 +1292,24 @@ func (h *HeapStore) Stats() StoreStats { } // HoleRatio reports the dead bytes in the files against the live -// bytes: what the mover has yet to reclaim. +// bytes: what the mover has yet to reclaim. Live includes the delta +// entries a snapshot has not yet superseded. func (h *HeapStore) HoleRatio() (dead, live int64) { h.mu.RLock() defer h.mu.RUnlock() return h.deadBytes, h.liveBytes } +// deltaBytes is the bytes of delta entries not yet superseded. +func (h *HeapStore) deltaBytes() (n int64) { + h.mu.RLock() + defer h.mu.RUnlock() + for _, hf := range h.files { + n += hf.deltas + } + return n +} + // SyncCost reports the block syncs so far: how many, the bytes their // data fsyncs covered, and the time spent in the data files' fsyncs // and in the delta's write and fsync. diff --git a/database/heap_test.go b/database/heap_test.go index 065f216..f015152 100644 --- a/database/heap_test.go +++ b/database/heap_test.go @@ -55,7 +55,7 @@ func TestHeapRewriteReusesWithinTheBlockAndMovesOneSyncLate(t *testing.T) { require.NotEqual(t, first, h.index[key(1)], "a durable slot is never rewritten") dead, live := h.HoleRatio() require.EqualValues(t, entrySize(3), dead, "the old slot is dead where it lies") - require.EqualValues(t, entrySize(3), live) + require.EqualValues(t, entrySize(3), live-h.deltaBytes(), "one live entry besides the deltas") // Fill the block's file past its size so it rolls, then kill most // of what the first file holds for i := byte(10); i < 40; i++ { @@ -68,6 +68,7 @@ func TestHeapRewriteReusesWithinTheBlockAndMovesOneSyncLate(t *testing.T) { require.NoError(t, h.Put(key(i), make([]byte, 64))) } syncHeap(t, h) + require.NoError(t, h.Snapshot()) // The deltas so far are superseded: dead where they lie h.AdvanceBlock(4) moved, err := h.clean(1 << 20) @@ -110,7 +111,7 @@ func TestHeapReopenKeepsTheDurableAndDropsTheRest(t *testing.T) { h.AdvanceBlock(2) require.NoError(t, h.Put(key(1), []byte("rewritten in block 2"))) syncHeap(t, h) - size := h.cur.size + size := h.deltaAt.off // Where block 2's delta ended: the durable append point // Block 3: written, never synced -- the crash h.AdvanceBlock(3) require.NoError(t, h.Put(key(2), []byte("lost"))) @@ -140,17 +141,20 @@ func TestHeapReopenKeepsTheDurableAndDropsTheRest(t *testing.T) { require.EqualValues(t, 2, r.height, "the durable height: block 3 never synced") } -// A torn delta at the end of the generation is dropped whole. -func TestHeapTornLogTailIsDropped(t *testing.T) { +// A torn entry at the end of the data file -- a crash mid-write -- +// is cut, and the deltas before it stand. +func TestHeapTornTailIsCut(t *testing.T) { dir := heapDir(t) h, err := NewHeapStore(dir) require.NoError(t, err) h.AdvanceBlock(1) require.NoError(t, h.Put(key(1), []byte("one"))) + syncHeap(t, h) + end := h.deltaAt.off require.NoError(t, h.Close()) - f, err := os.OpenFile(filepath.Join(dir, indexName(1)), os.O_WRONLY|os.O_APPEND, 0o644) + f, err := os.OpenFile(filepath.Join(dir, dataName(0)), os.O_WRONLY|os.O_APPEND, 0o644) require.NoError(t, err) - _, err = f.Write([]byte{0x50, 0x41, 0x45, 0x48, 9, 9}) // A marker and six bytes of nothing + _, err = f.Write([]byte{9, 0, 0, 0, 1, 2, 3, 4, 5, 6}) // A length and a few bytes of nothing require.NoError(t, err) require.NoError(t, f.Close()) @@ -160,9 +164,9 @@ func TestHeapTornLogTailIsDropped(t *testing.T) { v, err := r.Get(key(1)) require.NoError(t, err) require.Equal(t, "one", string(v)) - st, err := os.Stat(filepath.Join(dir, indexName(1))) + st, err := os.Stat(filepath.Join(dir, dataName(0))) require.NoError(t, err) - require.EqualValues(t, 2*(heapIndexHdr+4)+heapIndexRec, st.Size(), "the empty snapshot and one whole delta of one key remain") + require.Equal(t, end, st.Size(), "cut back to the last delta") } // A snapshot starts a new generation in its own file and retires the @@ -214,6 +218,9 @@ func TestHeapChecksumCatchesADamagedSlot(t *testing.T) { h.AdvanceBlock(1) require.NoError(t, h.Put(key(1), []byte("intact"))) s := h.index[key(1)] + syncHeap(t, h) + h.AdvanceBlock(2) + require.NoError(t, h.Put(key(2), []byte("later"))) // A later block's sync covers block 1 require.NoError(t, h.Close()) f, err := os.OpenFile(filepath.Join(dir, dataName(s.file)), os.O_WRONLY, 0o644) require.NoError(t, err) @@ -311,7 +318,7 @@ func TestHeapShardRoundTrip(t *testing.T) { require.EqualValues(t, 60*200, dyna.PutTotal) require.NoError(t, kvs.SealBlock(61)) // The sync that deletes what the last pass emptied dead, live := kvs.Shards[0].Heap.HoleRatio() - require.Less(t, dead, 2*live, "the mover keeps dead bytes under twice the live set") + require.Less(t, dead, 4*live, "the mover keeps dead bytes bounded (deltas are dead once superseded, and reclaimed)") require.NoError(t, kvs.Close()) re, err := OpenKVShard(dir) @@ -348,6 +355,10 @@ func TestHeapMoveIsDeadOnArrivalIfTheKeyWasRewritten(t *testing.T) { } syncHeap(t, h) } + require.NoError(t, h.Snapshot()) // The deltas so far are superseded: the file is mostly dead + files := HeapCleanFiles + HeapCleanFiles = 64 // One pass takes every eligible file, key 20's half-dead one included + defer func() { HeapCleanFiles = files }() h.AdvanceBlock(4) moverHook = func() { require.NoError(t, h.Put(key(20), []byte("rewritten while moving"))) From 309d1e398e805e2175db109aa5228561b41663cc Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 20:19:01 -0500 Subject: [PATCH 32/58] The permanent store's block delta lives in its data file: one barrier per block The block's delta run, filter included, is appended to the block's data file behind its entries under the reserved key the heap's deltas use, so the permanent layer costs a block one fsync per shard instead of two; the separate delta files are gone. The window and the pending deltas read their runs from the data files; open replays the deltas after the manifest's point from the data files, verifies the last one's entries by checksum before trusting it, and cuts the newest data file back to the last delta admitted. Maintenance keeps its own run files. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 290 +++++++++++++++++++++++------------------- database/perm_test.go | 7 +- 2 files changed, 162 insertions(+), 135 deletions(-) diff --git a/database/perm.go b/database/perm.go index 72b5e53..281a60f 100644 --- a/database/perm.go +++ b/database/perm.go @@ -4,6 +4,7 @@ import ( "encoding/json" "errors" "fmt" + "io" "os" "path/filepath" "sort" @@ -22,14 +23,19 @@ import ( // perm-N.dat entries, [len][height][key][value][crc] (heap.go's // layout), appended in the order written, rolled at // PermFileBytes; never rewritten -// deltas-N.dat the seal's run file: a block's delta appended at every -// seal; open replays the deltas after the manifest from -// these files alone // runs-N.dat maintenance's run file: a bucket's run at every merge, -// a retired run at every pack; never the seal's, so their -// barriers never share an inode; both kinds are rolled -// at PermFileBytes and deleted once no run in them is -// referenced +// a retired run at every pack; rolled at PermFileBytes +// and deleted once no run in it is referenced +// +// A block's delta -- its records' run, with its filter -- is appended +// to the block's data file right behind the block's entries, as an +// entry under the reserved key heap.go's deltas use, so a block is +// ONE fsync per shard for the permanent layer: entries and delta +// together. Open replays the deltas in the data files after the +// point the manifest records, verifies the last one's entries by +// their checksums before trusting it, and cuts the newest data file +// back to the last delta. +// // perm.json the manifest: which runs are the window, the buckets // and the retired history, and where in the run files // the deltas not yet in the manifest begin @@ -71,7 +77,6 @@ type PermStore struct { dirty map[uint32]*heapFile runs map[uint32]*runFile // Run files, by id - curRun *runFile // The run file the seal appends deltas to maintRun *runFile // The run file maintenance appends to: never the seal's, so their barriers never share an inode nextRun uint32 live map[[32]byte]permRecord // This block's records @@ -117,7 +122,8 @@ var PermFoldRatio = 0.25 type permDelta struct { height uint64 run *permRun - rf *runFile + f *os.File // The data file the delta lies in + data uint32 // Its id } type permBucket struct { @@ -128,23 +134,15 @@ type permBucket struct { // runFile is a file of runs and how many runs still reference it. type runFile struct { - id uint32 - f *os.File - size int64 - refs int - deltas bool // The seal's, replayed on open; else maintenance's + id uint32 + f *os.File + size int64 + refs int } -func permDataName(id uint32) string { return fmt.Sprintf("perm-%06d.dat", id) } -func permRunName(id uint32) string { return fmt.Sprintf("runs-%06d.dat", id) } -func permDeltaName(id uint32) string { return fmt.Sprintf("deltas-%06d.dat", id) } - -func (rf *runFile) name() string { - if rf.deltas { - return permDeltaName(rf.id) - } - return permRunName(rf.id) -} +func permDataName(id uint32) string { return fmt.Sprintf("perm-%06d.dat", id) } +func permRunName(id uint32) string { return fmt.Sprintf("runs-%06d.dat", id) } +func (rf *runFile) name() string { return permRunName(rf.id) } // NewPermStore creates an empty store in directory, replacing anything // there. @@ -182,11 +180,10 @@ type permManifest struct { Retired []permRunRef `json:"retired"` DataFiles []uint32 `json:"dataFiles"` RunFiles []uint32 `json:"runFiles"` - DeltaFiles []uint32 `json:"deltaFiles"` } type permRunRef struct { - File uint32 `json:"file"` + File uint32 `json:"file"` // A data file for a delta, a run file otherwise Off int64 `json:"off"` Height uint64 `json:"height,omitempty"` } @@ -219,9 +216,6 @@ func (p *PermStore) Open() (err error) { if p.cur, err = p.newDataFile(); err != nil { return err } - if p.curRun, err = p.newRunFile(true); err != nil { - return err - } return p.commitManifest() case err != nil: return err @@ -249,30 +243,6 @@ func (p *PermStore) Open() (err error) { st, _ := f.Stat() p.runs[id] = &runFile{id: id, f: f, size: st.Size()} } - // Delta files: the manifest's, and any the seal created after it - deltaIDs := append([]uint32(nil), m.DeltaFiles...) - if entries, err := os.ReadDir(p.Directory); err == nil { - for _, e := range entries { - var id uint32 - if n, _ := fmt.Sscanf(e.Name(), "deltas-%06d.dat", &id); n == 1 && id > m.NextRun-1 { - deltaIDs = append(deltaIDs, id) - } - } - } - for _, id := range deltaIDs { - if p.runs[id] != nil { - continue - } - f, err := os.OpenFile(filepath.Join(p.Directory, permDeltaName(id)), os.O_RDWR, 0o644) - if err != nil { - return fmt.Errorf("perm: delta file %s: %w", permDeltaName(id), err) - } - st, _ := f.Stat() - p.runs[id] = &runFile{id: id, f: f, size: st.Size(), deltas: true} - if id >= p.nextRun { - p.nextRun = id + 1 - } - } load := func(ref permRunRef, resident bool) (*permRun, *runFile, error) { rf := p.runs[ref.File] if rf == nil { @@ -285,19 +255,30 @@ func (p *PermStore) Open() (err error) { rf.refs++ return r, rf, nil } + loadDelta := func(ref permRunRef) (*permDelta, error) { + hf := p.files[ref.File] + if hf == nil { + return nil, fmt.Errorf("perm: data file %d not open", ref.File) + } + r, err := openPermRun(hf.f, ref.File, ref.Off, true) + if err != nil { + return nil, err + } + return &permDelta{height: ref.Height, run: r, f: hf.f, data: ref.File}, nil + } for _, ref := range m.Window { - r, rf, err := load(ref, true) + d, err := loadDelta(ref) if err != nil { return err } - p.window = append(p.window, &permDelta{height: ref.Height, run: r, rf: rf}) + p.window = append(p.window, d) } for _, ref := range m.Pending { - r, rf, err := load(ref, true) + d, err := loadDelta(ref) if err != nil { return err } - p.pending = append(p.pending, &permDelta{height: ref.Height, run: r, rf: rf}) + p.pending = append(p.pending, d) } for i, bm := range m.Buckets { if i >= PermBuckets { @@ -327,58 +308,100 @@ func (p *PermStore) Open() (err error) { p.cur = hf } } - p.curRun = p.runs[m.DeltasFile] - if p.curRun == nil { - if p.curRun, err = p.newRunFile(true); err != nil { - return err - } - } p.deltasFrom.file, p.deltasFrom.off = m.DeltasFile, m.DeltasOff return p.replayDeltas() } -// replayDeltas reads every whole delta after the manifest's offset -// into the window (and pending), cutting a torn tail. The caller -// holds the lock. +// replayDeltas applies every delta entry in the data files after the +// manifest's point, oldest first, into the window and pending. The +// last delta is trusted only if every entry it names checks -- one +// fsync covered it and its entries together -- and the newest data +// file is cut back to the last delta admitted. The caller holds the +// lock. func (p *PermStore) replayDeltas() error { - ids := make([]uint32, 0, len(p.runs)) - for id, rf := range p.runs { - if rf.deltas && id >= p.deltasFrom.file { + ids := make([]uint32, 0, len(p.files)) + for id := range p.files { + if id >= p.deltasFrom.file { ids = append(ids, id) } } sort.Slice(ids, func(i, j int) bool { return ids[i] < ids[j] }) + type found struct { + hf *heapFile + off, end int64 + run *permRun + } + var deltas []found for _, id := range ids { - rf := p.runs[id] - off := int64(0) + hf := p.files[id] + buf := make([]byte, hf.size) + if _, err := hf.f.ReadAt(buf, 0); err != nil && !errors.Is(err, io.EOF) { + return err + } + at := int64(0) if id == p.deltasFrom.file { - off = p.deltasFrom.off + at = p.deltasFrom.off } - for off < rf.size { - r, err := openPermRun(rf.f, id, off, true) - if err != nil { - // Torn: cut the file here - if err := rf.f.Truncate(off); err != nil { - return err - } - rf.size = off + for at < int64(len(buf)) { + size, _, key, _, ok := decodeEntry(buf[at:]) + if size == 0 { break } - rf.refs++ - p.admit(&permDelta{height: r.height, run: r, rf: rf}) - if r.height > p.height { - p.height = r.height + if ok && key == heapDeltaKey { + r, err := openPermRun(hf.f, id, at+heapHeader, true) + if err == nil { + deltas = append(deltas, found{hf: hf, off: at, end: at + size, run: r}) + } } - off += int64(r.bytes) + at += size + } + } + var last *found + for i := range deltas { + d := &deltas[i] + if i == len(deltas)-1 && !p.deltaEntriesCheck(d.run, d.hf.f) { + break // The block did not commit } - // The newest delta file is the seal's current one - if p.curRun == nil || rf.id > p.curRun.id { - p.curRun = rf + p.admit(&permDelta{height: d.run.height, run: d.run, f: d.hf.f, data: d.hf.id}) + if d.run.height >= p.height { + p.height = d.run.height + 1 } + last = d + } + // The newest data file is cut back to what is committed: the last + // delta admitted, or the manifest's point + newest := p.files[ids[len(ids)-1]] + end := int64(0) + if newest.id == p.deltasFrom.file { + end = p.deltasFrom.off + } + if last != nil && last.hf == newest { + end = last.end } + if newest.size > end { + if err := newest.f.Truncate(end); err != nil { + return err + } + newest.size = end + } + p.cur = newest return nil } +// deltaEntriesCheck reads every entry a delta names and checks it. +func (p *PermStore) deltaEntriesCheck(r *permRun, f *os.File) bool { + recs, err := r.records(f) + if err != nil { + return false + } + for _, rec := range recs { + if _, err := p.readEntry(rec, rec.key); err != nil { + return false + } + } + return true +} + // admit puts a delta in the window and moves what falls out of it to // pending. The caller holds the lock. func (p *PermStore) admit(d *permDelta) { @@ -401,10 +424,10 @@ func (p *PermStore) newDataFile() (*heapFile, error) { return hf, nil } -func (p *PermStore) newRunFile(deltas bool) (*runFile, error) { +func (p *PermStore) newRunFile() (*runFile, error) { id := p.nextRun p.nextRun++ - rf := &runFile{id: id, deltas: deltas} + rf := &runFile{id: id} f, err := os.OpenFile(filepath.Join(p.Directory, rf.name()), os.O_RDWR|os.O_CREATE|os.O_EXCL, 0o644) if err != nil { return nil, err @@ -436,7 +459,7 @@ func (p *PermStore) Close() error { err = cerr } } - p.files, p.runs, p.cur, p.curRun = nil, nil, nil, nil + p.files, p.runs, p.cur, p.maintRun = nil, nil, nil, nil return err } @@ -457,7 +480,7 @@ func (p *PermStore) PutIfAbsent(key [32]byte, value []byte) (existing []byte, ex } for i := len(p.window) - 1; i >= 0; i-- { d := p.window[i] - rec, found, err := d.run.lookup(d.rf.f, key) + rec, found, err := d.run.lookup(d.f, key) if err != nil { return nil, false, err } @@ -517,7 +540,7 @@ func (p *PermStore) Get(key [32]byte) ([]byte, error) { } for i := len(p.window) - 1; i >= 0; i-- { d := p.window[i] - rec, found, err := d.run.lookup(d.rf.f, key) + rec, found, err := d.run.lookup(d.f, key) if err != nil { return nil, err } @@ -540,7 +563,7 @@ func (p *PermStore) GetDeep(key [32]byte) ([]byte, error) { defer p.mu.RUnlock() for i := len(p.pending) - 1; i >= 0; i-- { d := p.pending[i] - rec, found, err := d.run.lookup(d.rf.f, key) + rec, found, err := d.run.lookup(d.f, key) if err != nil { return nil, err } @@ -625,31 +648,33 @@ func (s *permSeal) finish() error { p.mu.Unlock() return nil } + // The delta's run goes into the block's data file behind the + // block's entries, under the reserved key, and the one fsync + // covers both p.mu.Lock() - rf := p.curRun - if rf.size > PermFileBytes { - var err error - if rf, err = p.newRunFile(true); err != nil { - p.mu.Unlock() + defer p.mu.Unlock() + var w bufWriterAt + run, err := writePermRun(&w, 0, p.cur.id, s.recs, s.height) + if err != nil { + return err + } + entry := encodeEntry(s.height, heapDeltaKey, w.buf) + if p.cur.size+int64(len(entry)) > PermFileBytes { + if p.cur, err = p.newDataFile(); err != nil { return err } - p.curRun = rf } - at := rf.size - p.mu.Unlock() - run, err := writePermRun(rf.f, at, rf.id, s.recs, s.height) - if err != nil { + hf, at := p.cur, p.cur.size + if _, err = hf.f.WriteAt(entry, at); err != nil { return err } - if err = fsync(rf.f); err != nil { + hf.size += int64(len(entry)) + run.file, run.off, run.bloomAt = hf.id, at+heapHeader, at+heapHeader+run.bloomAt + if err = fsync(hf.f); err != nil { return err } p.indexBytes.Add(uint64(run.bytes)) - p.mu.Lock() - defer p.mu.Unlock() - rf.size = at + int64(run.bytes) - rf.refs++ - p.admit(&permDelta{height: s.height, run: run, rf: rf}) + p.admit(&permDelta{height: s.height, run: run, f: hf.f, data: hf.id}) p.live = map[[32]byte]permRecord{} if s.height >= p.height { p.height = s.height + 1 @@ -657,6 +682,17 @@ func (s *permSeal) finish() error { return nil } +// bufWriterAt collects what a run writer writes at offset 0. +type bufWriterAt struct{ buf []byte } + +func (w *bufWriterAt) WriteAt(b []byte, off int64) (int, error) { + if need := int(off) + len(b); need > len(w.buf) { + w.buf = append(w.buf, make([]byte, need-len(w.buf))...) + } + copy(w.buf[off:], b) + return len(b), nil +} + // AdvanceBlock sets the block new writes belong to. func (p *PermStore) AdvanceBlock(height uint64) { p.mu.Lock() @@ -682,7 +718,7 @@ func (p *PermStore) LiveCount() int { // PermFileBytes. The caller holds the lock. func (p *PermStore) maintFile() (*runFile, error) { if p.maintRun == nil || p.maintRun.size > PermFileBytes { - rf, err := p.newRunFile(false) + rf, err := p.newRunFile() if err != nil { return nil, err } @@ -734,7 +770,7 @@ func (p *PermStore) Merge() error { p.mu.Unlock() byBucket := map[int][][]permRecord{} for _, d := range pendingCopy { - recs, err := d.run.records(d.rf.f) + recs, err := d.run.records(d.f) if err != nil { return err } @@ -810,10 +846,8 @@ func (p *PermStore) Merge() error { } var keep []*permDelta for _, d := range p.pending { - if d.height <= minMerged { - d.rf.refs-- - } else { - keep = append(keep, d) + if d.height > minMerged { + keep = append(keep, d) // A delta in a data file costs nothing to drop: the file stays } } p.pending = keep @@ -936,13 +970,12 @@ func (p *PermStore) Pack() error { var inputs [][]permRecord var release []*runFile for _, d := range p.pending { - recs, err := d.run.records(d.rf.f) + recs, err := d.run.records(d.f) if err != nil { p.mu.RUnlock() return err } inputs = append(inputs, recs) - release = append(release, d.rf) } for b := range p.buckets { for i, r := range p.buckets[b].runs { @@ -1022,10 +1055,10 @@ func (p *PermStore) encodeManifest() ([]byte, error) { return permRunRef{File: rf.id, Off: r.off, Height: height} } for _, d := range p.window { - m.Window = append(m.Window, ref(d.run, d.rf, d.height)) + m.Window = append(m.Window, permRunRef{File: d.data, Off: d.run.off, Height: d.height}) } for _, d := range p.pending { - m.Pending = append(m.Pending, ref(d.run, d.rf, d.height)) + m.Pending = append(m.Pending, permRunRef{File: d.data, Off: d.run.off, Height: d.height}) } for b := range p.buckets { bm := permBucketM{Merged: p.buckets[b].merged} @@ -1040,19 +1073,14 @@ func (p *PermStore) encodeManifest() ([]byte, error) { for id := range p.files { m.DataFiles = append(m.DataFiles, id) } - for id, rf := range p.runs { - if rf.deltas { - m.DeltaFiles = append(m.DeltaFiles, id) - } else { - m.RunFiles = append(m.RunFiles, id) - } + for id := range p.runs { + m.RunFiles = append(m.RunFiles, id) } sort.Slice(m.DataFiles, func(i, j int) bool { return m.DataFiles[i] < m.DataFiles[j] }) sort.Slice(m.RunFiles, func(i, j int) bool { return m.RunFiles[i] < m.RunFiles[j] }) - sort.Slice(m.DeltaFiles, func(i, j int) bool { return m.DeltaFiles[i] < m.DeltaFiles[j] }) - // Deltas sealed after this point append to the current delta file - // from its end; open replays from there - m.DeltasFile, m.DeltasOff = p.curRun.id, p.curRun.size + // Deltas sealed after this point lie in the data files from the + // current one's end; open replays from there + m.DeltasFile, m.DeltasOff = p.cur.id, p.cur.size p.deltasFrom.file, p.deltasFrom.off = m.DeltasFile, m.DeltasOff return json.MarshalIndent(m, "", " ") } @@ -1088,7 +1116,7 @@ func (p *PermStore) writeManifest(buf []byte) error { // holds the lock. func (p *PermStore) dropUnreferencedRunFiles() error { for id, rf := range p.runs { - if rf.refs > 0 || rf == p.curRun || rf == p.maintRun { + if rf.refs > 0 || rf == p.maintRun { continue } rf.f.Close() diff --git a/database/perm_test.go b/database/perm_test.go index a7a6958..bf409f4 100644 --- a/database/perm_test.go +++ b/database/perm_test.go @@ -90,9 +90,8 @@ func TestPermStoreTiersAndReopen(t *testing.T) { } sealPerm(t, p, b) } - // A torn delta after the last seal: the crash - rf := p.curRun - _, err = rf.f.WriteAt([]byte("PRUNgarbage"), rf.size) + // A torn entry after the last seal: the crash + _, err = p.cur.f.WriteAt([]byte{9, 0, 0, 0, 1, 2, 3, 4, 5, 6}, p.cur.size) require.NoError(t, err) last := keys[len(keys)-perBlock:] for _, hf := range p.files { @@ -104,7 +103,7 @@ func TestPermStoreTiersAndReopen(t *testing.T) { r, err := OpenPermStore(dir) require.NoError(t, err) defer r.Close() - require.EqualValues(t, 3*MinFilterBlocks+5, r.height-1+1, "the replayed deltas set the height") + require.EqualValues(t, 3*MinFilterBlocks+6, r.height, "the replayed deltas leave the height at the next block, as the seal does") for i, k := range last { v, err := r.Get(k) require.NoError(t, err, "a delta replayed after the manifest") From 1d4635323cb9fdbc7ff6c4e85c7029007e37f5d7 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 20:24:55 -0500 Subject: [PATCH 33/58] The heap's delta names ranges, not keys A block's entries are contiguous in its file and each carries its key, so the delta is the ranges the block wrote -- the mover's named copies first, then the block's own appends, so a later write of a key wins on replay -- and the few copies that arrived dead: a few dozen bytes a block in place of 44 bytes a record, which was a fifth of the heap's writes and all of it dead at the next snapshot (the one-barrier run: store 6.1 GB at five minutes against 3.5). Replay scans the ranges; the last delta is trusted only if every entry in its ranges checks; a damaged entry of a committed block is named all the same, so a read reports the damage rather than absence. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 235 +++++++++++++++++++++++++++++------------- database/heap_test.go | 9 +- 2 files changed, 171 insertions(+), 73 deletions(-) diff --git a/database/heap.go b/database/heap.go index 7f2df67..57e7209 100644 --- a/database/heap.go +++ b/database/heap.go @@ -36,16 +36,20 @@ import ( // new generation in a file of its own, switched to by // rename. // -// A block's delta -- the keys it touched and where they are -- is -// appended to the block's data file right behind the block's -// entries, as an entry under a reserved key, so a block is ONE fsync -// per shard (entries and delta together) rather than two (measured: -// the second barrier was a third of the sync's cost, and the barrier -// count is what the device queue charges for). One fsync does not -// order the delta's bytes after the entries', so open verifies the -// last delta's entries by their checksums before trusting it; an -// earlier delta was followed by a later block's fsync, which covers it. -// A delta's bytes are dead once a snapshot supersedes them. +// A block's delta is appended to the block's data file right behind +// the block's entries, as an entry under a reserved key, so a block is +// ONE fsync per shard (entries and delta together) rather than two +// (measured: the second barrier was a third of the sync's cost, and +// the barrier count is what the device queue charges for). The delta +// names no key: a block's entries are contiguous in its file and each +// carries its key, so the delta is the RANGES the block wrote -- its +// own appends, and the mover's copies named this block -- and the few +// copies that arrived dead, a few dozen bytes a block (a delta of +// 44-byte records was a fifth of the heap's writes, all of it dead at +// the next snapshot and churned by the mover). Replay scans the +// ranges in order, later naming winning, and the last delta is +// trusted only if every entry in its ranges checks; an earlier delta +// was followed by a later block's fsync, which covers it. // // The key map is in memory for the live key set (spec 1.2: memory that // scales with the working set). Open loads the newest whole @@ -79,12 +83,15 @@ type HeapStore struct { index map[[32]byte]slot height uint64 // The block being written; slots taken in it may be rewritten in place - // touched is the block's delta in the making; dirty the files - // written since the last sync; release the files the mover - // emptied, deleted once the delta naming their copies is durable. - touched map[[32]byte]struct{} - dirty map[uint32]*heapFile - release []uint32 + // The block's delta in the making: the ranges the block appended + // (the mover's named copies first, then the block's own) and the + // copies that arrived dead; dirty the files written since the last + // sync; release the files the mover emptied, deleted once the + // delta naming their copies is durable. + ranges []heapRange + excluded []heapRange + dirty map[uint32]*heapFile + release []uint32 // deltaAt is where the last delta ended: the replay point a // snapshot records, and what a file is cut back to on open deltaAt struct { @@ -127,6 +134,12 @@ type heapFile struct { releasing bool // Emptied by a pass; deleted by the next sync } +// heapRange is a stretch of one data file: what a delta names. +type heapRange struct { + file uint32 + from, to uint32 +} + // slot is where an entry lives: its file, its offset there, the value // length, and the block that took it, which decides whether a rewrite // may reuse it in place. @@ -276,10 +289,9 @@ func (h *HeapStore) Open() (err error) { return err } h.index = map[[32]byte]slot{} - h.touched = map[[32]byte]struct{}{} h.dirty = map[uint32]*heapFile{} h.files = map[uint32]*heapFile{} - h.release = nil + h.release, h.ranges, h.excluded = nil, nil, nil h.closed = false h.liveBytes, h.deadBytes = 0, 0 // The newest whole generation is the index; older ones are what a @@ -399,11 +411,8 @@ func (h *HeapStore) replayDeltas(dataIDs []uint32) error { } } for i, d := range deltas { - if i == len(deltas)-1 && !h.deltaEntriesCheck(d.recs) { - break // The block did not commit - } - if _, err := h.applyRecord(d.recs); err != nil { - break + if !h.applyDelta(d.recs, i == len(deltas)-1) { + break // Torn, or the last block's entries do not all check: it did not commit } h.deltaAt.file, h.deltaAt.off = d.file, d.end h.files[d.file].deltas += d.end - d.off @@ -411,37 +420,6 @@ func (h *HeapStore) replayDeltas(dataIDs []uint32) error { return nil } -// deltaEntriesCheck reads every entry a delta names and checks it. -func (h *HeapStore) deltaEntriesCheck(recs []byte) bool { - if len(recs) < heapIndexHdr { - return false - } - count := int(binary.LittleEndian.Uint32(recs[12:])) - end := heapIndexHdr + count*heapIndexRec - if len(recs) < end+4 || crc32.ChecksumIEEE(recs[:end]) != binary.LittleEndian.Uint32(recs[end:]) { - return false - } - for at := heapIndexHdr; at < end; at += heapIndexRec { - var key [32]byte - copy(key[:], recs[at:]) - file := binary.LittleEndian.Uint32(recs[at+32:]) - off := binary.LittleEndian.Uint32(recs[at+36:]) - n := binary.LittleEndian.Uint32(recs[at+40:]) - hf := h.files[file] - if hf == nil { - return false - } - buf := make([]byte, heapHeader+int(n)+heapTrailer) - if _, err := hf.f.ReadAt(buf, int64(off)); err != nil { - return false - } - if _, err := heapEntryValue(buf, key); err != nil { - return false - } - } - return true -} - // startGeneration begins an index generation: a snapshot of the map // with the replay point -- where the last delta ended -- written // aside and fsynced, renamed into place, the directory fsynced, and @@ -527,15 +505,107 @@ func fsyncDir(directory string) error { return fsync(d) } -// encodeIndex is one index record: a snapshot of the whole map or the -// block's delta of the keys it touched, checksummed. The caller holds -// the lock. -func (h *HeapStore) encodeIndex(magic uint32, keys map[[32]byte]struct{}) []byte { - return h.encodeIndexOf(magic, func(emit func(key [32]byte)) { - for key := range keys { - emit(key) +// encodeDelta is the block's delta: marker, height, the ranges the +// block wrote (the mover's named copies first, then the block's own +// appends, so a later write of a key wins on replay) and the copies +// that arrived dead, checksummed. The caller holds the lock. +func (h *HeapStore) encodeDelta() []byte { + const rangeRec = 4 + 4 + 4 + buf := make([]byte, 4+8+4+4+len(h.ranges)*rangeRec+len(h.excluded)*rangeRec+4) + binary.LittleEndian.PutUint32(buf, heapMagic) + binary.LittleEndian.PutUint64(buf[4:], h.height) + binary.LittleEndian.PutUint32(buf[12:], uint32(len(h.ranges))) + binary.LittleEndian.PutUint32(buf[16:], uint32(len(h.excluded))) + at := 20 + for _, r := range append(append([]heapRange(nil), h.ranges...), h.excluded...) { + binary.LittleEndian.PutUint32(buf[at:], r.file) + binary.LittleEndian.PutUint32(buf[at+4:], r.from) + binary.LittleEndian.PutUint32(buf[at+8:], r.to) + at += rangeRec + } + binary.LittleEndian.PutUint32(buf[at:], crc32.ChecksumIEEE(buf[:at])) + return buf +} + +// decodeDelta reads a delta's ranges and exclusions. +func decodeDelta(buf []byte) (height uint64, ranges, excluded []heapRange, ok bool) { + const rangeRec = 4 + 4 + 4 + if len(buf) < 20 || binary.LittleEndian.Uint32(buf) != heapMagic { + return + } + nr, ne := int(binary.LittleEndian.Uint32(buf[12:])), int(binary.LittleEndian.Uint32(buf[16:])) + end := 20 + (nr+ne)*rangeRec + if len(buf) < end+4 || crc32.ChecksumIEEE(buf[:end]) != binary.LittleEndian.Uint32(buf[end:]) { + return + } + height = binary.LittleEndian.Uint64(buf[4:]) + at := 20 + for i := 0; i < nr+ne; i++ { + r := heapRange{file: binary.LittleEndian.Uint32(buf[at:]), from: binary.LittleEndian.Uint32(buf[at+4:]), to: binary.LittleEndian.Uint32(buf[at+8:])} + if i < nr { + ranges = append(ranges, r) + } else { + excluded = append(excluded, r) + } + at += rangeRec + } + return height, ranges, excluded, true +} + +// applyDelta names every entry in the delta's ranges, in order, later +// naming winning, except the excluded copies; verify makes it refuse +// a delta any of whose entries does not check. The caller holds the +// lock; the files are open. +func (h *HeapStore) applyDelta(recs []byte, verify bool) bool { + height, ranges, excluded, ok := decodeDelta(recs) + if !ok { + return false + } + skip := map[heapRange]bool{} + for _, e := range excluded { + skip[heapRange{file: e.file, from: e.from}] = true + } + type naming struct { + key [32]byte + s slot + } + var names []naming + for _, r := range ranges { + hf := h.files[r.file] + if hf == nil || int64(r.to) > hf.size { + return false + } + buf := make([]byte, r.to-r.from) + if _, err := hf.f.ReadAt(buf, int64(r.from)); err != nil { + return false + } + at := int64(0) + for at < int64(len(buf)) { + size, _, key, value, ok := decodeEntry(buf[at:]) + if size == 0 || !ok && verify { + return false // Torn; or the last block's entries do not all check + } + off := int64(r.from) + at + // A damaged entry of a committed block is named all the + // same, so a read of it reports the damage rather than + // answering that the key is absent + n := uint32(len(value)) + if !ok { + n = binary.LittleEndian.Uint32(buf[at:]) + } + if key != heapDeltaKey && !skip[heapRange{file: r.file, from: uint32(off)}] { + names = append(names, naming{key: key, s: slot{file: r.file, off: uint32(off), n: n}}) + } + at += size } - }, len(keys)) + } + for _, n := range names { + h.index[n.key] = n.s + } + if height > h.height { + h.height = height + } + return true } func (h *HeapStore) encodeIndexOf(magic uint32, each func(emit func(key [32]byte)), count int) []byte { @@ -566,7 +636,7 @@ func (h *HeapStore) applyRecord(buf []byte) (int, error) { return 0, errHeapTorn } magic := binary.LittleEndian.Uint32(buf) - if magic != heapMagic && magic != heapSnapshot { + if magic != heapSnapshot { return 0, errHeapTorn } count := int(binary.LittleEndian.Uint32(buf[12:])) @@ -689,8 +759,9 @@ func decodeEntry(buf []byte) (size int64, height uint64, key [32]byte, value []b if n == 0 && binary.LittleEndian.Uint64(buf[4:]) == 0 || size > int64(len(buf)) { return 0, 0, key, nil, false } + copy(key[:], buf[12:]) if crc32.ChecksumIEEE(buf[4:heapHeader+n]) != binary.LittleEndian.Uint32(buf[heapHeader+n:]) { - return size, 0, key, nil, false // Damaged: its size lets a scan step over it + return size, 0, key, nil, false // Damaged: its size lets a scan step over it, its key lets replay name it } height = binary.LittleEndian.Uint64(buf[4:]) copy(key[:], buf[12:]) @@ -735,10 +806,21 @@ func (h *HeapStore) reserve(mover bool, size int64) (hf *heapFile, off int64, er hf.inflight++ } else { h.dirty[hf.id] = hf + h.extendRange(hf.id, off, off+size) } return hf, off, nil } +// extendRange adds [from, to) of a file to the block's delta, +// extending the last range when it abuts. The caller holds the lock. +func (h *HeapStore) extendRange(file uint32, from, to int64) { + if n := len(h.ranges); n > 0 && h.ranges[n-1].file == file && int64(h.ranges[n-1].to) == from { + h.ranges[n-1].to = uint32(to) + return + } + h.ranges = append(h.ranges, heapRange{file: file, from: uint32(from), to: uint32(to)}) +} + // kill accounts a slot its key stopped naming. The caller holds the // lock. func (h *HeapStore) kill(s slot) { @@ -788,7 +870,6 @@ func (h *HeapStore) Put(key [32]byte, value []byte) error { return err } h.index[key] = s - h.touched[key] = struct{}{} return nil } @@ -872,12 +953,13 @@ func (h *HeapStore) beginBlockSync() (blockSync, error) { } p := &heapSync{h: h, release: h.release} h.release = nil - if len(h.touched) > 0 || len(p.release) > 0 { + if len(h.ranges) > 0 || len(p.release) > 0 { // The delta goes into the block's data file behind its entries, // under the reserved key, so the block's one fsync covers both - p.delta = h.encodeIndex(heapMagic, h.touched) - h.touched = map[[32]byte]struct{}{} + p.delta = h.encodeDelta() + h.ranges, h.excluded = nil, nil hf, off, err := h.reserve(false, entrySize(len(p.delta))) + h.ranges = nil // The delta entry itself is not part of the block's range if err != nil { h.syncMu.Unlock() return nil, err @@ -1106,17 +1188,29 @@ func (h *HeapStore) clean(budget int64) (bool, error) { // the files the pass emptied h.mu.Lock() defer h.mu.Unlock() + // The copies were reserved in order, so they form ranges of the + // mover's files; a copy that arrived dead is excluded from them. + // The mover's ranges go before the block's own in the delta, so + // that a put of the same key later in the block wins on replay + var mine []heapRange for _, m := range moves { m.hf.inflight-- + size := entrySize(int(m.to.n)) + if n := len(mine); n > 0 && mine[n-1].file == m.to.file && int64(mine[n-1].to) == int64(m.to.off) { + mine[n-1].to = uint32(int64(m.to.off) + size) + } else { + mine = append(mine, heapRange{file: m.to.file, from: m.to.off, to: uint32(int64(m.to.off) + size)}) + } s, live := h.index[m.key] if live && s == m.from { h.index[m.key] = m.to - h.touched[m.key] = struct{}{} h.kill(m.from) } else { h.kill(m.to) // Reserved and written, but no longer wanted + h.excluded = append(h.excluded, heapRange{file: m.to.file, from: m.to.off}) } } + h.ranges = append(mine, h.ranges...) for _, hf := range taken { if hf.live == 0 && hf != h.cur && hf != h.mov { hf.releasing = true @@ -1221,8 +1315,7 @@ func RepairHeapStore(directory string, committed uint64) (*HeapStore, error) { for _, g := range gens { os.Remove(filepath.Join(directory, indexName(g))) } - h := &HeapStore{Directory: directory, index: map[[32]byte]slot{}, touched: map[[32]byte]struct{}{}, - dirty: map[uint32]*heapFile{}, files: map[uint32]*heapFile{}, height: committed} + h := &HeapStore{Directory: directory, index: map[[32]byte]slot{}, dirty: map[uint32]*heapFile{}, files: map[uint32]*heapFile{}, height: committed} heights := map[[32]byte]uint64{} for _, id := range dataIDs { data, err := os.ReadFile(filepath.Join(directory, dataName(id))) diff --git a/database/heap_test.go b/database/heap_test.go index f015152..ad8dd20 100644 --- a/database/heap_test.go +++ b/database/heap_test.go @@ -316,9 +316,14 @@ func TestHeapShardRoundTrip(t *testing.T) { } _, dyna := kvs.Stats() require.EqualValues(t, 60*200, dyna.PutTotal) - require.NoError(t, kvs.SealBlock(61)) // The sync that deletes what the last pass emptied + // A pass for what the last ten blocks left dead, and the sync that + // deletes what it emptied + require.NoError(t, kvs.Compress()) + require.NoError(t, kvs.SealBlock(61)) + require.NoError(t, kvs.Compress()) + require.NoError(t, kvs.SealBlock(62)) dead, live := kvs.Shards[0].Heap.HoleRatio() - require.Less(t, dead, 4*live, "the mover keeps dead bytes bounded (deltas are dead once superseded, and reclaimed)") + require.Less(t, dead, 2*live+HeapFileBytes, "dead bytes are bounded: at most the current file, which the mover never takes, beyond the live set") require.NoError(t, kvs.Close()) re, err := OpenKVShard(dir) From 0a2982215056e42432957cc33ddd17d7efb5fb3c Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 20:34:11 -0500 Subject: [PATCH 34/58] The mover releases wholly dead files outright and scans 128 MB a pass A pass took at most four files, whether they needed a copy or not, and at 16 MB a file that paced release at about the rate dead bytes appeared: the store floated at 5.5 GB with the size bound engaged and the movers in their heavy mode every few minutes. A file with nothing live is now released without a scan, any number a pass; a pass takes up to eight files or 128 MB scanned, still copying at most 4 MB. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/database/heap.go b/database/heap.go index 57e7209..b9cea35 100644 --- a/database/heap.go +++ b/database/heap.go @@ -185,8 +185,17 @@ var HeapFileBlocks uint64 = 64 // cadence (~4 MB, of which a quarter to a third is live). var HeapCleanBytes int64 = 4 << 20 -// HeapCleanFiles bounds a pass by files taken as well. -var HeapCleanFiles = 4 +// HeapCleanFiles and HeapScanBytes bound a pass by the files it takes +// and the bytes it reads: with hot keys most of a file is dead and +// costs nothing to copy, so what limits the mover's pace is how much +// it looks at. At the soak's rate a shard makes ~50 MB of dead bytes +// per cadence; a pass that scans 128 MB releases more than that with +// room to spare (four 16 MB files did not, and the store floated with +// the size bound engaged). +var ( + HeapCleanFiles = 8 + HeapScanBytes int64 = 128 << 20 +) // HeapCleanRatio is the dead fraction a file must reach before the // mover takes it -- unless the heap is over its size bound, when the @@ -1084,18 +1093,33 @@ func (h *HeapStore) clean(budget int64) (bool, error) { h.mu.Unlock() return false, nil // The last pass's deletions are still waiting on a sync } + // A file with nothing live in it needs no scan and no copy: it is + // released outright, any number of them a pass, one sync late as + // every release is. Without this the pass's file count paced the + // mover at about the rate dead bytes appeared, and the store + // floated with the size bound engaged. + emptied := 0 + for _, hf := range h.files { + if hf.live == 0 && hf != h.cur && hf != h.mov && !hf.cleaning && hf.inflight == 0 && !hf.releasing && hf.size > 0 { + hf.releasing = true + h.release = append(h.release, hf.id) + emptied++ + } + } var taken []*heapFile - for len(taken) < HeapCleanFiles { + var scan int64 + for len(taken) < HeapCleanFiles && scan < HeapScanBytes { hf := h.pickFile() if hf == nil { break } hf.cleaning = true taken = append(taken, hf) + scan += hf.size } h.mu.Unlock() if len(taken) == 0 { - return false, nil + return emptied > 0, nil } // 2. Without the lock: read and decode them. A picked file is // neither the block's nor the mover's, so its bytes do not change; From b8aae7f621ac4882cb93353a5c979b091ad63ecb Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 20:42:27 -0500 Subject: [PATCH 35/58] The mover finds a file's live entries through the index A pass scanned each picked file end to end, decoding and checksumming up to 128 MB a shard to find the few live entries, and nine stores' passes together pushed gigabytes through the CRC at once: load 27 on 24 cores and every block's seal at 200 ms for the minute (heap runs with range deltas, minute 2). The index already knows which slots are live in a file; a pass now takes them from one walk of the map under the lock and reads only those entries. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 75 ++++++++++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 35 deletions(-) diff --git a/database/heap.go b/database/heap.go index b9cea35..db668d5 100644 --- a/database/heap.go +++ b/database/heap.go @@ -1121,30 +1121,52 @@ func (h *HeapStore) clean(budget int64) (bool, error) { if len(taken) == 0 { return emptied > 0, nil } - // 2. Without the lock: read and decode them. A picked file is - // neither the block's nor the mover's, so its bytes do not change; - // only what the index says of them can, and that is checked under - // the lock. Decoding here, checksums included, keeps 16 MB of - // CRC per file off the lock + // 2. Under the lock, briefly: the picked files' live slots, from + // the index -- one pass over the map, not a decode of the files. + // Scanning a file to find its live entries read and checksummed + // up to 128 MB a shard per pass, and nine stores' passes together + // starved the block loops for CPU (load 27 on 24 cores, seal p50 + // 200 ms for the minute). Then without the lock: read those + // entries, and only those + type liveSlot struct { + key [32]byte + s slot + } + h.mu.Lock() + picked := map[uint32]int{} + for i, hf := range taken { + picked[hf.id] = i + } + slots := make([][]liveSlot, len(taken)) + for key, s := range h.index { + if i, ok := picked[s.file]; ok { + slots[i] = append(slots[i], liveSlot{key: key, s: s}) + } + } + h.mu.Unlock() entries := make([][]heapEntry, len(taken)) for i, hf := range taken { - buf := make([]byte, hf.size) - if _, err := hf.f.ReadAt(buf, 0); err != nil { - h.mu.Lock() - for _, hf := range taken { - hf.cleaning = false + sort.Slice(slots[i], func(a, b int) bool { return slots[i][a].s.off < slots[i][b].s.off }) + for _, ls := range slots[i] { + buf := make([]byte, entrySize(int(ls.s.n))) + if _, err := hf.f.ReadAt(buf, int64(ls.s.off)); err != nil { + h.mu.Lock() + for _, hf := range taken { + hf.cleaning = false + } + h.mu.Unlock() + return false, err } - h.mu.Unlock() - return false, err + size, _, key, value, ok := decodeEntry(buf) + if !ok || key != ls.key { + continue // Damaged or stale: left where it is, and a read of it reports the damage + } + entries[i] = append(entries[i], heapEntry{off: ls.s.off, size: size, key: key, value: value}) } - entries[i] = decodeFile(buf) h.cleanedBytes.Add(uint64(hf.size)) } - // 3. Under the lock, a chunk of entries at a time: decide what is - // live and reserve each copy's slot in the mover's file. A pass - // walks up to a million entries; taking the lock per chunk keeps - // each hold to a millisecond or so. A block sync may begin - // between chunks; the mover's file is never in its dirty set + // 3. Under the lock, a chunk of entries at a time: confirm each is + // still named and reserve its copy's slot in the mover's file var moves []heapMove var copied int64 for i, hf := range taken { @@ -1280,23 +1302,6 @@ type heapEntry struct { value []byte } -// decodeFile walks a file's bytes into its entries, stopping at the -// first unwritten, torn or damaged slot. -func decodeFile(buf []byte) (entries []heapEntry) { - var at int64 - for at < int64(len(buf)) { - size, _, key, value, ok := decodeEntry(buf[at:]) - if size == 0 { - break - } - if ok { - entries = append(entries, heapEntry{off: uint32(at), size: size, key: key, value: value}) - } - at += size // A damaged entry stays where it is: not moved, not named by the mover - } - return entries -} - // planFile reserves, in the mover's file, a slot for each entry of a // picked file the index still names, up to budget bytes; the rest // wait for the next pass. The caller holds the lock; the entries From 0de9459ba4ef3c68b599d6cc874897b08f852621 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 20:42:50 -0500 Subject: [PATCH 36/58] The proposal records step 3 as built: deltas in the data files, range deltas, the index-driven mover Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- .../2026-09-16-entries-written-once.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index 62a7c11..7620761 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -289,5 +289,20 @@ point" and closes #33. remeasured next.* Measured alone first (`-stores 9 -dyna 0`), then with the heap under the full load, which is the acceptance run. -3. The block's deltas in the data files and the store-level commit: - one barrier round per shard, one per store. +3. The block's deltas in the data files: one barrier per shard per + layer. *Built for both layers.* The heap's delta names no key: + a block's entries are contiguous in its file and carry their + keys, so the delta is the ranges the block wrote (the mover's + named copies first, then the block's own) and the copies that + arrived dead, a few dozen bytes a block; a delta of 44-byte + records was a fifth of the heap's writes. Replay scans the + ranges and trusts the last delta only if every entry in it + checks. The permanent layer's delta stays a run with its filter, + since it is the window's search structure, but lives in the data + file too. Alone, the heap seals at 44-54 ms p50 at nine stores + (57-64 with two barriers). The mover finds a file's live entries + through the index rather than by scanning the file, because nine + stores' scans together starved the block loops for CPU. Still to + do: the store-level commit (one block record naming every + shard's deltas) and the per-store data file, so that a hundred + shards cost a block one barrier. From beb5c6a99a3cd7f61d1668be074598bba4e4cb74 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 20:59:13 -0500 Subject: [PATCH 37/58] The per-minute row carries the heap's split: fsync, releases, snapshots Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/main.go | 75 ++++++++++++++++++++++++++++++++++++--------- database/heap.go | 32 ++++++++++++++----- 2 files changed, 85 insertions(+), 22 deletions(-) diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index 90c4d1c..e4a20e5 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -438,23 +438,14 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { } } var height uint64 - var holes, live int64 - var scanned, moved, syncs, syncBytes uint64 - var heapFsync, deltaSync time.Duration + hs := sumHeap(stores) + holes, live, scanned, moved, syncs, syncBytes, heapFsync, deltaSync := hs.holes, hs.live, hs.scanned, hs.moved, hs.syncs, hs.bytes, hs.fsync, hs.delta var permMerges, permFolds, permPacks, permIndexBytes uint64 for _, s := range stores { if s.height > height { height = s.height } for _, sh := range s.kv.Shards { - if sh.Heap != nil { - h, l := sh.Heap.HoleRatio() - holes, live = holes+h, live+l - sc, mv := sh.Heap.Cleaned() - scanned, moved = scanned+sc, moved+mv - n, b, hf, ds := sh.Heap.SyncCost() - syncs, syncBytes, heapFsync, deltaSync = syncs+n, syncBytes+b, heapFsync+hf, deltaSync+ds - } if sh.Perm != nil { m, f, pk, ib := sh.Perm.Counters() permMerges, permFolds, permPacks, permIndexBytes = permMerges+m, permFolds+f, permPacks+pk, permIndexBytes+ib @@ -472,10 +463,47 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { "heapSyncs": syncs, "heapSyncKBAvg": float64(syncBytes) / 1e3 / float64(max(syncs, 1)), "permMerges": permMerges, "permFolds": permFolds, "permPacks": permPacks, "permIndexMB": float64(permIndexBytes) / 1e6, "heapFsyncMsAvg": float64(heapFsync) / 1e6 / float64(max(syncs, 1)), "heapDeltaMsAvg": float64(deltaSync) / 1e6 / float64(max(syncs, 1)), + "heapReleases": hs.releases, "heapReleaseMs": hs.release.Milliseconds(), "heapSnapshots": hs.snapshots, "heapSnapshotMs": hs.snapshot.Milliseconds(), }) return b } +// heapSplit sums the heap layer's counters across the stores: the +// mover's and the sync's cost, cumulative since the start. +type heapSplit struct { + holes, live int64 + scanned, moved, syncs, bytes uint64 + fsync, delta time.Duration + releases, snapshots uint64 + release, snapshot time.Duration +} + +func (a heapSplit) minus(b heapSplit) heapSplit { + return heapSplit{holes: a.holes, live: a.live, + scanned: a.scanned - b.scanned, moved: a.moved - b.moved, syncs: a.syncs - b.syncs, bytes: a.bytes - b.bytes, + fsync: a.fsync - b.fsync, delta: a.delta - b.delta, releases: a.releases - b.releases, snapshots: a.snapshots - b.snapshots, + release: a.release - b.release, snapshot: a.snapshot - b.snapshot} +} + +func sumHeap(stores []*store) (hs heapSplit) { + for _, s := range stores { + for _, sh := range s.kv.Shards { + if sh.Heap == nil { + continue + } + h, l := sh.Heap.HoleRatio() + hs.holes, hs.live = hs.holes+h, hs.live+l + sc, mv := sh.Heap.Cleaned() + hs.scanned, hs.moved = hs.scanned+sc, hs.moved+mv + n, b, hf, ds := sh.Heap.SyncCost() + hs.syncs, hs.bytes, hs.fsync, hs.delta = hs.syncs+n, hs.bytes+b, hs.fsync+hf, hs.delta+ds + r, rt, sn, st := sh.Heap.MoverCost() + hs.releases, hs.release, hs.snapshots, hs.snapshot = hs.releases+r, hs.release+rt, hs.snapshots+sn, hs.snapshot+st + } + } + return hs +} + func fail(what string, err error) { fmt.Fprintln(os.Stderr, "bdbench:", what+":", err) os.Exit(1) @@ -548,7 +576,9 @@ func main() { _ = csvw.Write([]string{"minute", "blocks", "over_budget", "block_p50_ms", "block_p90_ms", "block_max_ms", "seal_p50_ms", "seal_p90_ms", "seal_max_ms", "dyna_put_p99_us", "perm_put_p99_us", "read_p99_us", "compress_passes", "compress_s", "merge_passes", "merge_s", "pack_passes", "pack_s", "skipped", - "read_MBps", "write_MBps", "store_MB", "files", "perm_history", "dyna_history", "bloom_MB", "mismatches"}) + "read_MBps", "write_MBps", "store_MB", "files", "perm_history", "dyna_history", "bloom_MB", "mismatches", + "heap_syncs", "heap_fsync_ms_avg", "heap_sync_MB_avg", "heap_moved_MB", "heap_hole_MB", "heap_live_MB", + "heap_releases", "heap_release_ms", "heap_snapshots", "heap_snapshot_ms"}) csvw.Flush() fmt.Printf("bdbench: %d store(s) x %d shards, seal limit %d, window %d, maintenance every %d blocks, pack every %d; per block per store %d dyna + %d perm puts, %d reads; %s blocks for %s; %s\n", @@ -559,6 +589,7 @@ func main() { deadline := start.Add(c.duration) ioR0, ioW0 := procIO() period := start + var heap0 heapSplit report := func() { elapsed := time.Since(period) @@ -592,11 +623,20 @@ func main() { } minute := int(time.Since(start) / time.Minute) mism := t.mismatches.Load() - fmt.Printf("%3dm blocks %4d | block p50/p90/max %s/%s/%s ms | seal p50/p90/max %s/%s/%s ms | put p99 dyna %s perm %s us | read p99 %s us | maint compress %d (%.1fs) merge %d (%.1fs) pack %d (%.1fs) skipped %d | disk r %.0f w %.0f MB/s | store %.0f MB %d files | history perm %d dyna %d | bloom %.0f MB | mismatches %d%s\n", + heap1 := sumHeap(stores) + hs := heap1.minus(heap0) + heap0 = heap1 + heapNote := "" + if hs.syncs > 0 { + heapNote = fmt.Sprintf(" | heap fsync %.1f ms x %d (%.1f MB) moved %.0f MB hole %.0f MB live %.0f MB releases %d (%d ms) snapshots %d (%d ms)", + float64(hs.fsync)/1e6/float64(hs.syncs), hs.syncs, float64(hs.bytes)/1e6/float64(hs.syncs), float64(hs.moved)/1e6, float64(hs.holes)/1e6, float64(hs.live)/1e6, + hs.releases, hs.release.Milliseconds(), hs.snapshots, hs.snapshot.Milliseconds()) + } + fmt.Printf("%3dm blocks %4d | block p50/p90/max %s/%s/%s ms | seal p50/p90/max %s/%s/%s ms | put p99 dyna %s perm %s us | read p99 %s us | maint compress %d (%.1fs) merge %d (%.1fs) pack %d (%.1fs) skipped %d | disk r %.0f w %.0f MB/s | store %.0f MB %d files | history perm %d dyna %d | bloom %.0f MB | mismatches %d%s%s\n", minute, len(bt), ms(pct(bt, .5)), ms(pct(bt, .9)), ms(pct(bt, 1)), ms(pct(st, .5)), ms(pct(st, .9)), ms(pct(st, 1)), us(pct(dp, .99)), us(pct(pp, .99)), us(pct(rt, .99)), passes["compress"], spent["compress"].Seconds(), passes["merge"], spent["merge"].Seconds(), passes["pack"], spent["pack"].Seconds(), skipped, - rMB, wMB, float64(bytes)/1e6, files, permHist, dynaHist, float64(bloom)/1e6, mism, note) + rMB, wMB, float64(bytes)/1e6, files, permHist, dynaHist, float64(bloom)/1e6, mism, heapNote, note) _ = csvw.Write([]string{strconv.Itoa(minute), strconv.Itoa(len(bt)), strconv.FormatUint(over, 10), ms(pct(bt, .5)), ms(pct(bt, .9)), ms(pct(bt, 1)), ms(pct(st, .5)), ms(pct(st, .9)), ms(pct(st, 1)), us(pct(dp, .99)), us(pct(pp, .99)), us(pct(rt, .99)), @@ -606,7 +646,12 @@ func main() { strconv.FormatFloat(rMB, 'f', 1, 64), strconv.FormatFloat(wMB, 'f', 1, 64), strconv.FormatFloat(float64(bytes)/1e6, 'f', 0, 64), strconv.Itoa(files), strconv.Itoa(permHist), strconv.Itoa(dynaHist), strconv.FormatFloat(float64(bloom)/1e6, 'f', 1, 64), - strconv.FormatUint(mism, 10)}) + strconv.FormatUint(mism, 10), + strconv.FormatUint(hs.syncs, 10), strconv.FormatFloat(float64(hs.fsync)/1e6/float64(max(hs.syncs, 1)), 'f', 1, 64), + strconv.FormatFloat(float64(hs.bytes)/1e6/float64(max(hs.syncs, 1)), 'f', 1, 64), strconv.FormatFloat(float64(hs.moved)/1e6, 'f', 0, 64), + strconv.FormatFloat(float64(hs.holes)/1e6, 'f', 0, 64), strconv.FormatFloat(float64(hs.live)/1e6, 'f', 0, 64), + strconv.FormatUint(hs.releases, 10), strconv.FormatInt(hs.release.Milliseconds(), 10), + strconv.FormatUint(hs.snapshots, 10), strconv.FormatInt(hs.snapshot.Milliseconds(), 10)}) csvw.Flush() } diff --git a/database/heap.go b/database/heap.go index db668d5..8329447 100644 --- a/database/heap.go +++ b/database/heap.go @@ -119,6 +119,10 @@ type HeapStore struct { // The sync's cost, split: nanoseconds in the data files' fsyncs // and in the delta's write and fsync, and the syncs and bytes syncs, syncHeapNs, syncLogNs, syncBytes atomic.Uint64 + // The mover's cost on and off the block's path: files unlinked by + // a block's finish and the time in those unlinks; snapshots + // written and the time in them + releases, releaseNs, snapshotsN, snapshotNs atomic.Uint64 } // heapFile is one data file and its accounting. @@ -1007,14 +1011,19 @@ func (p *heapSync) finish() (err error) { h.syncBytes.Add(uint64(p.bytes)) h.mu.Lock() defer h.mu.Unlock() - for _, id := range p.release { - hf := h.files[id] - hf.f.Close() - delete(h.files, id) - h.deadBytes -= hf.dead - if err = os.Remove(filepath.Join(h.Directory, dataName(id))); err != nil { - return err + if len(p.release) > 0 { + t = time.Now() + for _, id := range p.release { + hf := h.files[id] + hf.f.Close() + delete(h.files, id) + h.deadBytes -= hf.dead + if err = os.Remove(filepath.Join(h.Directory, dataName(id))); err != nil { + return err + } } + h.releases.Add(uint64(len(p.release))) + h.releaseNs.Add(uint64(time.Since(t))) } return nil } @@ -1035,7 +1044,10 @@ func (h *HeapStore) compact() (bool, error) { } h.mu.Unlock() if due { + t := time.Now() err = h.Snapshot() + h.snapshotsN.Add(1) + h.snapshotNs.Add(uint64(time.Since(t))) } return moved, err } @@ -1445,6 +1457,12 @@ func (h *HeapStore) Cleaned() (scanned, moved uint64) { return h.cleanedBytes.Load(), h.movedBytes.Load() } +// MoverCost reports the files a block's finish has unlinked and the +// time in those unlinks, and the snapshots written and their time. +func (h *HeapStore) MoverCost() (releases uint64, release time.Duration, snapshots uint64, snapshot time.Duration) { + return h.releases.Load(), time.Duration(h.releaseNs.Load()), h.snapshotsN.Load(), time.Duration(h.snapshotNs.Load()) +} + // SetFilterBlocks and SetSealLimit are the segment layer's knobs; a // heap has neither a window nor a tail. func (h *HeapStore) SetFilterBlocks(uint64) error { return nil } From 0d0174db3d5e9d957e75838d747468f1fedaab75 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 21:09:08 -0500 Subject: [PATCH 38/58] bdbench records a block's samples under one lock Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/main.go | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index e4a20e5..37212c2 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -135,6 +135,15 @@ func (s *samples) add(d time.Duration) { s.mu.Unlock() } +// addAll takes a block's worth of samples under one lock: forty +// thousand reads a block, nine stores, made the recorder's lock a +// fifth of the benchmark's CPU. +func (s *samples) addAll(v []time.Duration) { + s.mu.Lock() + s.v = append(s.v, v...) + s.mu.Unlock() +} + func (s *samples) take() []time.Duration { s.mu.Lock() v := s.v @@ -209,6 +218,8 @@ type store struct { last map[[32]byte][]byte height uint64 phase uint64 // Blocks this store's maintenance cadence is offset by + // A block's samples, handed to the tallies once per block + dynaT, permT, readT []time.Duration maintaining atomic.Bool maintWG sync.WaitGroup } @@ -257,6 +268,12 @@ func openStore(c config, id int) (*store, error) { func (s *store) block(c config, t *tallies) error { s.height++ start := time.Now() + s.dynaT, s.permT, s.readT = s.dynaT[:0], s.permT[:0], s.readT[:0] + defer func() { + t.dynaPut.addAll(s.dynaT) + t.permPut.addAll(s.permT) + t.readT.addAll(s.readT) + }() for i := 0; i < c.dynaPuts; i++ { r := float64(s.rnd.UintN(1<<20)) / (1 << 20) k := s.hot[int(r*r*float64(c.hotKeys))%c.hotKeys] @@ -265,7 +282,7 @@ func (s *store) block(c config, t *tallies) error { if err := s.kv.PutDyna(k, v); err != nil { return fmt.Errorf("store %d PutDyna: %w", s.id, err) } - t.dynaPut.add(time.Since(at)) + s.dynaT = append(s.dynaT, time.Since(at)) if len(s.last) < checked || s.last[k] != nil { s.last[k] = v } @@ -277,7 +294,7 @@ func (s *store) block(c config, t *tallies) error { if err := s.kv.PutPerm(k, v); err != nil { return fmt.Errorf("store %d PutPerm: %w", s.id, err) } - t.permPut.add(time.Since(at)) + s.permT = append(s.permT, time.Since(at)) if len(s.permKeys) < permSample { s.permKeys = append(s.permKeys, k) } else if s.rnd.UintN(64) == 0 { // Keep the sample spread across every age @@ -298,7 +315,7 @@ func (s *store) block(c config, t *tallies) error { } at := time.Now() v, err := get(k) - t.readT.add(time.Since(at)) + s.readT = append(s.readT, time.Since(at)) if err != nil && !errors.Is(err, os.ErrNotExist) && !strings.Contains(err.Error(), "not found") { return fmt.Errorf("store %d read: %w", s.id, err) } From c76f02bcbaf294eabaa717aa6bf5801ce2adafdb Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 21:09:49 -0500 Subject: [PATCH 39/58] The live state carries each store's seal p90 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/main.go | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index 37212c2..c28ba94 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -169,6 +169,7 @@ func us(d time.Duration) string { return strconv.FormatInt(int64(d/time.Microsec type recent struct { at time.Time block, seal time.Duration + store int } // tallies is what every store adds to and the report takes from. @@ -215,13 +216,13 @@ type store struct { // A few hot keys are checked on read: the value the store returns // must be the value last written. A platform that only times // answers cannot tell a fast wrong answer from a fast right one. - last map[[32]byte][]byte - height uint64 - phase uint64 // Blocks this store's maintenance cadence is offset by + last map[[32]byte][]byte + height uint64 + phase uint64 // Blocks this store's maintenance cadence is offset by // A block's samples, handed to the tallies once per block dynaT, permT, readT []time.Duration - maintaining atomic.Bool - maintWG sync.WaitGroup + maintaining atomic.Bool + maintWG sync.WaitGroup } const ( @@ -333,7 +334,7 @@ func (s *store) block(c config, t *tallies) error { t.blockTimes.add(took) t.blocks.Add(1) t.ringMu.Lock() - t.ring[t.ringN%uint64(len(t.ring))] = recent{at: time.Now(), block: took, seal: sealTook} + t.ring[t.ringN%uint64(len(t.ring))] = recent{at: time.Now(), block: took, seal: sealTook, store: s.id} t.ringN++ t.ringMu.Unlock() if took > c.interval { @@ -433,6 +434,7 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { t.ringMu.Lock() cut := time.Now().Add(-10 * time.Second) var bt, st []time.Duration + var perStore [][]time.Duration n := t.ringN if n > uint64(len(t.ring)) { n = uint64(len(t.ring)) @@ -444,8 +446,19 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { } bt = append(bt, r.block) st = append(st, r.seal) + for len(perStore) <= r.store { + perStore = append(perStore, nil) + } + perStore[r.store] = append(perStore[r.store], r.seal) } t.ringMu.Unlock() + // Each store's seal p90 over the same window: a tail that is one + // store's looks different from a tail every store shares + storeP90 := make([]float64, len(perStore)) + for i, v := range perStore { + sort.Slice(v, func(a, b int) bool { return v[a] < v[b] }) + storeP90[i] = float64(pct(v, .9)) / 1e6 + } sort.Slice(bt, func(i, j int) bool { return bt[i] < bt[j] }) sort.Slice(st, func(i, j int) bool { return st[i] < st[j] }) over := 0 @@ -473,7 +486,8 @@ func (t *tallies) liveState(c config, stores []*store, start time.Time) []byte { "elapsedSec": int(time.Since(start).Seconds()), "blocks": t.blocks.Load(), "height": height, "last10s": map[string]any{"blocks": len(bt), "over": over, "blockP50ms": float64(pct(bt, .5)) / 1e6, "blockP90ms": float64(pct(bt, .9)) / 1e6, "blockMaxMs": float64(pct(bt, 1)) / 1e6, - "sealP50ms": float64(pct(st, .5)) / 1e6, "sealP90ms": float64(pct(st, .9)) / 1e6, "sealMaxMs": float64(pct(st, 1)) / 1e6}, + "sealP50ms": float64(pct(st, .5)) / 1e6, "sealP90ms": float64(pct(st, .9)) / 1e6, "sealMaxMs": float64(pct(st, 1)) / 1e6, + "storeSealP90ms": storeP90}, "maintenanceInFlight": t.inFlight.Load(), "mismatches": t.mismatches.Load(), "heapHoleMB": float64(holes) / 1e6, "heapLiveMB": float64(live) / 1e6, "heapScannedMB": float64(scanned) / 1e6, "heapMovedMB": float64(moved) / 1e6, From 06ad2ccc6cd00ecab7877a8605866ccf555a6d24 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 22:05:54 -0500 Subject: [PATCH 40/58] A data file rolled between manifest commits is replayed on reopen The seal rolls to a new data file when one fills, but only the manifest names data files and the manifest is committed at merges, so every block sealed into a rolled file was lost on reopen. Open now takes unnamed data files in id order past the manifest's next id, and removes unnamed run files so their ids are free for O_EXCL again. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 28 ++++++++++++++++++ database/perm_roll_test.go | 59 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) create mode 100644 database/perm_roll_test.go diff --git a/database/perm.go b/database/perm.go index 281a60f..5734ea6 100644 --- a/database/perm.go +++ b/database/perm.go @@ -243,6 +243,34 @@ func (p *PermStore) Open() (err error) { st, _ := f.Stat() p.runs[id] = &runFile{id: id, f: f, size: st.Size()} } + // Data files the seals rolled after the manifest's commit are not + // named by it, but they hold committed deltas: they are taken in + // id order from the manifest's next id for as long as they exist, + // and the replay below covers them. Run files past the manifest's + // next id are maintenance output that was never named; nothing + // durable refers to them, so they go, and their ids are free for + // O_EXCL creation again. + for { + f, err := os.OpenFile(filepath.Join(p.Directory, permDataName(p.nextID)), os.O_RDWR, 0o644) + if errors.Is(err, os.ErrNotExist) { + break + } + if err != nil { + return err + } + st, _ := f.Stat() + p.files[p.nextID] = &heapFile{id: p.nextID, f: f, size: st.Size()} + p.nextID++ + } + for id := p.nextRun; ; id++ { + err := os.Remove(filepath.Join(p.Directory, (&runFile{id: id}).name())) + if errors.Is(err, os.ErrNotExist) { + break + } + if err != nil { + return err + } + } load := func(ref permRunRef, resident bool) (*permRun, *runFile, error) { rf := p.runs[ref.File] if rf == nil { diff --git a/database/perm_roll_test.go b/database/perm_roll_test.go new file mode 100644 index 0000000..4e56fad --- /dev/null +++ b/database/perm_roll_test.go @@ -0,0 +1,59 @@ +package blockchainDB + +import ( + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +// A data file rolled by a seal between two manifest commits is still +// replayed on reopen: every block sealed before Close is found, and +// the store keeps writing after (spec 1.7: what a seal made durable +// stays durable). +func TestPermStoreReopenAcrossRolledFiles(t *testing.T) { + size := PermFileBytes + PermFileBytes = 64 << 10 + defer func() { PermFileBytes = size }() + dir := filepath.Join(t.TempDir(), "perm") + p, err := NewPermStore(dir, MinFilterBlocks) + require.NoError(t, err) + fr := NewFastRandom([]byte{7}) + var keys [][32]byte + value := make([]byte, 1000) + for b := uint64(1); b <= 8; b++ { // 8 blocks x 40 KB: several rolls, no merge + p.AdvanceBlock(b) + for i := 0; i < 40; i++ { + k := fr.NextHash() + keys = append(keys, k) + value[0] = byte(b) + require.NoError(t, p.Put(k, value)) + } + sealPerm(t, p, b) + } + require.Greater(t, len(p.files), 1, "the test must roll a data file") + require.NoError(t, p.Close()) + + p, err = OpenPermStore(dir) + require.NoError(t, err) + for i, k := range keys { + v, err := p.GetDeep(k) + require.NoError(t, err, "key %d after reopen", i) + require.Equal(t, byte(i/40+1), v[0]) + } + require.Equal(t, uint64(9), p.BlockHeight()) + // And the store keeps going: a new block, sealed, merged, reopened + p.AdvanceBlock(9) + k := fr.NextHash() + require.NoError(t, p.Put(k, value)) + sealPerm(t, p, 9) + require.NoError(t, p.Merge()) + require.NoError(t, p.Close()) + p, err = OpenPermStore(dir) + require.NoError(t, err) + _, err = p.GetDeep(k) + require.NoError(t, err) + _, err = p.GetDeep(keys[0]) + require.NoError(t, err) + require.NoError(t, p.Close()) +} From cf0bb6f43af2bd57f5f7f3311c952c36b202c89e Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 22:07:29 -0500 Subject: [PATCH 41/58] The perm seal is one fsync per file outside the lock; the heap's unlinks are the mover's; the mover's budget is a store's Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 77 +++++++++++++++++++++++++++++++++++-------- database/heap_test.go | 8 ++++- database/kv_shard.go | 12 +++++++ database/perm.go | 74 ++++++++++++++++++++++++----------------- 4 files changed, 126 insertions(+), 45 deletions(-) diff --git a/database/heap.go b/database/heap.go index 8329447..a1df4fe 100644 --- a/database/heap.go +++ b/database/heap.go @@ -123,6 +123,27 @@ type HeapStore struct { // a block's finish and the time in those unlinks; snapshots // written and the time in them releases, releaseNs, snapshotsN, snapshotNs atomic.Uint64 + unlink []uint32 // Released files not yet deleted + // The bytes one mover pass may copy; 0 means HeapCleanBytes. A + // store's budget is HeapStoreCleanBytes however many shards it + // has, so a shard gets its share (SetCleanBudget). + budget int64 +} + +// SetCleanBudget sets the bytes one mover pass may copy. +func (h *HeapStore) SetCleanBudget(n int64) { + h.mu.Lock() + h.budget = n + h.mu.Unlock() +} + +func (h *HeapStore) cleanBudget() int64 { + h.mu.RLock() + defer h.mu.RUnlock() + if h.budget > 0 { + return h.budget + } + return HeapCleanBytes } // heapFile is one data file and its accounting. @@ -189,6 +210,13 @@ var HeapFileBlocks uint64 = 64 // cadence (~4 MB, of which a quarter to a third is live). var HeapCleanBytes int64 = 4 << 20 +// HeapStoreCleanBytes is a store's mover budget per pass, shared out +// among its shards: a store sharded eight ways moves the same bytes a +// pass as one sharded once. Measured: with one shard and the +// per-shard budget alone the store grew to 6 GB in five minutes +// where eight shards held it under 4. +var HeapStoreCleanBytes int64 = 32 << 20 + // HeapCleanFiles and HeapScanBytes bound a pass by the files it takes // and the bytes it reads: with hot keys most of a file is dead and // costs nothing to copy, so what limits the mover's pace is how much @@ -731,6 +759,9 @@ func (h *HeapStore) Close() error { if err := p.finish(); err != nil { return err } + if err := h.unlinkReleased(); err != nil { + return err + } h.mu.Lock() defer h.mu.Unlock() h.closed = true @@ -1011,20 +1042,37 @@ func (p *heapSync) finish() (err error) { h.syncBytes.Add(uint64(p.bytes)) h.mu.Lock() defer h.mu.Unlock() - if len(p.release) > 0 { - t = time.Now() - for _, id := range p.release { - hf := h.files[id] - hf.f.Close() - delete(h.files, id) - h.deadBytes -= hf.dead - if err = os.Remove(filepath.Join(h.Directory, dataName(id))); err != nil { - return err - } + // The files the delta no longer names leave the map now; their + // unlinks are the mover's, off the block's path. Until then they + // are unnamed files on disk, which an open deletes as such. + for _, id := range p.release { + hf := h.files[id] + hf.f.Close() + delete(h.files, id) + h.deadBytes -= hf.dead + h.unlink = append(h.unlink, id) + } + return nil +} + +// unlinkReleased deletes the files the block syncs have released. +// Called without the lock. +func (h *HeapStore) unlinkReleased() error { + h.mu.Lock() + ids := h.unlink + h.unlink = nil + h.mu.Unlock() + if len(ids) == 0 { + return nil + } + t := time.Now() + for _, id := range ids { + if err := os.Remove(filepath.Join(h.Directory, dataName(id))); err != nil && !errors.Is(err, os.ErrNotExist) { + return err } - h.releases.Add(uint64(len(p.release))) - h.releaseNs.Add(uint64(time.Since(t))) } + h.releases.Add(uint64(len(ids))) + h.releaseNs.Add(uint64(time.Since(t))) return nil } @@ -1032,7 +1080,10 @@ func (p *heapSync) finish() (err error) { // bounded mover pass, and every HeapSnapshotEvery calls a new index // generation, which bounds the replay on open. func (h *HeapStore) compact() (bool, error) { - moved, err := h.clean(HeapCleanBytes) + if err := h.unlinkReleased(); err != nil { + return false, err + } + moved, err := h.clean(h.cleanBudget()) if err != nil { return moved, err } diff --git a/database/heap_test.go b/database/heap_test.go index ad8dd20..84deb3d 100644 --- a/database/heap_test.go +++ b/database/heap_test.go @@ -82,8 +82,14 @@ func TestHeapRewriteReusesWithinTheBlockAndMovesOneSyncLate(t *testing.T) { released := append([]uint32(nil), h.release...) syncHeap(t, h) for _, id := range released { + require.Nil(t, h.files[id], "out of the map after the sync") _, err := os.Stat(filepath.Join(h.Directory, dataName(id))) - require.ErrorIs(t, err, os.ErrNotExist, "deleted after the sync") + require.NoError(t, err, "the unlink is the mover's, off the block's path") + } + require.NoError(t, h.unlinkReleased()) + for _, id := range released { + _, err := os.Stat(filepath.Join(h.Directory, dataName(id))) + require.ErrorIs(t, err, os.ErrNotExist, "deleted by the mover") } _, copied := h.Cleaned() require.Greater(t, copied, uint64(0), "the live entries left in it were copied out") diff --git a/database/kv_shard.go b/database/kv_shard.go index e4d0061..c32a39f 100644 --- a/database/kv_shard.go +++ b/database/kv_shard.go @@ -172,6 +172,7 @@ func OpenKVShard(directory string) (kVShard *KVShard, err error) { return nil, err } } + kVShard.shareCleanBudget() kVShard.useSharedBlockRecord() if err = kVShard.adoptBlockHeight(); err != nil { return nil, err @@ -249,6 +250,16 @@ func NewKVShardN(directory string, shards int, sealLimit uint64) (kvs *KVShard, return newKVShardN(directory, shards, sealLimit, NewKV2) } +// shareCleanBudget gives each heap shard its share of the store's +// mover budget (HeapStoreCleanBytes). +func (k *KVShard) shareCleanBudget() { + for _, shard := range k.Shards { + if shard.Heap != nil { + shard.Heap.SetCleanBudget(HeapStoreCleanBytes / int64(len(k.Shards))) + } + } +} + func newKVShardN(directory string, shards int, sealLimit uint64, newShard func(string, uint64) (*KV2, error)) (kvs *KVShard, err error) { if shards < 1 { return nil, fmt.Errorf("a database needs at least one shard, asked for %d", shards) @@ -267,6 +278,7 @@ func newKVShardN(directory string, shards int, sealLimit uint64, newShard func(s return nil, err } } + kvs.shareCleanBudget() kvs.useSharedBlockRecord() if kvs.Sets, err = NewSetStore(kvs.setDir()); err != nil { return nil, err diff --git a/database/perm.go b/database/perm.go index 5734ea6..34bec1a 100644 --- a/database/perm.go +++ b/database/perm.go @@ -633,6 +633,8 @@ type permSeal struct { dirty []*heapFile recs []permRecord height uint64 + run *permRun // The delta as written, admitted once durable + hf *heapFile // The data file it was written to } // beginSeal takes the block's records under the lock; finish makes @@ -654,56 +656,66 @@ func (p *PermStore) beginSeal(height uint64) (*permSeal, error) { for _, rec := range p.live { s.recs = append(s.recs, rec) } + p.live = map[[32]byte]permRecord{} // A put from here on is the next block's sortPermRecords(s.recs) - return s, nil -} - -// finish: entries durable, then the delta appended and durable, then -// the delta admitted to the window and the live map cleared. -func (s *permSeal) finish() error { - p := s.p - defer p.syncMu.Unlock() - for _, hf := range s.dirty { - if err := fsync(hf.f); err != nil { - return err - } - } if len(s.recs) == 0 { - p.mu.Lock() - if s.height >= p.height { - p.height = s.height + 1 - } - p.mu.Unlock() - return nil + return s, nil } // The delta's run goes into the block's data file behind the - // block's entries, under the reserved key, and the one fsync - // covers both - p.mu.Lock() - defer p.mu.Unlock() + // block's entries, under the reserved key: one fsync of that file + // covers both, and replay trusts the last delta only if every + // entry it names checks (spec 1.7) var w bufWriterAt run, err := writePermRun(&w, 0, p.cur.id, s.recs, s.height) if err != nil { - return err + p.syncMu.Unlock() + return nil, err } entry := encodeEntry(s.height, heapDeltaKey, w.buf) if p.cur.size+int64(len(entry)) > PermFileBytes { if p.cur, err = p.newDataFile(); err != nil { - return err + p.syncMu.Unlock() + return nil, err } } hf, at := p.cur, p.cur.size if _, err = hf.f.WriteAt(entry, at); err != nil { - return err + p.syncMu.Unlock() + return nil, err } hf.size += int64(len(entry)) run.file, run.off, run.bloomAt = hf.id, at+heapHeader, at+heapHeader+run.bloomAt - if err = fsync(hf.f); err != nil { - return err + s.run, s.hf = run, hf + if _, dirty := p.dirty[hf.id]; !dirty { + for _, d := range s.dirty { + if d == hf { + dirty = true + } + } + if !dirty { + s.dirty = append(s.dirty, hf) + } + } + return s, nil +} + +// finish: the block's entries and its delta durable, one fsync per +// file touched, with the lock released; then the delta admitted to +// the window. Releases syncMu. +func (s *permSeal) finish() error { + p := s.p + defer p.syncMu.Unlock() + for _, hf := range s.dirty { + if err := fsync(hf.f); err != nil { + return err + } + } + p.mu.Lock() + defer p.mu.Unlock() + if s.run != nil { + p.indexBytes.Add(uint64(s.run.bytes)) + p.admit(&permDelta{height: s.height, run: s.run, f: s.hf.f, data: s.hf.id}) } - p.indexBytes.Add(uint64(run.bytes)) - p.admit(&permDelta{height: s.height, run: run, f: hf.f, data: hf.id}) - p.live = map[[32]byte]permRecord{} if s.height >= p.height { p.height = s.height + 1 } From 8dfe1fcf084ed503d35df7b618425953a5c4ff2f Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 22:08:50 -0500 Subject: [PATCH 42/58] A merge reads only its due buckets' slice of each delta and commits the manifest every eighth merge Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 57 ++++++++++++++++++++++++++++++++++---- database/perm_roll_test.go | 57 ++++++++++++++++++++++++++++++++++++++ database/permindex.go | 52 ++++++++++++++++++++++++++++++++++ 3 files changed, 160 insertions(+), 6 deletions(-) diff --git a/database/perm.go b/database/perm.go index 34bec1a..d741f62 100644 --- a/database/perm.go +++ b/database/perm.go @@ -95,14 +95,19 @@ type PermStore struct { file uint32 off int64 } - syncMu sync.Mutex // Serializes seals with each other and with manifest commits - closed bool + syncMu sync.Mutex // Serializes seals with each other and with manifest commits + mergesSince int // Merges since the manifest was last committed + closed bool putTotal, putDuplicate, lookups, windowHits, deepHits atomic.Uint64 mergeRuns, foldRuns, packRuns atomic.Uint64 indexBytes atomic.Uint64 } +// PermManifestEvery is how many merges may pass between manifest +// commits when no fold made one necessary. +var PermManifestEvery = 8 + // PermBuckets is how many buckets a shard's history above the // watermark is kept in, by the key's first byte. const PermBuckets = 256 @@ -476,6 +481,14 @@ func (p *PermStore) Close() error { } p.mu.Lock() defer p.mu.Unlock() + if p.mergesSince > 0 { + if err = p.commitManifest(); err != nil { + return err + } + if err = p.dropUnreferencedRunFiles(); err != nil { + return err + } + } p.closed = true for _, hf := range p.files { if cerr := hf.f.Close(); err == nil { @@ -805,14 +818,34 @@ func (p *PermStore) Merge() error { } rotation := (p.rotation + due) % PermBuckets // Read the deltas' records for these buckets outside the lock: - // runs are immutable + // runs are immutable. The due buckets are a run of the rotation, + // so their records are one or two slices of each sorted delta + oldest := ^uint64(0) + for _, j := range jobs { + if p.buckets[j.b].merged < oldest { + oldest = p.buckets[j.b].merged + } + } + var ranges [][2]byte + if last := p.rotation + due - 1; last < PermBuckets { + ranges = append(ranges, [2]byte{byte(p.rotation), byte(last)}) + } else { + ranges = append(ranges, [2]byte{byte(p.rotation), PermBuckets - 1}, [2]byte{0, byte(last - PermBuckets)}) + } pendingCopy := append([]*permDelta(nil), p.pending...) p.mu.Unlock() byBucket := map[int][][]permRecord{} for _, d := range pendingCopy { - recs, err := d.run.records(d.f) - if err != nil { - return err + if d.height <= oldest { + continue // Every due bucket has this delta already + } + var recs []permRecord + for _, rg := range ranges { + part, err := d.run.recordsBucketRange(d.f, rg[0], rg[1]) + if err != nil { + return err + } + recs = append(recs, part...) } for _, j := range jobs { if d.height <= p.buckets[j.b].merged { @@ -910,8 +943,20 @@ func (p *PermStore) Merge() error { return err } } + // The manifest names the new runs. It is committed when a fold + // left run files to drop, and otherwise every PermManifestEvery + // merges: the runs are durable in their file already, and a + // merge a crash loses is done again from the pending deltas, so a + // commit per merge -- two barriers a shard -- bought nothing but + // a shorter replay. Open removes the run files a lost merge left + // unnamed. p.mu.Lock() defer p.mu.Unlock() + p.mergesSince++ + if len(folds) == 0 && p.mergesSince < PermManifestEvery { + return nil + } + p.mergesSince = 0 if err := p.commitManifest(); err != nil { return err } diff --git a/database/perm_roll_test.go b/database/perm_roll_test.go index 4e56fad..b37f71d 100644 --- a/database/perm_roll_test.go +++ b/database/perm_roll_test.go @@ -57,3 +57,60 @@ func TestPermStoreReopenAcrossRolledFiles(t *testing.T) { require.NoError(t, err) require.NoError(t, p.Close()) } + +// Merges between manifest commits survive a crash: the store reopened +// from the stale manifest finds every key, does the lost merges again +// from the pending deltas, and creates run files without colliding +// with the ones the lost merges left behind. +func TestPermStoreReopenWithoutManifestCommit(t *testing.T) { + every, size := PermMergeEvery, PermFileBytes + PermMergeEvery, PermFileBytes = 4, 64<<10 + defer func() { PermMergeEvery, PermFileBytes = every, size }() + dir := filepath.Join(t.TempDir(), "perm") + p, err := NewPermStore(dir, MinFilterBlocks) + require.NoError(t, err) + fr := NewFastRandom([]byte{9}) + var keys [][32]byte + value := make([]byte, 500) + for b := uint64(1); b <= 3*MinFilterBlocks; b++ { + p.AdvanceBlock(b) + for i := 0; i < 50; i++ { + k := fr.NextHash() + keys = append(keys, k) + require.NoError(t, p.Put(k, value)) + } + sealPerm(t, p, b) + if b%3 == 0 { + require.NoError(t, p.Merge()) + } + } + // Keep going until a merge is left uncommitted (a fold commits) + last := uint64(3 * MinFilterBlocks) + for p.mergesSince == 0 { + last++ + require.Less(t, last, uint64(6*MinFilterBlocks), "a merge without a fold must come") + p.AdvanceBlock(last) + k := fr.NextHash() + keys = append(keys, k) + require.NoError(t, p.Put(k, value)) + sealPerm(t, p, last) + require.NoError(t, p.Merge()) + } + // No Close: the manifest on disk is the last commit's + q, err := OpenPermStore(dir) + require.NoError(t, err) + for i, k := range keys { + _, err := q.GetDeep(k) + require.NoError(t, err, "key %d after reopen", i) + } + q.AdvanceBlock(last + 1) + require.NoError(t, q.Put(fr.NextHash(), value)) + sealPerm(t, q, last+1) + require.NoError(t, q.Merge(), "a new run file must not collide with a lost merge's") + require.NoError(t, q.Close()) + q, err = OpenPermStore(dir) + require.NoError(t, err) + _, err = q.GetDeep(keys[0]) + require.NoError(t, err) + require.NoError(t, q.Close()) +} diff --git a/database/permindex.go b/database/permindex.go index 6c09ac6..c34fc05 100644 --- a/database/permindex.go +++ b/database/permindex.go @@ -195,6 +195,58 @@ func (r *permRun) records(f *os.File) ([]permRecord, error) { return recs, nil } +// recordsBucketRange reads the records whose first key byte lies in +// [lo, hi]: the run is sorted, so they are one slice, found by a +// binary search that probes a byte per step and read at once. A +// merge takes its due buckets' records from every pending delta this +// way rather than reading each delta whole -- with buckets due in +// rotation that was a read of every pending delta, up to +// PermMergeEvery blocks of them, at every pass. +func (r *permRun) recordsBucketRange(f *os.File, lo, hi byte) ([]permRecord, error) { + first := func(i int) (byte, error) { + var b [1]byte + _, err := f.ReadAt(b[:], r.off+permRunHdr+int64(i)*permRecSize) + return b[0], err + } + // The first record whose key's first byte is not below target + search := func(target int) (int, error) { + i, j := 0, int(r.count) + for i < j { + m := int(uint(i+j) >> 1) + b, err := first(m) + if err != nil { + return 0, err + } + if int(b) < target { + i = m + 1 + } else { + j = m + } + } + return i, nil + } + i, err := search(int(lo)) + if err != nil { + return nil, err + } + j, err := search(int(hi) + 1) + if err != nil { + return nil, err + } + if j <= i { + return nil, nil + } + buf := make([]byte, (j-i)*permRecSize) + if _, err := f.ReadAt(buf, r.off+permRunHdr+int64(i)*permRecSize); err != nil { + return nil, err + } + recs := make([]permRecord, j-i) + for k := range recs { + recs[k] = getPermRecord(buf[k*permRecSize:]) + } + return recs, nil +} + // mergePermRuns merges sorted record lists, oldest first, into one // sorted list; a key present in several takes the newest. Permanent // keys are written once, so a duplicate is a replay or a fault, and From 75c0b2886b1ba278d3163d2103f572c6d5b3542d Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 22:09:50 -0500 Subject: [PATCH 43/58] The proposal records the seal's one fsync, the mover's unlinks, replay past the manifest, and what the bad minutes were Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- .../2026-09-16-entries-written-once.md | 66 +++++++++++++++---- 1 file changed, 55 insertions(+), 11 deletions(-) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index 7620761..b0d7ee0 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -229,13 +229,34 @@ point" and closes #33. ## Durability and crash consistency (1.8) -- **A file is deleted one seal late.** A file emptied by the mover - is deleted only after the delta naming the mover's copies out of it - is durable. Until then the durable index still names its slots, - and a crash must find them intact: never unlink what a durable - index names. The adapter never asks the store for an old version - (its pre-images are memoized on its side), so deletion waits on the - seal and on nothing else. +- **A file is deleted one seal late, by the mover.** A file emptied + by the mover leaves the map only after the delta naming the mover's + copies out of it is durable. Until then the durable index still + names its slots, and a crash must find them intact: never unlink + what a durable index names. The unlink itself is the mover's, at + its next pass, so no block's seal holds the store lock over a + directory operation; between the two the file is unnamed on disk, + which an open deletes as such. The adapter never asks the store + for an old version (its pre-images are memoized on its side), so + deletion waits on the seal and on nothing else. +- **A seal is one fsync per file it touched, outside the lock.** The + block's delta is written into the data file behind the block's + entries before the fsync, so one barrier covers both, in both + layers; replay trusts the last delta only if every entry it names + checks. A put that lands during the seal belongs to the next + block: the seal takes the block's records under the lock and + releases it before the barrier. +- **What the manifest does not name is still replayed if a seal + wrote it.** The permanent layer's manifest is committed by + maintenance, not by seals, so the seals roll data files the + manifest has not seen. Open takes data files in id order past the + manifest's next id for as long as they exist, and replays their + deltas; a run file past the manifest's next run id is maintenance + output a crash left unnamed, which open removes, so the id is free + for exclusive creation again and no file is ever published over + (1.7). A merge lost this way is done again from the pending + deltas, which is why the manifest need only be committed every + eighth merge, or when a fold leaves run files to drop. - **A torn slot is detected, not misread.** Every entry carries its length and a checksum; an entry above the committed height is a block that never synced. An index entry is durable only after the @@ -302,7 +323,30 @@ point" and closes #33. file too. Alone, the heap seals at 44-54 ms p50 at nine stores (57-64 with two barriers). The mover finds a file's live entries through the index rather than by scanning the file, because nine - stores' scans together starved the block loops for CPU. Still to - do: the store-level commit (one block record naming every - shard's deltas) and the per-store data file, so that a hundred - shards cost a block one barrier. + stores' scans together starved the block loops for CPU. The + mover's budget is the store's (`HeapStoreCleanBytes`), shared + among its shards: with one shard and a shard's budget the store + grew to 6 GB in five minutes. A merge reads only its due + buckets' slice of each pending delta (the run is sorted, the + slice is found by a binary search on the first key byte) instead + of every pending delta whole at every pass. + + *Measured, 2026-09-17:* every "bad minute" of the day's runs -- + seal p50 unchanged, p90 200-450 ms, the heap's fsync average + 3-8x its usual 12-22 ms -- was the disk, not the store. A + 2-second timeline showed every fsync on the box stepping to + 100-330 ms for 30-40 s at once, with nothing in the store + changing (no snapshot, the same mover volume and release cadence) + and with one shard per store (9 barriers/s) exactly as with eight + (72/s). The instrument's root volume is on LUKS without discards + and has never been trimmed. Until it is, runs are compared by + their clean minutes; the per-minute row carries the heap's split + (fsync average and bytes, snapshots, releases, moved bytes) and + each store's seal p90 so a stall can be told from a tail. + + Still to do: the store-level commit (one block record naming + every shard's deltas) and the per-store data file, so that a + hundred shards cost a block one barrier. With one shard per + store, which is that layout by another name, the heap sealed at + 40-42 ms p50 and 55-58 ms p90 in clean minutes, and the files + store at 46-49 / 53-57. From a2ddec5f70519de7ce5a387116ce9bd0bbe0bd4f Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 23:01:43 -0500 Subject: [PATCH 44/58] Maintenance is a slice per call: the shards next in rotation, sized by the blocks elapsed A store's mover rate is HeapStoreCleanBytes per HeapCleanPeriod blocks. A Compress call takes as many shards as the blocks since the last call earn, in rotation, each with its share, so a call every block moves a little on one shard and the copies reach the device as a trickle rather than every shard's pass in one second. Snapshots are by block count, or early when the files held only by their deltas outweigh two data files; the perm manifest is committed by block count. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 73 ++++++++++++++++---------------- database/heap_crash_test.go | 2 +- database/heap_test.go | 23 +++++----- database/kv_2.go | 9 ++-- database/kv_shard.go | 84 ++++++++++++++++++++++++++++++------- database/perm.go | 24 ++++++----- database/perm_roll_test.go | 2 +- database/segstore.go | 2 +- 8 files changed, 142 insertions(+), 77 deletions(-) diff --git a/database/heap.go b/database/heap.go index a1df4fe..6dbcddd 100644 --- a/database/heap.go +++ b/database/heap.go @@ -103,10 +103,10 @@ type HeapStore struct { // snapshot, so the map a snapshot writes is exactly the state of // the last finished delta and no delta is in flight into a log // about to be retired. - syncMu sync.Mutex - log *os.File - gen uint64 - snapshots int + syncMu sync.Mutex + log *os.File + gen uint64 + snapAt uint64 // The block the key map was last snapshotted at closed bool liveBytes int64 @@ -124,26 +124,6 @@ type HeapStore struct { // written and the time in them releases, releaseNs, snapshotsN, snapshotNs atomic.Uint64 unlink []uint32 // Released files not yet deleted - // The bytes one mover pass may copy; 0 means HeapCleanBytes. A - // store's budget is HeapStoreCleanBytes however many shards it - // has, so a shard gets its share (SetCleanBudget). - budget int64 -} - -// SetCleanBudget sets the bytes one mover pass may copy. -func (h *HeapStore) SetCleanBudget(n int64) { - h.mu.Lock() - h.budget = n - h.mu.Unlock() -} - -func (h *HeapStore) cleanBudget() int64 { - h.mu.RLock() - defer h.mu.RUnlock() - if h.budget > 0 { - return h.budget - } - return HeapCleanBytes } // heapFile is one data file and its accounting. @@ -243,10 +223,22 @@ var ( HeapBoundOn = 1.5 ) -// HeapSnapshotEvery is how many compact calls pass between key-map +// HeapSnapshotBlocks is how many blocks pass between key-map // snapshots; between them the generation's deltas are what open -// replays. -var HeapSnapshotEvery = 5 +// replays. Counted in blocks, not maintenance calls, so the cadence +// of the caller does not set the cadence of the snapshots. +var HeapSnapshotBlocks uint64 = 100 + +// HeapSnapshotPinnedFiles is how many files' worth of bytes may be +// held only by their deltas before a snapshot is taken early, +// whatever the block count. +var HeapSnapshotPinnedFiles int64 = 2 + +// HeapCleanPeriod is the blocks over which a store spends +// HeapStoreCleanBytes: the store's mover rate is the one divided by +// the other, and a maintenance call moves what the blocks since the +// last call earned (KVShard.Compress). +var HeapCleanPeriod uint64 = 20 // entrySize is the bytes an entry of n value bytes takes, aligned. func entrySize(n int) int64 { @@ -1076,22 +1068,33 @@ func (h *HeapStore) unlinkReleased() error { return nil } -// compact is the heap's maintenance on the adapter's cadence: one -// bounded mover pass, and every HeapSnapshotEvery calls a new index -// generation, which bounds the replay on open. -func (h *HeapStore) compact() (bool, error) { +// compact is one slice of the heap's maintenance: the released +// files unlinked, a mover pass bounded by budget bytes copied, and +// every HeapSnapshotBlocks a new index generation, which bounds the +// replay on open. +func (h *HeapStore) compact(budget int64) (bool, error) { if err := h.unlinkReleased(); err != nil { return false, err } - moved, err := h.clean(h.cleanBudget()) + moved, err := h.clean(budget) if err != nil { return moved, err } + // A snapshot is due by age, or when the files it would free + // outweigh it: a file whose only live bytes are deltas cannot go + // until a snapshot supersedes them, so the space those files hold + // is what a snapshot buys, and it is taken when that exceeds + // HeapSnapshotPinned h.mu.Lock() - h.snapshots++ - due := h.snapshots >= HeapSnapshotEvery + var pinned int64 + for _, hf := range h.files { + if hf != h.cur && hf != h.mov && hf.deltas > 0 && hf.live == hf.deltas { + pinned += hf.size + } + } + due := h.height-h.snapAt >= HeapSnapshotBlocks || pinned >= HeapSnapshotPinnedFiles*HeapFileBytes if due { - h.snapshots = 0 + h.snapAt = h.height } h.mu.Unlock() if due { diff --git a/database/heap_crash_test.go b/database/heap_crash_test.go index 01a4992..1a469bb 100644 --- a/database/heap_crash_test.go +++ b/database/heap_crash_test.go @@ -38,7 +38,7 @@ func TestHeapCrashChild(t *testing.T) { require.NoError(t, p.finish(), "child: finish") fmt.Printf("CHECKPOINT %d\n", b) if b%4 == 0 { - _, err := h.compact() + _, err := h.compact(HeapCleanBytes) require.NoError(t, err, "child: compact") } } diff --git a/database/heap_test.go b/database/heap_test.go index 84deb3d..9ef48d2 100644 --- a/database/heap_test.go +++ b/database/heap_test.go @@ -182,9 +182,9 @@ func TestHeapSnapshotStartsAGenerationSafely(t *testing.T) { dir := heapDir(t) h, err := NewHeapStore(dir) require.NoError(t, err) - every := HeapSnapshotEvery - HeapSnapshotEvery = 1 - defer func() { HeapSnapshotEvery = every }() + every := HeapSnapshotBlocks + HeapSnapshotBlocks = 1 + defer func() { HeapSnapshotBlocks = every }() for b := uint64(1); b <= 30; b++ { h.AdvanceBlock(b) for i := byte(1); i <= 20; i++ { @@ -192,7 +192,7 @@ func TestHeapSnapshotStartsAGenerationSafely(t *testing.T) { } syncHeap(t, h) if b == 20 { - _, err := h.compact() + _, err := h.compact(HeapCleanBytes) require.NoError(t, err) _, err = os.Stat(filepath.Join(dir, indexName(1))) require.ErrorIs(t, err, os.ErrNotExist, "generation 1 retired") @@ -322,12 +322,15 @@ func TestHeapShardRoundTrip(t *testing.T) { } _, dyna := kvs.Stats() require.EqualValues(t, 60*200, dyna.PutTotal) - // A pass for what the last ten blocks left dead, and the sync that - // deletes what it emptied - require.NoError(t, kvs.Compress()) - require.NoError(t, kvs.SealBlock(61)) - require.NoError(t, kvs.Compress()) - require.NoError(t, kvs.SealBlock(62)) + // Maintenance is a slice per call, sized by the blocks since the + // last: a call a block over a few blocks visits every shard with + // budget to spare for what the last ten blocks left dead, and each + // seal releases what the pass before it emptied + for b := uint64(61); b <= 64; b++ { + require.NoError(t, kvs.SealBlock(b)) + require.NoError(t, kvs.Compress()) + } + require.NoError(t, kvs.SealBlock(65)) dead, live := kvs.Shards[0].Heap.HoleRatio() require.Less(t, dead, 2*live+HeapFileBytes, "dead bytes are bounded: at most the current file, which the mover never takes, beyond the live set") require.NoError(t, kvs.Close()) diff --git a/database/kv_2.go b/database/kv_2.go index 390d1e4..37c0d7e 100644 --- a/database/kv_2.go +++ b/database/kv_2.go @@ -175,7 +175,7 @@ type dynaLayer interface { AdvanceBlock(height uint64) LiveRecords() uint64 beginBlockSync() (blockSync, error) - compact() (bool, error) + compact(budget int64) (bool, error) // A heap moves at most budget bytes; a segment store ignores it Stats() StoreStats } @@ -706,8 +706,11 @@ func (k *KV2) Put(key [32]byte, value []byte) (writes int, err error) { // weight rather than a wrong answer. // // TODO: Cleanse PermKV of keys in DynaKV -func (k *KV2) Compress() error { - if _, err := k.dyna().compact(); err != nil { +func (k *KV2) Compress() error { return k.compress(HeapCleanBytes) } + +// compress is Compress with the mover's budget for this call. +func (k *KV2) compress(budget int64) error { + if _, err := k.dyna().compact(budget); err != nil { return err } k.Mutex.Lock() diff --git a/database/kv_shard.go b/database/kv_shard.go index c32a39f..7d847c5 100644 --- a/database/kv_shard.go +++ b/database/kv_shard.go @@ -82,6 +82,13 @@ type KVShard struct { // found" for data that is on disk. Shards []*KV2 + // Maintenance in rotation (Compress): the height last sealed, the + // height at the last call, and the shard next in line + maintMu sync.Mutex + sealed uint64 + maintAt uint64 + maintNext int + // Sets holds the finalized Perm data that has left the shards: one // block-set file per completed set of blocks, packed from every // shard's merged segment (blockset.go). Each shard's Perm layer @@ -172,7 +179,6 @@ func OpenKVShard(directory string) (kVShard *KVShard, err error) { return nil, err } } - kVShard.shareCleanBudget() kVShard.useSharedBlockRecord() if err = kVShard.adoptBlockHeight(); err != nil { return nil, err @@ -250,16 +256,6 @@ func NewKVShardN(directory string, shards int, sealLimit uint64) (kvs *KVShard, return newKVShardN(directory, shards, sealLimit, NewKV2) } -// shareCleanBudget gives each heap shard its share of the store's -// mover budget (HeapStoreCleanBytes). -func (k *KVShard) shareCleanBudget() { - for _, shard := range k.Shards { - if shard.Heap != nil { - shard.Heap.SetCleanBudget(HeapStoreCleanBytes / int64(len(k.Shards))) - } - } -} - func newKVShardN(directory string, shards int, sealLimit uint64, newShard func(string, uint64) (*KV2, error)) (kvs *KVShard, err error) { if shards < 1 { return nil, fmt.Errorf("a database needs at least one shard, asked for %d", shards) @@ -278,7 +274,6 @@ func newKVShardN(directory string, shards int, sealLimit uint64, newShard func(s return nil, err } } - kvs.shareCleanBudget() kvs.useSharedBlockRecord() if kvs.Sets, err = NewSetStore(kvs.setDir()); err != nil { return nil, err @@ -480,6 +475,9 @@ func (k *KVShard) writeBlockHeight(height uint64) (err error) { // with the block it belongs to func (k *KVShard) adoptBlockHeight() error { height, err := k.readBlockHeight() + k.maintMu.Lock() + k.sealed = height + k.maintMu.Unlock() if err != nil { return err } @@ -555,7 +553,13 @@ func (k *KVShard) SealBlock(height uint64) (err error) { return err } } - return k.writeBlockHeight(height + 1) + if err = k.writeBlockHeight(height + 1); err != nil { + return err + } + k.maintMu.Lock() + k.sealed = height + 1 + k.maintMu.Unlock() + return nil } // MergeFinalized @@ -777,11 +781,59 @@ func (k *KVShard) Stats() (perm, dyna StoreStats) { // Compress // Compress all the shards func (k *KVShard) Compress() (err error) { - for i, kvs := range k.Shards { - if err = kvs.Open(); err != nil { + if len(k.Shards) > 0 { + if err = k.Shards[0].Open(); err != nil { + return fmt.Errorf("shard 0: %w", err) + } + } + if len(k.Shards) == 0 || k.Shards[0].Heap == nil { + // Segment-store shards: a bounded pass on every shard, as before + for i, kvs := range k.Shards { + if err = kvs.Open(); err != nil { + return fmt.Errorf("shard %d: %w", i, err) + } + if err = kvs.Compress(); err != nil { + return fmt.Errorf("shard %d: %w", i, err) + } + } + return nil + } + // Heap shards: one slice of the store's maintenance, sized by the + // blocks since the last call. The store's mover rate is + // HeapStoreCleanBytes per HeapCleanPeriod blocks; a call takes the + // shards next in rotation, as many as the elapsed blocks earn, each + // with its share of what those blocks earned. Called every + // HeapCleanPeriod blocks it moves the whole budget over every + // shard; called every block it moves a little on the next shard, + // and the copies reach the device as a trickle rather than every + // shard's pass at once -- which, at nine stores in lockstep, was + // 300 MB and 72 barriers in one second, and queued the seals' + // fsyncs behind them for the next four. A hundred shards spread + // the same rate a hundred ways, and each call locks one. + k.maintMu.Lock() + n := len(k.Shards) + elapsed := k.sealed - k.maintAt + if k.maintAt == 0 || elapsed > HeapCleanPeriod { + elapsed = HeapCleanPeriod + } + k.maintAt = k.sealed + due := int(uint64(n) * elapsed / HeapCleanPeriod) + if due < 1 { + due = 1 + } + if due > n { + due = n + } + budget := HeapStoreCleanBytes * int64(elapsed) / int64(HeapCleanPeriod) / int64(due) + start := k.maintNext + k.maintNext = (start + due) % n + k.maintMu.Unlock() + for j := 0; j < due; j++ { + i := (start + j) % n + if err = k.Shards[i].Open(); err != nil { return fmt.Errorf("shard %d: %w", i, err) } - if err = kvs.Compress(); err != nil { + if err = k.Shards[i].compress(budget); err != nil { return fmt.Errorf("shard %d: %w", i, err) } } diff --git a/database/perm.go b/database/perm.go index d741f62..5431d58 100644 --- a/database/perm.go +++ b/database/perm.go @@ -95,18 +95,20 @@ type PermStore struct { file uint32 off int64 } - syncMu sync.Mutex // Serializes seals with each other and with manifest commits - mergesSince int // Merges since the manifest was last committed - closed bool + syncMu sync.Mutex // Serializes seals with each other and with manifest commits + manifestAt uint64 // The height the manifest was last committed at + manifestDirty bool // Merges since then + closed bool putTotal, putDuplicate, lookups, windowHits, deepHits atomic.Uint64 mergeRuns, foldRuns, packRuns atomic.Uint64 indexBytes atomic.Uint64 } -// PermManifestEvery is how many merges may pass between manifest -// commits when no fold made one necessary. -var PermManifestEvery = 8 +// PermManifestBlocks is how many blocks may pass between manifest +// commits when no fold made one necessary. Counted in blocks so the +// caller's cadence does not set the commit cadence. +var PermManifestBlocks uint64 = 160 // PermBuckets is how many buckets a shard's history above the // watermark is kept in, by the key's first byte. @@ -481,7 +483,7 @@ func (p *PermStore) Close() error { } p.mu.Lock() defer p.mu.Unlock() - if p.mergesSince > 0 { + if p.manifestDirty { if err = p.commitManifest(); err != nil { return err } @@ -952,11 +954,10 @@ func (p *PermStore) Merge() error { // unnamed. p.mu.Lock() defer p.mu.Unlock() - p.mergesSince++ - if len(folds) == 0 && p.mergesSince < PermManifestEvery { + p.manifestDirty = true + if len(folds) == 0 && p.height-p.manifestAt < PermManifestBlocks { return nil } - p.mergesSince = 0 if err := p.commitManifest(); err != nil { return err } @@ -1130,6 +1131,9 @@ func (p *PermStore) commitManifest() error { p.mu.Unlock() err = p.writeManifest(buf) p.mu.Lock() + if err == nil { + p.manifestAt, p.manifestDirty = p.height, false + } return err } diff --git a/database/perm_roll_test.go b/database/perm_roll_test.go index b37f71d..832ee06 100644 --- a/database/perm_roll_test.go +++ b/database/perm_roll_test.go @@ -86,7 +86,7 @@ func TestPermStoreReopenWithoutManifestCommit(t *testing.T) { } // Keep going until a merge is left uncommitted (a fold commits) last := uint64(3 * MinFilterBlocks) - for p.mergesSince == 0 { + for !p.manifestDirty { last++ require.Less(t, last, uint64(6*MinFilterBlocks), "a merge without a fold must come") p.AdvanceBlock(last) diff --git a/database/segstore.go b/database/segstore.go index 0543e5d..b842c31 100644 --- a/database/segstore.go +++ b/database/segstore.go @@ -2732,7 +2732,7 @@ func (s *SegmentStore) beginBlockSync() (blockSync, error) { return p, nil } -func (s *SegmentStore) compact() (bool, error) { return s.CompactHistory() } +func (s *SegmentStore) compact(int64) (bool, error) { return s.CompactHistory() } // beginPermSeal and mergeBelow are the permLayer surface (kv_2.go) // over beginSeal and MergeBelow. From 3d7885477ed4cced2efcd424f4bad99348404a15 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 23:02:06 -0500 Subject: [PATCH 45/58] The proposal records maintenance in rotation and the files store's closed tail Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- .../2026-09-16-entries-written-once.md | 31 ++++++++++++++++--- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index b0d7ee0..6ad6139 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -344,9 +344,32 @@ point" and closes #33. (fsync average and bytes, snapshots, releases, moved bytes) and each store's seal p90 so a stall can be told from a tail. + *Maintenance is a slice per call, in rotation.* The heap's + remaining tail in clean minutes was the mover's own delivery: on + the adapter's cadence every shard of every store ran its pass in + the same second -- 300 MB and 72 barriers at once at nine stores + -- and the seals' fsyncs queued behind it for the next four + seconds (device write time 24 s in a 2-second window against 1-3 + otherwise). A store's mover rate is now `HeapStoreCleanBytes` per + `HeapCleanPeriod` blocks, and a `Compress` call takes the shards + next in rotation, as many as the blocks since the last call earn, + each with its share: called every block it moves a little on one + shard, and a hundred shards spread the same rate a hundred ways + with one shard locked at a time. The permanent layer's merge + already worked this way (buckets due by blocks elapsed). + Snapshots are by block count, or early when the files held only + by their deltas outweigh two data files; the manifest commits by + block count. + + *Measured on that build (before the every-block cadence), nine + stores, eight shards, clean minutes:* the heap alone 36-38 ms p50 + / 48-60 p90 in every minute; the files store 36-38 / 50-52, where + the day before it was 60 / 370-450 in every minute after the first + -- the seal's second barrier, the merge's three, and its reread of + every pending delta were the tail. One shard per store: 37-39 / + 51-54. + Still to do: the store-level commit (one block record naming every shard's deltas) and the per-store data file, so that a - hundred shards cost a block one barrier. With one shard per - store, which is that layout by another name, the heap sealed at - 40-42 ms p50 and 55-58 ms p90 in clean minutes, and the files - store at 46-49 / 53-57. + hundred shards cost a block one barrier; one shard per store is + that layout by another name and measures the same as eight now. From 8acb7e01e02c156f9d530b3266bc914c62185b03 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 23:07:55 -0500 Subject: [PATCH 46/58] A store's shards snapshot their key maps on different blocks Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 19 ++++++++++++++----- database/kv_shard.go | 13 +++++++++++++ 2 files changed, 27 insertions(+), 5 deletions(-) diff --git a/database/heap.go b/database/heap.go index 6dbcddd..a8060e7 100644 --- a/database/heap.go +++ b/database/heap.go @@ -103,10 +103,11 @@ type HeapStore struct { // snapshot, so the map a snapshot writes is exactly the state of // the last finished delta and no delta is in flight into a log // about to be retired. - syncMu sync.Mutex - log *os.File - gen uint64 - snapAt uint64 // The block the key map was last snapshotted at + syncMu sync.Mutex + log *os.File + gen uint64 + snapAt uint64 // The block the key map was last snapshotted at + snapPhase uint64 // Blocks this shard's snapshot cadence is offset by closed bool liveBytes int64 @@ -1047,6 +1048,14 @@ func (p *heapSync) finish() (err error) { return nil } +// SetSnapshotPhase offsets this shard's snapshot cadence by blocks, so +// that a store's shards do not all snapshot on the same block. +func (h *HeapStore) SetSnapshotPhase(blocks uint64) { + h.mu.Lock() + h.snapPhase = blocks + h.mu.Unlock() +} + // unlinkReleased deletes the files the block syncs have released. // Called without the lock. func (h *HeapStore) unlinkReleased() error { @@ -1092,7 +1101,7 @@ func (h *HeapStore) compact(budget int64) (bool, error) { pinned += hf.size } } - due := h.height-h.snapAt >= HeapSnapshotBlocks || pinned >= HeapSnapshotPinnedFiles*HeapFileBytes + due := h.height+h.snapPhase-h.snapAt >= HeapSnapshotBlocks || pinned >= HeapSnapshotPinnedFiles*HeapFileBytes if due { h.snapAt = h.height } diff --git a/database/kv_shard.go b/database/kv_shard.go index 7d847c5..504e237 100644 --- a/database/kv_shard.go +++ b/database/kv_shard.go @@ -179,6 +179,7 @@ func OpenKVShard(directory string) (kVShard *KVShard, err error) { return nil, err } } + kVShard.phaseSnapshots() kVShard.useSharedBlockRecord() if err = kVShard.adoptBlockHeight(); err != nil { return nil, err @@ -256,6 +257,17 @@ func NewKVShardN(directory string, shards int, sealLimit uint64) (kvs *KVShard, return newKVShardN(directory, shards, sealLimit, NewKV2) } +// phaseSnapshots spreads the heap shards' key-map snapshots over the +// snapshot period, so that a store's shards do not all write theirs +// on the same block. +func (k *KVShard) phaseSnapshots() { + for i, shard := range k.Shards { + if shard.Heap != nil { + shard.Heap.SetSnapshotPhase(uint64(i) * HeapSnapshotBlocks / uint64(len(k.Shards))) + } + } +} + func newKVShardN(directory string, shards int, sealLimit uint64, newShard func(string, uint64) (*KV2, error)) (kvs *KVShard, err error) { if shards < 1 { return nil, fmt.Errorf("a database needs at least one shard, asked for %d", shards) @@ -274,6 +286,7 @@ func newKVShardN(directory string, shards int, sealLimit uint64, newShard func(s return nil, err } } + kvs.phaseSnapshots() kvs.useSharedBlockRecord() if kvs.Sets, err = NewSetStore(kvs.setDir()); err != nil { return nil, err From b53affc9a6d8eccef8c4233ddc88b20b3e2d5603 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 23:18:19 -0500 Subject: [PATCH 47/58] bdbench checks every sampled permanent read and reads every store back after a reopen A permanent value is derived from its key, so any read of the key is checked for presence and content; a sampled permanent key that is missing is a mismatch; permanent keys of all ages go through the deep read the adapter uses. At the end every store is closed, reopened, and every sampled key of both layers read back; a run that lost one fails. Until now the platform timed permanent reads and tolerated "not found" on every one of them. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/main.go | 65 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 63 insertions(+), 2 deletions(-) diff --git a/cmd/bdbench/main.go b/cmd/bdbench/main.go index c28ba94..f0e7d46 100644 --- a/cmd/bdbench/main.go +++ b/cmd/bdbench/main.go @@ -20,6 +20,7 @@ package main import ( + "bytes" _ "embed" "encoding/csv" "encoding/json" @@ -290,7 +291,7 @@ func (s *store) block(c config, t *tallies) error { } for i := 0; i < c.permPuts; i++ { k := s.rnd.NextHash() - v := s.rnd.RandBuff(c.valueMin, c.valueMax) + v := permValue(k, c.valueMin, c.valueMax) at := time.Now() if err := s.kv.PutPerm(k, v); err != nil { return fmt.Errorf("store %d PutPerm: %w", s.id, err) @@ -306,11 +307,15 @@ func (s *store) block(c config, t *tallies) error { for i := 0; i < c.reads; i++ { var k [32]byte var get func([32]byte) ([]byte, error) + perm := false switch pick := s.rnd.UintN(6); { case pick < 3: k, get = s.hot[int(s.rnd.UintN(uint(c.hotKeys)))], s.kv.GetDyna case pick < 5 && len(s.permKeys) > 0: - k, get = s.permKeys[int(s.rnd.UintN(uint(len(s.permKeys))))], s.kv.GetPerm + // A permanent key of any age, through the deep read the + // adapter uses for anything older than the window (Get + // answers the permanent layer's window only, by design) + k, get, perm = s.permKeys[int(s.rnd.UintN(uint(len(s.permKeys))))], s.kv.GetDeep, true default: k, get = s.rnd.NextHash(), s.kv.Get } @@ -320,6 +325,14 @@ func (s *store) block(c config, t *tallies) error { if err != nil && !errors.Is(err, os.ErrNotExist) && !strings.Contains(err.Error(), "not found") { return fmt.Errorf("store %d read: %w", s.id, err) } + // A permanent key that was written must come back, with the + // value its key derives; a checked hot key must come back as + // last written. A platform that only times answers cannot + // tell a fast wrong answer, or a fast "not found", from a + // right one. + if perm && (err != nil || !bytes.Equal(v, permValue(k, c.valueMin, c.valueMax))) { + t.mismatches.Add(1) + } if want, ok := s.last[k]; ok && err == nil && string(v) != string(want) { t.mismatches.Add(1) } @@ -535,6 +548,21 @@ func sumHeap(stores []*store) (hs heapSplit) { return hs } +// permValue is the value a permanent key carries: derived from the +// key, so that any read of the key can be checked without remembering +// what was written. Its length spreads over [min, max] by the key. +func permValue(k [32]byte, min, max uint) []byte { + n := int(min) + if max > min { + n += int(k[31]) * int(max-min) / 255 + } + v := make([]byte, n) + for i := 0; i < n; i += len(k) { + copy(v[i:], k[:]) + } + return v +} + func fail(what string, err error) { fmt.Fprintln(os.Stderr, "bdbench:", what+":", err) os.Exit(1) @@ -752,6 +780,39 @@ wait: fail("close", err) } } + // Every store is reopened and read back: every sampled permanent + // key with its derived value, every checked hot key with its last + // value. What a seal made durable must be there after a close + // and an open, and a run that loses one fails. + var wrong int + for _, s := range stores { + re, err := blockchainDB.OpenKVShard(s.kv.Directory) + if err != nil { + fail(fmt.Sprintf("store %d reopen", s.id), err) + } + bad := 0 + for i, k := range s.permKeys { + if v, err := re.GetDeep(k); err != nil || !bytes.Equal(v, permValue(k, c.valueMin, c.valueMax)) { + if bad < 3 { + fmt.Printf(" sampled key %d of %d: err=%v got %d bytes want %d\n", i, len(s.permKeys), err, len(v), len(permValue(k, c.valueMin, c.valueMax))) + } + bad++ + } + } + for k, want := range s.last { + if v, err := re.GetDyna(k); err != nil || !bytes.Equal(v, want) { + bad++ + } + } + fmt.Printf("reopen store %d: %d permanent + %d dynamic keys read back, %d wrong\n", s.id, len(s.permKeys), len(s.last), bad) + wrong += bad + if err := re.Close(); err != nil { + fail("close after reopen", err) + } + } + if wrong > 0 { + fail("reopen", fmt.Errorf("%d keys lost or wrong after close and reopen", wrong)) + } if runErr != nil { fail("run", runErr) } From e7d08186cc181d5e8a777940bec62ef3085038f3 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 23:19:08 -0500 Subject: [PATCH 48/58] A merge's run files are fsynced at the manifest commit, not per call Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 75 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 55 insertions(+), 20 deletions(-) diff --git a/database/perm.go b/database/perm.go index 5431d58..5906b31 100644 --- a/database/perm.go +++ b/database/perm.go @@ -141,10 +141,11 @@ type permBucket struct { // runFile is a file of runs and how many runs still reference it. type runFile struct { - id uint32 - f *os.File - size int64 - refs int + id uint32 + f *os.File + size int64 + refs int + unsynced bool // Runs written since the file was last fsynced } func permDataName(id uint32) string { return fmt.Sprintf("perm-%06d.dat", id) } @@ -484,6 +485,9 @@ func (p *PermStore) Close() error { p.mu.Lock() defer p.mu.Unlock() if p.manifestDirty { + if err = p.syncRunFiles(); err != nil { + return err + } if err = p.commitManifest(); err != nil { return err } @@ -896,12 +900,15 @@ func (p *PermStore) Merge() error { run *permRun }{b: j.b, run: run}) } - if err := fsync(rf.f); err != nil { - return err - } + // No barrier here: a bucket's run is named by the manifest alone, + // so it needs to be durable before the manifest commit and not + // before. Every merge call once ended with an fsync of its run + // file, which at a call per shard per block was 72-144 barriers + // a second at nine stores and doubled every seal's fsync. // Swap: the new runs join their buckets; then fold what the ratio // says, one bucket at a time p.mu.Lock() + rf.unsynced = true for _, w := range written { bk := &p.buckets[w.b] bk.runs = append(bk.runs, w.run) @@ -928,20 +935,10 @@ func (p *PermStore) Merge() error { p.pending = keep folds := p.planFolds() p.mu.Unlock() - // The folds' runs are written without a barrier each and synced - // once, before the manifest that names them - var synced []*runFile + // The folds' runs are written without a barrier too; the manifest + // commit syncs every run file written since the last for _, f := range folds { - rf, err := p.fold(f) - if err != nil { - return err - } - if rf != nil && (len(synced) == 0 || synced[len(synced)-1] != rf) { - synced = append(synced, rf) - } - } - for _, rf := range synced { - if err := fsync(rf.f); err != nil { + if _, err := p.fold(f); err != nil { return err } } @@ -958,6 +955,9 @@ func (p *PermStore) Merge() error { if len(folds) == 0 && p.height-p.manifestAt < PermManifestBlocks { return nil } + if err := p.syncRunFiles(); err != nil { + return err + } if err := p.commitManifest(); err != nil { return err } @@ -1033,6 +1033,7 @@ func (p *PermStore) fold(f permFold) (*runFile, error) { p.mu.Lock() defer p.mu.Unlock() rf.size = at + int64(run.bytes) + rf.unsynced = true bk = &p.buckets[f.b] if f.at+f.count > len(bk.runs) { return rf, nil @@ -1110,12 +1111,46 @@ func (p *PermStore) Pack() error { for b := range p.buckets { p.buckets[b] = permBucket{merged: height} } + if err := p.syncRunFiles(); err != nil { + return err + } if err := p.commitManifest(); err != nil { return err } return p.dropUnreferencedRunFiles() } +// syncRunFiles makes every run file written since its last fsync +// durable, with the lock released for the barriers. Maintenance is +// one pass at a time, so nothing writes a run file meanwhile. The +// caller holds the lock and gets it back. +func (p *PermStore) syncRunFiles() error { + var dirty []*runFile + for _, rf := range p.runs { + if rf.unsynced { + dirty = append(dirty, rf) + } + } + if len(dirty) == 0 { + return nil + } + p.mu.Unlock() + var err error + for _, rf := range dirty { + if err = fsync(rf.f); err != nil { + break + } + } + p.mu.Lock() + if err != nil { + return err + } + for _, rf := range dirty { + rf.unsynced = false + } + return nil +} + // commitManifest writes perm.json: encoded under the lock, written, // fsynced and renamed with the lock released, so that a merge's // manifest commit -- every twenty blocks, per shard -- is not a From e60c6f9bada4a1b32a42b64da9a472966822b4b7 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 23:19:30 -0500 Subject: [PATCH 49/58] The proposal records the platform's checks and the merge's barrier Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- .../2026-09-16-entries-written-once.md | 23 ++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index 6ad6139..994cb59 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -278,7 +278,18 @@ point" and closes #33. ## What it is measured with -`cmd/bdbench -stores 9`, the run above, is the acceptance test: +`cmd/bdbench -stores 9`, the run above, is the acceptance test. It +checks answers, not only times: a permanent value is derived from its +key, so every sampled permanent read (through the deep read, as the +adapter reads anything older than the window) is checked for presence +and content, a checked hot key must read as last written, and at the +end every store is closed, reopened and every sampled key of both +layers read back -- a run that lost one fails. Until 2026-09-17 the +platform tolerated "not found" on every permanent read, and a data +file rolled between manifest commits was being lost on reopen without +a number moving. With the check in place: 200,000 sampled permanent +keys per store read back correctly after reopen, zero mismatches +under load. The acceptance run must show: - seal p90 under `-seal-budget` (100 ms) and flat from minute 1 to minute 30; @@ -369,6 +380,16 @@ point" and closes #33. every pending delta were the tail. One shard per store: 37-39 / 51-54. + *The cadence per layer.* On the every-block cadence the heap's + slice is cheap, but the permanent layer's merge ended every call + with an fsync of its run file: at a call per shard per block that + was 72-144 barriers a second at nine stores, merges cost 190-250 s + of work a minute, and every seal's fsync doubled. A bucket's run + is named by the manifest alone, so it needs durability before the + manifest commit and not before: the run files are fsynced at the + commit (every PermManifestBlocks, or at a fold), and a merge call + has no barrier. + Still to do: the store-level commit (one block record naming every shard's deltas) and the per-store data file, so that a hundred shards cost a block one barrier; one shard per store is From 040dc5d5936efbaafe5171910b8d09b27d15dcda Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 23:23:38 -0500 Subject: [PATCH 50/58] A fold commits the manifest only when it left a run file to drop, and the manifest names only the run files it keeps With a merge every block a bucket folds on nearly every call, and a commit at every fold was two barriers a shard a block. The manifest listed every open run file, unreferenced ones included, which the drop then deleted: a durable manifest could name a file that was gone. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/database/perm.go b/database/perm.go index 5906b31..429f9ee 100644 --- a/database/perm.go +++ b/database/perm.go @@ -952,7 +952,19 @@ func (p *PermStore) Merge() error { p.mu.Lock() defer p.mu.Unlock() p.manifestDirty = true - if len(folds) == 0 && p.height-p.manifestAt < PermManifestBlocks { + // A commit is due by age, or when a fold left a run file that + // nothing references, since only a commit can drop it. A fold + // alone is not a reason: with a merge every block a bucket folds + // on nearly every call, and a commit each time was two barriers + // a shard a block. + droppable := false + for _, rf := range p.runs { + if rf.refs == 0 && rf != p.maintRun { + droppable = true + break + } + } + if !droppable && p.height-p.manifestAt < PermManifestBlocks { return nil } if err := p.syncRunFiles(); err != nil { @@ -1197,8 +1209,13 @@ func (p *PermStore) encodeManifest() ([]byte, error) { for id := range p.files { m.DataFiles = append(m.DataFiles, id) } - for id := range p.runs { - m.RunFiles = append(m.RunFiles, id) + // Only the run files kept: a file nothing references is dropped + // once this manifest is durable, and a durable manifest must never + // name a file that is gone (1.7, 1.8) + for id, rf := range p.runs { + if rf.refs > 0 || rf == p.maintRun { + m.RunFiles = append(m.RunFiles, id) + } } sort.Slice(m.DataFiles, func(i, j int) bool { return m.DataFiles[i] < m.DataFiles[j] }) sort.Slice(m.RunFiles, func(i, j int) bool { return m.RunFiles[i] < m.RunFiles[j] }) From c80b24bfd796d4fe9ed935203c7b7ba7ba1ed968 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 23:30:14 -0500 Subject: [PATCH 51/58] A file the current generation's deltas name is not released, whatever is live in it The mover's files are named by the deltas as the destination of their ranges. When every copy in one died it was released and unlinked while those deltas still named it; on reopen the replay stopped at that delta and the derivation deleted the files it then took for unnamed. Found by the platform's reopen check. A file now carries the bytes of ranges the generation's deltas name in it, is released only when that is zero as well, and the snapshot that supersedes the deltas clears it; the pinned-bytes rule counts such files so the snapshot comes early when they pile up. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/heap.go | 46 +++++++++++++++++++++++++++++++++---- database/heap_test.go | 53 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 4 deletions(-) diff --git a/database/heap.go b/database/heap.go index a8060e7..166ac19 100644 --- a/database/heap.go +++ b/database/heap.go @@ -138,6 +138,27 @@ type heapFile struct { cleaning bool // Taken by the pass in progress inflight int // Copies reserved in it and not yet written: its size runs ahead of its bytes releasing bool // Emptied by a pass; deleted by the next sync + // Bytes of ranges in it that the deltas of the current generation + // name, and where the last of them ends. A durable delta names + // the file, so the file stays until a snapshot supersedes the + // delta, whatever is live in it (spec 1.7): never unlink what a + // durable index names. + named, namedEnd int64 +} + +// nameRanges records that a delta of the current generation names +// these ranges. The caller holds the lock. +func (h *HeapStore) nameRanges(ranges []heapRange) { + for _, r := range ranges { + hf := h.files[r.file] + if hf == nil || r.to <= r.from { + continue + } + hf.named += int64(r.to - r.from) + if int64(r.to) > hf.namedEnd { + hf.namedEnd = int64(r.to) + } + } } // heapRange is a stretch of one data file: what a delta names. @@ -479,8 +500,10 @@ func (h *HeapStore) startGeneration() error { } snap := h.encodeIndexOf(heapSnapshot, all, len(h.index)) superseded := map[uint32]int64{} + namedThen := map[uint32]int64{} for id, hf := range h.files { superseded[id] = hf.deltas + namedThen[id] = hf.named } h.mu.Unlock() f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) @@ -525,6 +548,16 @@ func (h *HeapStore) startGeneration() error { h.deadBytes += n } } + // The ranges those deltas named are no longer named by anything + // durable but the snapshot's own slots + for id, n := range namedThen { + if hf := h.files[id]; hf != nil { + hf.named -= n + if hf.named == 0 { + hf.namedEnd = 0 + } + } + } return nil } @@ -636,6 +669,7 @@ func (h *HeapStore) applyDelta(recs []byte, verify bool) bool { for _, n := range names { h.index[n.key] = n.s } + h.nameRanges(ranges) if height > h.height { h.height = height } @@ -717,6 +751,9 @@ func (h *HeapStore) deriveFiles(dataIDs []uint32) error { if id == h.deltaAt.file && h.deltaAt.off > end { end, live = h.deltaAt.off, true } + if hf.namedEnd > end { + end, live = hf.namedEnd, true // A delta of this generation names it + } if !live && hf.deltas == 0 { hf.f.Close() delete(h.files, id) @@ -994,6 +1031,7 @@ func (h *HeapStore) beginBlockSync() (blockSync, error) { // The delta goes into the block's data file behind its entries, // under the reserved key, so the block's one fsync covers both p.delta = h.encodeDelta() + h.nameRanges(h.ranges) h.ranges, h.excluded = nil, nil hf, off, err := h.reserve(false, entrySize(len(p.delta))) h.ranges = nil // The delta entry itself is not part of the block's range @@ -1097,8 +1135,8 @@ func (h *HeapStore) compact(budget int64) (bool, error) { h.mu.Lock() var pinned int64 for _, hf := range h.files { - if hf != h.cur && hf != h.mov && hf.deltas > 0 && hf.live == hf.deltas { - pinned += hf.size + if hf != h.cur && hf != h.mov && hf.live == hf.deltas && (hf.deltas > 0 || hf.named > 0) { + pinned += hf.size // Nothing live in it but what the generation's deltas need } } due := h.height+h.snapPhase-h.snapAt >= HeapSnapshotBlocks || pinned >= HeapSnapshotPinnedFiles*HeapFileBytes @@ -1175,7 +1213,7 @@ func (h *HeapStore) clean(budget int64) (bool, error) { // floated with the size bound engaged. emptied := 0 for _, hf := range h.files { - if hf.live == 0 && hf != h.cur && hf != h.mov && !hf.cleaning && hf.inflight == 0 && !hf.releasing && hf.size > 0 { + if hf.live == 0 && hf.named == 0 && hf != h.cur && hf != h.mov && !hf.cleaning && hf.inflight == 0 && !hf.releasing && hf.size > 0 { hf.releasing = true h.release = append(h.release, hf.id) emptied++ @@ -1333,7 +1371,7 @@ func (h *HeapStore) clean(budget int64) (bool, error) { } h.ranges = append(mine, h.ranges...) for _, hf := range taken { - if hf.live == 0 && hf != h.cur && hf != h.mov { + if hf.live == 0 && hf.named == 0 && hf != h.cur && hf != h.mov { hf.releasing = true h.release = append(h.release, hf.id) } diff --git a/database/heap_test.go b/database/heap_test.go index 9ef48d2..4577d88 100644 --- a/database/heap_test.go +++ b/database/heap_test.go @@ -396,3 +396,56 @@ func TestHeapMoveIsDeadOnArrivalIfTheKeyWasRewritten(t *testing.T) { require.NoError(t, err) require.Equal(t, "rewritten while moving", string(v)) } + +// A file the mover copied into is named by the deltas of the current +// generation, as the destination of their ranges. When every copy in +// it dies it must not be unlinked until a snapshot supersedes those +// deltas: never unlink what a durable index names (spec 1.7). Found +// by the platform's reopen check: a store closed cleanly would not +// open, "the index names heap-000006.dat: missing". +func TestHeapMoverFileNamedByDeltasOutlivesItsEntries(t *testing.T) { + was, blocks, pinned := HeapFileBytes, HeapSnapshotBlocks, HeapSnapshotPinnedFiles + HeapFileBytes, HeapSnapshotBlocks, HeapSnapshotPinnedFiles = 64<<10, 1<<20, 1<<30 // No snapshot at all + defer func() { HeapFileBytes, HeapSnapshotBlocks, HeapSnapshotPinnedFiles = was, blocks, pinned }() + dir := heapDir(t) + h, err := NewHeapStore(dir) + require.NoError(t, err) + value := make([]byte, 400) + last := map[byte]byte{} + // Half the keys are rewritten every block, half every seventh: a + // file is soon half dead with live entries the mover copies out, + // and the copies die within seven blocks, emptying the mover's file + put := func(b uint64) { + h.AdvanceBlock(b) + for i := byte(1); i <= 100; i++ { + if i > 50 && b%7 != 0 { + continue + } + value[0] = byte(b) + require.NoError(t, h.Put(key(i), value)) + last[i] = byte(b) + } + syncHeap(t, h) + } + for b := uint64(1); b <= 70; b++ { + put(b) + if b%3 == 0 { + _, err := h.compact(HeapCleanBytes) // Copies live entries into mover files, releases what emptied + require.NoError(t, err) + } + } + _, moved := h.Cleaned() + require.Greater(t, moved, uint64(0), "the mover must have copied") + releases, _, snapshots, _ := h.MoverCost() + t.Logf("moved %d bytes, %d files unlinked, %d snapshots, %d files open, gen %d", moved, releases, snapshots, len(h.files), h.gen) + require.NoError(t, h.Close()) + h, err = OpenHeapStore(dir) + require.NoError(t, err, "a store closed cleanly reopens") + t.Logf("after open: %d files, gen %d, replay point %v", len(h.files), h.gen, h.deltaAt) + for i := byte(1); i <= 100; i++ { + v, err := h.Get(key(i)) + require.NoError(t, err, "key %d", i) + require.Equal(t, last[i], v[0], "key %d", i) + } + require.NoError(t, h.Close()) +} From 148bb88dd363e61d117f1990b74a99ce99a66b58 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 23:30:43 -0500 Subject: [PATCH 52/58] A bucket is merged once a window, so no more than a window of deltas waits unmerged Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/perm.go | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/database/perm.go b/database/perm.go index 429f9ee..7831ebf 100644 --- a/database/perm.go +++ b/database/perm.go @@ -116,7 +116,12 @@ const PermBuckets = 256 // PermMergeEvery is how many blocks pass between merges of one // bucket: PermBuckets/PermMergeEvery buckets are merged each block. -var PermMergeEvery uint64 = 256 +// It bounds how many deltas wait unmerged, which every deep read +// probes and every merge reads a slice of: at 256, read p99 climbed +// from 7 to 47 us over five minutes and merge work from 157 to 305 s +// a minute. One window's worth keeps both flat; the extra folds +// cost a fraction of the index bytes. +var PermMergeEvery uint64 = MinFilterBlocks // PermFileBytes is the size data files and run files are rolled at. var PermFileBytes int64 = 64 << 20 From 06f051abead82053683940c41668b58b47eba4e9 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Wed, 16 Sep 2026 23:31:02 -0500 Subject: [PATCH 53/58] The proposal records the named-file rule and the merge period Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- .../2026-09-16-entries-written-once.md | 29 +++++++++++++++---- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index 994cb59..329ea78 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -229,11 +229,21 @@ point" and closes #33. ## Durability and crash consistency (1.8) -- **A file is deleted one seal late, by the mover.** A file emptied - by the mover leaves the map only after the delta naming the mover's - copies out of it is durable. Until then the durable index still - names its slots, and a crash must find them intact: never unlink - what a durable index names. The unlink itself is the mover's, at +- **A file is deleted one seal late, by the mover, and never while a + delta names it.** A file emptied by the mover leaves the map only + after the delta naming the mover's copies out of it is durable. + Until then the durable index still names its slots, and a crash + must find them intact: never unlink what a durable index names. + The deltas name ranges, and the mover's files are the destination + of theirs: a file every copy in which has died is still named by + the generation's deltas, and stays until the snapshot that + supersedes them, however dead it is. (Measured the other way + first: the platform's reopen check found a store closed cleanly + that would not open, its replay stopped at a delta naming an + unlinked mover file, and the derivation deleting what it then took + for unnamed.) Each file carries the bytes such deltas name in it; + the pinned-bytes rule counts those files, so the snapshot comes + early when they pile up. The unlink itself is the mover's, at its next pass, so no block's seal holds the store lock over a directory operation; between the two the file is unnamed on disk, which an open deletes as such. The adapter never asks the store @@ -380,6 +390,15 @@ under load. The acceptance run must show: every pending delta were the tail. One shard per store: 37-39 / 51-54. + *What waits unmerged is bounded by the window.* A bucket was + merged once per 256 blocks, so up to 256 deltas waited unmerged, + and every deep read probed each one's filter and every merge read + a slice of each: over five minutes read p99 climbed from 7 to 47 + us and merge work from 157 to 305 s a minute. A bucket is now + merged once a window (`PermMergeEvery` = `MinFilterBlocks`), so + no more than a window of deltas waits; the extra folds are a + fraction of the index bytes, themselves a sixth of the data. + *The cadence per layer.* On the every-block cadence the heap's slice is cheap, but the permanent layer's merge ended every call with an fsync of its run file: at a call per shard per block that From 32e0702fc8e63f3d6392bcabd9c8717863f05324 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Thu, 17 Sep 2026 00:04:12 -0500 Subject: [PATCH 54/58] A resident run keeps a fence of every 32nd key, so a lookup is one read Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- database/permindex.go | 50 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/database/permindex.go b/database/permindex.go index c34fc05..5157e09 100644 --- a/database/permindex.go +++ b/database/permindex.go @@ -72,6 +72,27 @@ type permRun struct { bloom *Bloom k int bytes uint32 + // Every permFenceEvery-th key, for a resident run: a lookup finds + // its chunk in memory and reads that chunk once, where a binary + // search over the file was a pread a probe -- a dozen a run, and + // 40% of a block's CPU under the load platform's reads. + fence [][32]byte +} + +// permFenceEvery is the records between fence keys: a chunk is one +// read of this many records, and the fence costs 32 bytes per chunk. +const permFenceEvery = 32 + +// buildFence takes every permFenceEvery-th key from records laid out +// in buf. +func buildFence(buf []byte, count uint32) [][32]byte { + fence := make([][32]byte, 0, (int(count)+permFenceEvery-1)/permFenceEvery) + for i := 0; i < int(count); i += permFenceEvery { + var k [32]byte + copy(k[:], buf[i*permRecSize:]) + fence = append(fence, k) + } + return fence } // writePermRun writes records, which must be sorted by key and free @@ -102,7 +123,8 @@ func writePermRun(w io.WriterAt, at int64, file uint32, recs []permRecord, heigh if _, err := w.WriteAt(buf, at); err != nil { return nil, err } - return &permRun{path: permRunName(file), file: file, off: at, count: uint32(len(recs)), height: height, bloomAt: at + int64(p), bloom: bloom, k: bloom.K, bytes: uint32(len(buf))}, nil + return &permRun{path: permRunName(file), file: file, off: at, count: uint32(len(recs)), height: height, bloomAt: at + int64(p), bloom: bloom, k: bloom.K, bytes: uint32(len(buf)), + fence: buildFence(buf[permRunHdr:], uint32(len(recs)))}, nil } // openPermRun reads a run's header at off in f and verifies the run. @@ -131,6 +153,7 @@ func openPermRun(f *os.File, file uint32, off int64, resident bool) (*permRun, e r.bloom = &Bloom{NumBytes: uint64(bloomBytes), SizeOfMap: float64(bloomBytes) / (1 << 20), K: r.k, Map: make([]byte, bloomBytes), Capacity: uint64(bloomBytes) * 8 / BloomBitsPerKey, Count: uint64(r.count)} copy(r.bloom.Map, body[int64(r.count)*permRecSize:]) + r.fence = buildFence(body, r.count) } return r, nil } @@ -145,6 +168,31 @@ func (r *permRun) lookup(f *os.File, key [32]byte) (rec permRecord, found bool, } else if ok, err := r.bloomTestCold(f, key); err != nil || !ok { return rec, false, err } + if r.fence != nil { + // The chunk whose first key is the greatest not above key + i := sort.Search(len(r.fence), func(i int) bool { return bytes.Compare(r.fence[i][:], key[:]) > 0 }) - 1 + if i < 0 { + return rec, false, nil + } + first := int64(i) * permFenceEvery + n := int64(permFenceEvery) + if first+n > int64(r.count) { + n = int64(r.count) - first + } + buf := make([]byte, n*permRecSize) + if _, err := f.ReadAt(buf, r.off+permRunHdr+first*permRecSize); err != nil { + return rec, false, err + } + for j := int64(0); j < n; j++ { + switch c := bytes.Compare(buf[j*permRecSize:j*permRecSize+32], key[:]); { + case c == 0: + return getPermRecord(buf[j*permRecSize:]), true, nil + case c > 0: + return rec, false, nil + } + } + return rec, false, nil + } lo, hi := int64(0), int64(r.count) buf := make([]byte, permRecSize) for lo < hi { From d5974bc82aa5342db76bf934e21b0ca2d27be400 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Thu, 17 Sep 2026 00:04:32 -0500 Subject: [PATCH 55/58] The proposal records the verified every-block run and the run fence Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- docs/proposals/2026-09-16-entries-written-once.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index 329ea78..035337d 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -409,7 +409,18 @@ under load. The acceptance run must show: commit (every PermManifestBlocks, or at a fold), and a merge call has no barrier. + *Measured with every fix above, both layers, nine stores, eight + shards, maintenance every block, 2026-09-17 00:05:* seal 38-42 ms + p50 and 57-65 p90 in the four clean minutes (the fifth was a disk + stall), merges 3-8 s of work a minute, zero mismatches, and every + store reopened with all 200,000 sampled permanent keys and 1,024 + checked dynamic keys read back correctly. Block time drifted from + 157 to 210 ms: the profile put 40% of the CPU in the deep read's + binary search over bucket runs, a pread a probe; a resident run + now keeps a fence of every 32nd key and a lookup is one read. + Still to do: the store-level commit (one block record naming every shard's deltas) and the per-store data file, so that a hundred shards cost a block one barrier; one shard per store is that layout by another name and measures the same as eight now. + The 30-minute acceptance run waits on the trimmed disk. From 1f6197cf87dda908134ed19498138b2cbe42423b Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Thu, 17 Sep 2026 00:10:43 -0500 Subject: [PATCH 56/58] The proposal records the fence's measurement Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- docs/proposals/2026-09-16-entries-written-once.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index 035337d..edd9bf5 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -418,6 +418,11 @@ under load. The acceptance run must show: 157 to 210 ms: the profile put 40% of the CPU in the deep read's binary search over bucket runs, a pread a probe; a resident run now keeps a fence of every 32nd key and a lookup is one read. + Measured: read p99 6-7 us instead of 9-10, block time flat at + 178-181 ms from minute 3 on instead of climbing, all nine stores + read back clean again. What remains in the deep read is one read + of the run's chunk and one of the entry, which is the floor for + an index that does not fit in memory. Still to do: the store-level commit (one block record naming every shard's deltas) and the per-store data file, so that a From 3dd27bc9d4b3ec6bb4b9c08f73bbb58ba5ce5730 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Thu, 17 Sep 2026 00:30:55 -0500 Subject: [PATCH 57/58] The platform's samplers, the disk runbook, and what the instrument must be (spec 2.11) Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5 --- cmd/bdbench/tools/README.md | 35 ++++++++++++++++ cmd/bdbench/tools/sys-timeline.sh | 12 ++++++ cmd/bdbench/tools/timeline.sh | 8 ++++ docs/SPEC.md | 19 ++++++++- docs/runbooks/disk-trim.md | 69 +++++++++++++++++++++++++++++++ 5 files changed, 142 insertions(+), 1 deletion(-) create mode 100644 cmd/bdbench/tools/README.md create mode 100755 cmd/bdbench/tools/sys-timeline.sh create mode 100755 cmd/bdbench/tools/timeline.sh create mode 100644 docs/runbooks/disk-trim.md diff --git a/cmd/bdbench/tools/README.md b/cmd/bdbench/tools/README.md new file mode 100644 index 0000000..e27bffd --- /dev/null +++ b/cmd/bdbench/tools/README.md @@ -0,0 +1,35 @@ +# bdbench tools + +Samplers that run beside a bdbench run and write one line every two +seconds, so that a minute's row can be taken apart afterwards. + +- `timeline.sh ` samples the run's `live.json` into + `.timeline`: the last ten seconds' seal p50/p90/max, the heap's + cumulative fsync average and bytes per sync, snapshots, releases, + moved bytes, hole and live bytes, perm merges and folds, and each + store's seal p90 (`ps`). +- `sys-timeline.sh ` samples the machine into + `.sys`: dirty and free memory, the NVMe device's write count, + write ticks and in-flight requests from `/proc/diskstats`, and the + load average. + +Both wait for `//live.json` to appear and stop when it +goes away (the run directory is removed). Start them detached before +the run: + + setsid nohup tools/timeline.sh $S heap-2a >/dev/null 2>&1 & + setsid nohup tools/sys-timeline.sh $S heap-2a >/dev/null 2>&1 & + +## Reading them + +Difference the cumulative fields between samples. The device's write +ticks per 2-second window is the queue depth times the latency: 1-3 s +on this disk when healthy, 100-500 s when the drive is stalling. + +A disk stall and a store tail look alike in the per-minute row (seal +p50 unchanged, p90 200-500 ms) and different here: a stall raises the +heap's fsync average for every store at once, the device's write ticks +with it, and the store's own counters (moved, snapshots, releases) show +no step. A store tail lines up with a step in one of those counters +and the device stays quiet. See `docs/runbooks/disk-trim.md` for the +disk this was learned on. diff --git a/cmd/bdbench/tools/sys-timeline.sh b/cmd/bdbench/tools/sys-timeline.sh new file mode 100755 index 0000000..646951f --- /dev/null +++ b/cmd/bdbench/tools/sys-timeline.sh @@ -0,0 +1,12 @@ +#!/bin/bash +# System counters every 2 s while a run's live.json exists: memory and the device +S=$1; run=$2 +until [ -f $S/$run/live.json ]; do sleep 2; done +while [ -f $S/$run/live.json ]; do + t=$(jq -r .elapsedSec $S/$run/live.json 2>/dev/null) + m=$(awk '/^(Dirty|Writeback|MemFree|MemAvailable|Cached):/{printf "%s=%d ", $1, $2/1024}' /proc/meminfo) + d=$(grep -w nvme0n1 /proc/diskstats | awk '{printf "wr=%d wr_ms=%d inflight=%d io_ms=%d", $8, $11, $12, $13}') + l=$(cut -d' ' -f1 /proc/loadavg) + echo "t=$t $m $d load=$l" >> $S/$run.sys + sleep 2 +done diff --git a/cmd/bdbench/tools/timeline.sh b/cmd/bdbench/tools/timeline.sh new file mode 100755 index 0000000..11ba052 --- /dev/null +++ b/cmd/bdbench/tools/timeline.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# Samples a run's live.json every 2 s into .timeline (one JSON line each) +S=$1; run=$2 +until [ -f $S/$run/live.json ]; do sleep 2; done +while [ -f $S/$run/live.json ]; do + jq -c '{t:.elapsedSec, s50:.last10s.sealP50ms, s90:.last10s.sealP90ms, smax:.last10s.sealMaxMs, fs:.heapFsyncMsAvg, kb:.heapSyncKBAvg, syncs:.heapSyncs, snaps:.heapSnapshots, snapms:.heapSnapshotMs, rel:.heapReleases, relms:.heapReleaseMs, mv:.heapMovedMB, hole:.heapHoleMB, live:.heapLiveMB, merges:.permMerges, folds:.permFolds, inflight:.maintenanceInFlight, ps:((.last10s.storeSealP90ms // [])|map(floor))}' $S/$run/live.json 2>/dev/null >> $S/$run.timeline + sleep 2 +done diff --git a/docs/SPEC.md b/docs/SPEC.md index 6a3559d..02f479b 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -527,9 +527,26 @@ resident filter memory; and the count of reads that returned a value other than the last written, which fails the run. A minute whose seal p90 exceeds `-seal-budget` (100 ms) is flagged. +It checks answers, not only times: permanent values derive from their +keys, so every sampled permanent read is checked for presence and +content; checked hot keys must read as last written; and at the end +every store is closed, reopened and every sampled key of both layers +read back. A run that loses one fails (1.8). + What a healthy store must show on it: block time under the interval, the seal bounded and flat with age, put and read p99 flat with age, -and maintenance passes bounded per pass (1.2). A change to the +maintenance passes bounded per pass (1.2), zero wrong answers, and +every store reading back after reopen. + +The instrument has to be sound before a number is the store's. The +disk is shared by every store under test, and an SSD that is never +trimmed stalls every fsync on the box for tens of seconds at a time +under a sustained write load; that reads as a seal tail and is not +one. A run is read with the per-minute heap split and the two-second +store and device timelines (`cmd/bdbench/tools`), which tell a device +stall from a store tail, and a run on a disk that stalls is compared +by its clean minutes or not at all. `docs/runbooks/disk-trim.md` +records the disk this was learned on and how it is verified. A change to the protocol path or to maintenance is measured here before it is measured under Accumulate, because here the seal's wait is the store's own and not the executor's. diff --git a/docs/runbooks/disk-trim.md b/docs/runbooks/disk-trim.md new file mode 100644 index 0000000..2d39679 --- /dev/null +++ b/docs/runbooks/disk-trim.md @@ -0,0 +1,69 @@ +# The measurement disk must be trimmed + +Written 2026-09-17, for the machine the load platform (`cmd/bdbench`, +spec 2.11) runs on. It applies to any SSD the store is measured on. + +## Why + +An SSD does not learn that a file was deleted; the filesystem only +marks the blocks free in its own tables. Until the filesystem sends a +discard ("trim") for them, the drive treats them as live data it must +preserve, and when it runs out of pre-erased flash under a sustained +write load it erases and rewrites around them. On this machine that +showed as every fsync on the box going from 12-22 ms to 100-330 ms for +30-40 seconds at a time, with nothing in the store changing, and the +device's write ticks going from 1-3 s to 100-500 s per 2-second window +(`cmd/bdbench/tools/README.md`). Half of a day's runs carried such +minutes. They are not the store's, and they make the 30-minute +acceptance run (spec 2.11) meaningless until the disk is trimmed. + +## How this disk is laid out + + nvme0n1p3 -> cryptdata (LUKS2, dm-0) -> data-root (LVM linear, dm-1) -> ext4 / + +Discards have to pass through every layer. As found: + +- The root filesystem is mounted without `discard`, so nothing is + trimmed on delete. Fine: the weekly `fstrim.timer` is the usual + way. +- The timer's log showed it trimming only `/boot/efi`: `fstrim` skips + a filesystem whose device reports no discard support. +- `lsblk -D` showed `cryptdata` and `data-root` both with `DISC-MAX` + 0: the LUKS mapping was opened without `allow-discards`, so the + encrypted layer refused discards and the volume above it inherited + that. + +## What was done + +1. `sudo cryptsetup refresh --allow-discards --persistent cryptdata` + reloaded the encrypted mapping with discards allowed and wrote the + flag into the LUKS2 header, so every future open has it without an + `/etc/crypttab` change. After it, `/sys/block/dm-0/queue/ + discard_max_bytes` read 2 TB. +2. `data-root` (dm-1) kept `discard_max_bytes` 0. A volume computes + its limits from the device beneath it when its table is loaded. + `sudo lvchange --refresh data/root` and an explicit + `dmsetup reload ... --table "0 7803994112 linear 252:0 2048" && + dmsetup resume` both ran without error and changed nothing: this + kernel (6.17) did not recompute the limits of the mounted root + volume on a live reload. +3. Remaining: a reboot, after which the volume activates over the + discard-capable encrypted device, then `sudo fstrim -v /`, which + should report over a terabyte discarded. The weekly timer keeps it + trimmed from then on. + +## How to verify + + lsblk -D -o NAME,DISC-GRAN,DISC-MAX # every layer non-zero + cat /sys/block/dm-1/queue/discard_max_bytes # non-zero + sudo fstrim -v / # reports bytes trimmed + journalctl -u fstrim.service # the timer trims / + +Then run the platform with the samplers and confirm the device's write +ticks stay at 1-3 s per window through a five-minute run. + +## The trade-off + +With discards on, someone holding the raw disk can tell which blocks +of the encrypted volume are unused. For a development machine that is +acceptable; it is the reason distributions leave it off by default. From f240ce2b0a3abc018540c14288c5d46feb48f862 Mon Sep 17 00:00:00 2001 From: Paul Snow Date: Thu, 17 Sep 2026 12:22:00 -0500 Subject: [PATCH 58/58] The disk is trimmed, the acceptance run holds, and the runbook moves out The measurement disk had never been trimmed: the root filesystem sits on a LUKS mapping opened without allow-discards, so `fstrim` skipped it and the drive still believed every deleted store was live data. Under a sustained write load it garbage-collected underneath us, and every fsync on the box went 5-10x slower for 30-40 seconds at a time. Half a day's runs carried minutes like that, and they are not the store's. Fixed on 2026-09-17: discards allowed on the mapping, a reboot so the volume above it recomputes its limits, then one `fstrim`, which discarded 1.5 TiB. With that done the 30-minute acceptance run of spec 2.11 could finally be made, and it holds: 16,164 blocks across nine files stores with maintenance every block, every minute's seal p90 between 42 and 57 ms against a 100 ms budget, the heap's fsync average flat, no mismatch, and all nine stores reopened and read every sampled key back with none wrong. The device's write ticks stayed at a median of 1.9 s per two-second window against the 100-500 s of a stall. The proposal records it, including the two minutes where the process read from the drive because the store had outgrown the page cache -- the floor for an index that does not fit in memory, and not a cost that grows with the store's age. The runbook for the disk itself moves out of the repository, at the user's direction, to the machine notes where computer maintenance is kept. Section 2.11 no longer points at a file: it states the requirement -- the measurement disk must be trimmed, and verified trimmed, before a run counts -- and says the notes for a given machine live with that machine. A spec that depends on a path outside itself is not a spec. Co-Authored-By: Claude Fable 5.1 --- cmd/bdbench/tools/README.md | 5 +- docs/SPEC.md | 7 +- .../2026-09-16-entries-written-once.md | 20 +++++- docs/runbooks/disk-trim.md | 69 ------------------- 4 files changed, 27 insertions(+), 74 deletions(-) delete mode 100644 docs/runbooks/disk-trim.md diff --git a/cmd/bdbench/tools/README.md b/cmd/bdbench/tools/README.md index e27bffd..3acbfef 100644 --- a/cmd/bdbench/tools/README.md +++ b/cmd/bdbench/tools/README.md @@ -31,5 +31,6 @@ p50 unchanged, p90 200-500 ms) and different here: a stall raises the heap's fsync average for every store at once, the device's write ticks with it, and the store's own counters (moved, snapshots, releases) show no step. A store tail lines up with a step in one of those counters -and the device stays quiet. See `docs/runbooks/disk-trim.md` for the -disk this was learned on. +and the device stays quiet. The disk this was learned on, and how it +was fixed, are in the operator's machine notes rather than here: on this +machine `~/infrastructure/disk-trim.md`. diff --git a/docs/SPEC.md b/docs/SPEC.md index 02f479b..214cdaa 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -545,8 +545,11 @@ under a sustained write load; that reads as a seal tail and is not one. A run is read with the per-minute heap split and the two-second store and device timelines (`cmd/bdbench/tools`), which tell a device stall from a store tail, and a run on a disk that stalls is compared -by its clean minutes or not at all. `docs/runbooks/disk-trim.md` -records the disk this was learned on and how it is verified. A change to the +by its clean minutes or not at all. A disk that never receives +discards will do this, so the measurement disk must be trimmed and +verified trimmed before a run counts; the operator's notes for the +machine a given run was made on are kept with that machine, not in +this repository. A change to the protocol path or to maintenance is measured here before it is measured under Accumulate, because here the seal's wait is the store's own and not the executor's. diff --git a/docs/proposals/2026-09-16-entries-written-once.md b/docs/proposals/2026-09-16-entries-written-once.md index edd9bf5..66e24ac 100644 --- a/docs/proposals/2026-09-16-entries-written-once.md +++ b/docs/proposals/2026-09-16-entries-written-once.md @@ -428,4 +428,22 @@ under load. The acceptance run must show: every shard's deltas) and the per-store data file, so that a hundred shards cost a block one barrier; one shard per store is that layout by another name and measures the same as eight now. - The 30-minute acceptance run waits on the trimmed disk. + The 30-minute acceptance run (2026-09-17, after the disk was + trimmed; nine files stores, maintenance every block, phase + offset): 16,164 blocks, every minute's seal p90 between 42 and + 57 ms against the 100 ms budget, the heap's fsync average flat + at 10-14 ms, no mismatch, and all nine stores reopened and read + every sampled key back with 0 wrong. The device's write ticks + per two-second window were p50 1.9 s, max 14.3 s: no stall. + Block p50 rose from 129 ms to ~170 ms over the first five + minutes as the window filled and then held at 170-193 ms. In + minutes 28 and 30 the process read 56 and 120 MB/s from the + drive and read p99 went from 7-8 us to 52-59 us, which at 40,000 + lookups a block put block p50 at 250-300 ms and p90 at 370-460 + ms; the seal did not move. The store was 68-74 GB on a 64 GB + box whose page cache had been full since minute 10: lookups of + permanent keys of all ages had begun to miss the cache. That + is the floor for an index that does not fit in memory, not a + cost that grows with the store's age in the store's own work, + and it is where a resident permanent index (or a bigger box) + would show. Run: accept-30m, 2026-09-17 08:00. diff --git a/docs/runbooks/disk-trim.md b/docs/runbooks/disk-trim.md deleted file mode 100644 index 2d39679..0000000 --- a/docs/runbooks/disk-trim.md +++ /dev/null @@ -1,69 +0,0 @@ -# The measurement disk must be trimmed - -Written 2026-09-17, for the machine the load platform (`cmd/bdbench`, -spec 2.11) runs on. It applies to any SSD the store is measured on. - -## Why - -An SSD does not learn that a file was deleted; the filesystem only -marks the blocks free in its own tables. Until the filesystem sends a -discard ("trim") for them, the drive treats them as live data it must -preserve, and when it runs out of pre-erased flash under a sustained -write load it erases and rewrites around them. On this machine that -showed as every fsync on the box going from 12-22 ms to 100-330 ms for -30-40 seconds at a time, with nothing in the store changing, and the -device's write ticks going from 1-3 s to 100-500 s per 2-second window -(`cmd/bdbench/tools/README.md`). Half of a day's runs carried such -minutes. They are not the store's, and they make the 30-minute -acceptance run (spec 2.11) meaningless until the disk is trimmed. - -## How this disk is laid out - - nvme0n1p3 -> cryptdata (LUKS2, dm-0) -> data-root (LVM linear, dm-1) -> ext4 / - -Discards have to pass through every layer. As found: - -- The root filesystem is mounted without `discard`, so nothing is - trimmed on delete. Fine: the weekly `fstrim.timer` is the usual - way. -- The timer's log showed it trimming only `/boot/efi`: `fstrim` skips - a filesystem whose device reports no discard support. -- `lsblk -D` showed `cryptdata` and `data-root` both with `DISC-MAX` - 0: the LUKS mapping was opened without `allow-discards`, so the - encrypted layer refused discards and the volume above it inherited - that. - -## What was done - -1. `sudo cryptsetup refresh --allow-discards --persistent cryptdata` - reloaded the encrypted mapping with discards allowed and wrote the - flag into the LUKS2 header, so every future open has it without an - `/etc/crypttab` change. After it, `/sys/block/dm-0/queue/ - discard_max_bytes` read 2 TB. -2. `data-root` (dm-1) kept `discard_max_bytes` 0. A volume computes - its limits from the device beneath it when its table is loaded. - `sudo lvchange --refresh data/root` and an explicit - `dmsetup reload ... --table "0 7803994112 linear 252:0 2048" && - dmsetup resume` both ran without error and changed nothing: this - kernel (6.17) did not recompute the limits of the mounted root - volume on a live reload. -3. Remaining: a reboot, after which the volume activates over the - discard-capable encrypted device, then `sudo fstrim -v /`, which - should report over a terabyte discarded. The weekly timer keeps it - trimmed from then on. - -## How to verify - - lsblk -D -o NAME,DISC-GRAN,DISC-MAX # every layer non-zero - cat /sys/block/dm-1/queue/discard_max_bytes # non-zero - sudo fstrim -v / # reports bytes trimmed - journalctl -u fstrim.service # the timer trims / - -Then run the platform with the samplers and confirm the device's write -ticks stay at 1-3 s per window through a five-minute run. - -## The trade-off - -With discards on, someone holding the raw disk can tell which blocks -of the encrypted volume are unused. For a development machine that is -acceptable; it is the reason distributions leave it off by default.