Skip to content

Proposal: entries are written once; keys are managed separately - #96

Draft
PaulSnow wants to merge 1 commit into
load-platformfrom
entries-written-once
Draft

PaulSnow wants to merge 1 commit into
load-platformfrom
entries-written-once

Conversation

@PaulSnow

Copy link
Copy Markdown
Contributor

Design proposal (draft PR for review; stacked on #95 whose platform run is its evidence). On acceptance it becomes spec text in 1.5, 1.8, 2.3 and 2.7, and #94 leaves the register.

docs/proposals/2026-09-16-entries-written-once.md:

  • The rule: an entry is written once; reclaiming space and bounding the walk are done on keys, never by rewriting entries. Entries move only to make a bigger hole, one per bounded step.
  • Dynamic layer: a heap with holes. A rewrite that fits reuses its slot; one that doesn't appends and frees a hole; holes by size class; a hole is reusable one seal late; entries carry length and checksum so a torn slot is a hole, not a wrong answer.
  • Permanent layer: append-only data file per shard; each block seals an index delta; merge and pack fold indexes (~40 B a record), never entries.
  • Seal: one commit point per store per block (parallel data sync, one block file of index deltas, the manifest) instead of four barriers per shard. Closes A seal costs six fsyncs and rewrites the whole manifest; imports pay the same per segment #33.
  • Acceptance: cmd/bdbench -stores 9 flat for 30 minutes: seal p90 under 100 ms, every block inside the interval, maintenance bytes near ingest, zero wrong answers. Today that run reaches seal p90 731 ms and misses 46 of 540 blocks by minute 3.

🤖 Generated with Claude Code

https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5

The store rewrites entries to manage keys, and the rewrites are what
the commit path waits for: on the platform, nine stores on one NVMe
put 518 of 519 seal goroutine samples in fsync by minute 4, behind
360 seal barriers a second and maintenance rewriting write-once data
three to four times.  The proposal: a heap with holes for the
dynamic layer (a rewrite that fits reuses its slot, a hole is reused
one seal late, one bounded move per pass), append-only data with
merged index deltas for the permanent layer, and one commit point per
store per block.  Acceptance is the nine-store platform run flat.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015wUZmWRdAPtcfFgYrLJCc5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant