[forhappy]

Cell images previously hit a 64 MiB implementation ceiling and whole-image memory/transfer paths. This change admits an opt-in 10,000,000,000-byte logical SQLite quota, keeps the 64 MiB default and independent command/value/response/concurrency bounds, and moves large physical snapshots to bounded file I/O.

Production 10 GB recovery is still unqualified. The materialized native path and the incremental extent data components have separate validation scopes below. This follows merged #119 and includes origin/main at 42be6037 (development WAL-index retention), following typed RPC hardening and the rusqlite 0.40.2 upgrade.

Implementation

• File-backed v0.5.2 snapshots use bounded I/O, a disk-spooled page index, checked u64 offsets and canonical zero bytes for SQLite’s reserved lock page. Default codec buffers, source callbacks and decoded destination writes remain 64 KiB. Native archives explicitly select 2 MiB codec buffers for actual images of at least 256 MiB, within their existing physical memory reservation. Caller-owned transport windows reach 4 MiB without another complete native read allocation; existing providers retain 64 KiB fallback operations. Both profiles preserve exact encoded bytes and all integrity/sync gates. Archives use authenticated indexes and immutable 4 MiB parts, conditional creation and exact read-back confirmation.
• Source, receive and verification spools share explicit disk/file admission. Physical owners retain charges through clones, failed writes, cancellation and joined shutdown. Large logical snapshots request optional streaming-cache advice, including compressed images; macOS uses F_NOCACHE, unsupported providers retain ordinary bounded I/O, and other advice errors abort cleanly.
• SQL owners freeze exact cuts/pins before encoding on owned blocking tasks. Late completion cannot release a newer SQL operation. Generation preparation uses exclusive COW clones where supported and bounded copying elsewhere; independent destination verification, file/directory sync and exact installation/epoch/membership/lineage gates remain required.
• Receipt/control headroom stays inside the quota: one sixteenth, capped at 32 MiB. Application growth returns StorageFull; receipt exhaustion reports overload. Cleanup, page reuse and retained-receipt retries preserve the output proof boundaries.
• Committed-image observation binds the exact live WAL-index cut, both matching/checksummed headers, geometry, salts and final-frame checksum. A short VFS lock protects the 96-byte metadata sample. Corrupt/incomplete committed frames cannot select an older same-size transaction. Readiness preserves coherent images across protocol-only applied entries.
• Fresh archive publication retains its confirmed source, checks the actual uploaded stream against its encoded SHA-256, and avoids downloading its own parts again. An identical complete image can rebind its validated index to a newer exact configuration cut without uploading image parts. Prior closure checks, independent destination replay/sealing and head CAS remain required.
• Snapshot part publication now overlaps a bounded two-part conditional PUT/read-back window. Source hashing and manifest order remain sequential; each owned buffer moves into its upload. Failed windows drain entered confirmations before releasing their source. Independent destination replay and exact head CAS/confirmation remain mandatory. Coarse archive-phase logs distinguish encoding, upload confirmation, material reads, replay and head confirmation.
• Native publication now uses a bounded conditional scheduling hint for actual images of at least 64 MiB. Each publisher binds its attempt to node/disk/process; exact CAS and full read-back reconcile lost replies and late releases. Peers age unchanged versions on local monotonic clocks. Renewal requires completed data I/O, so a live but stalled publisher cannot keep peers waiting forever. Entered renewal work joins before return. Invalid/unavailable hints leave ordinary publication available. Covering heads still require each replica's independent material reads, replay and sealing; hints grant no ownership or durability proof.
• Incremental data components add typed, domain-separated extent roots, independently verified extent files, committed SQLite/WAL observations and actual SQLite demand reads through the shared immutable VFS. A bounded content-addressed cache evicts outside its mutex; source I/O, verification and sync hold no cache lock. Downloads and live readers retain their physical charges.
• A persistent NodeExtentStore now shares one exclusively locked inventory and budget across groups, charges closed-root data, inventories names/lengths without body scans, and distinguishes exact metadata views from independently verified local closure seals. Complete authenticated-reference reconciliation precedes GC; pinned views, monotonic references and closed retained roots protect shared objects. Synced deletion precedes refunds; ambiguous creation, failed cleanup and interrupted mutations quarantine admission. Native execution/publication/recovery do not select these extent roots yet.

Internal cutover: Cellar-owned markers remain 1; native snapshot objects now contain the authenticated physical index. No legacy reader/migration is added. ExecutionLimits gains receipt_reserve_bytes. Extent roots remain distinct from flat SHA-256 and are exposed through the unstable verification API.

Validation

• Latest main merge (825fc5e7, main 42be6037): resolved the sole conflict in development-store API documentation, preserving both the idle WAL-index connection explanation and offline/online backup guidance. Main's implementation and regression fixtures are included. All nine native local-store/cell-limit cases pass, including WAL recycling across cloned handles and reopen/CAS identity. Clippy for the host library and both affected test targets with warnings denied passes, alongside formatting/layout (1,067 Rust files), dependency and documentation checks. GitHub reports this head mergeable; CI for this head remains a separate gate.

• Merge with origin/main (0c23ae0b): resolved development-store documentation/fixture conflicts and retained quota accounting plus bounded WAL image observation while adapting to rusqlite 0.40.2. The LTX suite, formatting/layout, documentation, native dependency checks and 19 planning-tool tests pass locally. 818 selected native cases pass (620 host library, three local-store, 68 archive, 21 execution, four ordinary large-image, 27 native LTX, nine multigroup, 22 read and 44 snapshot). Eleven child/full-size entries remain ignored by the parent invocations. Native/LTX all-target Clippy with warnings denied also passes. New-head GitHub CI remains pending. The preceding SDK run failed a stale whole-image receive-budget assertion in repair_budget_under_load; the fixture now checks the configured archive reservation plus bounded streaming memory and tightens the source ceiling from 384 MiB to 8 MiB. Repair concurrency and outcome assertions remain intact.

• Bounded archive I/O follow-up (d5f12630): all 858 ordinary cases pass: 605 host library, 68 archive, four native large-image, 27 native LTX, 22 native read, 44 snapshot and 88 LTX unit/integration cases. Eight native and two LTX qualification/child entries remain ignored by parent invocations. Exact byte/metadata comparisons cover both codec profiles, native and legacy file providers, three SQLite page sizes and incompressible overflow pages across 2/4 MiB boundaries. Fault checks preserve uncertain-write poisoning, physical scratch ownership, corruption refusal and independent destination verification. Real compressed SQLite images above 256 MiB preserve identity and integrity across cache policies/profiles. Native sparse reads and buffered index varints cross 4 GiB offsets; truncated, noncanonical and overflowing varints refuse. Rust 1.98.1 library/test Clippy (plus final LTX all-target Clippy), no-default-feature LTX all-target checks, native-only/no-Python libraries, formatting/layout (1,059 files), native dependencies and documentation checks pass. The source-bound d5f12630 release passes its 12 MiB preflight (9.39 s, 47,595,520-byte maximum RSS) and its explicitly invoked actual 272 MiB incompressible RustFS recovery/replacement-write/restart preflight (48.54 s, 89,866,240-byte maximum RSS). The native archive selects Buffered at its real 285,212,672-byte geometry. Read-only backend inspection finds 142 objects / 505,543,090 stored bytes and independently authenticates the selected head at 9/4. The subsequent 9,999,998,976-byte drill under frozen d5f12630 failed with VersionUnavailable after 4,080.31 seconds, with 91,586,560-byte maximum RSS. Warm takeover, owner-state erasure, retained-receipt retry and quota cleanup/reuse passed. It confirmed the required archive head at 9/4 after a 931.957-second wait, and an empty replacement served a historical local read after 912.264 seconds. Replacement first-write and restart proof were not reached; coarse logs do not locate the intervening refusal. The fixture now adds operation-specific error context and stage observations without changing requests, deadlines or assertions. Read-only process disk-I/O counters supplement phase/RSS observations; no debugger or private filesystem was substituted. This remains a failed full-size qualification. Documentation follow-up 5e0ca45a records the preflight and reproducible command without changing that frozen executable. These ordinary tests do not qualify production 10 GB recovery or an RTO.

• Component layout follow-up (eee82472): DB lifecycle/capture, committed image/extent observations and WAL inspection now live under crates/cellar-ltx/src/db/, with the entry point in db/mod.rs. Extent identities, files, cached images and the persistent inventory live under src/extent/ and extent/store/. Public Db, db and internal verification paths and integration-test targets are preserved; implementation algorithms are unchanged. All 82 LTX unit/integration cases pass (two child/full-size entries remain ignored by parent invocations), alongside LTX all-target Clippy, no-default-feature all-target checks, native-only/no-Python host/Python libraries, formatting/layout (1,057 files), dependency graph and documentation checks. All 98 Python tooling cases pass, and the standalone package validator passes. Source inventories/links follow the moves; baseline locators retain the original revision and hashes. The live full-size drill remains bound to the preceding frozen bbdc6b2a binary, separately from this layout validation.

• Publication scheduling follow-up (bbdc6b2a): all 848 ordinary cases pass: 605 host library, 68 archive, four large-image, 27 native LTX, 22 native read, 44 snapshot and 78 LTX integration cases. Seven native and two LTX child/full-size entries remain ignored by parent invocations. Six scheduling cases cover concurrent publishers, lost replies, active progress, abandoned work, a live stalled publisher, late release and corrupt hints; independent destination-sync refusal still exposes no new head. Completed-I/O observations do not change bytes, verification or sync requirements. An intermittent WAL spill fixture now pins its committed reader and flushes dirty uncommitted pages, preserving all original assertions; it passes 20 additional fresh processes. Native/LTX all-target Rust 1.98.1 Clippy, native-only/no-Python libraries, formatting/layout (1,056 files), dependency graph (465 packages) and documentation (750 references) checks pass. The source-bound release build passes its 12 MiB native recovery/write/restart preflight (9.48 s wall, 46,989,312-byte maximum RSS) and the explicitly invoked above-64-MiB RustFS/native takeover/empty-worker seed drill (13.70 s wall, 166,707,200-byte maximum RSS). Read-only inspection of its fresh disposable RustFS prefix finds a valid 327-byte publication record at sequence 8, attempt 3, released=true, confirming native conditional scheduling against the real backend. These small/compressible drills do not qualify full-size recovery or an RTO. The actual 9,999,998,976-byte rerun under frozen bbdc6b2a failed its unchanged deadline after 2,567.14 seconds, with 59,375,616-byte maximum RSS. Owner-state erasure, warm takeover/read/write, receipt retry and quota cleanup/reuse passed. Repair admitted index 9 in 0.894 seconds, but the subsequent 1,800-second archive wait retained selected head 2/2. One index-2 encoder and one index-9 encoder completed their uploads; index 9 reached neither independent replay completion nor confirmed head selection before failure. A read-only RustFS inventory found 4,813 objects / 20,116,303,274 stored bytes and independently authenticated the selected 327-byte head at 2/2. Uploaded objects do not establish recoverability. No cold restore, replacement write or restart was reached. Concurrent validation load and cache state were not controlled, so these timings are not a controlled performance comparison or an RTO qualification.

• 873 ordinary cases pass at baseline eb459388: 603 host library, 60 archive, 21 execution, 32 native host, four ordinary large-image, 44 snapshot, 27 native LTX, 22 native read and 60 LTX integration cases. Ten child/qualification entries remain ignored by their parent invocations.

• Post-seal integrity follow-up (ddc604c4): all 62 LTX integration cases pass, including a reproduced post-seal read-corruption regression and actual SQLite cached-read refusal. Each extent retains at most 2 KiB of block digests derived during independent destination verification; every later read checks its containing blocks before returning any output, including a faulty second block. LTX all-target Clippy and formatting/layout/docs checks pass. The 813 native cases above remain prior-revision evidence; no native extent installation or full-size extent restore is claimed.

• Native publication follow-up (29817781): all 110 ordinary cases pass: 62 archive, 44 snapshot and four large-image cases; five child/qualification entries remain ignored by parent invocations. A gate reproduces serial publication before the fix and verifies two exact confirmations after it. Failure while the second confirmation is held retains scratch and waits for entered work, then exposes no head and supports exact retry. Existing source-corruption, missing confirmation, independent destination sync refusal, peer/S3 seeding and crash matrices pass. Native all-target Rust 1.98.1 Clippy, native-only/no-Python libraries, formatting/layout (1,052 files), dependency/docs checks and release library build pass. These ordinary results do not qualify full-size recovery or an RTO. The source-bound release fixture passes its 12 MiB recovery/replacement-write/restart/receipt preflight in 9.77 seconds, with 47,382,528-byte maximum RSS. The actual 9,999,998,976-byte RustFS attempt under frozen 29817781 failed its unchanged deadline after 3,128.52 seconds, with 78,954,496-byte maximum RSS. Repair admitted required index 9 in 0.130 seconds, but the subsequent 1,800-second archive wait expired with selected head 2/2. No cold restore, replacement write or restart was reached. Coarse phases show three overlapping encoders for the same index-2 cut and two completed uploads/replays before retained-material reads. This establishes duplicate cross-replica work; it does not establish a lower-level filesystem bottleneck. This attempt used an isolated RustFS instance on the workspace volume, unlike the preceding store, so timings are not a controlled comparison.

• Persistent-owner follow-up (99bb6018): the checked scope at that revision passes 126 ordinary cases: 77 LTX integration cases and 49 native LTX/read cases. Two LTX child/full-size entries and two native child entries remain ignored by parent invocations. The final owner invocation adds a source-panic case to the preceding all-integration pass. Fifteen owner cases include real SQLite restart/demand reads without startup body scans, missing local closures, pre-I/O quota refusal, monotonic retention, shared objects, uncertain creation, independent destination corruption, file/directory sync failures, conservative cleanup accounting and joined work. A child exits abruptly at root-file sync before directory sync; reopening keeps exact old-root selection. This small component cut does not qualify native marker atomicity, power loss or full-size recovery. LTX/native all-target Clippy, native-only/no-Python libraries, formatting/layout (1,052 Rust files), dependency and documentation checks pass. The remaining 764 native baseline cases are prior-revision evidence.

• Extent tests include an independent hash vector, canonical manifest corruption, quota/4 GiB geometry, actual SQLite/WAL observations and physical file faults. Actual SQLite hot queries fetch one 4 MiB extent from both 8 and 40 MiB payloads; broad reads, cross-extent reads, integrity checking and eviction/refetch work with a one-extent cache. Missing/corrupt/incomplete data, source panic, destination corruption, admission pressure and held/cancelled work expose no partial data.

• Cache-policy tests round-trip real >256 MiB compressed SQLite images across accepted, unsupported and native hints with byte-identical encoded identity, independently hashed restored data and SQL/integrity checks. Archive, committed-WAL and readiness regressions fail before their respective fixes. Existing corruption, lost-response, replay, batching and process-kill suites remain intact.

• Rust 1.98.1 native/LTX all-target Clippy with warnings denied, native-only/no-Python libraries, formatting/layout (1,047 files), native dependency and documentation checks pass. New-head GitHub CI is queued/pending and remains a separate gate.

• The source-bound production release fixture at 3573f8e3 passes its 12 MiB recovery/promotion/new-write/restart/receipt preflight in 10.11 seconds, with 45,793,280 bytes maximum RSS. The added repair/admission and archive-head observations retain the same requests, checks and deadlines.

• Earlier physical-codec qualification passed at 9,999,998,976 SQLite bytes and 10,083,173,257 encoded bytes, including lock-page/4 GiB crossings, quota refusal, cleanup/reuse, payload hashes and SQLite integrity. The earlier native e46b0577 baseline passed at that logical size, but did not erase the old disk or promote/write on the replacement.

• The controlled 3c3e68e5 full-size archive diagnostic failed a deadline after 2,875.96 seconds. It passed owner-state erasure, warm takeover/read/write, receipt retries and quota cleanup/reuse, but reached no cold restore/restart. Retained data contains two full snapshots: the selected head remains index/term 2/2, while a newer 9/4 manifest is unselected. Uploaded objects alone do not prove a recovery head.

• The production-filesystem full-size RustFS run under frozen 3573f8e3 (runtime code ce65404e) failed its unchanged deadline after 2,990.24 seconds, with 77,807,616-byte maximum RSS and no debugger/private filesystem substitution. First read/write took 4.549/174.868 s and warm first read/write 7.818/179.113 s. Owner-state erasure, retained-receipt retry and quota cleanup/reuse passed. Repair admission reached required index 9 in 0.127 s; the subsequent 1,800-second archive wait expired with the selected head at 2/2. No archive-ready, cold read, replacement promotion/write or restart was reached. This isolates the pending phase to archive coverage, without establishing a more specific root cause. This frozen run predates the later extent components and is not a production RTO qualification.

CI follow-up

The direct filesystem lock guard now explicitly unlocks from its creating process before closing the descriptor. Closing alone could leave a finished owner's lock held by a forked child's inherited descriptor until exec/exit, refusing immediate node reopen. A deterministic Unix regression keeps that child alive, fails before the fix and passes after it; the probe is joined on assertion failure and exits if its parent dies. An inherited guard cannot unlock a live parent's lock. On PR head 4bd8c123, the lock regression and all 21 native execution cases pass, along with LTX all-target Clippy (and a final probe-target check), formatting/layout (1,068 Rust files) and documentation checks. New-head CI remains a separate gate.

The reported constant-chunk lint uses fixed array chunks. Development-store reads are read-only/non-creating and no longer checkpoint unrelated WAL on close; normal writers retain FULL commits and automatic checkpoints, with close-time checkpoints disabled after the merge from main. The reported event-owner test passed its 93-case suite and 20 additional local processes without changing lease deadlines.

At preceding 29817781, the JavaScript and Node SDK workflow passed. The native SDK matrix was still running when this follow-up was pushed; new-head GitHub CI remains a separate gate. A different earlier Linux membership-repair timeout remains unresolved locally despite 20 fresh passing repetitions and the full membership suite; timeout observations remain without extending deadlines.

Remaining work

Full-size peer/RustFS recovery after local-state loss, replacement writes/restart, interruption and Linux COW qualification remain open. Full-size fixtures cancel/join execution domains before erasing the old disk; abrupt process/power loss at that size is separate qualification.

The persistent extent owner still needs native marker/reference wiring. Authenticated archive/peer placement, cold writer bases, incremental checkpoint continuity and native installation/publication remain unimplemented. The materialized path still performs full-image scans; the measured first-write work remains substantial. No fast-recovery or flawless-production claim is made.

See the implementation checklist, extent contract and large-cell recovery design.