Skip to content

Raise opt-in Raft image limit to 256 MiB and verify recovery - #119

Merged
forhappy merged 2 commits into
mainfrom
codex/large-cell-recovery
Oct 25, 2026
Merged

forhappy merged 2 commits into
mainfrom
codex/large-cell-recovery

Conversation

@forhappy

@forhappy forhappy commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Raft execution, committed-image markers and retained-snapshot recovery previously imposed independent 64 MiB limits. Larger configurations could not execute, seed or reopen the same physical state consistently.

Add RaftCellsConfig::with_materialized_image_limit(bytes) to configure logical images up to 256 MiB while retaining the 64 MiB defaults. The profile coordinates decoder/apply bounds, worst-case snapshot encoding, archive objects/sets, generation disk admission, archive verification scratch space and transport working-memory reservations. Archive manifests enforce a separate logical image bound before downloading compressed bodies; retained-snapshot recovery uses the destination store’s configured image policy.

let config = config.with_materialized_image_limit(256 * 1024 * 1024)?;

The new native drill builds 72 MiB of SQLite payload from ordinary 1 MiB values, removes the leader, checks the exact retry receipt/version, seeds an empty authorized learner from a peer or object-store archive, replicates another write, and restarts the seeded worker before its peers start. An explicit ignored variant runs against S3-compatible storage in an isolated random prefix; it passed against the running local RustFS instance.

plans/multi-raft/large-cell-recovery.md records a concrete design for both fast warm takeover and cold activation at the 10 GB (10,000,000,000-byte) product target: immutable extent roots and copy-on-write writers, bounded streamed transfers, verified demand loading, retention and qualification gates. The product target matches Cloudflare’s SQLite-backed Durable Objects Paid quota. The plan separates that decimal logical quota from physical replica/retention budgets and 10 GiB engine stress tests, with page-rounded enforcement and cleanup/receipt headroom. Those storage changes are proposed work; this PR does not implement or qualify 10 GB recovery. Physical large-image drills cover 72 MiB under a 128 MiB policy; configuration/worst-case bounds cover the 256 MiB ceiling. Large-image drills stop services; existing snapshot/execution/LTX suites retain their process-kill matrices at ordinary fixture sizes.

Validation: 739 passing tests across the library, native archive/execution/host/operations/snapshot/LTX suites, both large-image drills and the explicit RustFS run. Existing child-process fixtures remain ignored as standalone tests and are exercised by their owning matrices. Also passed Clippy for cellar/cellar-ltx and the new test with -D warnings, cargo format --check, cargo check-layout, native dependency checks and documentation checks.

Documentation follow-up: aligned the product quota with Cloudflare’s published limit and units; documentation checks and git diff --check passed. Runtime code and its supported 256 MiB opt-in ceiling are unchanged.

Coordinate execution, archive, snapshot, transport and recovery bounds without increasing default admissions. Verify warm takeover and fresh peer/RustFS recovery above 64 MiB.

Document the incremental storage work required for fast 10 GiB recovery.
Define the planned per-cell quota as 10 GB in decimal bytes. Separate product quota enforcement from engine stress sizes and the current materialized ceiling.
@forhappy
forhappy merged commit 7b38f67 into main Oct 25, 2026
14 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant