diff --git a/README.md b/README.md index 3ff0f19..c931caa 100644 --- a/README.md +++ b/README.md @@ -203,7 +203,7 @@ cargo test --locked --test server_binary -- --include-ignored --test-threads=1 See [contributor invariants](AGENTS.md), [server qualification boundaries](docs/cellar-server.md#qualification-still-required) and -[independent protocol qualification](docs/implementation-status.md#independent-client-qualification). +[independent protocol runner](scripts/qualify-upstream.py). ## Documentation diff --git a/diagram/operations/gsi-ordered-ranges.svg b/diagram/operations/gsi-ordered-ranges.svg new file mode 100644 index 0000000..bfa9343 --- /dev/null +++ b/diagram/operations/gsi-ordered-ranges.svg @@ -0,0 +1,54 @@ + +A table and its GSI use separate ordered ranges +Table metadata refers to independent base and GSI directories. Base ranges order the complete base primary key. GSI ranges order the GSI partition key and sort key followed by the complete base key. The base-key suffix keeps duplicate GSI keys distinct. The displayed three-range plans are illustrative, not a fixed range count or a mapping of one table to one Cell. Cellar owns physical placement and fenced ownership. + + + + + + + + + + + + +A table and its GSI use separate ordered ranges +Separate directories, range identities and owners; one table spans multiple Cells. + + + + + +Table generation + index specification +Account metadata anchors each independent directory. + +BASE DIRECTORY +Order: base PK, then base SK when present + +GSI DIRECTORY +Order: GSI PK/SK, then complete base key + +Base Cell A [−∞, b1) + +Base Cell B [b1, b2) + +Base Cell C [b2, +∞) + +GSI Cell X [−∞, g1) + +GSI Cell Y [g1, g2) + +GSI Cell Z [g2, +∞) +Duplicate GSI keys remain separate rows because the full base key breaks ties. +b1/b2 and g1/g2 are unrelated ordered boundaries; the three-range plans are illustrative. +Cellar assigns physical owners. Several Cells on one node still share that node’s failure domain. + diff --git a/diagram/operations/gsi-split-lifecycle.svg b/diagram/operations/gsi-split-lifecycle.svg new file mode 100644 index 0000000..3491b99 --- /dev/null +++ b/diagram/operations/gsi-split-lifecycle.svg @@ -0,0 +1,73 @@ + +GSI split: verify before publication, open before finish +The controller durably reserves one exact directory transfer plan. Children enter importing and the source seals. Export copies full keys, original projection versions, images and tombstones. Both children activate only after their complete fingerprints match, while still blocking serving traffic. The directory publishes replacement routes before the controller opens both children. Only after verifying both opened states does it finish the plan and release reservations. An interrupted plan stays discoverable across publication. Base journals retain work while GSI routes are stale. + + + + + + + + + + + + +GSI split: verify before publication, open before finish +The source version survives transfer; directory cutover does not invent a new projected value. + + + + + + + + +1 +Persist the exact directory plan +Reserve the source and both child identities before transfer. + + +2 +Prepare children; seal the source +Children enter IMPORTING; sealed source rejects serving traffic. + + +3 +Copy full transfer records +Keep keys, original versions, images and deletion tombstones. + + +4 +Verify fingerprints; activate children +Both imports must match. ACTIVATED still blocks client traffic. + + +5 +Publish replacement directory routes +Keep the plan discoverable through the routed children. + + +6 +Open and verify both children +Controller observes publication before opening either child. + + +7 +Finish plan; release split reservations +Only after both OPENED states are verified. +PLAN +COPY +CUT OVER + +During the pause, stale-route responses keep base journals available for retry. +Finishing a split does not authorize deletion of historical storage or tombstones. + diff --git a/diagram/operations/gsi-version-replay.svg b/diagram/operations/gsi-version-replay.svg new file mode 100644 index 0000000..bfed88f --- /dev/null +++ b/diagram/operations/gsi-version-replay.svg @@ -0,0 +1,57 @@ + +Replay compares source versions, not arrival order +For one complete index and base key, compare lexicographic source epoch and source sequence. If stored version is (7,42) with a tombstone, incoming old (7,41) image is superseded; identical (7,42) tombstone replays; different content at (7,42) is a version conflict; (8,1) is newer and applies if valid. Tombstones prevent old images from resurrecting removed entries. Scheduling deferral never changes the immutable projection version. Import uses stricter exact-replay rules. + + + + + + + + + + + + +Replay compares source versions, not arrival order +Version = (source epoch, source sequence), compared lexicographically for one full key. + +Stored row: version (7, 42), tombstone +A tombstone retains the deletion version even though Query and Scan omit the row. +INCOMING DELIVERY +RESULT AT THIS KEY + + +(7, 41) · old projected image + +Superseded +Keep the newer tombstone. + + +(7, 42) · identical tombstone + +Replay +Same version and content are idempotent. + + +(7, 42) · different content + +VersionConflict +Reject; equal version cannot change content. + + +(8, 1) · valid newer image + +Applied +A higher source epoch wins before sequence. +Deferral changes retry scheduling, not this version. Full tombstone retirement remains unqualified. +Split import requires exact replay and rejects conflicting versions instead of replacing them. + diff --git a/diagram/operations/ordered-key-encoding.svg b/diagram/operations/ordered-key-encoding.svg new file mode 100644 index 0000000..dd8cab1 --- /dev/null +++ b/diagram/operations/ordered-key-encoding.svg @@ -0,0 +1,52 @@ + +Ordered key bytes preserve logical key order +Within a table generation, encode the partition key followed by its sort key when present. Each component has a schema type tag, an order-preserving escaped payload and a two-byte terminator. Strings use UTF-8 byte order, binary uses unsigned byte order, and numbers use exact normalized decimal order. Embedded zero escapes as 00 ff and a complete component ends with 00 00. Table generation and tenant scope are outside user-key comparison. A GSI appends the complete ordered base key to break ties. + + + + + + + + + + + + +Ordered key bytes preserve logical key order +One encoding feeds directory routing, Cell ownership checks and local ordered storage. + +Logical primary key: PK + optional SK +Schema types determine byte, string or exact decimal comparison. + + + + + +Partition-key component +type tag | escaped ordered payload | 00 00 +Embedded 00 → 00 ff; full component ends with 00 00. + +Sort-key component, when present +type tag | escaped ordered payload | 00 00 +Embedded 00 → 00 ff; full component ends with 00 00. + + + +Base key = encoded PK || encoded SK +PK-only tables use encoded PK. Tenant and table generation scope the directory. +String and binary order +Unsigned bytes; strings are UTF-8. +Number order +Exact normalized decimal; no floating-point conversion. +GSI ordered key = encoded index PK/SK || complete encoded base key. +Wire primary keys remain DynamoDB AttributeValue objects; clients do not send internal routing bytes. + diff --git a/diagram/operations/ordered-pagination-split.svg b/diagram/operations/ordered-pagination-split.svg new file mode 100644 index 0000000..fec0958 --- /dev/null +++ b/diagram/operations/ordered-pagination-split.svg @@ -0,0 +1,51 @@ + +Resume from the logical key after a split +Page one stops after sort key 12 for customer-42. Its LastEvaluatedKey contains logical PK/SK attributes, not a physical Cell identity. Between requests the range [10,20) splits at observed key 15 into two new children [10,15) and [15,20). The next request uses the same logical ExclusiveStartKey and current directory, visiting greater keys in the left and then right child. This illustrates forward resume; reverse resumes exclusively in the opposite direction. It is not a whole-table snapshot or a complete split-during-page qualification. Ambiguous mutations resolve through their original identity instead of being blindly resent to children. + + + + + + + + + + + + +Resume from the logical key after a split +Continuation identifies the last evaluated user key; physical range identities can change. + +Page 1 stops at (customer-42, 12) +LastEvaluatedKey = logical PK/SK attributes + + +Old Cell B: [10, 20) +Between requests, split at observed SK = 15. + + + + + +New left child: [10, 15) +Resume strictly after SK 12: e.g. 13, 14. + +New right child: [15, 20) +Continue with remaining keys: e.g. 15, 16, … + + + +Page 2: same ExclusiveStartKey, current directory +No client-side Cell-ID translation; remaining logical keys are routed afresh. +Read continuation does not make an uncertain mutation safe to submit to a new child. +Resolve a submitted write through its original Cell/identity; retain coordinator participant authority. +Illustrative resume between requests. Concurrent writes do not provide a whole-table snapshot. + diff --git a/diagram/operations/ordered-query-span.svg b/diagram/operations/ordered-query-span.svg new file mode 100644 index 0000000..5f674f4 --- /dev/null +++ b/diagram/operations/ordered-query-span.svg @@ -0,0 +1,55 @@ + +One Query can cross multiple Cells of the same PK +An illustrative populated directory has Cell A before (customer-42,10), Cell B from (customer-42,10) to (customer-42,20), Cell C from (customer-42,20) to the start of customer-90, and Cell D beyond that. A Query for customer-42 with numeric sort key BETWEEN 12 and 25 intersects B and C. Forward traversal visits B then C, reverse visits C then B. Only matching logical keys are returned, clipped to the requested span. Boundary widths do not represent data size or load, and the layout is not a bootstrap promise. + + + + + + + + + + + + +One Query can cross multiple Cells of the same PK +Illustrative populated layout without LSIs; each range is half-open [lower, upper). + +PK = customer-42 AND numeric SK BETWEEN 12 AND 25 +Compile one logical span, then visit only intersecting ranges. + + + + + +Cell A +Earlier logical keys +Outside this Query + +Cell B +10 ≤ SK < 20 +Read only 12 ≤ SK < 20 + +Cell C +From (customer-42, 20) +Read only 20 ≤ SK ≤ 25 + +Cell D +customer-90 onward +Outside this Query +Cell C can also hold later logical keys; the Query remains clipped to customer-42. + +Forward: B → C | Reverse: C → B +One request limit and evaluation budget across every visited Cell. +A PK equality condition does not imply one Cell or one local transaction. +These schematic intervals are not byte/load proportions or workload-derived bootstrap boundaries. + diff --git a/diagram/operations/scaling-capacity.svg b/diagram/operations/scaling-capacity.svg new file mode 100644 index 0000000..7d0447f --- /dev/null +++ b/diagram/operations/scaling-capacity.svg @@ -0,0 +1,53 @@ + +Capacity has three separate budgets +Per-Cell database and capture limits, per-node resources and resident Cell admission, and shared account, directory, coordinator and object-store capacity must all be considered. Metadata and coordinator Cells count against the node residency limit. Raising one limit does not qualify aggregate capacity. + + + + + + + + + + + + +Capacity has three separate budgets +Count retained work, maintenance and recovery alongside user items. + +DATA / GSI CELL +Bound one range +Items and local indexes +Transaction state and receipts +Projection / stream work +512 MiB database budget +64 MiB capture budget + +PER NODE +Share host resources +CPU, memory and local disk +Resident data + system Cells +Queues and durability work +Restore and split scratch +max_active_cells: default 128 + +SHARED PATHS +Measure contention +Account and table metadata +Directory and route lookups +Transaction coordinators +GSI projection backlogs +Object-store requests + +Plan headroom for owner recovery, range copies and pending durable work. +A larger resident-Cell limit is not a throughput or memory guarantee. + diff --git a/diagram/operations/scaling-move-split.svg b/diagram/operations/scaling-move-split.svg new file mode 100644 index 0000000..24090dc --- /dev/null +++ b/diagram/operations/scaling-move-split.svg @@ -0,0 +1,57 @@ + +Moving a Cell and splitting a range +Moving a logical Cell keeps its identity and ordered interval while native Cellar changes its fenced physical owner. A range split replaces one source with two child identities and intervals and publishes a new application route after verification. Placement is illustrative; this is not a live-copy or zero-downtime claim. + + + + + + + + + + + + +Moving a Cell and splitting a range +Logical ownership and physical placement change through different protocols. + +MOVE +Same Cell ID +Same interval +Native owner fence + + +Node 1 owns Cell A +Interval [a, z) + +Node 2 owns Cell A +Interval [a, z) +The application directory still names the same Cell and range. + +SPLIT +New child IDs +New range epochs +Replacement route + + + +Source Cell A +Interval [a, z) + +Child Cell B +Interval [a, m) + +Child Cell C +Interval [m, z) +Verify copies before publishing the children; then open them. +Illustrative boundaries. Source sealing can interrupt access; merges remain design work. + diff --git a/diagram/operations/server-boundaries.svg b/diagram/operations/server-boundaries.svg new file mode 100644 index 0000000..4f2afc7 --- /dev/null +++ b/diagram/operations/server-boundaries.svg @@ -0,0 +1,60 @@ + +Server boundaries: public, private, persistent +AWS clients use the public HTTP or HTTPS endpoint through ExtendDB. BeyondDB owns routing and account maintenance. Cellar owns Cell execution, placement, ownership and authenticated private peer traffic. Each node retains local state; fleet mode additionally uses an S3-compatible conditional object store. Peer authentication does not encrypt transport; the credential encryption key protects credential secrets only. + + + + + + + + + + + + +Server boundaries: public, private, persistent +Each node runs one beyonddb process; every table spans ordered range Cells. + +ONE NODE · UNIQUE IDENTITY + LOCAL DIRECTORY + + + + + + + +AWS SDK / CLI +Signed DynamoDB calls + +Public listener → ExtendDB +HTTP on loopback; HTTPS for external serving + +BeyondDB routing + maintenance +Ordered ranges, transactions, indexes, TTL and retention + +Fleet peers +Private authenticated RPC + +Cellar execution + ownership +Placement, fencing, SQLite, LTX and durability + +Local data directory +Retain on every node + +Conditional object store +Fleet mode only +Separate controls +Public TLS +Shared peer key +Credential-encryption key +Peer authentication is not transport encryption; protect the private network externally. + diff --git a/diagram/operations/server-durability.svg b/diagram/operations/server-durability.svg new file mode 100644 index 0000000..cfe2c4f --- /dev/null +++ b/diagram/operations/server-durability.svg @@ -0,0 +1,65 @@ + +Choose the acknowledgement boundary +Without fleet configuration, acknowledgement relies on local Cellar persistence and does not cover machine loss. Fleet mode defaults to bucket publication. With follower durability enabled, an eligible native follower fsync proof may race bucket publication. Every mode retains local data. Follower mode does not establish independent-host follower-loss tolerance. Preserve shutdown and handoff are separate choices. + + + + + + + + + + + + +Choose the acknowledgement boundary +Storage mode selects the durability proof. Shutdown mode is a separate choice. + +LOCAL +No fleet object + +Cell command commits +SQLite + LTX + + +Local persistence +Proof → acknowledge +Retain the node directory. +Machine loss is not covered. + +FLEET · DEFAULT +follower_durability: false + +Cell command commits +SQLite + LTX + + +Bucket publication +Proof → acknowledge +Use a dedicated S3 prefix. +Conditional operations required. + +FLEET · FOLLOWER +follower_durability: true + +Cell command commits +SQLite + LTX + + +Native follower fsync +Proof → acknowledge +Eligible proof races the bucket. +Qualification remains bounded. +Every mode retains local node data. Follower proof is available only when eligible. +This is a proof-selection diagram, not a machine-loss or fleet availability guarantee. +Shutdown: preserve retains reload state; handoff needs a fleet and an eligible live peer. + diff --git a/diagram/operations/server-lifecycle.svg b/diagram/operations/server-lifecycle.svg new file mode 100644 index 0000000..23c110c --- /dev/null +++ b/diagram/operations/server-lifecycle.svg @@ -0,0 +1,69 @@ + +Server lifecycle: readiness and clean shutdown +Startup validates configuration, starts Cellar, optionally bootstraps credentials and policy, completes initial transaction recovery for configured accounts, then starts maintenance and public serving. SIGINT or SIGTERM drains the public listener, cancels and joins maintenance, requests Cellar shutdown, checks the complete drain report and retains state. A fence or maintenance failure withdraws public serving; failures must not be treated as a clean stop. + + + + + + + + + + + + +Server lifecycle: readiness and clean shutdown +Read top to bottom. A failed gate does not authorize serving or a clean-exit claim. +STARTUP +SIGINT / SIGTERM + + + + + +Validate configuration +Check fields, paths, key sizes and TLS. + +Start Cellar +Wait for native node readiness. + +Bootstrap when requested +Persist credential and operator policy. + +Initial account recovery +Complete a pass for every listed account. + +Start maintenance + serve +Expose /health and signed requests. + + + + + +Drain public listener +ExtendDB stops accepting public work. + +Cancel + join maintenance +Keep both executors alive. + +Request Cellar shutdown +Use preserve or fleet-only handoff. + +Validate the stop report +Drain complete; durability closed. + +Exit + retain persistent state +Fleet preserve needs clean-reload proof. + +While serving: a node fence or supervised maintenance failure withdraws the public server. +A killed process or incomplete drain is not a successful shutdown. + diff --git a/diagram/operations/transaction-flow.svg b/diagram/operations/transaction-flow.svg new file mode 100644 index 0000000..bff20dc --- /dev/null +++ b/diagram/operations/transaction-flow.svg @@ -0,0 +1,73 @@ + +One transaction, three participant Cells +BeyondDB validates and groups the request by original participant. A coordinator stores durable BEGIN and immutable operations before any participant prepares. Each participant atomically evaluates conditions, locks keys, stages images and reserves apply capacity. All durable prepare evidence permits COMMIT; a definitive failure requests ABORT while BEGIN. The driver resolves every original participant from the immutable decision and records durable resolution receipts before returning success or cancellation. A coordinator is separate from the table routes; logical keys do not guarantee distinct Cells. + + + + + + + + + + + + +One transaction, three participant Cells +Example placement: Accounts/A, Accounts/B and Transfers/id route to three Cells. +1 · ADMIT IMMUTABLE WORK + + + +TransactWriteItems +Signed request + stable token + +BeyondDB driver +Validate; group original keys + +Coordinator: BEGIN +Immutable targets + payloads + +2 · PREPARE INDEPENDENT CELLS + + + +Participant A +Accounts/A +Conditions, locks, images, capacity + + + +Participant B +Accounts/B +Conditions, locks, images, capacity + + + +Participant T +Transfers/id +Conditions, locks, images, capacity + + + +3 · PUBLISH ONE DECISION + +COMMIT or ABORT +COMMIT requires every durable prepare receipt. + +4 · RESOLVE + CONFIRM + +Resolve the original participant set +Apply or discard; record every durable resolution receipt. + + +Return success or ordered cancellation + diff --git a/diagram/operations/transaction-payloads.svg b/diagram/operations/transaction-payloads.svg new file mode 100644 index 0000000..58aedcc --- /dev/null +++ b/diagram/operations/transaction-payloads.svg @@ -0,0 +1,62 @@ + +Payloads have three different lifetimes +Temporary wire-upload staging transfers bounded chunks without acquiring transaction locks or deciding a transaction. Sealing phase input atomically consumes it and writes durable BEGIN or prepare payloads. Participant prepare atomically writes phase records, images, locks and apply reservations. Resolution compacts write payloads and releases locks and claims; committed read images require durable consumption acknowledgement before cleanup. Decisions, identities and terminal tombstones remain retained. Chunking does not bound total disk, WAL, heap or history. + + + + + + + + + + + + +Payloads have three different lifetimes +Wire staging, prepared work and terminal evidence have different cleanup rules. + +1 · WIRE STAGING + +2 · DURABLE PHASE + +3 · TERMINAL EVIDENCE + + +Bounded upload chunks +Immutable upload identity +No transaction locks yet +Sealed with phase admission; +partial upload is not BEGIN. +BEGIN / prepared payloads +Locks + staged images +Physical apply reservations +Commit or roll back together +inside the owning command. +Decision + identities +Resolution receipts +Participant replay markers +Keep delayed messages fenced; +full retirement remains open. +Transport success ≠ decision. +Size includes apply work. +Token expiry ≠ history GC. + +Write payloads +Compact through terminal resolution. +Keep identity, digest and terminal markers. + +Committed read images +Retain until durable consumption intent. +Recovery completes acknowledged cleanup. +Chunk limits do not bound aggregate disk, WAL, heap, reservations or retained history. +A timeout cannot reclaim an unresolved participant’s apply reservation. + diff --git a/diagram/operations/transaction-reads.svg b/diagram/operations/transaction-reads.svg new file mode 100644 index 0000000..0717f2e --- /dev/null +++ b/diagram/operations/transaction-reads.svg @@ -0,0 +1,66 @@ + +Transactional reads: capture before release +Ordinary reads must not return an old image behind an unresolved committed write intent; they resolve and reread or fail retryably. A cross-Cell transactional read captures existing or absent images and shared locks atomically in each participant. All captures coexist once the last shared prepare succeeds. The coordinator commits and resolves participants to release locks while retaining saved images. The initiating reader assembles original positions, then durably acknowledges consumption before cleanup. Cleanup can finish through recovery; unacknowledged images remain retained. + + + + + + + + + + + + +Transactional reads: capture before release +Locks protect the capture; saved images protect the response after locks are released. +ORDINARY READ VS. PENDING WRITE +COORDINATED TRANSACTGETITEMS + + + + +COMMIT is already durable +Some Cells may still be PREPARED. + +A has applied; B has not +B retains its exclusive write intent. + +Read B through the lock barrier +Resolve and re-read, or fail retryably. + +Never return B’s stale image +Missing creates and deletes need barriers too. + + + + + +Prepare shared read locks +Atomically capture existing or absent images. + +Last shared prepare succeeds +All captures coexist while keys remain locked. + +COMMIT + resolve participants +Release locks; preserve immutable images. + +Assemble every original position +Read saved images, including absent items. + +Acknowledge, then clean up +Durable consumption intent permits deletion. + +Read-result lifetime is explicit +Only the initiating reader acknowledges complete assembly. Recovery can finish acknowledged cleanup. +Cancellation before acknowledgement retains images; terminal markers still prevent late recreation. + diff --git a/diagram/operations/transaction-recovery.svg b/diagram/operations/transaction-recovery.svg new file mode 100644 index 0000000..9d169b2 --- /dev/null +++ b/diagram/operations/transaction-recovery.svg @@ -0,0 +1,62 @@ + +Recovery follows the durable decision +Recovery observes the coordinator after establishing the required native ownership authority. BEGIN resumes immutable prepare work; startup does not alone authorize abort. A definitive failure or atomic ten-minute expiry decision may win ABORT only while BEGIN. COMMIT applies all original participants. ABORT discards staged state and records tombstones even for unprepared participants. Terminal decisions cannot change to their opposite. Unavailable owners and lost replies do not prove abort. No recovery path rewrites participants to current table routes. + + + + + + + + + + + + +Recovery follows the durable decision +Read authoritative state; a timeout, lost reply or unavailable owner does not prove ABORT. + +Observe the original coordinator +Use durable state and native ownership fences. + + + + +BEGIN + + +COMMIT + + +ABORT +Resume original operations. +Keep identities and epochs. +ABORT needs a valid decision: +definitive failure, or atomic +ten-minute expiry check. +Apply original participants. +Release locks and claims. +Retry resolution idempotently. +Retain newer writes on replay. +Record durable receipts. +Discard staged work. +Release locks and claims. +Resolve unprepared Cells too. +ABORTED tombstones reject +a delayed prepare. +Startup alone does not abort. +Never change to ABORT. +Never change to COMMIT. + +Stable recovery authority +Table splits and route changes do not rewrite an admitted participant set. +An unavailable coordinator can keep affected keys locked until its durable decision is recovered. + diff --git a/diagram/reference/architecture-1.svg b/diagram/reference/architecture-1.svg index 6f1521e..824887c 100644 --- a/diagram/reference/architecture-1.svg +++ b/diagram/reference/architecture-1.svg @@ -1 +1,77 @@ -Detailed original graph. Its containing guide states the applicable revision scope.architecture detailed reference diagram 1Leased BeyondDB nodesPublic request pathAWS SDK or CLIExtendDB HTTP + SigV4ExtendDB validation +operation engineBeyondDB StorageEngine /CatalogStoreCell client + authenticatedpeer transportAccount Cell: table catalog,policy, TTL settingsDirectory Cells: base andGSI routesCredential Cells: encryptedkeysCoordinator Cells:transaction decisionsData Cells: items, LSIs,journals, streamsGSI Cells: projected entriesCellar SQLite + LTXConditional object store + +Components and ownership +AWS SDK or CLI requests pass through ExtendDB HTTP and SigV4, validation and the operation engine, then the BeyondDB StorageEngine and CatalogStore. The Cell client uses authenticated peer transport to independently owned Account, Directory, Credential, Coordinator, Data and GSI Cells on leased nodes. All Cell types execute through Cellar SQLite and LTX. Fleet durability uses a conditional object store; nodes retain local data. + + + + + + + + + + + + +Components and ownership +One executable embeds ExtendDB, BeyondDB and Cellar. +PUBLIC REQUEST PATH + + + +AWS SDK or CLI +DynamoDB API requests + +ExtendDB HTTP + SigV4 +Protocol and authentication + +ExtendDB engine +Validation and operations + + +BeyondDB StorageEngine / CatalogStore +Application adapters, metadata and ordered routing + +LEASED BEYONDDB NODES +Ownership is per Cell + + +Cell client + authenticated peer transport +Dispatch to the current owner + + +INDEPENDENT CELL TYPES + +Account Cell +Table catalog, policy, TTL + +Directory Cells +Base-table and GSI routes + +Credential Cells +Encrypted keys and tokens + +Coordinator Cells +Durable transaction decisions + +Data Cells +Items, LSIs, journals, streams + +GSI Cells +Projected index entries + + +Cellar SQLite + LTX +Shared execution and durability layer for every Cell type + + +Fleet mode: conditional object store + diff --git a/diagram/reference/architecture-2.svg b/diagram/reference/architecture-2.svg index 55530d0..6dfef03 100644 --- a/diagram/reference/architecture-2.svg +++ b/diagram/reference/architecture-2.svg @@ -1 +1,78 @@ -Detailed original graph. Its containing guide states the applicable revision scope.architecture detailed reference diagram 2Serving source rangeDurable split planSeal source after intentsand journals drainCopy and verify two childrangesPublish replacementdirectory routeOpen children for servingRetire source residencywhen safe + +Range split: prepare, verify, publish +A serving source range creates a durable split plan. After prepared intents and projection journals drain, the source seals. Two child ranges are copied and verified before a replacement directory route is published. Both children open for serving before source residency retires when safe. Split plans remain durable and replayable; broad split qualification is separate. + + + + + + + + + + + + +Range split: prepare, verify, publish +Publish the replacement route only after both child ranges are verified. + +PREPARE + +VERIFY + +CUT OVER +Durable plan +and source seal +Publish, open, +then retire + + + + + + + + +1 +Serving source range +The existing range owns and serves its ordered interval. + + +2 +Record durable split plan +Keep split progress replayable across owner recovery. + + +3 +Seal source range +Prepared intents and pending projection journals must drain. + + +4 +Copy and verify both children +Verify two child ranges before changing the directory route. + + +5 +Publish replacement directory route +Retain the split plan until both child ranges are open. + + +6 +Open children for serving +Serve through the replacement route; fence stale source work. + + +7 +Retire source residency when safe +Reclaim only when the split protocol permits it. +Safety order shown here; broad split behavior and Scan continuation remain under qualification. + diff --git a/diagram/reference/architecture-3.svg b/diagram/reference/architecture-3.svg index 312c2b6..d900b46 100644 --- a/diagram/reference/architecture-3.svg +++ b/diagram/reference/architecture-3.svg @@ -1 +1,59 @@ -Detailed original graph. Its containing guide states the applicable revision scope.architecture detailed reference diagram 3durable base commitworker reads journalidempotent updatedurable receiptafter all required receiptsBase data Cellitem + projection journalAcknowledge writeProjection workerGSI CellPrune journal entry + +GSI projection: commit, deliver, confirm +The base data Cell atomically commits the item and immutable projection journal. The write is acknowledged after durable base commit. A projection worker reads the journal and sends idempotent versioned entries or tombstones to each required GSI Cell. Durable receipts return to the worker. The journal entry is pruned only after all required index receipts. GSI reads are eventually consistent; base transactions do not create an atomic GSI view. + + + + + + + + + + + + +GSI projection: commit, deliver, confirm +Base acknowledgement and index delivery have separate completion points. + +BASE WRITE · SYNCHRONOUS + +Durable base commit + +Base data Cell +Item + immutable projection journal +Committed atomically + +Acknowledge write +After the base durability gate + +INDEX PROJECTION · ASYNCHRONOUS + + +Read committed journal + +Idempotent, versioned update + +Durable receipt from each index + +Projection worker +Replay pending journal work + +Required GSI Cells +Versioned entry or tombstone + +All required receipts + +Prune journal entry +Only after every required index confirms durability +An unavailable index delays its own projection; healthy indexes can advance. +GSI reads are eventually consistent. A base transaction does not create an atomic GSI view. + diff --git a/diagram/reference/implementation-status-1.svg b/diagram/reference/implementation-status-1.svg deleted file mode 100644 index c7d6e63..0000000 --- a/diagram/reference/implementation-status-1.svg +++ /dev/null @@ -1 +0,0 @@ -Detailed original graph. Its containing guide states the applicable revision scope.implementation-status detailed reference diagram 1Native Cell behaviorSigned SDK requestDurable owner restartFull protocol suiteopen gateSustained fleet loadopen gate diff --git a/diagram/reference/implementation-status-2.svg b/diagram/reference/implementation-status-2.svg deleted file mode 100644 index 5164186..0000000 --- a/diagram/reference/implementation-status-2.svg +++ /dev/null @@ -1 +0,0 @@ -Detailed original graph. Its containing guide states the applicable revision scope.implementation-status detailed reference diagram 2one Cell commandread before visibility cutoffafter 24 hoursItem or eligible TTL changeStream journal recordStreams read APIBounded retention sweepActive owner orcatalog-discovered Cell diff --git a/docs/README.md b/docs/README.md index d86ad7a..22d251d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -32,7 +32,7 @@ below to evaluate a deployment; these guides do not establish a production SLA. | Streams and retention | [Streams](streams-contract.md) | | Follower acknowledgement and recovery | [Follower durability](follower-durability.md) | | Capacity planning | [Scaling](scaling.md) | -| Tests, open gates and historical evidence | [Implementation status](implementation-status.md) | +| Tests, open gates and historical evidence | [Qualification workflow](../.github/workflows/qualification.yml) · [Server evidence](cellar-server.md#detailed-qualification-record) | ## Reading conventions diff --git a/docs/api.md b/docs/api.md index 4d3f412..c077061 100644 --- a/docs/api.md +++ b/docs/api.md @@ -3,7 +3,7 @@ See the [server guide](cellar-server.md#qualification-still-required) for current Cellar qualification boundaries. The matrix describes implemented paths and their limits. Use the current -[registered targets](implementation-status.md#current-verified-slice) to select +[qualification workflow](../.github/workflows/qualification.yml) to select evidence; a historical fixture count is not a new qualification result. Use this page to decide whether a DynamoDB operation fits your workload. BeyondDB exposes ExtendDB's DynamoDB JSON endpoint, but a protocol handler alone does not establish Cell-backed support. A **verified path** has a signed AWS SDK or CLI request, a durable Cell commit, and a result checked after owner or process restart. Each fixture proves only its stated cases, not every DynamoDB option or error response. @@ -14,7 +14,7 @@ Use this page to decide whether a DynamoDB operation fits your workload. BeyondD | **Partial** | Some options work, but the named alternatives are rejected or unqualified. | | **Unsupported** | No compatible Cell path is available through the public endpoint. | -The [AWS CLI guide](user-guide.md) shows requests you can try. The [implementation record](implementation-status.md) contains the test evidence and open qualification gates. +The [AWS CLI guide](user-guide.md) shows requests you can try. The [server guide](cellar-server.md#qualification-still-required) records focused test evidence and open qualification gates. ## Tables, items, and reads @@ -30,9 +30,9 @@ Core table, item, read, and transaction operations have Cell paths. Read the bou | `TransactWriteItems`, `TransactGetItems` | Durable coordinator and participant paths; cross-Cell SDK hard-restart coverage | Conflict and capacity behavior at sustained fleet load is unqualified. GSI propagation remains asynchronous after base commit. | | `TagResource`, `UntagResource`, `ListTagsOfResource` | Cell-backed; signed SDK restart coverage | Table deletion removes tag rows in bounded generation cleanup. | -The [implementation record](implementation-status.md#current-verified-slice) -maps these paths to tests. The full upstream protocol suite remains an -[acceptance gate](implementation-status.md#acceptance-proof-for-a-server-claim). +The [qualification workflow](../.github/workflows/qualification.yml) selects the +registered native and process fixtures. Full upstream protocol compatibility +remains an open [qualification gate](cellar-server.md#qualification-still-required). ### Creation-time Cell model extension @@ -70,7 +70,7 @@ The serving worker configures a fixed expiry index in data Cells, backfills older items in bounded commands, and conditionally removes expired items. Settings and sweep progress survive owner restart. Expiry is asynchronous. The global TTL listing traits remain unsupported; the worker enumerates each -configured account directly. [TTL implementation status](implementation-status.md#running-the-current-server) +configured account directly. [Account maintenance](cellar-server.md#account-maintenance) ## Streams @@ -87,7 +87,7 @@ change the view type. Bounded retention sweeps cover active routed owners and catalog-discovered dormant or deleted data Cells. Expired stream catalog rows, full split-lineage and iterator qualification, and fleet-scale retention work remain open. See [Streams contract](streams-contract.md) and -[implementation status](implementation-status.md#streams-dependency-contract). +[recorded restart evidence](cellar-server.md#detailed-qualification-record). ## Authentication and administration @@ -138,5 +138,5 @@ Use the same evidence path for each new operation or option: and verify idempotent participant resolution. Until those checks pass, describe a path as implemented or partial rather than -generally compatible. The current [independent qualification runner](implementation-status.md#independent-client-qualification) +generally compatible. The current [independent qualification runner](../scripts/qualify-upstream.py) supports unchanged ExtendDB Python client tests. diff --git a/docs/architecture.md b/docs/architecture.md index 1a6a635..7b7f8aa 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -19,7 +19,7 @@ execution and durability. Every node embeds these layers. The request path separates protocol work from state execution. The Cell types below have different ownership and recovery policies, even when they run on the same node. -![Components and ownership: detailed diagram retained from the original guide](../diagram/reference/architecture-1.svg) +![Request layers and independently owned Cell types](../diagram/reference/architecture-1.svg) Fleet mode configures an S3-compatible bucket with conditional create/update semantics. Each node retains its own local data directory. Native peer requests @@ -101,7 +101,7 @@ The original coordinator and participant identities survive routing changes. If Splitting changes a route only after the source has sealed and both children have copied and verified its contents. Global secondary index (GSI) projection follows a different path: a base write commits a journal entry, then an index worker applies it asynchronously. -![Range split and GSI projection: detailed diagram retained from the original guide](../diagram/reference/architecture-2.svg) +![Seven-stage range split: prepare, verify, publish and retire](../diagram/reference/architecture-2.svg) Split plans are durable and replayable. The source cannot seal while prepared transactions or pending GSI projection work would be lost. Directory @@ -112,7 +112,7 @@ under qualification; see [metadata ownership](metadata-sharding.md). For a GSI, the base command writes an immutable projection journal entry. A worker applies a versioned entry or tombstone to the index Cell. It removes the journal entry only after every required index acknowledges it durably. An unavailable index delays its own projection while healthy indexes can advance. GSI reads are therefore eventually consistent with base writes; a base transaction does not produce an atomic GSI view. See the [GSI design and limits](global-indexes.md). -![Failure recovery and current limits: detailed diagram retained from the original guide](../diagram/reference/architecture-3.svg) +![GSI projection: durable base acknowledgement, asynchronous delivery and receipt-gated journal pruning](../diagram/reference/architecture-3.svg) ## Failure recovery and current limits diff --git a/docs/cellar-server.md b/docs/cellar-server.md index e851899..34991c0 100644 --- a/docs/cellar-server.md +++ b/docs/cellar-server.md @@ -1,36 +1,117 @@ # BeyondDB server reference -The `beyonddb` executable embeds ExtendDB and Cellar. BeyondDB owns configuration, -public TLS and application maintenance. Cellar owns execution, placement, -ownership, private peer transport and durability. See [deployment](deployment.md) -for setup and [operations](operations.md) for failure handling. +Run one `beyonddb` executable per node. It embeds ExtendDB for the DynamoDB +frontend and Cellar for state execution and durability. Use this guide to choose +storage, configure a node, follow its lifecycle and inspect qualification evidence. + +| I need to… | Start here | +| --- | --- | +| Understand node boundaries | [Server layout](#server-layout) | +| Select local or fleet persistence | [Persistence and shutdown](#persistence-and-shutdown) | +| Set listener, key and capacity fields | [Configuration](#configuration) | +| Bootstrap, start or stop a process | [Starting and stopping](#starting-and-stopping) | +| Understand maintenance, caching and retries | [Serving and caching](#serving-and-caching) | +| Select checks and interpret their limits | [Qualification still required](#qualification-still-required) | +| Inspect the original fault cases and measurements | [Detailed qualification record](#detailed-qualification-record) | + +## Server layout + +![Public clients, private peers and persistent storage around one embedded BeyondDB server](../diagram/operations/server-boundaries.svg) + +| Layer | Owns | +| --- | --- | +| ExtendDB | Public HTTP, SigV4, IAM evaluation, validation and operation handling | +| BeyondDB | Configuration, public TLS setup, ordered routing and application maintenance | +| Cellar | Cell execution, placement, fenced ownership, private peer transport and durability | + +Every table spans ordered range Cells. The `accounts` list selects accounts for +maintenance; it does not place a table into one Cell. See [architecture](architecture.md) +for the Cell types and [deployment](deployment.md) for provisioning. + +## Persistence and shutdown + +Choose the **acknowledgement boundary** first, then choose how a node stops. + +![Local persistence, default fleet bucket publication and optional native follower durability](../diagram/operations/server-durability.svg) + +| Storage configuration | Acknowledgement / recovery boundary | Retain | +| --- | --- | --- | +| No `fleet` object | Local Cellar persistence; does not cover machine loss | Node data directory | +| `fleet.follower_durability: false` | Bucket-publication proof is the default acknowledgement path | Node data, bucket prefix and deployment keys | +| `fleet.follower_durability: true` | An eligible native follower fsync proof may race bucket publication; see [follower durability](follower-durability.md) | Node data, bucket prefix and deployment keys | + +| `shutdown_mode` | Use when | Completion requirement | +| --- | --- | --- | +| `preserve` — default in both modes | Restarting the same logical node | Drain work and close local durability; fleet mode also requires the SDK's clean-reload proof and retained local directory | +| `handoff` — fleet only | Removing a node while an eligible live peer can adopt its Cells | Complete native transfer; cannot complete on the last eligible node; rejected without `fleet` | + +Local-only nodes have no replication marker: they reopen closed local state through +normal recovery. A process kill is not a clean stop. Keep persistent state after exit. ## Configuration -Use [config/cellar.example.json](../config/cellar.example.json). The parser -rejects unknown fields. Current fields are defined in -[src/bin/beyonddb.rs](../src/bin/beyonddb.rs). +Start with [config/cellar.example.json](../config/cellar.example.json). The parser +rejects unknown fields; [the executable's configuration structs](../src/bin/beyonddb.rs) +are the source of truth. Paths below are examples to provision for the service account. -| Field | Meaning / default | +### Node, network and keys + +| Field | Requirement / default | | --- | --- | -| `node_name` | Unique running node identity | +| `node_name` | Unique identity for each running node | | `data_dir` | Persistent local directory; engine state is under `cellar-ordered-v1` | | `region` | Deployment and request-signing region | -| `public_bind`, `public_endpoint` | Listener address and advertised public URL | -| `public_certificate`, `public_private_key` | Optional PEM pair; required for non-loopback public serving | -| `peer_bind`, `peer_advertise` | Private listener and reachable `HOST:PORT` | -| `peer_key_file` | Exactly 32 raw bytes; matching key across fleet nodes | -| `encryption_key_file` | Exactly 32 raw bytes; preserve to decrypt credential secrets | -| `accounts` | Accounts maintained by this deployment; default empty | -| `initial_partitions` | Initial base/GSI data-range count, 2–256; default 4 | -| `max_active_cells` | Positive native residency limit for all cell types; default 128 | -| `split_threshold_bytes` | Positive threshold supplied to range maintenance; default 268435456 | -| `auth_cache_enabled` | Credential, authorization, table metadata and route caching; default false | +| `public_bind` | Public listener address; plaintext HTTP requires loopback | +| `public_endpoint` | Advertised URL; scheme must match the listener's TLS configuration | +| `public_certificate`, `public_private_key` | Optional PEM pair; both required for non-loopback public serving | +| `peer_bind`, `peer_advertise` | Private listener and reachable advertised `HOST:PORT` | +| `peer_key_file` | Exactly 32 raw bytes; matching peer key across fleet nodes | +| `encryption_key_file` | Exactly 32 raw bytes; preserve it to decrypt stored credential secrets | + +### Application policy and capacity + +| Field | Requirement / default | +| --- | --- | +| `accounts` | Accounts maintained by this deployment; default `[]` | +| `initial_partitions` | Initial base/GSI range count, **2–256**; default **4** | +| `max_active_cells` | Positive native residency limit for **all Cell types**; default **128** | +| `split_threshold_bytes` | Positive threshold supplied to range maintenance; default **268435456** (256 MiB) | +| `auth_cache_enabled` | Credential, authorization, table metadata and route caching; default **false** | | `shutdown_mode` | `preserve` (default) or fleet-only `handoff` | -| `fleet` | Optional S3-compatible fleet configuration; absent means local persistence | -| `bootstrap` | Optional account/key/principal/policy settings, used with `--bootstrap` | +| `fleet` | Optional S3-compatible storage settings; absent means local persistence | +| `bootstrap` | Optional initial account, access key, principal and policy settings; applied with `--bootstrap` | + +### Local node example + +This complete configuration starts an **already bootstrapped** local node. Create +both key files and provision the account first using the [bootstrap steps](deployment.md#bootstrap-and-start). +For initial provisioning, use the checked-in example's `bootstrap` object. + +```json +{ + "node_name": "bdb-node-1", + "data_dir": "/var/lib/beyonddb", + "region": "us-east-1", + "public_bind": "127.0.0.1:8000", + "public_endpoint": "http://127.0.0.1:8000", + "peer_bind": "127.0.0.1:9000", + "peer_advertise": "127.0.0.1:9000", + "peer_key_file": "/etc/beyonddb/peer.key", + "encryption_key_file": "/etc/beyonddb/encryption.key", + "accounts": ["123456789012"], + "initial_partitions": 4, + "max_active_cells": 128, + "split_threshold_bytes": 268435456, + "auth_cache_enabled": false, + "shutdown_mode": "preserve" +} +``` -The optional fleet object is: +### Add fleet storage + +Merge the following object into the node configuration. Create the bucket first +and provide its credentials in the **server process environment**. Keep the +DynamoDB client's bootstrapped identity separate from that storage identity. ```json { @@ -43,355 +124,336 @@ The optional fleet object is: } ``` -`endpoint` may be omitted for the provider's standard endpoint. The server builds -S3 clients from its environment. It disables conditional-operation retries to -preserve ambiguous ownership evidence. GCS/Azure URL configuration, peer mTLS -fields, `storage_url`, `sql_workers`, `disk_budget_bytes` and -`node_retained_bytes` are not fields in this executable. +| Fleet setting | Meaning | +| --- | --- | +| `bucket` | S3-compatible bucket supporting conditional create/update | +| `endpoint` | Optional provider override; omit for the provider's standard endpoint | +| `prefix` | Nonempty, dedicated prefix for one compatible Cellar deployment | +| `follower_durability` | Default `false`; opt into eligible native follower fsync proofs | +| S3 client configuration | Built from the server environment; conditional-operation retries are disabled to preserve ambiguous ownership evidence | + +### Security and unsupported knobs + +| Control | What it protects | Operational boundary | +| --- | --- | --- | +| Public TLS | Client-to-public-listener traffic | Supply both PEM files and matching HTTPS URL for external serving | +| Shared peer key | Private peer authentication | Does **not** encrypt transport; isolate and encrypt the private network externally | +| Credential-encryption key | Stored access secrets and session tokens | Does **not** establish encryption of every application row or storage file | + +The parser does **not** accept GCS/Azure URL configuration, peer mTLS fields, +`storage_url`, `sql_workers`, `disk_budget_bytes` or `node_retained_bytes`. +See [deployment](deployment.md#expose-https-and-add-nodes) before adding peers. + +## Starting and stopping + +![Startup readiness gates and the ordered SIGINT or SIGTERM clean shutdown path](../diagram/operations/server-lifecycle.svg) + +### Build and bootstrap + +Run from the repository root. Provision directories, keys and the operator-authored +IAM policy using [the deployment guide](deployment.md#build-and-provision). + +```sh +cargo build --locked --bin beyonddb +``` + +For **initial provisioning only**, include the `bootstrap` object from the example +configuration. Its `account_id` must appear in `accounts`; `policy_file` must point +to the operator's IAM policy. Supply the new credential secret on stdin, for example +from a protected file mounted by your secret manager: + +```sh +"${CARGO_TARGET_DIR:-target}/debug/beyonddb" /etc/beyonddb/config.json \ + --bootstrap < /run/secrets/beyonddb-bootstrap +``` + +Later starts omit `--bootstrap`: + +```sh +"${CARGO_TARGET_DIR:-target}/debug/beyonddb" /etc/beyonddb/config.json +``` + +### Readiness gates + +| Order | Gate | Failure / recovery behavior | +| --- | --- | --- | +| 1 | Configuration and Cellar startup | Public serving stays closed until native readiness | +| 2 | Optional bootstrap | Persist the credential and policy before serving | +| 3 | Initial transaction recovery for each configured account | Use each shard's observed pending boundary; continue healthy records/shards after failures, but report unresolved admission/resolution failure before maintenance starts | +| 4 | Maintenance and public serving | Start only after the initial pass succeeds; fleet observation can take longer than local startup | + +Check from a second terminal after serving starts. These commands assume the local +HTTP example and a **client shell configured with the bootstrapped DynamoDB key**: + +```sh +curl --fail http://127.0.0.1:8000/health +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 dynamodb list-tables +``` + +`/health` checks listener readiness. Follow signed `ListTables` with a write/read +and restart check; health alone does not qualify application durability. + +### Coordinator recovery rules + +| Observed state | Allowed recovery action | Preserved invariant | +| --- | --- | --- | +| Durable COMMIT / ABORT | Resolve the recorded original participants | Do not change the admitted participant set | +| Fresh BEGIN | Resume from immutable coordinator payloads | Keep original participant identities and epochs; startup does not authorize abort | +| Abandoned BEGIN aged ten minutes | Atomically check age and record ABORT in the coordinator command | The deadline does not authorize decision collection | +| Unavailable owner or uncertain mutation reply | Recover through the durable protocol | Neither observation establishes ABORT | + +Coordinator discovery pages durable account registrations and retained pending +records. See [the transaction protocol](cross-cell-transactions.md) for resolution +and replay details. + +### Clean shutdown checklist + +SIGINT/SIGTERM drains ExtendDB's public listener, then BeyondDB cancels and joins +maintenance before requesting Cellar shutdown. Embedding callers can own a +`TaskGroup` and call its bounded `shutdown` first. Keep both caller-owned executors +alive until the node outcome returns. -The encryption key protects stored credentials; it is not a claim of encryption -for every application row or storage file. Public TLS and peer authentication -are separate controls. Isolate and encrypt the private network externally. +A successful exit requires **every** condition below, as checked by +[the executable](../src/bin/beyonddb.rs): + +| Stop-report condition | Required outcome | +| --- | --- | +| Public ingress, application work and Cells | Drained | +| Pending application events and dispatches | Zero | +| Logs | Quiesced | +| Local durability | Closed | +| Drain timeout / elapsed deadline | Neither occurred | +| Fleet `preserve` only | Clean-reload proof prepared and consistent with retained local state | + +A node fence or failed supervised maintenance task withdraws public serving. +Check the process outcome and logs before restart; keep the node directory. +See [failure handling](operations.md#shutdown-and-replacement). ## Serving and caching -Readiness waits for Cellar startup, optional bootstrap and an initial transaction -recovery pass for each configured account. A node fence or failed supervised -maintenance task withdraws public serving. Use `/health` for listener readiness -and signed requests for application checks. +### Account maintenance `accounts` drives table lifecycle, projection, TTL, retention and coordinator -maintenance. Empty or incomplete account lists do not provide maintenance for -omitted accounts. Table data remains in separately routed cells. - -Caching is off by default. When enabled, the frontend uses the pinned ExtendDB -cache defaults and BeyondDB's bounded route metadata cache. Cross-node changes -can remain cached until expiry; local invalidation is not a fleet-wide immediate -revocation guarantee. Leave caching disabled when testing fresh authorization -and inspect the pinned implementation before relying on a specific TTL. +maintenance. Empty or incomplete lists leave omitted accounts without those +workers. Base and GSI data remain in separately routed Cells. -The current binary fixes native per-cell concurrency at 16 and the handler budget -at 30 seconds. These are source settings, not JSON knobs. Pure queries use bounded -backpressure; commands retain single-attempt submission semantics. More request -concurrency can worsen latency or cause admission errors. +| Maintenance responsibility | State it advances | +| --- | --- | +| Table lifecycle and range maintenance | Creation/deletion repair, capacity and split work | +| GSI projection and statistics | Published index ranges, pending projections and metadata samples | +| Transaction recovery | Discovered BEGINs and durable decisions | +| TTL and retention | Expiry work and retained stream records, including inventoried ranges | -## Persistence and shutdown +### Cache policy -| Mode | Acknowledgement and restart boundary | +| Setting / event | Behavior | | --- | --- | -| Local, no `fleet` | Local Cellar persistence; retain node data; does not cover machine loss | -| Fleet, follower durability false | Bucket-publication proof is the default acknowledgement path | -| Fleet, follower durability true | Native follower fsync proof may race bucket publication; see [follower durability](follower-durability.md) | -| `preserve` shutdown | Drain work and local durability; fleet mode also requires clean-reload proof | -| `handoff` shutdown | Fleet-only transfer to an eligible live peer; cannot complete on the last eligible node | +| `auth_cache_enabled: false` — default | Use when testing fresh authorization and revocation | +| `auth_cache_enabled: true` | Use pinned ExtendDB cache defaults plus BeyondDB's bounded route metadata cache | +| Cross-node credential, policy or metadata change | May remain cached until expiry; local invalidation is not fleet-wide immediate revocation | +| Need a specific cache TTL | Inspect the pinned implementation before relying on one | -SIGINT/SIGTERM drains the public server, stops application maintenance and invokes -Cellar shutdown. A successful exit requires drained work/cells, no pending -dispatches, quiesced logs, closed local durability and no shutdown deadline failure. -Keep persistent state after exit; never confuse a process kill with a clean stop. +### Admission and retry boundaries -## Starting and stopping +These limits are **source settings**, not JSON knobs. Raising client concurrency +can increase latency or admission errors. -Build with `cargo build --locked --bin beyonddb`. Run `beyonddb config.json --bootstrap` -once with the new credential secret supplied on stdin; the bootstrap policy -must be an operator-authored IAM policy file. Later starts omit `--bootstrap`. -The bootstrap account must be listed in `accounts`. - -The public server starts after Cellar reports ready, bootstrap completes and -an initial transaction recovery pass succeeds for every configured account. -The pass uses each shard's observed pending boundary, continues healthy work -after a failure, and reports any unresolved admission or resolution failure -before maintenance starts. Fresh BEGINs resume normally; startup alone does -not authorize their abort. Initial fleet observation can take longer than -local startup; the listener is not published while that readiness gate is closed. -ExtendDB handles SIGINT/SIGTERM and drains its public listener. The application -then cancels/joins maintenance and requests the Cellar stop protocol. A node -fence or failed maintenance task withdraws public serving. Shutdown succeeds -only with drained ingress, application work and cells, zero pending events and -dispatches, quiesced logs, closed local durability and no drain timeout or elapsed -deadline. Keep both caller-owned executors alive until that outcome is returned. - -`shutdown_mode` defaults to `"preserve"`, including in fleet mode. Fleet Preserve -prepares a clean reload of the same logical node using its retained local -directory and requires the SDK's clean-reload proof. Local-only nodes have no -replication marker and reopen their closed local state through normal recovery. -Use `"handoff"` when removing a fleet node -and another eligible node is running to adopt its cells. Handoff cannot complete -on the last node; it is rejected without fleet configuration. +| Boundary | Current behavior | +| --- | --- | +| Native per-Cell concurrency | **16** | +| Native handler budget | **30 seconds** | +| One HTTP gateway | **128** queued/submitted dispatches; **8** per target, with permits held through response completion | +| Pure SDK query backpressure | Bounded **30-second** window for HTTP 429/503 and the recognized Cellar 0.1.0 owner-resolution failure | +| Recognized HTTP 500 | Only the exact bounded `Worker failed: route failed: ResolveFailed` body; other 500s retain their failure | +| Commands, including transaction read-image capture | One transport attempt; HTTP status does not prove a mutation was unstarted | -Coordinator discovery pages durable account registrations and retained pending -records. Published decisions resolve their original participants. Fresh BEGINs -resume from the coordinator's immutable payloads, retaining original participant -identities and epochs. An abandoned BEGIN may be atomically changed to ABORT -after ten minutes; the age check and -decision share the coordinator command. This deadline does not authorize -collecting decisions or changing an admitted participant set. Recovery cannot -infer ABORT from an unavailable owner or an uncertain mutation reply. +The dispatch wrapper preserves the SDK body-size hint and retains permits through +completion. Read pacing does not cache credentials, authorization or item values. +Cellar still owns admission across gateways, execution and durability. Never infer +mutation rollback from cancellation or a lost reply. ## Qualification still required -[Current test targets](implementation-status.md#current-verified-slice) cover -selected public operations, native peer/fencing faults, owner restoration and -process loss. They do not establish full DynamoDB compatibility, automatic fleet -recovery within a bound, independent-host follower-loss tolerance, sustained -capacity, backup/PITR or safe upgrades of existing development roots. +The [qualification workflow](../.github/workflows/qualification.yml) defines +current native and process checks. The expandable record below preserves older +measurements and fault cases; it is not a report of the latest CI run. + +| Evidence available | Still not established | +| --- | --- | +| Selected signed public operations and local restart | Full DynamoDB compatibility | +| Native peer/fencing faults and fresh-owner restoration | Automatic fleet recovery within a bound | +| Specific follower recovery process fixture | Independent-host follower-loss tolerance or a complete fleet fencing/follower-log matrix | +| Capacity and admission fixtures | Sustained throughput, overload behavior or fleet capacity | +| Retained state and process-loss cases | Backup/PITR or safe upgrades of development roots | +| Migrated native contracts | Removed contracts listed in [the retirement inventory](retired-tests.md) | + +Use fresh storage for incompatible formats. Do not open roots from another runtime +or reuse a hash-layout root as ordered-layout state. Record binary revision, +`Cargo.lock`, configuration, storage and filesystem with every qualification. + +Examples from the repository root; process cases require Docker, OpenSSL and AWS CLI: + +```sh +cargo test --locked --test cellar_sdk +cargo test --locked --test cellar_transactions -- --test-threads=1 +cargo test --locked --test cellar_process -- --ignored --test-threads=1 +``` -Use fresh storage for incompatible formats. Do not open roots from a different -runtime or reuse an older hash-layout root as ordered-layout state. Capture the -binary revision, dependency lockfile and configuration with every qualification. +See the [Rust workflow](../.github/workflows/rust.yml) and +[signed SDK workflow](../.github/workflows/qualification.yml) for the full matrix. ## Detailed qualification record -> **Recorded evidence:** These detailed native and process cases are retained from -> the [original report](https://github.com/crabbuild/beyonddb/blob/a1eb96a0e87d4473f1bec89c0f3b4f2b02a21afb/docs/cellar-server.md). -> Counts and timing describe those runs, not the latest CI result. Earlier-runtime -> comparisons are not additional Cellar coverage. Current configuration remains -> defined by the reference above. - -The new `tests/cellar_sdk.rs` uses signed SDK requests against the real process, -including typed/large escaped item values, conditional writes, token replay, -Query pages and restart. The Put/Get/Update/Delete, transaction replay and -query/restart cases pass. It is local qualification only. The separate -`tests/cellar_process.rs` target has passing runs for nine bucket-backed cases: -coordinator history, GSI projection recovery, large transactional reads on a -fresh owner, metadata caching, segmented LSI scans, and signed Streams/TTL recovery with -retained generations after table deletion and recreation, native capacity and -authenticated peer identity, the broad SDK restart scenario and native follower -recovery with bundle/LTX publication withheld. The broad SDK scenario -keeps batch/transaction/index/large-value checks across ordered ranges through -hard and graceful restarts, then deletes and recreates a table. The -account-domain fixture also passes SDK handoff to a second node. Twenty-two cases in -`tests/cellar_transactions.rs` qualify token retention, cross-cell write skew, -bounded resume, delayed resolution, shared read snapshots, lost transaction -phases, caller admission refusal, competing drivers, fair background discovery, -acknowledged read-result cleanup, TTL lock races, ordered range-read barriers and -prepared-commit/mixed-participant handoff through native SDK ingress. -The native coordinator cases also retain history across 24 shards under a -16-cell resident cap, recover a partial read on a fresh owner, discover a later -BEGIN after settled passes, and admit four table ranges after 32 coordinator -shards have exceeded residency. A caller-injected child-install refusal retains -the same pending split until capacity returns. Proven pre-submission Capacity -maps to LimitExceeded; native HTTP failure remains ambiguous. The native recovery -admission cases cover BEGIN/COMMIT/ABORT at startup and while serving, progress -past failed records and shards, retry without overwriting newer values, and -deferred discovery beyond the initial pending boundary. A server regression -checks failure before maintenance starts and successful retry. These use an -explicit caller policy before SDK submission, not physical network faults. -`tests/cellar_peer.rs` adds actual native peer sockets behind an opaque TCP -cut. With SDK route caching both disabled and enabled, signed requests retain -numeric pagination, two-range reads and 50 concurrent increments without AWS -client retries. Removing the old directory before lease-expiry recovery retains -values and advances native owner epochs. Authentication fails closed during a -credential-owner cut while healthy local data remains readable, then recovers -after expiry. Unknown keys create no credential cells; revocation remains fresh. - -The public frontend paces pure SDK queries on HTTP 429/503 and Cellar 0.1.0 -owner-resolution failures within a 30-second backpressure window. Only the exact -bounded `Worker failed: route failed: ResolveFailed` 500 body is recognized; -other 500s retain their original failure. The dispatch wrapper preserves the -SDK body size hint while retaining its permits through completion. This permits bursts through native per-cell admission -without caching credential, authorization or item values. Commands keep their -single-attempt transport contract, including transaction read-image capture; -HTTP status still does not prove a mutation was unstarted. Separate policy -checks cover read retry hints, deadline failure and mutation non-retry. -Eight `tests/node_log_authority.rs` cases exercise the native single lease -writer and folded log through real storage faults. They cover normal and slow -renewals, an applied CAS reply lost before delivery, delayed/stalled ordinary -reads, graceful sealing, natural expiry after sustained renewal failure and -closed ingress after fencing. An expired unavailable follower cannot prove a -new write while both bundle and LTX publication are withheld; that same signed -request completes when publication resumes. The fixtures never edit authority -records or require epoch rotation from TTL expiry alone. -Four additional native shutdown cases verify immediate certified same-node -restart, incomplete drain with unpublished signed work, a real provider CAS -that commits after the node closes, and bounded read/write storage stalls. -A clean report must match the actual local certificate; incomplete stops -cannot grant one and must close ingress. After ordinary expiry, signed reads -retain acknowledged rows in both data ranges. The tests do not retain -Cellule's separate application-owned session retirement API. -Six `tests/cellar_indexes.rs` cases retain local/global index mutation, -transaction, split, replay and restart contracts. LSI splits keep PK collections -together and retain logical query cursors. GSI replay preserves large chunked -images, versions and tombstones. Maintenance visits every published GSI range -through native ingress even when its base outbox is empty, so idle and empty -ranges recover on a new owner. Fixtures observe actual SDK factories and owner -epochs before issuing user index reads, after deleting the former directories. -Backfill cases preserve the historical delivery barrier, prepare reservations -and unfinished scan policy across owner replacement and range splits. The GSI -transfer case retains 70 wide-key records, binary images and tombstones across -partial-import and post-publication replacements. Equal index PK/SK entries -span both children via the complete base-key suffix; signed forward/reverse -queries retain all rows and full continuation keys after recovery. -Embedding callers can own a `TaskGroup` and call its bounded `shutdown` before -stopping the SDK node; the executable follows that same shutdown order. -Two multipart cases preserve escaped inputs larger than one command, -immutable/independent upload identities, exact receipt replay and durable -payload recovery. Complete SDK snapshots verify expired staging cleanup and -consumed-input collection. A fresh owner restores partial chunks after removal -of the former directory. Another replacement occurs after ABORT but before -the remaining account/data pieces arrive: delayed prepares remain rejected, -normal writes succeed and signed reads contain no aborted payload. The -controlled fixture clock governs application deadlines; SDK lease time stays -independent. -Three `tests/cellar_capacity.rs` cases verify physical reservation occupancy, -near-full unrelated writes, durable Throttled prepare rejection and reservation -release on COMMIT/ABORT. The original 144/32/2 MiB fixture budgets and item -sizes are retained. Complete SDK snapshots audit rollback and cleanup; signed -reads and exact rejected receipts survive fresh-owner recovery. Submitted SQL -or transport errors remain ambiguous. A forced physical SQLite FULL upload is -still unqualified on the public SDK and remains a separate migration gate. -The native serving target retains the large ordered numeric/IAM contract: -two initial ranges grow through automatic and explicit splits, one PK spans -actual cells, token replay survives the new routes, and signed calls cross a -separate native peer. Policy boundaries, revocation and encrypted credential -state survive deletion of both owners' directories. Exact-limit Query pages -now retain the owner's exhaustion proof when its range finishes the requested -span, avoiding a redundant terminal page without another cell RPC. Logical -target validation is available before admission; native authority stays in -the SDK. CI executes these contracts alongside the other migrated suites. -The data-range contract also retains activation fencing, durable transaction -visibility, sealed export/import, resumable split publication and child stream -lineage. Ordered fixture rows occupy both children. Complete non-owning SDK -snapshots audit all four data journals after shutdown. Deleting the old local -directory and observing native lease/sample expiry precede another node's -restoration of all seven state scopes at higher ownership epochs, including -the retired source and prepared child intent. Unfinished BEGINs remain pending -until explicit fenced recovery. The physical SQLite FULL upload fixture still -requires its separate SDK equivalent. -Five `tests/cellar_metadata.rs` cases retain table-class change history and -limits, deletion protection, capacity and logical byte counters, compacted -transaction replay and sparse LSI/GSI statistics. They run under 8/11/16-cell -resident budgets with signed SDK retries disabled. Fresh owners recover the -same account scope at higher epochs after local-file deletion and observed -native lease/sample departure. Real base/GSI splits during sampling reject -stale statistics before publication. Public tables retain two initial ordered -ranges; a second observed key is inserted before splitting a one-item source. -Three `tests/cellar_batches.rs` cases preserve per-child returned-update -conditions, stream ordinals, exact receipt replay and the 390,000-NUL escaped -image case. A real SDK peer forwards prepare batches, then recovers prepared -intents and their shared receipt after the original owner loses its lease. -Another owner retains independent COMMIT/ABORT states. Each handoff removes -the former local directory and verifies the same data scope at a higher epoch; -signed reads and ordinary writes prove final visibility and released locks. -Three `tests/cellar_images.rs` cases retain immutable saved images, bounded -wide-read fallback and exact large conditional-failure images. Original -380/384 KiB binary and NUL payloads remain intact. Signed cross-cell reads -preserve response order and absent positions. Canceled-token retry and terminal -replay preserve later writes. Separate SDK peers serve initial requests; -fresh owners recover the same data scopes at higher epochs after deletion of -all former local directories and observed lease/sample departure. Cellule -mailbox measurements remain a separate migration gate. -Two `tests/cellar_domain.rs` cases verify metadata-cache query counts and -generation fencing, and usage counters against the compiled data schema. -Directory fixtures exercise full-contract transfer, immutable initial-count -activation, 1,024-entry metadata copy/cutover and interrupted retirement. A -48-root/16-resident-cell case retains metadata fences and actual two-cell data -through fresh-owner recovery. Those development-bucket cases qualify metadata -behavior; they do not establish fleet capacity or 1,024 active data partitions. -Interrupted creation also recovers its original two-cell plan before ACTIVE, -then retains more than 64 MiB of data through another owner handoff. -Paged serving also runs over 65 distinct installed data cells, checking Scan -and TTL continuation through a 64-entry page boundary and the 16-table TTL -schedule bound. A separate metadata-only fixture grows a valid 256-entry plan -to 1,024 entries, splits its directory, and retains 1,025 pageable entries and -an unfinished leaf transfer on a fresh owner after deleting the old directory. -It does not claim 1,025 installed data cells. -Two table-lifecycle fixtures also exercise repeated deletion/recreation with -independent two-cell base/GSI directories under a 16-cell resident limit, and -recovery of a committed retirement whose reply was lost before acknowledgement. -Retired directory fences and old base items survive a fresh-owner handoff. -Creating new cells after handoff waits for the departed node's observed native -lease to expire and the native placement sample to observe departure; the -fixture does not assert immediate fresh-cell placement. -Signed coalescing qualification also retains 64 returned updates, eight -same-key increments and 32 concurrent two-item transactions, including a lost -prepare-batch reply and fresh-owner recovery. Registration batches preserve -caller transport/admission isolation, durable discovery and cancellation -semantics. One HTTP gateway bounds dispatch to 128 queued/submitted calls and -eight per target through response completion. This is local backpressure; -Cellar still owns cross-gateway admission, execution and durability. Mutations -retain one transport attempt. Development/test builds optimize LTX checksum -and compression dependencies so physical reservation traffic can meet the -same durability deadlines; no reservation or workload bounds were reduced. -Three native public-placement cases also reject Single selection, retain -four-range plans despite later creation-policy and tag changes, preserve atomic -reads within one range of a multi-range table, and split GSI ranges independently -of the base plan. Token replay, exact stream counts and signed index values -survive fresh-owner recovery after local-file deletion. SDK factory identities -and ownership rows prove higher epochs for the restored data/index scopes. -Native registration publication also holds actual account LTX provider PUTs -through caller cancellation. Later admission waits for acknowledgement; both -discovery rows survive native owner replacement without admitting BEGIN. -Four-way publication coalesces below four durable account commands. Scope and -batch rejection leave no partial row, while the maximal account/batch encoding -survives restore. These cases use the SDK's publication and ownership proofs. -Five native creation cases also retain partial GSI/base installation under -10/11-cell resident budgets, worker-only repair, concurrent deletion, immutable -initial counts and unpublished-directory retirement. Known pre-submission -admission failures establish the partial checkpoints; SDK ownership and actual -installed contracts remain authoritative. Signed values and index contents -survive fresh owners after local-file deletion. -Two native deletion cases retain the 160-item/160-LSI workload under an -eight-cell resident limit, with worker-only cleanup after owner replacement. -An internal account fixture retains bounded row cleanup; the public fixture -spans two data cells, verifies retained base/LSI history and a retired directory, -and recovers both SDK scopes at higher epochs. Stale deletion cannot affect the -replacement generation, whose signed contents survive another fresh owner. -Five native reclamation cases keep the 4/5/11-cell resident limits while SDK -samples show more owned cells than resident cells. They cover first-transaction -replay, repeated directory/data restoration, cold participants without account -activation and durable base/GSI split seals. Completed split replay issues no -historical-source ingress; signed data and exact index values survive fresh -owners after local-file deletion. SDK Sticky ownership replaces the old explicit -owner-release policy. The creation-time race pauses a real CreateTable retry, -observes destruction of both original data activations under ordinary SDK -pressure, then completes the same pending generation and restores the old data. -Those values and the new table write survive another fresh owner. -Six native coordinator-admission cases cover publication, hot reads and SDK owner claims. -Two hold actual coordinator LTX payload PUTs. -Concurrent replay shares one held cold publication after both requests reach -SDK ingress; independent coordinator scopes overlap before release. Account -registration survives owner replacement, and replay retains a newer signed -value after local-file deletion. These cases keep four caller threads and the -original 8/16-cell budgets and 150 ms/one-second observation bounds. A cold -claim reuses one absent-owner read before its conditional PUT. A delayed SDK -Create-mode claim receives a clean provider rejection after a real peer wins; -the peer's owner record, coordinator sequence and newer values remain intact. -Exact seed items and token replay survive another owner after both old local -directories are removed. Both contenders are ready with free capacity before the -timed claim, keeping membership startup outside that operation window. Registered -admission issues four canonical SDK queries with no hot managed-cell provider -GETs or account mutations. Fresh provisioners still issue that query; fresh SDK -owners recover the same registration and coordinator state at higher epochs. -The pending-slot case pins four SDK callbacks under a five-cell limit. Two -coordinators reach SDK owner lookup, but the second creation CAS stays blocked -while the first accepted CAS is held, including after its caller is cancelled. -Once the first CAS settles at the provider, the queued claim progresses. The -cancelled caller invokes no application handler; Cellar may retain dormant -ownership from its accepted claim. Cancellation alone does not establish rollback. -Registration and exact values survive another fresh owner. The shared activation -observer is also checked by a fresh run of all five reclamation cases. -No ownership record or CAS payload is fabricated. -Two native ownership-race cases also retain the original two caller threads, -eight-cell limits, two data ranges and ten-second bounds. They hold actual SDK -owner lookup results or conditional PUTs while another SDK wins, and drain a live -owner after a lookup. The signed reads preserve exact items and the winner's -fence; later values survive another owner after local-file deletion. Cellar's -public handoff supplies the release/durability proof, and a departing SDK may -transfer directly to its successor. No fixture rewrites owner or lease records. -The latest complete fixture run passed 155 focused native tests across 24 -SHA-256-verified executables with normal parallelism, zero failed/ignored. This -includes all native discovery, creation, deletion, reclamation and admission -ports and the stale-owner/competing-claim regressions. All nine explicitly -executed RustFS process cases also pass on the rebuilt server, zero failed/ignored, -with its SHA-256 unchanged throughout the run. Current dependency pins are recorded -in [Cargo.toml](../Cargo.toml). -Caller refusal is checked before SDK submission; a native HTTP error remains -ambiguous. The caller/provider faults qualify their stated local contracts; -they do not establish a fleet fencing or follower-log matrix. The separate process fixture supplies the -specific follower recovery evidence described above. The two unported Cellule -test targets were removed at the user's request. All-target build and strict -Clippy gates remain enabled and pass; native and process coverage is retained. -The removed contracts remain unqualified and are listed with their source -revision in [the retirement inventory](retired-tests.md). Further API, -index and fleet fault qualification remains separate from this SDK migration. +> **Recorded evidence:** The following tables preserve the cases and measurements +> from the [original report](https://github.com/crabbuild/beyonddb/blob/a1eb96a0e87d4473f1bec89c0f3b4f2b02a21afb/docs/cellar-server.md). +> Counts, budgets and timing belong to those runs. Earlier-runtime comparisons +> are not additional Cellar coverage; recorded build/test successes are not the +> latest CI result. Configuration and operation follow the reference above. + +| Recorded run | Result and scope | +| --- | --- | +| Focused native run | **155 tests**, **24 SHA-256-verified executables**, normal parallelism, zero failed/ignored | +| RustFS process run | **9 explicitly executed cases**, zero failed/ignored; rebuilt server SHA-256 unchanged throughout | +| Build gates in that record | All-target build and strict Clippy passed; current dependency pins are in [Cargo.toml](../Cargo.toml) | +| Fault model | Caller refusal precedes SDK submission; native HTTP failure remains ambiguous. Caller/provider faults qualify their stated local contracts, not a complete fleet fault matrix | + +Expand a domain to inspect its recorded assertions, fixture sizes and limits. + +
+Public SDK, process recovery and private peers + +| Scope | Recorded evidence | +| --- | --- | +| [Signed SDK](../tests/cellar_sdk.rs) | Real-process typed/large escaped values, conditional writes, token replay, Query pages and restart; Put/Get/Update/Delete, transaction replay and query/restart passed. Local qualification only. | +| [Bucket-backed process cases](../tests/cellar_process.rs) | Nine cases covered coordinator history, GSI projection recovery, large transactional reads on a fresh owner, metadata caching, segmented LSI scans, signed Streams/TTL recovery with retained generations after deletion/recreation, native capacity and authenticated peer identity, broad SDK restart, and follower recovery with bundle/LTX publication withheld. | +| Broad SDK scenario | Batch, transaction, index and large-value checks crossed ordered ranges through hard and graceful restarts, followed by table deletion/recreation. The account-domain fixture also passed SDK handoff to a second node. | +| [Native peer sockets](../tests/cellar_peer.rs) | Opaque TCP cut with route caching disabled and enabled; numeric pagination, two-range reads and **50 concurrent increments** without AWS client retries. Removing the former directory before lease-expiry recovery preserved values and advanced owner epochs. | +| Credential-owner cut | Authentication failed closed while healthy local data remained readable, then recovered after expiry. Unknown keys created no credential Cells; revocation remained fresh. | +| Query pacing policy | Tests covered read retry hints, deadline failure and mutation non-retry. Bursts passed native per-Cell admission without caching credential, authorization or item values; command attempts remained single-shot. | + +
+ +
+Transactions, coordinator discovery and admission failures + +| Scope | Recorded evidence | +| --- | --- | +| [Transaction suite](../tests/cellar_transactions.rs) | **22 cases** covered token retention, cross-Cell write skew, bounded resume, delayed resolution, shared read snapshots, lost phases, caller admission refusal, competing drivers, fair background discovery, acknowledged read-result cleanup, TTL lock races, ordered range-read barriers and prepared-COMMIT/mixed-participant handoff through native SDK ingress. | +| Coordinator residency | History across **24 shards** under a **16-Cell** resident cap; partial read recovery on a fresh owner; discovery of a later BEGIN after settled passes; four table ranges admitted after **32 coordinator shards** exceeded residency. | +| Split install refusal | Caller-injected child-install refusal retained the same pending split until capacity returned. Proven pre-submission Capacity mapped to LimitExceeded; native HTTP failure remained ambiguous. | +| Recovery admission | BEGIN/COMMIT/ABORT at startup and while serving; progress past failed records/shards; retry without overwriting newer values; discovery deferred beyond the initial pending boundary. A server regression failed before maintenance, then succeeded on retry. | +| Fault boundary | These admission cases used an explicit caller policy **before SDK submission**, not physical network faults. | + +
+ +
+Lease authority, follower proof and clean shutdown + +| Scope | Recorded evidence | +| --- | --- | +| [Node log authority](../tests/node_log_authority.rs) | **8 cases** exercised the native single lease writer and folded log through real storage faults: normal/slow renewal, applied CAS reply lost before delivery, delayed/stalled ordinary reads, graceful sealing, natural expiry after sustained renewal failure and closed ingress after fencing. | +| Expired follower | An expired unavailable follower could not prove a new write while both bundle and LTX publication were withheld. The same signed request completed when publication resumed. Fixtures never edited authority records or required epoch rotation from TTL expiry alone. | +| Additional shutdown cases | **4 cases** covered immediate certified same-node restart, incomplete drain with unpublished signed work, a real provider CAS committing after node close, and bounded read/write storage stalls. | +| Stop proof | The clean report matched the actual local certificate; incomplete stops could not grant a certificate and had to close ingress. After ordinary expiry, signed reads preserved acknowledged rows in both data ranges. | +| Historical distinction | These tests do not retain the earlier runtime's separate application-owned session-retirement API. Its removed contracts remain unqualified. | + +
+ +
+Indexes, split state and ordered serving + +| Scope | Recorded evidence | +| --- | --- | +| [Index suite](../tests/cellar_indexes.rs) | **6 cases** retained local/global index mutation, transaction, split, replay and restart contracts. LSI splits kept PK collections together and logical Query cursors valid; GSI replay retained large chunked images, versions and tombstones. | +| Idle GSI recovery | Maintenance visited every published GSI range through native ingress even with an empty base outbox. Idle/empty ranges recovered on a new owner; fixtures observed SDK factories and higher epochs after deleting former directories, before user index reads. | +| Backfill | Historical delivery barrier, prepare reservations and unfinished scan policy survived owner replacement and range splits. | +| GSI transfer | **70 wide-key records**, binary images and tombstones survived partial-import and post-publication replacements. Equal index PK/SK entries spanned both children via the full base-key suffix; signed forward/reverse queries retained all rows and complete continuation keys. | +| [Ordered numeric/IAM serving](../tests/cellar_serving.rs) | Two initial ranges grew through automatic and explicit splits; one PK spanned actual Cells; token replay survived new routes; signed calls crossed a separate native peer. Policy boundaries, revocation and encrypted credential state survived deletion of both owners' directories. | +| Query exhaustion | Exact-limit Query pages retained the owner's exhaustion proof when the range finished the requested span, avoiding a redundant terminal page without another Cell RPC. Logical-target validation preceded admission; native authority stayed in the SDK. CI included these contracts with the migrated suites. | +| Data-range state | Activation fencing, durable transaction visibility, sealed export/import, resumable split publication and child stream lineage; ordered rows occupied both children. Complete non-owning SDK snapshots audited **all four data journals** after shutdown. | +| Data-range recovery | Old directory deletion and observed native lease/sample expiry preceded restoration of **all seven state scopes** at higher epochs, including the retired source and prepared child intent. Unfinished BEGINs remained pending until explicit fenced recovery. | + +
+ +
+Large payloads, reservations, batches and saved images + +| Scope | Recorded evidence | +| --- | --- | +| Multipart payloads | **2 cases** retained escaped inputs larger than one command, immutable/independent upload identities, exact receipt replay and durable payload recovery. Complete SDK snapshots verified expired staging cleanup and consumed-input collection. | +| Multipart owner loss | A fresh owner restored partial chunks after former-directory removal. Another replacement occurred after ABORT but before remaining account/data pieces arrived: delayed prepares stayed rejected, ordinary writes succeeded, and signed reads contained no aborted payload. Fixture time governed application deadlines independently of SDK lease time. | +| [Capacity](../tests/cellar_capacity.rs) | **3 cases** verified physical reservation occupancy, near-full unrelated writes, durable Throttled prepare rejection and reservation release on COMMIT/ABORT. Original **144/32/2 MiB** fixture budgets and item sizes were retained. | +| Capacity recovery boundary | Complete SDK snapshots audited rollback/cleanup; signed reads and exact rejected receipts survived fresh-owner recovery. Submitted SQL/transport errors remained ambiguous. Forced physical SQLite FULL upload still lacked a public-SDK equivalent and remained unqualified. | +| [Batches](../tests/cellar_batches.rs) | **3 cases** preserved per-child returned-update conditions, stream ordinals, exact receipt replay and the **390,000-NUL** escaped image. A real SDK peer forwarded prepare batches, then recovered prepared intents and their shared receipt after the original owner's lease loss. | +| Batch handoff | Another owner retained independent COMMIT/ABORT states. Every handoff removed the former directory and verified the same data scope at a higher epoch; signed reads/writes proved visibility and released locks. | +| [Saved images](../tests/cellar_images.rs) | **3 cases** retained immutable images, bounded wide-read fallback and exact large conditional-failure images with original **380/384 KiB** binary/NUL payloads. Signed cross-Cell reads preserved response order and absent positions; canceled-token retry and terminal replay preserved later writes. | +| Image recovery boundary | Separate SDK peers served initial requests. Fresh owners recovered identical data scopes at higher epochs after all former directories were removed and lease/sample departure was observed. Earlier-runtime mailbox measurements remained a separate unqualified gate. | + +
+ +
+Metadata, directories, paging and table lifecycle + +| Scope | Recorded evidence | +| --- | --- | +| [Metadata](../tests/cellar_metadata.rs) | **5 cases** retained table-class history/limits, deletion protection, capacity/logical byte counters, compacted transaction replay and sparse LSI/GSI statistics under **8/11/16-Cell** resident budgets with signed SDK retries disabled. | +| Metadata recovery | Same account scope, higher owner epochs after local-file deletion and observed lease/sample departure. Real base/GSI splits during sampling rejected stale statistics before publication. Public tables kept two initial ranges; a second observed key was inserted before splitting a one-item source. | +| [Account domain](../tests/cellar_domain.rs) | **2 cases** checked metadata-cache query counts, generation fencing and usage counters against the compiled data schema. | +| [Directory transfer](../tests/cellar_directories.rs) | Full-contract transfer, immutable initial-count activation, **1,024-entry** metadata copy/cutover and interrupted retirement. A **48-root / 16-resident-Cell** case preserved metadata fences and actual two-Cell data through fresh-owner recovery. Development-bucket evidence does not establish fleet capacity or 1,024 active data partitions. | +| Interrupted creation | Original two-Cell plan recovered before ACTIVE, then more than **64 MiB** of data survived another owner handoff. | +| Scan and TTL paging | **65 distinct installed data Cells** crossed the **64-entry** page boundary and the **16-table** TTL schedule bound. | +| Metadata-only paging | A valid **256-entry** plan grew to **1,024** entries, split its directory and retained **1,025 pageable entries** plus unfinished leaf transfer on a fresh owner after old-directory deletion. This did not claim 1,025 installed data Cells. | +| Lifecycle recovery | **2 fixtures** exercised repeated deletion/recreation with independent two-Cell base/GSI directories under a **16-Cell** resident cap, and recovery of committed retirement with a lost reply. Retired-directory fences and old base items survived fresh-owner handoff. | +| Fresh placement boundary | New Cell creation waited for observed native lease expiry and placement-sample departure; immediate fresh-Cell placement was not asserted. | + +
+ +
+Coalescing, placement and durable registration + +| Scope | Recorded evidence | +| --- | --- | +| [Signed coalescing](../tests/cellar_coalescing.rs) | **64 returned updates**, **8 same-key increments** and **32 concurrent two-item transactions**, including a lost prepare-batch reply and fresh-owner recovery. Registration batches preserved caller transport/admission isolation, durable discovery and cancellation semantics. | +| Gateway and build conditions | One gateway bounded **128** queued/submitted calls and **8** per target through response completion; Cellar retained cross-gateway admission/execution/durability. Mutations kept one attempt. Development/test builds optimized LTX checksum/compression dependencies to meet unchanged durability deadlines; reservation/workload bounds were not reduced. | +| [Public placement](../tests/cellar_placement.rs) | **3 cases** rejected Single selection, retained four-range plans across later creation-policy/tag changes, preserved atomic reads within one range of a multi-range table, and split GSI ranges independently of the base plan. | +| Placement restoration | Token replay, exact stream counts and signed index values survived fresh-owner recovery after local-file deletion. SDK factory identities and ownership rows proved higher epochs for restored data/index scopes. | +| Registration publication | Actual account LTX provider PUTs stayed held through caller cancellation. Later admission waited for acknowledgement; both discovery rows survived owner replacement without admitting BEGIN. | +| Registration batching | Four-way publication coalesced below four durable account commands. Scope/batch rejection left no partial row; maximal account/batch encoding survived restore. Cases used SDK publication and ownership proofs. | + +
+ +
+Creation, deletion and residency reclamation + +| Scope | Recorded evidence | +| --- | --- | +| [Creation](../tests/cellar_creation.rs) | **5 cases** retained partial GSI/base installation under **10/11-Cell** resident budgets, worker-only repair, concurrent deletion, immutable initial counts and unpublished-directory retirement. Known pre-submission admission failures established partial checkpoints; SDK ownership and installed contracts remained authoritative. | +| Creation restoration | Signed values/index contents survived fresh owners after local-file deletion. | +| [Deletion](../tests/cellar_deletion.rs) | **2 cases** retained a **160-item / 160-LSI** workload under an **8-Cell** cap, with worker-only cleanup after owner replacement. An internal account fixture retained bounded row cleanup. | +| Deletion generation fence | The public fixture spanned two data Cells, retained base/LSI history and a retired directory, and recovered both scopes at higher epochs. Stale deletion could not affect the replacement generation; its signed contents survived another fresh owner. | +| [Reclamation](../tests/cellar_reclamation.rs) | **5 cases** kept **4/5/11-Cell** caps while SDK samples showed more owned than resident Cells: first-transaction replay, repeated directory/data restoration, cold participants without account activation and durable base/GSI split seals. | +| Split replay and ownership | Completed split replay issued no historical-source ingress; signed data and exact index values survived fresh owners after local-file deletion. SDK Sticky ownership replaced explicit application owner release. | +| Creation-time race | Paused a real CreateTable retry, observed destruction of both original data activations under ordinary SDK pressure, then completed the same pending generation and restored old data. Old values and the new table write survived another fresh owner. | + +
+ +
+Coordinator admission, cancellation and competing owners + +| Scope | Recorded evidence | +| --- | --- | +| [Coordinator admission](../tests/cellar_admission.rs) | **6 cases** covered publication, hot reads and SDK owner claims. Two held actual coordinator LTX payload PUTs. Concurrent replay shared one held cold publication after both requests reached ingress; independent coordinator scopes overlapped before release. | +| Admission budgets | **4 caller threads**, original **8/16-Cell** caps and **150 ms / 1-second** observation bounds. Account registration survived replacement; replay preserved a newer signed value after local-file deletion. | +| Cold claim and lost race | Cold claim reused one absent-owner read before conditional PUT. A delayed SDK Create-mode claim received clean provider rejection after a real peer won; the winner's owner record, coordinator sequence and newer values stayed intact. | +| Claim recovery | Exact seed items and token replay survived another owner after both former directories were removed. Both contenders were ready with free capacity before timing began, excluding membership startup from the operation window. | +| Hot registration reads | Registered admission issued **4 canonical SDK queries**, with no hot managed-Cell provider GETs or account mutations. Fresh provisioners still queried; fresh owners recovered identical registration/coordinator state at higher epochs. | +| Pending-slot case | **4 SDK callbacks** were pinned under a **5-Cell** cap. Two coordinators reached owner lookup, but the second creation CAS remained blocked while the first accepted CAS was held, even after caller cancellation. Provider settlement released the queued claim. | +| Cancellation boundary | The canceled caller invoked no application handler; Cellar could retain dormant ownership from the accepted claim. Cancellation alone did not prove rollback. Registration/exact values survived another fresh owner; all **5 reclamation cases** rechecked the shared activation observer. No owner record or CAS payload was fabricated. | +| [Ownership races](../tests/cellar_peer/ownership_race.rs) | **2 cases**, original **2 caller threads**, **8-Cell** caps, **2 data ranges** and **10-second** bounds. Held actual SDK owner lookup results or conditional PUTs while another SDK won; also drained a live owner after lookup. | +| Race outcome | Signed reads preserved exact items and the winner's fence; later values survived another owner after local-file deletion. Public Cellar handoff supplied release/durability proof and could transfer directly to the successor. Fixtures never rewrote owner/lease records. | + +
+ +The two unported earlier-runtime test targets were removed; their contracts remain +unqualified and retain source revisions in [the retirement inventory](retired-tests.md). +Further API, index and fleet fault qualification remains separate from the SDK migration. diff --git a/docs/cross-cell-transactions.md b/docs/cross-cell-transactions.md index c4b4a9b..182ca56 100644 --- a/docs/cross-cell-transactions.md +++ b/docs/cross-cell-transactions.md @@ -1,44 +1,338 @@ # Cross-Cell transaction protocol +A coordinated transaction has one durable decision and a fixed participant set. +Participants lock keys and save proposed or read images; the coordinator records +COMMIT or ABORT; resolution finishes the original work after retries or owner loss. + +| I need to… | Start here | +| --- | --- | +| Choose the local or coordinated path | [Execution paths](#current-cellar-reference) | +| Follow a multi-Cell write | [Coordinated write path](#coordinated-write-path) | +| Understand what readers can observe | [Read visibility and saved images](#read-visibility-and-saved-images) | +| Handle a timeout, rejection or owner loss | [Retry and capacity rules](#retry-and-capacity-rules) and [recovery](#recovery-and-stable-identities) | +| Size payloads and prepared work | [Payload and capacity boundaries](#payload-and-capacity-boundaries) | +| Try a signed transaction | [Request example](#request-example) | +| Inspect safety arguments and reproductions | [Detailed design and recorded evidence](#detailed-design-and-recorded-evidence) | + ## Current Cellar reference -BeyondDB uses durable prepare/decision/resolution records and immutable participant -identities. The original protocol explanations, safety arguments, capacity -analysis, regression cases and reproduction steps remain below. +Use this first part for the checked-in Cellar adapter. The expandable sections +retain the full original protocol analysis, numerical costs, failure reproductions +and revision-specific evidence. They describe successive implementations; their +older scheduling, cleanup and runtime claims do not override this reference. + +| Request placement | Execution path | Boundary | +| --- | --- | --- | +| Write, one participant, no client token | One atomic local command | Every operation must resolve to that participant | +| Write with a client token, or multiple participants | Durable coordinator + prepare/decision/resolution | Account-scoped identity and original participant targets survive routing changes | +| Read, one participant, encoded reply fits | One atomic Cell query | Check locks and read items in one serialized callback | +| Read spanning participants, or local reply too large | Durable coordinator + shared locks + saved images | Assemble immutable images through bounded queries | +| GSI view of a committed base write | Asynchronous projection | A base transaction does not create an atomic GSI view | + +Public tables span **multiple ordered range Cells**. The `single` creation option +is rejected; a request can still happen to touch only one participant. Several +keys in the same range share one prepare and one resolution. See the +[local paths](../src/backend/data.rs), [driver](../src/backend/transaction.rs) and +[read adapter](../src/backend/transaction_read.rs). + +## Coordinated write path + +The example assumes three keys route to three participant Cells. The coordinator +is another Cell selected independently of the current table routes. + +![A coordinated write publishes BEGIN, prepares three Cells, records one decision and resolves every original participant](../diagram/operations/transaction-flow.svg) + +| Phase | Durable work | Required guarantee | +| --- | --- | --- | +| BEGIN | Transaction identity, fingerprint, original participants/epochs and immutable operations | Publish before preparing any participant; a digest alone cannot reconstruct a lost request | +| PREPARE | Evaluate conditions; capture images; acquire locks and apply reservations | These changes share one participant command; a failed prepare cannot leave partial state | +| DECIDE | Record durable prepare evidence and one terminal decision | COMMIT requires every prepare receipt; COMMIT and ABORT cannot change to their opposite | +| RESOLVE | Apply or discard, release locks/claims, record terminal markers and receipts | Resolve idempotently against the original participants; acknowledge completion before success/cancellation | + +The successful current driver combines prepare evidence and COMMIT in one +`CommitPreparedTransaction` command. Historical sections also describe earlier +per-receipt commands and sequential prepares; those command-count formulas are +recorded costs, not the current execution plan. + +```mermaid +sequenceDiagram + participant Client as AWS client + participant Driver as BeyondDB driver + participant Coord as Coordinator Cell + participant Cells as Original participant Cells + Client->>Driver: TransactWriteItems + optional client token + Driver->>Coord: Durable BEGIN + immutable targets and operations + par Prepare independent participants + Driver->>Cells: Evaluate, lock, stage and reserve + Cells-->>Driver: Durable prepare receipts + end + Driver->>Coord: CommitPreparedTransaction(receipts) + Coord-->>Driver: Durable COMMIT, or authoritative competing decision + loop Resolve original participants, bounded concurrency + Driver->>Cells: Apply or discard from the durable decision + Cells-->>Driver: Durable resolution receipt + Driver->>Coord: Record resolution progress + end + Driver-->>Client: Success or ordered cancellation after resolution +``` + +The sequence shows the all-prepared path. A definitive condition, conflict or +admission failure requests ABORT while the coordinator is still BEGIN; an already +published terminal decision wins. ABORT must resolve **every original participant**, +including those whose prepare has not arrived. -| Current Cellar boundary | Contract | +### What commits together + +| Participant action | One local atomic boundary | | --- | --- | -| Public table layout | Multiple ordered data cells; `single` creation is rejected | -| Local fast paths | Eligible operations can still resolve to one participant | -| Retry | Submitted errors are ambiguous; preserve mutation/token identity | -| Decision | Durable COMMIT/ABORT and idempotent participant resolution | -| Recovery | Resume immutable work; a fresh BEGIN is not aborted merely because startup ran | +| Prepare | Condition evaluation, proposed/read images, locks, phase record and physical apply reservation | +| COMMIT resolution | Base items, local indexes, TTL metadata, GSI projection journal, stream records, terminal marker and lock/claim release | +| ABORT resolution | Intent/lock cleanup and ABORTED tombstone; no committed write/stream effect | +| Delayed prepare after ABORT | Reject using the retained terminal marker; do not reacquire locks | + +`ConditionCheck` keys need locks too. A preflight read cannot prevent a concurrent +change between condition evaluation and commit. The [write-skew reproduction](#condition-checks-must-prevent-write-skew) +retains the concrete two-transaction schedule and its safety argument. + +## Read visibility and saved images + +![Ordinary reads respect unresolved write intents; coordinated transactional reads capture locked images before release and acknowledge them before cleanup](../diagram/operations/transaction-reads.svg) + +| Reader | What it may observe | What it must not do | +| --- | --- | --- | +| Individual Get/Put/Update/Delete | Operate through the common item-lock boundary | Bypass a prepared intent | +| Read behind a durable COMMIT whose apply is incomplete | Resolve and re-read, or return a retryable error | Return the old image from a still-locked participant | +| Coordinated `TransactGetItems` | Immutable existing **or absent** images captured with shared locks | Re-read live items after releasing those locks | +| Two separate `GetItem` calls | Calls may straddle a transaction | Claim an atomic multi-key view | +| Query, Scan and batch reads | Per-item read-committed barriers | Claim one snapshot for the whole response, or miss prepared creates/deletes absent from live rows | +| GSI read | An eventually consistent projected view | Infer atomic visibility from the base transaction | + +Shared read locks coexist and exclude writers. Once the last shared prepare +succeeds, all captured images coexist while the keys remain locked. Resolution +releases those locks; saved images preserve the response during assembly. +An absent item needs a lock just as an existing item does. + +### Saved read-image lifetime + +| Step | Who can advance it | Durable boundary | +| --- | --- | --- | +| Capture | Participant prepare | Image and shared lock commit together | +| Assemble | Initiating read adapter | Fetch saved images from original targets and restore every response position | +| Acknowledge consumption | Initiating reader only | `BeginReadResultRelease` requires committed, fully resolved participants | +| Delete and compact | Driver or recovery | Delete saved images idempotently, then record release and compact the operation mapping | +| Interrupted before acknowledgement | Retain for the reader | Cancellation alone does not authorize image deletion | +| Interrupted after acknowledgement | Recovery finishes cleanup | Original targets remain discoverable until every cleanup receipt is durable | + +This is an internal consumption acknowledgement after complete assembly, not proof +that the HTTP client received the response. Cleanup failure after that durable +acknowledgement leaves recoverable work; acknowledgement failure returns an error +and starts no deletion. Terminal identity/digest markers remain to reject delayed +phase messages. See the [cleanup protocol and regressions](#acknowledged-transactional-read-cleanup). ## Retry and capacity rules -| Observation | What it proves | +Classify evidence before deciding what can be retried. + +| Observation | What it proves | Next step | +| --- | --- | --- | +| Explicit caller refusal before SDK submission | That attempt did not start | Follow the bounded admission policy; retain transaction authority | +| Durable rejected receipt | The recorded rejection for that mutation | Use the rejection contract; do not confuse it with a lost reply | +| Complete committed receipt | Result covered by the configured Cellar durability gate | Use the durable outcome | +| HTTP error, timeout, lost body or driver cancellation after submission | The outcome may be ambiguous | Resolve/query durable state; do not infer rollback or assign a fresh identity to the same uncertain invocation | +| Matching client token and request fingerprint | Replay of the original transaction | Use its coordinator and immutable participants | +| Same token with changed request content | Mismatch | Reject the changed request | + +Token lookup occurs before backend data routing; ExtendDB's frontend metadata +validation may already have run. A matching replay cannot retarget an admitted +transaction to new range Cells after a split. Preserve invocation identity/input +when retrying the same uncertain command; explicit durable fallback outcomes have +their own opcode/identity rules in [the transport](../src/backend/transaction_transport.rs). + +| Token / history boundary | Rule | +| --- | --- | +| Successful token replay | Ten-minute window starts at durable completion; repeated resolution receipts do not move completion time | +| Fully resolved ABORT | Unlink the token only after all abort resolutions, allowing a corrected condition or released lock to be retried | +| Expired token slot | May admit a new transaction; old decision and participant records still fence late messages | +| Unresolved record | Retain regardless of token age | +| History reclamation | Token expiry alone is insufficient; safe full retirement remains open | + +## Recovery and stable identities + +![BEGIN resumes immutable work, COMMIT applies it and ABORT discards it; recovery never rewrites participants to current routes](../diagram/operations/transaction-recovery.svg) + +| Failure point | Authoritative evidence | Recovery rule | +| --- | --- | --- | +| Lost BEGIN or prepare reply | Coordinator and participant state, receipts and immutable payload | Establish what published; a sent prepare is not a prepare receipt | +| Lost decision reply | Original coordinator's durable state | Read the decision before resolving; never invent the opposing decision | +| Driver loss after COMMIT | COMMIT plus participant PREPARED/COMMITTED states | Complete idempotent apply and record receipts | +| ABORT racing a delayed prepare | ABORT and participant tombstones | Resolve unprepared participants too; later prepare must reject | +| Fresh BEGIN at startup | Immutable coordinator input | Resume; startup alone does not authorize abort | +| Abandoned BEGIN aged ten minutes | Atomic age check and ABORT command | The coordinator decides; elapsed time alone cannot discard participant state | +| Unavailable coordinator | Prepared locks with unknown decision | Keep affected keys fenced and fail retryably while recovering the owner | +| Split or new table route | Original table generation, participant target and epoch | Do not rewrite an admitted participant set; sealing must respect outstanding locks | + +Current startup and serving recovery use configured accounts and bounded pending +pages. Initial recovery must succeed before public serving; a healthy participant +can progress while another resolution fails. See [server readiness](cellar-server.md#starting-and-stopping) +and the [recovery implementation](../src/backend/recovery.rs). + +**Trust boundary:** the authenticated internal driver supplies the durable +coordinator decision to participants. Participants do not independently verify a +decision certificate. Crash/retry safety does not establish tolerance of a faulty +or compromised driver. Cellar supplies native ownership fencing; a BeyondDB range +epoch does not replace it. + +## Payload and capacity boundaries + +![Temporary uploads, durable phase payloads and terminal evidence have different retention and cleanup rules](../diagram/operations/transaction-payloads.svg) + +| Layer | Current bound / behavior | What it does not bound | +| --- | --- | --- | +| Public transaction validation | ExtendDB validates the request, including unique item count and request-side size estimate | Aggregate evaluated Update images still need cloud-contract qualification | +| Cell operation codec | Global ceiling **4 MiB + 64 KiB**, with smaller operation-specific envelopes | Whether a raw request fits after JSON/base64 expansion; total resident memory | +| Small coordinator resume query | Up to **32 KiB** saved payload; complete encoded reply within **64 KiB** | Large/multi-chunk requests, which retain bounded chunk retrieval | +| Durable phase SQL payload | **768-KiB** chunks; owning phase and chunk counts commit together | Full wire payload, WAL, physical apply or retained history | +| Prepared participant | Staging, index/stream intents and physical apply reservations count toward capacity | Request bytes alone cannot size the reservation | +| Coalesced prepare publication | Current host batcher forms at most **2 prepares** per publication | Per-caller identity, admission or result isolation must still hold | + +| Current scheduling window | Source | +| --- | --- | +| Up to **8** independent participant prepares | [transaction driver](../src/backend/transaction.rs) | +| Up to **4** terminal participant resolutions | [recovery resolver](../src/backend/recovery.rs) | +| **2** coalesced prepares; **2-ms** batching window | [prepare batcher](../src/backend/prepare_batch.rs) | + +Do not reclaim an unresolved participant's reservation on timeout. Coordinator +capacity must also permit decision and receipt publication. Write payload +compaction and acknowledged read cleanup reduce some retained bytes; abandoned +readers, terminal records, command receipts and object-store history remain +separate bounds. Detailed [payload](#bounded-sql-payloads) and +[page-reservation](#durable-page-admission-for-participant-resolution) analyses follow below. + +## Request example + +Assume two ACTIVE tables, `Accounts` and `Transfers`, each with string partition +key `pk`. Seed `Accounts/A` and `Accounts/B` with numeric `balance` attributes. +Save this as `transfer.json`; conditions are evaluated during the participant +command that acquires the locks, not by a separate balance read. + +```json +{ + "ClientRequestToken": "transfer-example-001", + "TransactItems": [ + { + "Update": { + "TableName": "Accounts", + "Key": {"pk": {"S": "A"}}, + "UpdateExpression": "SET #b = #b - :amount", + "ConditionExpression": "#b >= :amount", + "ExpressionAttributeNames": {"#b": "balance"}, + "ExpressionAttributeValues": {":amount": {"N": "10"}} + } + }, + { + "Update": { + "TableName": "Accounts", + "Key": {"pk": {"S": "B"}}, + "UpdateExpression": "SET #b = #b + :amount", + "ExpressionAttributeNames": {"#b": "balance"}, + "ExpressionAttributeValues": {":amount": {"N": "10"}} + } + }, + { + "Put": { + "TableName": "Transfers", + "Item": {"pk": {"S": "transfer-example-001"}, "amount": {"N": "10"}}, + "ConditionExpression": "attribute_not_exists(pk)" + } + } + ] +} +``` + +Use a client shell configured with the bootstrapped DynamoDB credentials. This +command assumes the [local server example](cellar-server.md#configuration): + +```sh +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 \ + dynamodb transact-write-items --cli-input-json file://transfer.json +``` + +Retry the unchanged file with the same token to resolve/replay that transaction +within its replay window. A new transfer needs its own token and transfer-item +key. Logical keys do not guarantee three distinct Cells; the diagram's placement +is illustrative. This JSON/CLI example is explanatory; the signed fixtures below +supply execution and recovery evidence. + +For an atomic view of the three items, save this as `transfer-read.json`: + +```json +{ + "TransactItems": [ + {"Get": {"TableName": "Accounts", "Key": {"pk": {"S": "A"}}}}, + {"Get": {"TableName": "Accounts", "Key": {"pk": {"S": "B"}}}}, + {"Get": {"TableName": "Transfers", "Key": {"pk": {"S": "transfer-example-001"}}}} + ] +} +``` + +```sh +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 \ + dynamodb transact-get-items --cli-input-json file://transfer-read.json +``` + +Read results follow request order. This read has its own capture point; it can +observe later committed writes and does not replay the earlier write's token. + +## Qualification and source map + +| Area | Current entry points / evidence | | --- | --- | -| Explicit caller-side refusal before SDK submission | That attempt was not started | -| Durable rejected receipt | The recorded rejection for that mutation | -| Complete committed receipt | Result covered by the configured durability gate | -| HTTP error, timeout, lost body or driver cancellation | Outcome may be ambiguous after submission | - -Retain the original mutation/client-token identity and request content when -resolving uncertainty. Do not turn a submitted error into a fresh mutation or -assume it rolled back. Token mismatch must reject changed content; retries may -not reroute an existing decision to a different participant set. - -Prepare budgets include staged data, index/stream intent and physical apply -reservations. Encoded request size alone does not bound publication or memory -cost. Multipart uploads and replies are bounded separately. Coalescing must -preserve each caller's identity, admission and result; current batching limits -are in [prepare_batch.rs](../src/backend/prepare_batch.rs). +| Admission and stable identity | [admission](../src/backend/admission.rs), [token handling](../src/transaction_coordinator/token.rs) | +| Prepare/decision | [driver](../src/backend/transaction.rs), [coordinator phases](../src/transaction_coordinator/phase.rs), [participant](../src/participant.rs) | +| Read capture and cleanup | [read adapter](../src/backend/transaction_read.rs), [durable release](../src/transaction_coordinator/read_release.rs) | +| Recovery and discovery | [resolver](../src/backend/recovery.rs), [account maintenance](../src/cellar/server.rs) | +| Native signed transaction and batch regressions | [transaction suite](../tests/cellar_transactions.rs), [coalescing suite](../tests/cellar_coalescing.rs) | +| Process evidence and commands | [server qualification](cellar-server.md#qualification-still-required), [qualification workflow](../.github/workflows/qualification.yml) | + +```sh +cargo test --locked --test cellar_transactions -- --test-threads=1 +cargo test --locked --test cellar_coalescing -- --test-threads=1 +``` + +The fixtures cover selected schedules and restart boundaries. Full API +compatibility, serializable-history checking, bounded fleet recovery, sustained +capacity and safe history retirement remain qualification work. Historical test +names/pass counts below do not establish current Cellar coverage; consult the +[retirement inventory](retired-tests.md) for removed gates. ## Detailed design and recorded evidence -> **Revision scope:** The ordered range adapter and Cellar SDK are current. Runtime-specific admission mechanics, old transport tests, numerical costs and pass counts below belong to their recorded revisions; they are not new Cellar qualification. The original protocol invariants and failure analyses remain available in full. +> **Revision scope:** The complete original analysis remains below, grouped by topic. +> Runtime-specific admission mechanics, older scheduling, transport tests, numerical +> costs and pass counts belong to their recorded revisions. They are not new Cellar +> qualification. Use the [current reference](#current-cellar-reference) above for +> local fast paths, concurrent prepare, combined COMMIT and acknowledged read cleanup. > Snapshot: [original detailed document](https://github.com/crabbuild/beyonddb/blob/a1eb96a0e87d4473f1bec89c0f3b4f2b02a21afb/docs/cross-cell-transactions.md). +Expand a topic for the full safety arguments, failure schedules, fixture sizes, +measurements and reproduction steps. Original section headings and links are retained. + +| Topic | What is preserved | +| --- | --- | +| [Foundation, placement and isolation](#at-a-glance) | Execution choices, three-key example, atomicity, write skew and scale assessment. | +| [Durable records and protocol costs](#contract-and-recorded-boundary) | Ownership records, admission history, isolation targets and revision-specific command counts. | +| [Bounded SQL and wire payloads](#bounded-sql-payloads) | SQL chunking, escaped values, wire fallback and multipart input transport. | +| [Write state machine and read visibility](#write-state-machine) | Original sequence, token lifetime, lock barriers and saved-image serialization argument. | +| [Splits, serving recovery and residency](#splits-recovery-and-capacity) | Split locks, driver evidence, coordinator admission, account participants and read barriers. | +| [Foundation review and qualification plan](#foundation-review) | Trust boundary, payload reproduction, moved endpoints, apply capacity and API qualification order. | +| [Capacity failures and tenant isolation](#recovery-while-the-survivor-keeps-serving) | Survivor recovery, dense exhaustion, page reservation, BEGIN cancellation, tenant collision and SQLite FULL. | +| [Resolution progress and unavailable participants](#resolution-must-progress-past-an-unavailable-participant) | Continue healthy work, participant admission and bounded terminal resolution. | +| [Local/global indexes and settled observations](#local-indexes-join-the-transaction-boundary) | Index atomicity, readiness timings, settled checkpoints, replica-query regressions and GSI scope. | +| [Compaction and acknowledged read cleanup](#terminal-operation-compaction) | Terminal payload compaction, read-consumption acknowledgement and cleanup-residency regressions. | + +
+Foundation, placement and isolation + ## At a glance | Boundary | Contract | Present limit | @@ -236,6 +530,11 @@ failure reproductions follow below. +
+ +
+Durable records and protocol costs + ## Contract and recorded boundary This document describes the implemented write and read protocols and the @@ -486,6 +785,11 @@ cannot acquire locks after recovery declares the participant resolved. A rejected command cannot leave a partial prepare: the runtime rolls its application savepoint back while recording the rejection. +
+ +
+Bounded SQL and wire payloads + ## Bounded SQL payloads Coordinator participant operations, cancellation decisions, and participant @@ -714,6 +1018,11 @@ it uses a Blob-role namespace with asynchronous hydration, while application handlers expose synchronous bounded SQL. This transport keeps ownership inside the existing phase savepoint and adds no second decision authority. +
+ +
+Write state machine and read visibility + ## Write state machine ```mermaid @@ -958,6 +1267,11 @@ collection requires a completion/response-retention contract that also fences late fetches and recovery. This is an explicit capacity gap, especially for read-heavy workloads; it must be closed before production scale claims. +
+ +
+Splits, serving recovery and residency + ## Splits, recovery, and capacity The current split seals a source, exports live item rows, then opens children. @@ -1277,6 +1591,11 @@ The prior branch behavior returned live images without checking intents. addresses that unsafe visibility path, while the following API and recovery gates remain necessary. +
+ +
+Foundation review and qualification plan + ## Foundation review Historical review baseline: `88d06d986c9`. The SQL payload finding below is @@ -1636,6 +1955,10 @@ failures; it does not justify production readiness or unlimited scaling. Public writes must continue to wait for all participant resolutions: success after the decision alone could expose partial application; cancellation after an ambiguous decision could hide a committed transaction. +
+ +
+Capacity failures and tenant isolation ## Recovery while the survivor keeps serving @@ -2170,6 +2493,11 @@ and runtime, format, diff, and Cell/LTX layout checks passed. The new FULL regressions establish local/peer command refusal and driver cleanup; the process smoke protects the deployed server path and does not simulate a full device. +
+ +
+Resolution progress and unavailable participants + ## Resolution must progress past an unavailable participant At `b03f83f71c2`, `finish_decided_cross_cell_transaction` returned on the first @@ -2422,6 +2750,11 @@ API and persistence path without those injected pauses. Strict all-target Clippy, formatting, diff, and Cell/LTX layout checks passed. The source change adds 13 net production lines for bounded polling and the six shared descriptors. +
+ +
+Local/global indexes and settled observations + ## Local indexes join the transaction boundary ALL-projection local secondary indexes now share the base item's Cell command. @@ -2668,6 +3001,11 @@ revision, online index lifecycle and automatic GSI splitting remained open. Current split status is in [global indexes](global-indexes.md); retention, full DynamoDB compatibility, and 10,000-Cell/multi-TB qualification remain open. +
+ +
+Compaction and acknowledged read cleanup + ## Terminal operation compaction Inspected main (`311105eb864`) and the previous branch head retained every @@ -2835,3 +3173,5 @@ the other Linux peer failures or replace current-head CI qualification. Three further runs passed (7.04 s, 5.57 s, 5.53 s); the sibling settled-checkpoint regression passed (0.48 s). Strict Clippy for the native integration target passed (11.12 s), as did formatting and diff checks. Temporary diagnostics were removed. + +
diff --git a/docs/deployment.md b/docs/deployment.md index 8217e87..350af05 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -188,7 +188,7 @@ Qualify the exact binary, storage service, filesystem, failure domains and clien workload. Exercise owner loss, ambiguous mutations, capacity limits and restore from retained state. Backups/PITR, supported in-place upgrades and bounded fleet recovery are still open gates; a copied live data directory is not a table backup. -Use [implementation status](implementation-status.md) to select current tests. +Use the [qualification workflow](../.github/workflows/qualification.yml) to select current tests. ## State and credential retention @@ -216,5 +216,6 @@ signed endpoint, correct generation and durable result, then exercise restart: 5. Repeat the relevant transaction/index/Streams cases and record failures and the exact binary/storage/configuration. A local restart is not machine-loss proof. -[Implementation status](implementation-status.md) retains detailed fault evidence, -current target commands and the distinction between historical and current runs. +The [server qualification guide](cellar-server.md#qualification-still-required) +provides test commands, detailed fault evidence and the distinction between +historical and current runs. diff --git a/docs/global-indexes.md b/docs/global-indexes.md index ccc118f..4f7609f 100644 --- a/docs/global-indexes.md +++ b/docs/global-indexes.md @@ -1,22 +1,319 @@ # Global secondary indexes +A GSI is a separately routed, asynchronously maintained view of a base table. +The base command durably records what to project; a worker delivers versioned +changes to the index Cells. A successful base write can precede its index view. + +| I need to… | Start here | +| --- | --- | +| Choose a projection and understand placement | [Current Cellar reference](#current-cellar-reference) | +| Follow write acknowledgement and index delivery | [Projection delivery](#projection-delivery) | +| Understand duplicate delivery and deletion replay | [Versions and tombstones](#versions-and-tombstones) | +| Follow a split or owner replacement | [Split and recovery reference](#split-and-recovery-reference) | +| Create and query an indexed table | [Request examples](#request-examples) | +| Inspect original reasoning, faults and measurements | [Detailed design and recorded evidence](#detailed-design-and-recorded-evidence) | + ## Current Cellar reference -Creation-time GSIs use independent ordered ranges and asynchronous versioned -projection. `ALL`, `KEYS_ONLY` and `INCLUDE` are implemented; online lifecycle -changes remain rejected. No index read implies an atomic cross-index base view. +Creation-time GSIs use independent **ordered ranges**, directories and Cell +identities. The full base primary key breaks ties between duplicate GSI keys. +The range count and physical owners can differ from the base table's plan. + +![Base and GSI directories own separate ordered range Cells; full base keys distinguish duplicate index keys](../diagram/operations/gsi-ordered-ranges.svg) + +| Feature | Current boundary | +| --- | --- | +| GSI specified in `CreateTable` | Install independent index ranges before publishing the base route | +| `ALL`, `KEYS_ONLY`, `INCLUDE` | Store the selected projection in the index Cell | +| Query and Scan | Read projected index rows; do not fetch unprojected base attributes | +| Query ordering | Index PK equality and optional SK predicates, forward/reverse pagination, with full base-key tie breakers | +| Continuation | Pass the returned `LastEvaluatedKey` unchanged; it contains the union of base and index keys | +| Missing GSI key attributes | Sparse item: no projected entry for that index | +| Strongly consistent GSI reads | Rejected by the pinned frontend | +| `ALL_ATTRIBUTES` on a partial projection | Rejected; choose projected attributes | +| Online GSI create/delete/update | Rejected by `UpdateTable`; internal backfill primitives do not establish a supported public lifecycle | +| Base transaction followed by index reads | Eventual projection; no atomic cross-index or base/index read view | + +### What each projection contains + +Assume a base key `order_id`, index keys `status`/`created_at`, and non-key +attributes `total` and `internal_note`. + +| Projection | Stored attributes | Unavailable through that index | +| --- | --- | --- | +| `KEYS_ONLY` | Base key + index keys | `total`, `internal_note` | +| `INCLUDE` with `NonKeyAttributes: ["total"]` | Base key + index keys + `total` | `internal_note` | +| `ALL` | Complete projected item | No additional base fetch is needed | + +The current GSI route key orders `(index PK, index SK, complete base key)`; +omit absent sort-key components. DynamoDB JSON names those logical key roles +`HASH` and `RANGE`; those API names do **not** select physical hash routing. +See [ordered keys](ordered-cell-partitioning.md#primary-key-bytes) and +[the encoding](../src/partition/key.rs). + +## Projection delivery + +![Durable base acknowledgement is separate from asynchronous GSI delivery, with journal pruning gated on every required durable index result](../diagram/reference/architecture-3.svg) + +| Base operation | Projection effect in the same base command | +| --- | --- | +| Put / Update / Delete | Record immutable old/new images when at least one projection changes | +| Transaction PREPARE | Reserve journal/apply capacity; emit no projection entry yet | +| Transaction COMMIT resolution | Emit projection entries while applying staged base images | +| Transaction ABORT | Discard staged images; emit no projection entry | +| Split-copy import | Copy existing state; do not create a new user change or projection intent | +| No projected value changes | No journal work | + +A journal entry holds the table/index generation snapshot, old/new images and +immutable source version. The entry is shared across the table's indexes. +The source removes it only after every required mutation for the live index +generations has a durable `Applied`, `Replay` or `Superseded` result. + +| Change to one index | Delivery order | +| --- | --- | +| New projected item | Write the new entry | +| Same index key, changed projected attributes | Replace the projected image | +| Index key changes | Tombstone the old entry, then write the new entry | +| Item deleted or becomes sparse | Tombstone the old entry | +| This index's projection is unchanged | Skip its mutation | + +### When an index is unavailable + +| Observation | Worker action | Source journal | +| --- | --- | --- | +| One index fails | Continue attempts to other healthy indexes | Retain until all required mutations succeed | +| A reply is lost | Replay from immutable work and version rules | Retain uncertain work | +| GSI returns a stale route during cutover | Retry through current routing | Retain | +| Entry fails delivery | Durably defer it behind work already queued on that source | Keep the original projection version | +| Another worker already acknowledged it | Duplicate deferral is a no-op | Do not recreate the removed entry | + +Deferral changes scheduling order, not content or source version. Healthy indexes +can advance to later base writes while another index is unavailable. This does +not bound lag: sustained writes above projection capacity can fill the finite +base Cell budget. See [journal replay](../src/backend/global_index.rs) and +[enqueue/deferral](../src/global_index/outbox.rs). + +## Versions and tombstones -![Atomic source journal and asynchronous versioned delivery](../diagram/operations/projection.svg) +![Older source versions are superseded, identical versions replay, conflicting content is rejected, and a newer source epoch can apply](../diagram/operations/gsi-version-replay.svg) -The original journal schema/rationale, reservations, split protocol, replay and -transfer investigations remain below. Hash-layout and retired-fixture results -are scoped to their recorded implementation. +| Field | Meaning | +| --- | --- | +| `source_epoch` | Base range epoch that produced the change | +| `sequence` | Source Cell command sequence; together with epoch forms the immutable lexicographic version | +| Content digest | Distinguishes exact replay from different content at the same version | +| Projected image / tombstone | Live row, or retained deletion evidence that suppresses stale resurrection | +| Destination route epoch | Separate GSI range-routing fence; not a replacement for source version or native ownership fencing | + +| Incoming mutation | Result | Can count as delivered? | +| --- | --- | --- | +| Older than stored version | `Superseded`; keep stored content | Yes, after the durable result | +| Same version and same content | `Replay` | Yes, after the durable result | +| Same version and different content | `VersionConflict` | No | +| Newer valid version | `Applied` | Yes, after the durable result | +| Wrong route, generation or serving state | `StaleRoute` | No; retry routing | + +Versions compare only entries for the **same complete index/base key**. Separate +base items with equal index PK/SK values do not overwrite each other. Split import +uses stricter exact-replay checks: a conflicting imported version is rejected +rather than replacing the copied row. Tombstone collection still needs a safe +retirement protocol; elapsed time alone does not make old delivery impossible. + +## Split and recovery reference + +![GSI split lifecycle reserves an exact plan, copies and verifies both children, publishes replacement routes, opens both children and only then finishes the plan](../diagram/operations/gsi-split-lifecycle.svg) + +| State / boundary | Serving behavior | Durable rule | +| --- | --- | --- | +| Source `Serving` | Reads and projection apply allowed | Exact source/child plan is recorded before transfer | +| Source `Sealed` | Query, Scan and projection apply reject stale routes | Export only the sealed source; retain original keys, versions and tombstones | +| Child `Importing` | No client traffic | Row and fingerprint commit together; duplicate import does not increment its count | +| Child `Activated` | Still no client traffic | Complete imported fingerprint must match expected contents | +| Directory publication | Replacement routes become authoritative | Retain plan/reservations through both child opens | +| Child `Opened` | Reads and projection apply allowed | Trusted controller observes publication before opening | +| Plan complete | Both opened states verified | Finish reservations; this does not authorize historical storage deletion | + +Base splits have another gate: source sealing rejects unresolved transaction +locks **and pending projection entries**. A copied base image is already projected; +a later child mutation carries its newer source epoch. + +### Serving-time owner recovery + +| Discovery step | Current behavior | +| --- | --- | +| Account selection | Rotate the deployment's configured `accounts`; omitted accounts receive no such maintenance | +| Table discovery | Page live table generations; skip deleting generations in the projection sweep | +| Base ranges | Replay pending source journals | +| Index ranges | Visit every published range through native ingress, even with empty base journals | +| Idle or expired owner | Use Cellar's native routing, ownership fence and restoration | +| Live remote owner | Leave physical ownership with that node | +| Capacity refusal / transient range failure | Advance the cursor and revisit later; do not pin healthy work behind the failure | +| Worker restart | Rebuild transient sweep cursors; journal contents, versions and split plans remain durable | + +There is no separate GSI placement or lease mechanism. Peer authentication is not +transport encryption; see [server boundaries](cellar-server.md#server-layout). +A configured-account sweep and a successful local fixture do not establish a +fleet-wide recovery-time bound. + +### Resource bounds and open qualification + +| Current source setting / mechanism | Interpretation | +| --- | --- | +| Journal payload reads: **128-KiB** chunks | Bound each read; old/new images and journal retention still consume capacity | +| One sweep page: up to **64 ranges**, **4 range attempts** in flight | Bounded work, not a projection-lag or recovery SLA | +| Index Cell: **512-MiB database / 64-MiB capture** limits | Finite budgets, including retained tombstones; not a sustained-capacity claim | +| Split pressure | Occupied SQLite pages, configured byte threshold and available native admission | +| Online lifecycle / backfill | Internal primitives exist; public orchestration and SDK qualification remain unsupported | +| History and recovery | Safe tombstone/sealed-storage collection, sustained projection throughput and fleet fault/RTO qualification remain open | + +Use compatible retained roots or freshly provision incompatible formats. Earlier +unreleased index schemas are not a supported in-place migration path. + +## Request examples + +Use a client shell with bootstrapped DynamoDB credentials and a policy permitting +the table and its index. These examples assume the [local server](cellar-server.md#configuration). +They are explanatory JSON/CLI requests; selected executed cases are linked below. + +### Create an INCLUDE index with the table + +Save as `gsi-table.json`. `total` is an included non-key attribute, so it does not +appear in `AttributeDefinitions`. + +```json +{ + "TableName": "GsiOrders", + "BillingMode": "PAY_PER_REQUEST", + "AttributeDefinitions": [ + {"AttributeName": "order_id", "AttributeType": "S"}, + {"AttributeName": "status", "AttributeType": "S"}, + {"AttributeName": "created_at", "AttributeType": "S"} + ], + "KeySchema": [{"AttributeName": "order_id", "KeyType": "HASH"}], + "GlobalSecondaryIndexes": [ + { + "IndexName": "ByStatusTime", + "KeySchema": [ + {"AttributeName": "status", "KeyType": "HASH"}, + {"AttributeName": "created_at", "KeyType": "RANGE"} + ], + "Projection": {"ProjectionType": "INCLUDE", "NonKeyAttributes": ["total"]} + } + ] +} +``` + +```sh +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 \ + dynamodb create-table --cli-input-json file://gsi-table.json +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 \ + dynamodb wait table-exists --table-name GsiOrders +``` + +### Write a base item + +Save as `gsi-put.json` and submit it after the table becomes ACTIVE. + +```json +{ + "TableName": "GsiOrders", + "Item": { + "order_id": {"S": "O-001"}, + "status": {"S": "OPEN"}, + "created_at": {"S": "2026-10-19T10:00:00Z"}, + "total": {"N": "25"}, + "internal_note": {"S": "base-only attribute"} + }, + "ConditionExpression": "attribute_not_exists(order_id)" +} +``` + +```sh +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 \ + dynamodb put-item --cli-input-json file://gsi-put.json +``` + +### Query projected rows + +Save as `gsi-query.json`. `Limit: 1` makes the pagination boundary visible when +several rows match; an absent `LastEvaluatedKey` means no continuation is needed. + +```json +{ + "TableName": "GsiOrders", + "IndexName": "ByStatusTime", + "KeyConditionExpression": "#s = :status AND begins_with(#t, :month)", + "ExpressionAttributeNames": {"#s": "status", "#t": "created_at"}, + "ExpressionAttributeValues": {":status": {"S": "OPEN"}, ":month": {"S": "2026-10"}}, + "Select": "ALL_PROJECTED_ATTRIBUTES", + "ConsistentRead": false, + "ScanIndexForward": true, + "Limit": 1 +} +``` + +```sh +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 \ + dynamodb query --cli-input-json file://gsi-query.json --no-paginate +``` + +The first query can lag the base write. Re-query to observe eventual delivery; +there is no fixed propagation deadline. Once projected, the example row contains +`order_id`, `status`, `created_at` and `total`, not `internal_note`. Two orders with +the same `status` and `created_at` remain distinct because `order_id` breaks the tie. + +| Follow-up | How to express it | +| --- | --- | +| Next service page | Put the exact returned `LastEvaluatedKey` into `ExclusiveStartKey` in the next query; keep `--no-paginate` to inspect service pages | +| Reverse order | Set `ScanIndexForward: false` | +| Sparse item | Omit an index key attribute; the item remains in the base table but has no entry in this index | +| Move an existing order to another status | Update the base `status`; the worker tombstones the old key and projects the new key asynchronously | +| Delete an order | Delete the base item; projection retains a versioned tombstone | +| Strong read request | Do not set `ConsistentRead: true`; the frontend rejects it for GSIs | + +## Current source and verification + +| Boundary | Entry point / test | +| --- | --- | +| Creation and rejected online updates | [table adapter](../src/backend.rs) | +| Projection selection and versioned storage | [index module](../src/global_index.rs), [schema](../src/global_index_schema.sql) | +| Immutable journal and deferral | [outbox](../src/global_index/outbox.rs), [journal schema](../src/global_index_outbox_schema.sql) | +| Delivery, pagination and owner visits | [worker](../src/backend/global_index.rs), [read path](../src/global_index/read.rs) | +| Ordered range transfer | [controller](../src/provision/global_indexes.rs), [lifecycle commands](../src/global_index/transfer.rs) | +| Current registered native cases | [Cellar index target](../tests/cellar_indexes.rs) | +| Signed process scenario | [GSI process fixture](../tests/server_binary/global_indexes.rs); see [qualification workflow](../.github/workflows/qualification.yml) | + +```sh +cargo test --locked --test cellar_indexes -- --test-threads=1 +``` + +Process fixtures require the environment described in the +[server qualification guide](cellar-server.md#qualification-still-required). +The historical records below include an unresolved intermittent owner-observation +result. Those runs do not establish an all-green current GSI suite. Consult [the retirement inventory](retired-tests.md) for retired gates. ## Detailed design and recorded evidence -> **Revision scope:** The current GSI boundary is above. The original journal/version protocol, split analysis, transfer evidence and regression records follow. Older hash-layout and retired-test evidence remains historical; it does not qualify the present ordered runtime. +> **Revision scope:** The complete original journal/version protocol, reservation +> analysis, split investigations and regression records remain below. Older hash +> layouts, mTLS fixtures, runtime APIs, timing/pass counts and unfinished items +> describe their recorded revisions. They do not override the [current Cellar +> reference](#current-cellar-reference) or qualify present ordered routing. > Snapshot: [original detailed document](https://github.com/crabbuild/beyonddb/blob/a1eb96a0e87d4473f1bec89c0f3b4f2b02a21afb/docs/global-indexes.md). +Original headings and detailed evidence are preserved. Expand a topic or follow its link. + +| Topic | Detailed material retained | +| --- | --- | +| [Recorded feature boundary](#at-a-glance) | Original feature matrix, creation layout, projection semantics and scope. | +| [Journal, version and reservation protocol](#durable-asynchronous-maintenance) | Atomic enqueue, transaction effects, payload sizes, replay and fair deferral. | +| [Split protocol and owner recovery](#splits-and-recovery) | Transfer lifecycle, fingerprints, directory publication, discovery and capacity. | +| [Source map and serving-recovery investigations](#evidence-map) | Original source map, recorded suites/timings and empty-journal recovery regression. | +| [Transfer and capacity-pressure evidence](#transfer-evidence) | 70-entry transfer, interrupted publication, transient capacity and unresolved owner observation. | + +
+Recorded feature boundary + ## At a glance | Question | Current contract | @@ -53,6 +350,11 @@ sealed-source storage collection, and fleet-scale recovery qualification remain unfinished. Each index range has a 512-MiB database budget and a 64-MiB capture budget. Adding initial ranges is not an unlimited scaling claim. +
+ +
+Journal, version and reservation protocol + ## Durable asynchronous maintenance ```text @@ -122,6 +424,11 @@ GSI propagation is eventual. A successful base transaction does not imply that all of its GSI changes are already visible, or that index readers observe them atomically. This follows the [AWS propagation contract](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transaction-apis.html). +
+ +
+Split protocol and owner recovery + ## Splits and recovery The index Cell now has a durable transfer lifecycle: serving → sealed for the @@ -211,6 +518,11 @@ sustained write rate above projection capacity eventually consumes the finite base Cell budget; steady-state throughput and journal admission need further qualification. +
+ +
+Source map and serving-recovery investigations + ## Evidence map | Boundary | Source | @@ -288,6 +600,11 @@ The compiled-server signed SDK/RustFS smoke passed in 317.32 seconds, including unclean restart, recovered index state, and transaction-token replay. These results cover selected recovery schedules, not a fleet-scale bound. +
+ +
+Transfer and capacity-pressure evidence + ## Transfer evidence | Boundary | Evidence | @@ -341,3 +658,5 @@ the standalone server builds (19.21s), and format/layout/policy checks pass. The previously recorded intermittent idle-owner test has not been resolved by this work. Automatic GSI growth is demonstrated in bounded fixtures, not a full DynamoDB replacement or fleet-capacity guarantee. + +
diff --git a/docs/implementation-status.md b/docs/implementation-status.md deleted file mode 100644 index e117307..0000000 --- a/docs/implementation-status.md +++ /dev/null @@ -1,916 +0,0 @@ -# Review BeyondDB's implementation evidence - -## Current Cellar reference - -## Current verified slice - -Registered targets exercise the following boundaries. Read the individual cases -and run results before extending any claim beyond their workload/fault model. - -| Contract | Current target / source | -| --- | --- | -| Domain commands, table/item semantics | [cellar_domain](../tests/cellar_domain.rs), [account_cell](../tests/account_cell.rs) | -| Typed SDK/SQL adapter | [cellar_sdk](../tests/cellar_sdk.rs), [runtime crate](../crates/cellar-runtime/src/lib.rs) | -| Ordered serving, directories and splits | [cellar_serving](../tests/cellar_serving.rs), [cellar_directories](../tests/cellar_directories.rs), [keyspace](../crates/keyspace/src/lib.rs) | -| Transactions, lost replies and batching | [cellar_transactions](../tests/cellar_transactions.rs), [cellar_coalescing](../tests/cellar_coalescing.rs) | -| Native peer and owner fencing | [cellar_peer](../tests/cellar_peer.rs), [node_log_authority](../tests/node_log_authority.rs) | -| Lifecycle, admission and reclamation | [creation](../tests/cellar_creation.rs), [deletion](../tests/cellar_deletion.rs), [admission](../tests/cellar_admission.rs), [reclamation](../tests/cellar_reclamation.rs) | -| Indexes, images and metadata | [indexes](../tests/cellar_indexes.rs), [images](../tests/cellar_images.rs), [metadata](../tests/cellar_metadata.rs) | -| Actual executable, signed requests and hard restart | [cellar_process](../tests/cellar_process.rs), [server_binary](../tests/server_binary.rs), [process helper](../tests/support/process_fixture.rs) | -| Follower-acknowledged owner loss | [follower process case](../tests/server_binary/follower_durability.rs) | - -The exact CI matrix is in [Rust CI](../.github/workflows/rust.yml) and -[signed SDK qualification](../.github/workflows/qualification.yml). An ignored -process test has not run just because an ordinary `cargo test` succeeded. - -## Run the gates - -Use the toolchain and target-directory rules in [AGENTS.md](../AGENTS.md). - -```sh -cargo fmt --all --check -cargo check --workspace --all-targets --locked -cargo clippy --workspace --all-targets --locked -- -D warnings -cargo build --locked --bin beyonddb -cargo test --workspace --lib --locked -cargo test --locked --test cellar_coalescing --test cellar_transactions -- --test-threads=1 -``` - -Large transaction fixtures run separately on constrained hosts while retaining -internal concurrency. Process fixtures require Docker, OpenSSL and AWS CLI: - -```sh -cargo test --locked --test cellar_process -- --ignored --test-threads=1 -cargo test --locked --test server_binary -- --include-ignored --test-threads=1 -``` - -## Independent client qualification - -Run unchanged upstream client tests against an isolated, current server and retain -the upstream revision and complete result set. Broad protocol compatibility is -still an open gate. - -The existing [convenience runner](../scripts/qualify-upstream.py) emits legacy -configuration (`storage_url` and certificate-based peer fields). It needs a port -before it can qualify the current executable. Do not use that runner's historical -results as Cellar evidence. Current signed process fixtures use the native config. - -## Open qualification gates - -| Area | Work still required | -| --- | --- | -| Compatibility | Complete independent protocol/error matrix, including unsupported options | -| Recovery | Partitions, simultaneous failures, idle-cell scheduling and a bounded recovery objective | -| Resources | Sustained admission/fairness, memory/disk pressure and large payload histories | -| Operations | Coordinated backup/PITR, supported format upgrades and storage-history collection | -| Scale | Independent-machine throughput/latency and failure-domain evidence | - -Current local measurements and their failures are in -[the benchmark report](local-benchmark.md). They are not a fleet capacity target. - -## Detailed design and recorded evidence - -> **Revision scope:** Use the current target matrix above for this server. The original engineering and qualification record below is retained by revision, including old runtime APIs, commands and test counts. Historical pass counts and references to current main do not describe the present checkout. -> Snapshot: [original detailed document](https://github.com/crabbuild/beyonddb/blob/a1eb96a0e87d4473f1bec89c0f3b4f2b02a21afb/docs/implementation-status.md). - -> **Historical Cellule baseline:** this page records the earlier host and its -> qualification. The current server embeds Cellar, pinned to the crates.io -> `cellar-main =0.1.0` release. Read [the server guide](cellar-server.md) -> for current startup, deployment behavior and qualification boundaries. - -This is the detailed engineering and qualification record for BeyondDB. It explains what each path does, which tests exercise it, and which claims remain open. Start with the [API coverage table](api.md) for a workload decision, the [architecture guide](architecture.md) for Cell ownership, the [user guide](user-guide.md) for requests, or the [deployment guide](deployment.md) for a local node. Run commands on this page from the repository root. Some historical results used earlier dependency pins; check [Cargo.toml](../Cargo.toml) for the current revisions. - -BeyondDB connects ExtendDB's DynamoDB protocol, validation, expression, authentication, and operation layers to Cellule's Cell runtime, peer transport, SQLite execution, and LTX publication. Both dependencies are pinned to reviewed Git revisions. - -| Read this section | To understand | -| --- | --- | -| [Running the current server](#running-the-current-server) | Lease, placement, recovery, and process-test behavior | -| [Cell ownership](#cell-ownership) | Range routes, splits, transactions, and authority | -| [Streams dependency contract](#streams-dependency-contract) | Current journal and retention boundaries | -| [API coverage boundary](#api-coverage-boundary) | Specific compatibility gaps | -| [Current verified slice](#current-verified-slice) | Native, signed SDK, and restart evidence | -| [Independent client qualification](#independent-client-qualification) | Re-run unchanged ExtendDB client tests | - -**Evidence vocabulary:** A native Cell test checks a command or recovery path. A signed SDK test checks the public endpoint. A restart test checks that committed state survives owner loss. Passing one fixture does not qualify the full protocol suite or fleet-scale behavior. - -![Detailed design and recorded evidence: detailed diagram retained from the original guide](../diagram/reference/implementation-status-1.svg) - -The diagram orders evidence from a local command to broader compatibility and operations claims. A feature may pass one signed restart fixture while the later gates remain open. - -## Request path - -The path below shows the responsibility boundary. ExtendDB handles the public protocol; BeyondDB selects a Cell; Cellule executes and publishes its state. - -```text -AWS SDK / DynamoDB JSON client - -> ExtendDB HTTP handler, SigV4 verification, IAM, validation, expressions - -> ExtendDB StorageEngine and CatalogStore backed by BeyondDB - -> Cellule ApplicationHandle (one command/query per logical operation) - -> Cell actor, SQLite transaction, LTX publication, object store -``` - - - -## Running the recorded server - -This section records the earlier serving binary and its process fixtures. For a step-by-step setup, use the [deployment guide](deployment.md). - -### Request admission and node leases - -The server writes warnings and errors to stderr. -Peer requests reserve memory while awaiting storage or Cell dispatch. CPU slots -cover envelope decoding, signature verification and reply encoding; asynchronous -work does not retain a codec slot. The signed request deadline bounds enrollment, -dispatch and reply encoding. -Serving nodes publish a 15-second lease and renew every three seconds. -`shutdown_serving_node` drains the runtime and joins heartbeat maintenance before -conditionally retiring the boot-session advertisement. A replacement can reuse -the physical node ID immediately after successful retirement. If a canceled -heartbeat commits after the final load, retirement reconciles only a newer -advertisement from the same signed boot identity. Unsealed logs and recovery -claims still reject withdrawal. Retirement reads and writes share one lease -lifetime as their deadline; errors retain scratch state and failed drain never -starts retirement. Unclean exit still requires authoritative expiry before -takeover. See [measured lease qualification](scaling.md#graceful-session-retirement-and-immediate-restart). - -### Placement and range movement - -Each renewal measures RAM and scratch-filesystem availability on a blocking -worker, caps them by runtime reservations, and signs Cell/job counts and backlog -pressure for the placement planner. Failed measurements and shutdown advertise -no placement capacity. A probe cannot extend the lease or publish after fencing. -Linux placement measurement requires a readable, complete cgroup-v2 memory -hierarchy; it checks ancestor limits and usage, including usage above a limit. -Cgroup-v1, hidden ancestors in a container namespace, and unsupported platforms -are ineligible instead of advertising host-wide RAM. Native macOS uses host RAM. -Configured owned Cells can still serve when placement measurement is unavailable. -Requests for idle, previously published data/GSI Cells now select a destination -from signed capacity and activate it over pinned mTLS. Missing eligible capacity -rejects placement. Account, credential, and coordinator residency keep their -existing policies. Placement evaluates heartbeat freshness after fleet discovery, -so storage latency does not make a renewed sample appear to come from the future. -Initial data, GSI, and split-child provisioning use the same -signed placement measurements with a separate authenticated provisioning -capability. A serving-node loop now uses the runtime's transfer planner to move -settled data/GSI owners to available capacity. It samples every 15 seconds, -retains the planner's residence and settlement gates, and releases at most two -ranges per pass. Account, credential and coordinator Cells retain their existing -ownership policies. Fleet load and failure qualification remain open; see -[range movement](scaling.md#automatic-movement-of-settled-ranges). -When the local Cell pool is full, new range admission can release obsolete -base/GSI sources after current-owner metadata proves retirement. Restoration of -published owners and metadata admission can also release settled data/GSI owners, -preferring sealed sources, then live directory owners if no range can yield. -Restoration does not require the account to be resident. Runtime generation and -settled-work checks gate release; durable roots retain items, intents, directory -membership and unfinished transfers for later restoration. - -### Bootstrap and configuration - -`cargo run -p beyonddb --bin beyonddb -- config.json --bootstrap` starts one -leased Cell node, a private mTLS peer listener, and ExtendDB's public DynamoDB -listener. `--bootstrap` reads one access-key secret from stdin, stores it -encrypted in a credential Cell, and commits the configured inline user policy. -Run without `--bootstrap` after the first successful start. The encryption key -file contains exactly 32 raw bytes and must be retained across restarts. - -```json -{ - "storage_url": "s3://my-bucket/beyonddb", - "node_id": "01994f26-5966-7b20-8b58-2fddf198a321", - "data_dir": "/srv/beyonddb/scratch", - "disk_budget_bytes": 107374182400, - "max_active_cells": 128, - "sql_workers": 12, - "encryption_key_file": "/etc/beyonddb/encryption.key", - "region": "us-east-1", - "peer_bind": "0.0.0.0:9001", - "peer_endpoint": "https://node.example.com:9001", - "peer_certificate": "/etc/beyonddb/peer.crt", - "peer_private_key": "/etc/beyonddb/peer.key", - "peer_ca": "/etc/beyonddb/peer-ca.crt", - "peer_server_name": "node.example.com", - "public_bind": "0.0.0.0:8000", - "public_endpoint": "https://ddb.example.com:8000", - "public_certificate": "/etc/beyonddb/public.crt", - "public_private_key": "/etc/beyonddb/public.key", - "owned_accounts": ["123456789012"], - "owned_access_keys": ["AKIAIOSFODNN7EXAMPLE"], - "initial_partitions": 4, - "split_threshold_bytes": 268435456, - "bootstrap": { - "account_id": "123456789012", - "access_key_id": "AKIAIOSFODNN7EXAMPLE", - "principal_name": "operator", - "policy_name": "tables", - "policy_file": "/etc/beyonddb/operator-policy.json" - } -} -``` - -The peer certificate must be an Ed25519 leaf trusted by `peer_ca`, valid for -both client and server authentication, and cover `peer_server_name`. All nodes -in one fleet use the same CA, storage root, compiled release, and encryption -key. The public listener requires TLS unless bound to loopback. The object -store must support strict create and conditional updates; local `file://` -storage does not provide the required Cell authority semantics. A new node -session uses a fresh scratch directory; graceful shutdown removes it after -Cell drain. Crashed sessions may leave scratch directories for operator cleanup. - -### Signed process and restart fixtures - -The process smoke test starts RustFS, bootstraps a key, sends AWS SDK table and -item requests, kills the server without draining it, then restarts it at a new -peer address and reads the committed item after lease expiry and fenced Cell -takeover. It also verifies -BatchWriteItem, BatchGetItem, paginated parallel Scan, and TTL expiry across four -initial data Cells before the crash. It checks batch reads and TTL configuration -after recovery, then disables TTL and verifies that state through another -restart. Run it in a -dedicated environment with Docker, `aws`, and `openssl` available. The fixture -starts a digest-pinned RustFS 1.0 GA container with an isolated Docker volume -and a random loopback port; no native RustFS installation is used. Colima users -can select their daemon with `DOCKER_CONTEXT=colima`. - -When Docker's own volume filesystem is short of space or inodes, select a host -bind root that the daemon shares. Each test creates a fresh child directory -there. For example, with Colima's home-directory sharing: - -```bash -export BEYONDDB_TEST_RUSTFS_BIND_ROOT="$HOME/.codex/tmp/beyonddb-rustfs-tests" -CARGO_TARGET_DIR="$HOME/Workspace/crabbuild-target/beyonddb-sdk-verification" \ - cargo test --locked --test server_binary -- --ignored --test-threads=1 -``` - -Successful tests remove their owned bind data; failed tests retain it and print -the path. A bind mount still requires enough free Docker VM inodes to create a -container. Preserve failed fixture evidence before cleaning up owned test -resources. - -```bash -CARGO_TARGET_DIR=$HOME/Workspace/crabbuild-target/beyonddb-cellule \ - cargo test -p beyonddb --test server_binary -- --ignored --test-threads=1 -``` - -The `settled_history_beyond_residency` process test isolates coordinator recovery: -70 distinct coordinator shards update two small items in separate data Cells, -then the server is killed and restarted at another peer address. It uses the -same 45-second readiness gate as the full smoke test. On a readiness failure, -it retains the fixture and prints its path for startup-only replay. A failed -process test stops but retains its RustFS container and volume, printing the -container name; successful tests remove both. This test -does not cover large payloads or index restoration; both remain in the full -scenario. See [scaling qualification](scaling.md) for actual results and open gates. - -The standalone [Rust workflow](../.github/workflows/rust.yml) checks format, -types, Clippy, the server build, and library tests. The -[SDK qualification workflow](../.github/workflows/qualification.yml) runs native -elastic-Cell, peer-network, and process tests against pinned Cellule. The former -Crab monorepo's qualification results are historical. Ordinary `cargo test` -does not run the ignored process tests. - -### Startup recovery and request-driven takeover - -This server uses an explicit list of configured accounts and locally owned -credential Cells. On startup it recovers configured account and credential Cells, then -pages base/index directory leaves and the account coordinator registry. Idle or expired owners can -be recovered at a new peer endpoint; live remote owners remain in place. Every -takeover still requires node-session fencing and a Cell authority CAS. Recovery -also resumes a published root already claimed by this boot session if activation -was interrupted. Discovery verifies a local actor exists before reporting the -Cell recovered; configured admission and request routing share root restoration. -This requires the account to be configured on the replacement and enough local -capacity for its recovered ranges. While serving, it also discovers expired -coordinator owners through configured accounts and restores their original -participants. Requests also select capacity and recover expired data, index and -directory owners from their published roots, without waiting for transaction or -index maintenance to discover them. Unreachable owners with live leases remain -fenced against takeover. Background recovery of unaccessed data-only nodes still -needs a recovery scheduler. -Distributed recovery scheduling, fleet qualification, -management APIs, and the remaining DynamoDB operations are still required -before this is a complete service. A public node with no locally owned account -or credential Cells can forward signed requests to live owners through mTLS. - -### TTL sweeps and deferred expiry - -The account Cell commits TTL settings. The serving binary sweeps enabled tables, -configures a fixed expiry index in each routed data Cell, backfills old items in -bounded commands, and conditionally deletes expired items. Each tick processes -at most one 64-Cell route page per table and 16 tables per account. An owner -restart restores the settings, sweep and backfill cursors, and index state. The global TTL listing -trait remains unsupported; the worker lists tables through each configured account. -UpdateTimeToLive primes at most one route page and returns while the worker -reconciles the rest; disabling stops expiry sweeps without dropping the fixed -data Cell index. -TTL candidate selection skips shared and exclusive transaction locks. If a -prepare races selection, the conditional delete defers that item while the -sweep continues; later passes revisit it after transaction resolution. - -### Table creation, deletion, and tags - -The account capacity worker resumes incomplete table creation from its durable -catalog row, preserving any published GSI directories before publishing the base -route. It uses the same provisioning path as CreateTable and can continue after -account-owner restoration without a client retry. One creation attempt can install -all remaining initial ranges; this is not a one-Cell-per-tick operation. The global -table-transition hook has no account inventory and remains a no-op. -Table resource tags now have Cell-backed CreateTable, TagResource, UntagResource, -and ListTagsOfResource paths; DeleteTable fences the generation and removes their -rows in bounded cleanup batches. Large deletions remain DELETING until the -account capacity worker finishes; progress survives account-owner replacement. -The name remains reserved until catalog cleanup completes. This does not reclaim -the retired data/index Cells' object-store history. The RustFS -process test verifies these requests through the AWS SDK across a server -restart and verifies that a recreated table starts without the old tags. - -### Transaction decisions and replay - -TransactWriteItems accepts Put, Delete, Update, and ConditionCheck across -account and data Cells. Every public write transaction uses one durable -coordinator for admission, token lookup, prepare, decision, and resolution. -Matching tokens find the original participant set before consulting current -routes. Successful tokens replay for ten minutes after all participants -resolve; canceled tokens are released only after every abort resolves. -The signed SDK process test writes two primary keys in distinct data Cells, -checks replay, mismatch and rollback, then verifies replay and both values -after a hard kill and restart. Cross-Cell TransactGetItems uses shared key -locks and durable captured images; the same process test checks projected -results and missing items before and after restart. Get, Query, and Scan can -finish a blocking transaction's durable COMMIT or ABORT and repeat their read. -Once a terminal decision is known, resolution attempts every participant even -if an earlier participant or receipt fails, so reachable Cells can release their -locks. It keeps up to four attempts in flight per call and replenishes that -window as attempts finish, allowing progress while one owner remains slow. -The overall request stays incomplete until all receipts are durable; errors -remain retryable. Cancellation leaves durable progress for the next resolver. -Each Cell query helps at most one transaction; BEGIN and unavailable decisions -remain retryable conflicts. Transactional reads retain conflict cancellation. - -### Routed tables and splits - -Once a table's initial route is published, keyed CRUD and Scan use its data -Cells. Placement is committed with the table generation: account-local tables -use the account Cell, while routed tables remain CREATING until publication, -including for clients without provisioning capability. The host-backed -provisioner installs 1–256 initial data Cells per table during CreateTable and -resumes using the persisted count after an interrupted setup. Configuration -changes affect new generations only. UpdateTable rejects incomplete routed -creation atomically, preserving the specification already installed in owners. A host-backed controller can resume a recorded split. A cancellable -account capacity loop can trigger a split, and the serving binary starts that -loop for locally owned accounts. It visits base ranges and then GSI ranges in -order, advancing past transient failures and capacity refusals. GSI plans remain -discoverable through their source and children until both replacements open; -adding a node can resume a pending split onto remote owners. Prepared -transactions and pending index projections defer sealing without stopping the -serving task. Inspection and split replay use the routed client, so published sources and children can stay on remote owners. New table ranges, GSI ranges, and split -children use signed fleet placement in the serving binary. There is no merge controller or complete -`StorageEngine`/`CatalogStore` behavior, or account-management service yet. - -### HTTP authorization and credentials - -`build_http_state` now assembles ExtendDB's signed request path from a ready, -leased Cell node. ExtendDB requires a `CatalogStore` even for DynamoDB request -authorization. BeyondDB supplies a Cell-backed catalog for authorization reads; -management, admin login, settings, and metrics methods return explicit errors -until their Cell implementations exist. -Long-lived and externally provisioned temporary SigV4 credentials live in 256 -key-derived Cells. A server-held AES-256 key encrypts access secrets and session -tokens separately before Cell commands. Session records require a name, token, -and expiry; the credential store rejects expired sessions on lookup. BeyondDB -does not issue STS sessions yet. -Revocation is a durable Cell command; an inactive key is rejected by ExtendDB -authentication and stays inactive after owner recovery. - -### Capacity-driven split admission - -The provisioner can inspect a table's active data Cells, resume a pending split, -and split one range whose occupied SQLite pages cross a caller-supplied -threshold. That measurement includes indexes and runtime tables, but excludes -WAL and LTX files. Base-range item counts and stored JSON/key byte totals are -maintained atomically with item changes, so usage queries read one metadata row -instead of traversing all items. The account loop checks one table range per tick and can -trigger one split per tick. The provisioner can install it in the node's task -group, where failure closes readiness and shutdown cancels it before Cell drain. -The serving binary installs it for every locally admitted account. The loop -still needs disk and capture-pressure inputs before a Cell reaches its -admission limit. - -### Signed SDK and peer-network evidence - -One request-path gate now passes: a signed HTTP request reaches a durable Cell -write and survives owner restart in a loopback integration test using ExtendDB's -HTTP server and an AWS DynamoDB SDK client. The credential is encrypted and -committed in its own Cell before the request; the test verifies its ciphertext -does not contain the secret and reopens it after owner restart. The test uses -the Cell catalog to satisfy ExtendDB's server wiring. Developer authorization -is disabled: the signed request is evaluated -against inline user and role policies stored in the account Cell. The test verifies -denial without a policy, denial for an unlisted table, immediate denial after -policy removal, and policy recovery after owner restart. User and role permission -boundaries are committed in the account Cell and intersect with the principal's -inline policy on signed requests. The SDK tests verify user and role denial, -removal, and user-boundary recovery after account-owner restart. The test also -signs a request -with a session token, checks role-policy denial and grant, rejects wrong and -expired tokens, and reads the item after the credential and account owners -restart. Group, session-policy, and tag storage, STS issuance, and -Cell-backed management operations remain unfinished. Until policy-cache -invalidation is wired, serving composition must use ExtendDB's pass-through -authorization cache so removal takes effect immediately. -The same SDK client creates another table and writes through its newly admitted -data Cell without rebuilding the server's routed client. The HTTP state builder -accepts a caller-supplied `CellClient`. The signed SDK test supplies -`CellClient::runtime_with_peer` from a separate runtime; signed requests reach -account, credential, and data Cells through an authenticated loopback peer -round trip. `BeyonddbPeers::router` authenticates incoming requests against live node -advertisements, restricts targets to BeyondDB namespaces, and dispatches ordinary -operations only to the current local owner. A separate Describe-only capability -can activate published idle data/GSI Cells. A distinct provisioning capability -bootstraps cataloged data/GSI ranges; it cannot create catalog entries or acquire -account/credential/coordinator Cells. `BeyonddbPeers::client` binds the owner-resolving HTTP -transport and a fleet-scoped principal to account, credential, and data Cells. -It takes the matching node provisioner to restore cataloged ownerless Cells on -demand. An interrupted ownership claim by the same boot session resumes from -its published root; existing remote owners retain authority. SDK regressions -release data, account, and credential Cells and read the persisted item again, -including the claimed-but-not-yet-restored state. Data/GSI restoration selects -a destination from signed capacity; account, credential, and coordinator -restoration retains its existing policy. The range rebalancer uses the same -authenticated activation path after generation-fenced source release. -`tests/peer_network.rs` uses separate mTLS identities on two leased nodes, -denies a wrong peer principal, and sends signed AWS SDK CreateTable, PutItem, -and GetItem requests through ExtendDB's public listener and the private peer -listeners. The public node provisions and owns the data Cell; the other node -owns the account and credential Cells. After that owner's lease expires, a -replacement fences its session and discovers expired ranges at a new endpoint. -Startup coordinator discovery also finishes a COMMIT left after one participant -apply and before its receipt. Both Cells restore from object storage. -Both public endpoints then read the committed item, including a read that -forwards to the data owner. The replacement refuses data takeover while that -owner is live, then fences its expired node session after lease renewal stops, -restores the data Cell from object storage, and reads the item again. General -unattended takeover and fleet qualification remain unfinished. - -## Cell ownership - -This section records the durable ownership rules behind routes, splits, and transactions. A Cell command is the atomic boundary; multi-Cell work needs a coordinator decision. - -### Account and data Cell boundaries - -The ExtendDB adapter maps an account ID to one deterministic SQL Cell. That -Cell owns table key definitions and, until route activation, the table's items. -Route activation rejects a table with account-local items, and the old account -item commands are fenced after activation. A Cell command is the atomic -boundary. A rejected command rolls back its writes; a successful response is -returned only after LTX publication. - -### Range routes and split publication - -The data Cell module separately persists one table partition-key hash range per -Cell. Sort-key siblings have the same owner. -Its install contract binds the table ID, partition ID, range and Cell epoch; -keyed reads and writes reject a wrong range or stale epoch. The account Cell -can publish a durable initial route after the provisioner installs its data -Cells. The directory version can advance while unaffected data Cells retain -their own epochs. A durable split plan must replace one range with two fresh, -contiguous child ranges while leaving every other range unchanged. Plans are -owned by source range: independent splits can remain pending and publish -concurrently. Each publication advances its leaf version without changing -unrelated source or planned child epochs. Route validation requires complete, -nonoverlapping hash coverage and the table's immutable key schema. Route -publication currently trusts the provisioner to have installed and published -the data Cells; it does not verify -their receipts. The source data Cell can persist an idempotent split seal that -fences ordinary reads and writes, then serves bounded export pages after owner -restart. Import-only child Cells accept idempotent item copies and verify an -expected count and digest before activation. Activation closes imports while -keeping ordinary requests fenced. The host-backed split controller admits -children, seals the source, copies bounded export pages, checks both child -fingerprints against the sealed source, and atomically publishes the exact -durable plan against its exact predecessor source range. It then opens the -children for ordinary requests, verifies their durable open states, then -finishes the plan. Source and child reservations remain discoverable through -publication, so a restarted capacity sweep can resume from either published -child. An opened child cannot start a new split until its parent plan finishes. -The host provisioner can choose a range -midpoint, record the plan, and repeat the split on an already opened child. Repeated calls -resume this sequence after interruption. -The directory command cannot inspect other Cells; serving code must use -the controller rather than calling route publication directly. -`DescribeTable` reports periodically sampled item counts and logical item bytes -for the table and its indexes. Each mutation maintains local counters; the server -samples one range per tick and publishes a completed snapshot only if its table -and route generations still match. Published totals survive account-owner restart. -The values can lag writes and index projection, and do not measure billed storage -or SQLite/object-store usage. See [statistics semantics and proof](scaling.md#table-and-index-statistics). - -### Directory reads and transaction routing - -Routed keyed CRUD and Scan use independently owned base and GSI directory trees. -The account retains a publication anchor for each generation; bounded leaves own -range membership and full split plans. Point requests follow one tree path. -Scan reads at most 64 ranges per page, validates the previous leaf's version, -and adopts the neighbouring leaf's version when crossing its immutable boundary. -Query reads the HASH key's owner through its local ordered RANGE-key index, -including numeric sort keys and continuation. - -Public transactional writes use durable coordinator decisions and idempotent -participant resolution. Transactional reads within one Cell use one snapshot; -cross-Cell reads capture images under shared locks. An admitted transaction -retains its original coordinator and participants across directory changes. -Base and GSI splits advance only the source's epoch and the owning leaf's -membership version. Unfinished plans prevent metadata movement until both -children open. - -### Split evidence and remaining scale gates - -The base directory cutover is under verification. Focused signed SDK split, -restore, pagination and replay tests pass. Native fixtures compile and the -data-range owner-restart check passes; broader runtime and CI gates remain -incomplete. The account catalog and publication anchors still share -a 512 MiB Cell and one writer. Hot partition-key groups, coordinator/history -bounds and 10,000-Cell/multi-TB fleet qualification remain open. See -[metadata ownership](metadata-sharding.md) and [scaling requirements](scaling.md). - -### Bounded transaction payloads - -Transaction request/intent payloads use bounded SQL chunks while remaining in -one local command. Item and saved-read images also use bounded BLOB transfers, -including JSON images larger than 1 MiB after escaping. The signed SDK fixtures -cover multi-MiB transactions and replay across owner/process restart. Aggregate -Update accounting needs cloud-reference qualification: DynamoDB Local accepts -small Updates over more than 4 MiB of stored images. The protocol document and -`scripts/probe-transaction-size.py` record this distinction. Transactions -use bounded binary uploads for BEGIN and prepare inputs larger than one Cell -RPC, plus bounded recovery queries. Temporary uploads are capped and expire; -HTTP body limits, WAL/disk/memory headroom, and history collection remain separate -gaps. Prepare now reserves SQLite page capacity for resolution; unrelated -commands and runtime receipts cannot spend that claim. The conservative bound -reduces admitted transaction concurrency and still needs scale qualification. - -### Participant state and recovery - -The [cross-Cell transaction protocol](cross-cell-transactions.md) specifies -the decision, lock, visibility, and failure-recovery contract. -Start with its [foundation assessment](cross-cell-transactions.md#foundation-assessment-for-multiple-primary-keys) -for a three-key transfer, reader isolation, the driver trust boundary, and -the remaining 10,000-Cell qualification gates. -Account and data Cells persist prepared write images, immutable read images, -and exclusive write/shared read locks using one participant state machine. -Ordinary reads respect write locks; writes respect both lock modes. Data Cells refuse split sealing; account Cells refuse -table deletion and route activation while prepared intents remain. Sharded coordinator Cells can durably -record a participant set, prepare receipts, one commit or abort decision, and -resolution progress. The public write driver resumes published BEGIN records from -stored account/data participant payloads, handles prepare/decision ambiguity, and returns -only after participant resolution. A proven capacity refusal during participant -upload or prepare proposes ABORT and returns ordered `ThrottlingError` reasons -only after cleanup. This includes direct SQLite FULL errors with verified rollback; -unknown outcomes remain retryable and a competing COMMIT wins. -Recording a terminal participant resolution releases its coordinator operation -images for writes and aborted reads. Durable decisions, request digests, targets, -receipts and replay markers remain. Committed read operations stay available for -response assembly. Once all results are assembled, the reader durably acknowledges -consumption. The existing recovery loop deletes saved images in each original -participant Cell, then compacts its coordinator mapping after a durable receipt. -A driver whose chunk fetch races compaction re-reads and finishes the durable -decision. Unacknowledged read images, decisions, tombstones, and object-store -history still require safe retention and collection. -Coordinator token lookup preserves original -participants across route changes and starts the ten-minute replay window only -after all participants resolve. The old account claims and Cell-local token -receipts have been removed. Fenced startup -recovery discovers registered coordinators and their immutable participant owners, then resolves unfinished work before -public traffic starts. Keyed reads, Query, Scan, and same-Cell transactional -reads now reject unresolved intents; range checks include pending creates. -A mixed-participant host test restores account, data, and coordinator owners -and finishes a pending transaction with concurrent drivers. The Cell barriers -fail closed; the adapter helps resolve a blocking terminal decision before -retrying Get, Query, or Scan. An undecided or unavailable coordinator remains -a retryable error. - -### Serving coordinator recovery - -A supervised serving worker rotates through locally admitted coordinators and -discovers one registered shard per tick across configured accounts. It fences -expired coordinator/participant owners, resumes abandoned BEGIN records, and -finishes terminal decisions. It processes at most one pending transaction per -tick, advances past failures, and revisits them on a bounded pass. Discovery -uses an account-registry hint to skip a settled Idle root only when its root -digest, incarnation, ownership epoch, code, and schema still match. The hint -survives restart and is shared with startup recovery; an active owner, recovery -overlay, missing hint, or changed root takes the ordinary recovery path. Cell admission reclaims settled -coordinators at the active-Cell limit; idle shards restore before token lookup. -When published owners outnumber resident slots, restoration can release one -settled base or GSI owner and later restore its exact durable root. Metadata -admission, including a transaction's first coordinator, has the same allowance. -New data owners still require free capacity; -placement can choose another node instead of evicting live ranges to bootstrap. -Recovery reactivates released coordinators; startup resolves shards one at a -time. A participant admission failure no longer prevents a published decision -from resolving healthy Cells. Startup continues through the shard's pending -records but retains errors and fails readiness until recovery completes. Serving -recovery keeps undecided transactions behind successful admission. Distributed -recovery scheduling, bounded transaction/read-image retention, and -fleet qualification remain incomplete. Production admits 64 -active Cells per node; busy coordinators apply retryable backpressure. See -the [scaling requirements](scaling.md) for the unqualified 10,000-Cell, multi-TB target. - -### Admission limits and lease-backed readiness - -HTTP Cell work uses bounded FIFO mailbox admission per Cell, with independent -queues for different Cells. The client shares a 128-call/32-MiB encoded-input -budget and permits 50 seconds of admission waiting per transport stage. Unknown -write outcomes still require resolution. See [concurrent admission evidence and -remaining overload limits](scaling.md#concurrent-request-admission). - -The signed SDK host test uses `CellNodeBuilder::build`, a published node -advertisement, a renewing lease guard, and a task group. The lease task keeps -the authoritative advertisement fresh and fences the node on terminal failure; -normal task cancellation leaves the guard live while the node drains. Other offline host -tests use `build_unleased_for_maintenance`. The serving binary renews its -node lease through this task, supervises its tasks, and passes the host readiness gate before -accepting requests. It must route to the current owner or perform a fenced takeover; -bootstrapping another writer for an existing Cell is not valid. -The provisioner also admits account and credential Cells, including acquisition -from an idle published root after restart. The signed SDK test exercises both -initial admission and owner recovery through those paths. - -## Streams dependency contract - -This section connects the pinned ExtendDB stream contract to BeyondDB's journal, read, and retention paths. The public Streams API remains partial. - -The closed-shard completion contract and full Streams implementation boundaries -are in [streams-contract.md](streams-contract.md). The contract review is -recorded in [ExtendDB PR #372](https://github.com/ExtendDB/extenddb/pull/372). -BeyondDB's current crabbuild fork pin descends from the contract merged in -[ExtendDB fork PR #1](https://github.com/crabbuild/extenddb/pull/1), so upstream -release timing does not block this service. -The native Cell write path now journals item changes atomically for an -installed stream policy and exposes bounded owner Cell reads. CreateTable -can install a stream policy; ListStreams and DescribeStream discover current and -deleted generations for 24 hours after deletion, and GetShardIterator/GetRecords -read retained records during that window. Account Cell regression covers table -name reuse; a signed DynamoDB SDK write and AWS CLI Streams read survived a hard -server restart. The signed TTL sweep also emits the DynamoDB service identity on -its REMOVE record after the owner Cell verifies the configured expiry. - -![Streams dependency contract: detailed diagram retained from the original guide](../diagram/reference/implementation-status-2.svg) - -Supervised sweeps now delete expired account and routed Cell records in bounded -batches. Active owners are swept locally; a durable per-account cursor scans one -tenant catalog shard per cycle to reach idle Cells and deleted table generations. -Expired stream catalog rows, UpdateTable stream transitions, and fleet-scale -retention throughput remain open. - -## API coverage boundary - -These compatibility limits apply even when ExtendDB recognizes a request shape. See the [API matrix](api.md) for operation-by-operation status. - -CreateTable and UpdateTable persist `STANDARD` or -`STANDARD_INFREQUENT_ACCESS` as table-wide metadata; DescribeTable returns -`TableClassSummary`. The default is `STANDARD`. The account command enforces -[two class changes per trailing 30 days](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.tableclasses.html) -across owner restoration and rejects an invalid class without changing other -settings. Both classes use BeyondDB's existing Cell storage and capacity -mechanics; AWS pricing is not implemented. Signed SDK coverage includes a base -split and account restoration. The new record fields alter unreleased serialized -state; upgrading older roots remains unqualified. - -BeyondDB is not a complete DynamoDB replacement. In addition to the index, -Streams, backup/PITR, IAM, and scale gaps described here, the pinned ExtendDB -engine does not dispatch PartiQL or Global Tables operations. Its import/export -handlers use local filesystem extensions rather than the DynamoDB S3 workflow -([ImportTable](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_ImportTable.html), -[ExportTableToPointInTime](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_ExportTableToPointInTime.html)). -BeyondDB supplies empty import/export path lists in `src/cellar/server.rs`, so those -handlers reject requests before file access. Enabling filesystem paths would -not establish DynamoDB import/export compatibility. These operations need -upstream protocol support and Cell-backed orchestration, followed by signed SDK -and restart qualification, before they can be listed as supported. - -## ExtendDB contract to implement - -This is the implementation checklist for extending the backend without bypassing Cell authority or DynamoDB semantics. - -Use ExtendDB's existing `extenddb-server` and engine. The backend must supply -all six `StorageEngine` traits (`TableEngine`, `DataEngine`, `MetadataEngine`, -`StreamEngine`, `BackupEngine`, `WorkerStore`), plus `CatalogStore` and its -`CredentialStore`. The server must use the same Cell-backed state for HTTP, -workers, backup/restore, and management. There must be no separate SQLite or -Postgres write path for those operations. - -The backend needs these invariants: - -- Treat ExtendDB's `TableKeyInfo` and validated expression AST as the request - contract; evaluate conditions and updates inside the Cell command against - the item version that is being changed. -- Keep table metadata, base items, secondary-index entries, TTL state, and - stream records in the same atomic Cell command where those features require - it. Generate stream sequence numbers from durable Cell state. -- Keep account and resource authorization scoped to authenticated identity. - Credential lookup requires a global access-key index; secrets must never be - logged or stored in plaintext. -- Implement pagination against a stable ordering and use opaque, validated - continuation keys. Check item, request, transaction, and Cell wire limits - before dispatch. -- Derive backup points from published Cell snapshots. A backup spanning data - Cells needs a coordinated cut; independent snapshots are not a consistent - table backup. -- Make retries deterministic. ExtendDB's transaction client token is scoped - by account and its fingerprint must distinguish a replay from a different - request using the same token. - -## Recorded verified slice - -The cases below describe focused evidence, not blanket API support. The [API matrix](api.md) summarizes the resulting user-facing boundary. - -| Fixture | Evidence it supplies | Important limit | -| --- | --- | --- | -| `tests/account_cell.rs` | Account Cell commands, rollback, replay, and restoration from a published root | Local fixture behavior does not prove the public protocol. | -| Native `cellar_transactions`, `cellar_directories`, `cellar_indexes` and `cellar_serving` targets | Ordered cells, transactions, splits, indexes and owner restoration | Focused ranges and failures do not establish fleet-scale behavior. | -| `tests/cellar_peer.rs` | SDK peer transport, signed requests, stale-owner recovery and fenced replacement | A small peer fixture does not establish unattended fleet recovery. | -| `tests/server_binary.rs` | Signed SDK/CLI requests through a running process and hard-restart checks | Process tests are ignored by ordinary `cargo test`. | -| `scripts/qualify-upstream.py` | Unchanged ExtendDB Python client tests against a compiled BeyondDB server | Passing selected files does not prove the whole protocol suite. | - -### Table, item, query, and transaction paths - -`src/lib.rs` registers an account Cell with table create/describe/list/update/ -delete, keyed put/get/update/delete, transactional put/delete, and transactional -get operations and initial table route publication. It also registers -independently addressable data Cells with partition install, keyed CRUD and -bounded Scan. The `CellStorage` adapter implements the matching base-table -`TableEngine` and `DataEngine` methods. Conditional single-item writes and -update expressions execute inside the Cell transaction. Base-table Scan uses -bounded pages with stable continuation keys. Parallel Scan assigns contiguous -hash intervals to segments and skips data Cell ranges outside each interval; -cells crossing an interval boundary still scan and filter their items. Query -supports hash-only and sort-key tables on both account-local and routed storage. -Account-local sort-key Query uses a composite ordered key index and bounded -pages; its transaction-intent fence conservatively covers the account table. -The signed two-owner SDK test writes numeric sort keys, pages in both directions, -and repeats the query after account-owner restoration. -Low-level account and partition commands support local atomic transactions. -The public adapter routes all transactional writes through the coordinator, -including account participants before route activation. All transactional reads -use durable shared locks and captured participant images, retrieved individually -to avoid an aggregate Cell response limit. Same-Cell reads now pay the same -coordinator protocol cost. Online global-index changes and non-ALL local index -projections remain unsupported. Streamed writes use the installed Cell policy. - -### Index paths and backfill foundation - -Local secondary indexes with ALL projection support account/routed Query and Scan, -strong reads, numeric sort ordering with base-sort tie-breakers, and base-plus-index -continuation keys. -Sparse entries are absent until their index key exists. Ordinary writes, -transactions, TTL deletion, and split import maintain index entries in the -same Cell command as base items. Indexed Query fences the HASH group's prepared -write intents, including absent creates and sort-key moves; account-local Query -conservatively fences the table. See [the remaining LSI read contract](lsi-contract.md) -for KEYS_ONLY/INCLUDE, base-fetch capacity accounting, and scale limits. -Global indexes created with a table have independently owned ranges, durable -asynchronous maintenance, and ALL/KEYS_ONLY/INCLUDE projected Query and Scan. -See [global-index implementation and limits](global-indexes.md) for journal -replay, transaction boundaries, restart evidence, and unfinished index splitting -and collection. - -Data Cells also provide an internal foundation for online index backfill: -revisioned write policies, bounded durable scan cursors, a historical-journal -delivery barrier, and inheritance of unfinished work through splits. Policy -changes wait for prepared transactions to resolve, preserving their capacity -reservations. Native Cell tests cover restart, invalid historical keys, delivery -tracking, and split inheritance. Account lifecycle orchestration and SDK -`UpdateTable` create/delete remain unimplemented; these commands do not make -online index changes a supported API. The foundation and account-local ordered -query index extend unreleased initial SQL schemas; upgrading roots written by -earlier binaries is unqualified. - -### Native Cell and signed SDK fixtures - -The descriptions below retain historical Cellule fixture details. The unported -`elastic_cells` and `peer_network` targets were removed on 2026-10-18 at the -user's request; [their source and coverage inventory](retired-tests.md) -records what is no longer gated. Current native Cellar fixtures are described in -[the server guide](cellar-server.md#qualification-still-required). - -`tests/account_cell.rs` exercises them through a real -`CellNodeBuilder` and in-memory object store, including request replay, -receipt-based reads, conditional writes, update expressions, scan pagination, rollback of a -multi-table write, and restoration from the published object-store root on a -new host. `tests/elastic_cells.rs` exercises two distinct data Cells through a -real host, range and epoch rejection, route validation and publication, -partitioned adapter CRUD, Scan, same-Cell and cross-Cell writes, rollback and -cross-Cell snapshots, shared-read/write conflicts, account write fencing, and restoration of -both a data Cell and the route's account Cell from object storage. These tests -do not prove fleet-scale repartitioning, complete IAM, every index projection, -Streams, or backups. The -elastic test also verifies -sealed-source export, two import-only children, activation by source-derived -fingerprint, a delayed-copy fence, host-backed split resumption, atomic route -switch, adapter reads through the replacement children, and child recovery -after owner restart. Data Cells -expose their durable partition contract and lifecycle state so a trusted -controller can check split readiness across Cells before publication. The -provisioning test retries after installing two data Cells but before route -activation, writes more than 64 MiB of item payload across both ranges, and -reacquires both Cells after restart with the same measured payload bytes. -The 65-range host test checks TTL expiry in its first and last data Cells over -two bounded sweep ticks, deferred index setup during enable, disabled expiry, -and account-level cursor advance across 17 TTL tables. -The numeric Query host test checks ordered pages, reverse order, a sort-key -predicate, equivalent numeric key spellings, and two successive splits, the -first triggered by the account capacity loop, while preserving Query results -through the Cell adapter. It also sends -ExtendDB engine wire-format PutItem and GetItem requests through the Cell -backend, then restores the item owner from object storage and reads the exact -written image. The same test also sends signed SDK table creation, -DescribeTable, PutItem, GetItem, conditional UpdateItem/DeleteItem, and paginated -Query/Scan requests through ExtendDB's HTTP server, rejects an incorrectly -signed request, and verifies SDK-written items from both an existing table and -a newly created table after owner restart. It exercises Cell-backed inline user -authorization and permission boundaries, but not a production management -catalog. The SDK key is read -from a credential Cell; revocation denies a signed request immediately and -persists through recovery, and a wrong decryption key fails closed. - -The local-index integration fixture covers five ALL indexes on account and data -Cells, tied numeric keys, forward/reverse and scan pagination, sparse removal, -invalid/oversized images, mixed-participant commit/replay, prepared create/delete/ -sort-key-move read barriers, COMMIT/ABORT read helping, and split import. The -server-process fixture additionally exercises signed SDK index metadata, Query -projection/ranges, parallel Scan, transactional changes, and replay after an -unclean restart. These cases do not qualify the full upstream protocol suite. - -## Acceptance proof for a server claim - -Use this checklist before changing a feature from implemented or partial to supported: - -1. Run ExtendDB's protocol suite against the BeyondDB endpoint, then exercise the same endpoint with an AWS SDK. Include table/item CRUD, Query/Scan, indexes, batches, transactions, expressions, Streams, TTL, backups, auth failures, pagination, and error responses where applicable. -2. Restart the owner from object storage. Verify committed items and table metadata, replay requests, and inspect idempotency outcomes. -3. Repeat with two accounts that use the same table names to check isolation. -4. For partitioned work, inject owner loss before and after transaction prepare, decision, and apply. Verify atomic resolution. - - -## Development storage layout - -Unreleased Cell schemas have changed. Existing development roots can require reprovisioning before a new binary can read them. - -Cell catalog heads include the tenant as well as the application and shard. -This isolates account and credential catalogs sharing a BeyondDB store. Existing -unreleased roots with application-only catalog heads require reprovisioning; -there is no fallback reader. This does not enable multi-tenant backup or garbage -collection. See [the recovery finding](cross-cell-transactions.md#tenant-catalog-collision-found-during-recovery-qualification). - -Global-index journals and metadata, local-index schemas, principal permission boundaries, -required table/participant metadata, and settled-root -observations in the coordinator registry also change the unreleased Cell format. -Reprovision development roots created before these changes; there is no in-place -upgrade reader. - -## Serving global-index recovery - -This worker restores index owners as needed, but its existence does not establish bounded fleet recovery time. - -The supervised projection worker discovers base and global-index ranges for -configured accounts. It restores Idle owners and uses fenced takeover after -remote leases expire, including indexes with no pending journal. A failed -index does not prevent healthy indexes from applying the same source change; -the journal remains until all have applied. See [global index recovery and -qualification](global-indexes.md#splits-and-recovery). This does not establish -fleet placement, bounded recovery time, or 10,000-Cell capacity. - -## Independent client qualification - -Use `scripts/qualify-upstream.py` to compare BeyondDB against pinned ExtendDB protocol behavior. The runner starts a fresh RustFS store and the compiled BeyondDB binary, then runs ExtendDB's Python client tests unchanged. It requires -`uv`, `openssl`, and `rustfs`. Supply a read-only ExtendDB checkout at the pinned -dependency revision and an existing artifact directory on the workspace volume: - -```sh -uv run scripts/qualify-upstream.py \ - --binary "$HOME/Workspace/crabbuild-target/beyonddb-cellule/debug/beyonddb" \ - --tests-root /path/to/extenddb/tests/python \ - --artifacts "$HOME/Workspace/crabbuild-target/beyonddb-cellule" \ - test_transactions.py test_items.py -``` - -Build the binary first using this checkout's separate `CARGO_TARGET_DIR`. The -runner records the binary digest, upstream revision, client versions, JUnit -results, and service logs in a fresh artifact directory. It uses only local -fixture credentials and stops both services on exit. It disables bytecode and -pytest cache writes in the upstream checkout. Omit test selectors to collect the -whole Python suite; it stops at the first failure. Passing selected files does -not establish full DynamoDB compatibility. - -At the then-pinned ExtendDB revision `bdb7b3df4ace3b80a6e928f144036d056aec0327`, -the unchanged boto3 GSI file passed all 11 cases against the compiled server -and fresh RustFS after the live-table projection sweep correction. The account -Cell regression verifies that a deleted generation is excluded from serving -sweeps. A preceding combined transaction/GSI run -passed 16 transaction cases, then returned `ServiceUnavailable` on GSI table -creation as the node fenced under mailbox pressure. The isolated GSI run passed; -the longer-run admission and lease failure remains unqualified. - -The persisted creation-placement field changes the unreleased table-record -format, including embedded base/index specifications. Existing development -roots require reprovisioning before running this revision. diff --git a/docs/operations.md b/docs/operations.md index fb9bc31..bc74780 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -60,4 +60,4 @@ retained generations without a proven collection contract. In-place storage upgrades are unqualified. Retain the source/binary/configuration for existing roots; validate compatibility in a disposable clone or explicitly migrate to a fresh prefix. Reusing an incompatible development root is not an -upgrade plan. See [qualification gates](implementation-status.md#open-qualification-gates). +upgrade plan. See [qualification gates](cellar-server.md#qualification-still-required). diff --git a/docs/ordered-cell-partitioning.md b/docs/ordered-cell-partitioning.md index aabcebe..6b051ce 100644 --- a/docs/ordered-cell-partitioning.md +++ b/docs/ordered-cell-partitioning.md @@ -1,22 +1,356 @@ # Ordered table ranges on Cellar +A table is a directory of independently owned range Cells. Each range covers a +contiguous interval of the complete primary key `(PK, SK)`. A point request has +one range owner; a Query can visit several Cells for the same partition key. + +| I need to… | Start here | +| --- | --- | +| Understand logical ranges and physical owners | [Current Cellar reference](#current-cellar-reference) | +| Understand key comparison and byte encoding | [Encoding reference](#encoding-reference) | +| Follow Query across ranges | [Query traversal and budgets](#query-traversal-and-budgets) | +| Resume after a range split | [Logical continuation](#logical-continuation) | +| Choose initial counts and understand split limits | [Creation and split policy](#creation-and-split-policy) | +| Try Query and inspect an evaluated-item limit | [Request example](#request-example) | +| Inspect the complete design and acceptance matrices | [Detailed design and recorded evidence](#detailed-design-and-recorded-evidence) | + ## Current Cellar reference -The current Cellar executable uses encoding-v1 ordered ranges with 2–256 initial -cells, default four. The complete original ordering rationale, directory and -LSI invariants, split/replay analysis and acceptance work packages remain below. +![A logical table spans ordered range Cells whose physical owners are selected by Cellar](../diagram/operations/ordered-ranges.svg) + +| Boundary | Current contract | +| --- | --- | +| Public table creation | **2–256** initial data ranges; default **4**, recorded with the table generation | +| Ownership interval | Half-open **[lower, upper)**; adjacent ranges have no gaps or overlaps | +| Routing key | Complete ordered primary key; tenant and table generation scope the directory | +| Physical placement | Cellar chooses and fences the node owning each logical Cell | +| Node move | Retain Cell identity and interval; change physical owner | +| Range split | Create child identities/epochs and replace the source's directory interval | +| Metadata | Directory leaves hold routes; they do not hold the table's user items | +| Admission budget | `max_active_cells` counts metadata, credential, coordinator, base and GSI Cells together | + +Multiple Cells on one node share that node's failure domain. Creating four ranges +does not guarantee four nodes, equal data occupancy or balanced request traffic. +See [server configuration](cellar-server.md#application-policy-and-capacity) and +[physical ownership](architecture.md#components-and-ownership). + +## Encoding reference + +![Typed, escaped PK and SK components concatenate into one ordered base key; GSIs append the full base key](../diagram/operations/ordered-key-encoding.svg) + +| Key type | Comparison | Example | +| --- | --- | --- | +| String | Unsigned UTF-8 byte order | `"10"` sorts before `"2"` | +| Binary | Unsigned byte order | `61 00` sorts before `61 01` | +| Number | Exact normalized decimal order | `2` sorts before `10`; `1`, `1.00` and `1e0` identify the same numeric key | + +| Encoding-v1 rule | Boundary | +| --- | --- | +| One component | Type tag, order-preserving escaped payload, then `00 00` | +| Embedded zero | Encode `00` as `00 ff` so component boundaries remain unambiguous | +| Binary example (hex) | Payload `61 00 62` becomes `30 61 00 ff 62 00 00`; `30` is the binary type tag | +| Base key | `encoded(PK)` followed by `encoded(SK)`; omit SK for a PK-only schema | +| GSI key | Encoded index PK/SK followed by the **complete encoded base key** | +| Numeric limits | Up to 38 significant digits; magnitude exponent −130 through 125; no `f64` conversion | +| Encoded key bounds | Base: **6,150 bytes**; index plus base suffix: **12,300 bytes** | +| Version fence | Directory and installed range must agree on encoding version **1** | + +DynamoDB JSON still uses ordinary `S`, `N` and `B` attribute values. Clients do +not construct these routing bytes or submit Cellar invocation envelopes. +`HASH` and `RANGE` in a DynamoDB key schema name logical key roles; physical +ownership uses the ordered byte intervals. See [encoding](../crates/keyspace/src/encoding.rs) +and [range validation](../crates/keyspace/src/ranges.rs). + +## Query traversal and budgets + +DynamoDB Query requires **PK equality**. The optional SK predicate narrows the +logical span; it does not select one physical Cell. An arbitrary PK-range query +would require a separate API contract. + +![A numeric SK Query for one PK intersects and clips two ordered ranges, visiting them in forward or reverse order](../diagram/operations/ordered-query-span.svg) + +For the illustrative populated layout below, `K42(n)` means +`(customer-42, numeric SK n)`. `P90` is the boundary before all SKs of `customer-90`. +These intervals describe a possible layout without LSIs, not the initial +boundaries created by the CLI example. + +| Cell | Owned interval | Query: PK customer-42, SK BETWEEN 12 and 25 | +| --- | --- | --- | +| A | `[−∞, K42(10))` | Skip | +| B | `[K42(10), K42(20))` | Read SK 12 through values below 20 | +| C | `[K42(20), P90)` | Read SK 20 through 25; exclude other keys in C | +| D | `[P90, +∞)` | Skip | + +| Request | Traversal / result boundary | +| --- | --- | +| Point Get/Put/Update/Delete | Encode the full key; resolve its containing interval; validate generation, range and epoch at the destination | +| Forward Query | Visit intersecting ranges in ascending order and concatenate base-key results | +| Reverse Query | Visit intersecting ranges in descending order | +| SK equality / comparisons / BETWEEN | Compile the correct inclusive/exclusive logical endpoints into a half-open byte span | +| `begins_with` | String/binary prefix span; not a numeric SK operation | +| Scan | Traverse the ordered directory with bounded pages | +| Parallel Scan | Stable segment predicate on logical keys; its hash does not determine Cell ownership | + +### One request budget across all visited Cells + +| Boundary | Meaning | +| --- | --- | +| `Limit` | Evaluated items across the request; not returned items and not a fresh limit for each Cell | +| Public 1-MiB evaluation boundary | Account for logical item bytes before filtering and projection | +| Current 900,000-byte transport chunk target | Internal reply sizing; distinct from the frontend's public evaluation accumulator | +| Filtered empty response | Can still have `ScannedCount > 0` and `LastEvaluatedKey` | +| Projection expression | Does not reduce the bytes already evaluated for the public page budget | +| Exhausted local span | Continue to the next intersecting Cell when request budget remains | + +An ordinary read must respect unresolved transaction intents, including prepared +creates or deletes absent from live rows. Authoritative per-Cell reads do not +provide one table-wide snapshot. See [read visibility](cross-cell-transactions.md#read-visibility-and-saved-images) +and [the Query aggregator](../src/backend/data/query.rs). + +## Logical continuation + +![A continuation key remains logical when the old range is replaced by two child Cells between requests](../diagram/operations/ordered-pagination-split.svg) + +| Continuation field / event | Rule | +| --- | --- | +| `LastEvaluatedKey` | Logical primary-key attributes; no physical Cell or page identity | +| `ExclusiveStartKey` | Return the exact service key and keep the same Query conditions and direction | +| Forward resume | Start strictly after the evaluated key in the current directory | +| Reverse resume | Start strictly before the evaluated key | +| Range split between requests | Locate the remaining logical span using replacement ranges | +| Stale/sealed/not-ready range | May produce a retryable error; broad split-during-page behavior remains under qualification | +| New writes between requests | No whole-response or multi-page snapshot guarantee | +| Ambiguous submitted mutation | Resolve through the original Cell/identity before considering new-route submission | + +Do not turn a read-resume rule into a write retry rule. A lost increment or +conditional-write reply may already have committed in the source Cell. +Admitted transactions keep their original coordinator and participant identities; +a split cannot retarget that durable decision. + +## Creation and split policy + +### Bootstrap and route publication + +| Stage | Current requirement | +| --- | --- | +| Choose initial policy | Positive configured count from 2–256; counts need not be powers of two | +| Record creation | Persist the immutable generation and initial-count policy; retries finish that policy even after a node default changes | +| Install ranges | Match the generation, schema, encoding version, interval and range epoch | +| Publish GSI directories | Creation-time indexes must be installed and published before admitting base writes | +| Publish base directory / ACTIVE | Validate exact count, adjacency, full coverage and installation evidence | +| Failed provisioning | Keep creation recoverable; do not fall back to one account-local data store | + +The current [bootstrap implementation](../src/cellar/provision.rs) uses deterministic +PK-type-aware fences. String/binary templates use prefix values; numeric templates +use numeric fences. They do not predict future workload balance. The public JSON +configuration selects the count; the explicit schema/workload templates in the +retained work packages are design/acceptance material, not additional JSON knobs. + +`beyonddb:cell-model` accepts `auto` or `partitioned`; both use the configured +multi-range policy. `single` is rejected. Later tag changes do not change the +recorded creation policy or convert the table's placement. + +### Split safety and what remains open + +![Split order: durable plan, source seal, verified child copies, route publication, child opening and safe source retirement](../diagram/reference/architecture-2.svg) + +| Split boundary | Current behavior / qualification limit | +| --- | --- | +| Candidate selection | [Observed median key](../src/partition/split_point.rs); not an arithmetic midpoint of empty byte space | +| No LSI | May split a PK's SK siblings across Cells | +| PK-only table | Split only between distinct PKs; one item is indivisible | +| Table with LSIs | Choose only a PK-collection boundary; keep base and LSI state together | +| Source sealing | Reject unresolved participant work and pending GSI projection entries in the same command that seals | +| Copy / publication | Retain an exact plan, verify children, publish replacements, then open them | +| Old source | Retain evidence needed by replay, streams and recovery; full safe storage collection remains open | +| More advanced balancing | Byte/load-weighted sampling, hot-item handling and measured recovery headroom still need acceptance evidence | +| Merges / online delta catch-up | Later design work; not a supported operational shortcut | + +| Related subsystem | Invariant | +| --- | --- | +| LSI | PK collection locality; the finite Cell budget does not establish DynamoDB's full 10-GiB collection capacity | +| GSI | Independent ordered ranges; full base-key tie breaker, original projection versions and tombstones survive transfer | +| Transaction | Same PK does not imply one participant; group by full-key ownership before persisting immutable targets | +| Native ownership | Application range epochs supplement Cellar's owner fence; they do not replace it | + +See [GSI splits](global-indexes.md#split-and-recovery-reference), +[transaction recovery](cross-cell-transactions.md#recovery-and-stable-identities) and +[the split controller](../src/split.rs). + +Encoding-v1 roots cannot reinterpret older hash layouts or another runtime's +storage. Use fresh compatible storage, or a separately designed export/repartition/ +import migration with validation; do not reuse an incompatible prefix. + +## Request example + +Use a client shell with the bootstrapped DynamoDB credentials and the +[local server configuration](cellar-server.md#configuration). The following schema +uses a string PK and numeric SK. Save it as `ordered-table.json`: + +```json +{ + "TableName": "OrderedOrders", + "BillingMode": "PAY_PER_REQUEST", + "AttributeDefinitions": [ + {"AttributeName": "customer_id", "AttributeType": "S"}, + {"AttributeName": "order_no", "AttributeType": "N"} + ], + "KeySchema": [ + {"AttributeName": "customer_id", "KeyType": "HASH"}, + {"AttributeName": "order_no", "KeyType": "RANGE"} + ] +} +``` + +```sh +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 \ + dynamodb create-table --cli-input-json file://ordered-table.json +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 \ + dynamodb wait table-exists --table-name OrderedOrders +``` + +Save the following as `ordered-seed.json`. These four small items illustrate +logical ordering and filtering; they do not force a same-PK physical split. + +```json +{ + "RequestItems": { + "OrderedOrders": [ + { + "PutRequest": { + "Item": { + "customer_id": {"S": "customer-42"}, + "order_no": {"N": "12"}, + "visible": {"BOOL": false} + } + } + }, + { + "PutRequest": { + "Item": { + "customer_id": {"S": "customer-42"}, + "order_no": {"N": "15"}, + "visible": {"BOOL": false} + } + } + }, + { + "PutRequest": { + "Item": { + "customer_id": {"S": "customer-42"}, + "order_no": {"N": "20"}, + "visible": {"BOOL": true} + } + } + }, + { + "PutRequest": { + "Item": { + "customer_id": {"S": "customer-42"}, + "order_no": {"N": "25"}, + "visible": {"BOOL": true} + } + } + } + ] + } +} +``` + +```sh +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 \ + dynamodb batch-write-item --cli-input-json file://ordered-seed.json +``` + +Check `UnprocessedItems` and retry those entries before reading. Save the Query +as `ordered-query.json`; aliases avoid relying on attribute-name keyword status. + +```json +{ + "TableName": "OrderedOrders", + "KeyConditionExpression": "#pk = :customer AND #sk BETWEEN :lo AND :hi", + "FilterExpression": "#v = :yes", + "ProjectionExpression": "#pk, #sk", + "ExpressionAttributeNames": {"#pk": "customer_id", "#sk": "order_no", "#v": "visible"}, + "ExpressionAttributeValues": { + ":customer": {"S": "customer-42"}, + ":lo": {"N": "12"}, + ":hi": {"N": "25"}, + ":yes": {"BOOL": true} + }, + "ConsistentRead": true, + "ScanIndexForward": true, + "Limit": 2 +} +``` + +```sh +aws --endpoint-url http://127.0.0.1:8000 --region us-east-1 \ + dynamodb query --cli-input-json file://ordered-query.json --no-paginate +``` + +With all four seed writes completed and no concurrent changes: + +| Service page | Evaluated keys | Returned items | Continuation | +| --- | --- | --- | --- | +| First, forward | SK 12 and 15 | Empty; both fail the filter | Use the returned key for SK 15 | +| Next, same Query + returned `ExclusiveStartKey` | SK 20 and 25 | Both keys | Stop when `LastEvaluatedKey` is absent | + +`--no-paginate` exposes one service response. Copy the complete returned key into +`ExclusiveStartKey`; keep the other Query settings. For a **new** reverse traversal, +set `ScanIndexForward: false` and start without the forward cursor. Filtered empty +results do not mean the logical range is exhausted. + +The JSON and CLI examples explain the public request shape. They are not executed +fixture results or proof of the illustrative multi-Cell placement above. + +## Current source and qualification -![Logical ordered ranges mapped to physical owners](../diagram/operations/ordered-ranges.svg) +| Area | Entry point / evidence | +| --- | --- | +| Key encoding and span math | [keyspace](../crates/keyspace/src/lib.rs), [range tests](../crates/keyspace/tests/ranges.rs) | +| Full-key ownership and publication | [routing](../src/routing.rs), [creation path](../src/backend/table_creation.rs) | +| Lazy Query and Scan traversal | [Query aggregator](../src/backend/data/query.rs), [routed reads](../src/backend/data/routed.rs) | +| Split fence and observed key | [partition handlers](../src/partition.rs), [selector](../src/partition/split_point.rs) | +| Embedded range fixture | [native range Cells](../crates/cellar-runtime/tests/embedded/range_cells.rs) | +| Signed serving and owner recovery | [Cellar serving target](../tests/cellar_serving.rs), [ordered route fixture](../tests/support/ordered_routes.rs) | +| Complete CI matrix | [Rust checks](../.github/workflows/rust.yml), [signed SDK qualification](../.github/workflows/qualification.yml) | -The delivery-plan sections record the implementation sequence and acceptance -criteria. A work package is not proof of completion; consult current source and -[registered targets](implementation-status.md#current-verified-slice). +```sh +cargo test --locked -p keyspace +cargo test --locked --test cellar_serving -- --test-threads=1 +``` + +Focused ordering, paging and restart evidence does not establish balanced traffic, +all index/predicate combinations, every split interruption schedule, sustained +fleet throughput or a bounded recovery time. Use the +[server qualification boundaries](cellar-server.md#qualification-still-required) +when evaluating deployment claims. ## Detailed design and recorded evidence -> **Revision scope:** The current ordered-key contract is above. The complete original encoding rationale, directory/split invariants, work packages and acceptance criteria follow. Delivery instructions and unresolved items reflect the design revision; use current source and the test matrix to determine completion. +> **Revision scope:** The original ordering rationale, directory/LSI invariants, +> split/replay analysis, delivery sequence and acceptance work packages remain +> below. Recorded test counts and proposed steps describe their revisions. +> They do not turn workload-weighted splits, explicit bootstrap templates, +> merges or unqualified fault cases into current server features. Use the +> [current reference](#current-cellar-reference) above and the checked-in source. > Snapshot: [original detailed document](https://github.com/crabbuild/beyonddb/blob/a1eb96a0e87d4473f1bec89c0f3b4f2b02a21afb/docs/ordered-cell-partitioning.md). +Original substantive lines, headings and acceptance matrices are preserved. +Expand a topic for the complete design and revision-scoped evidence. + +| Topic | Material retained | +| --- | --- | +| [Ownership and encoding rationale](#ownership) | Complete key ordering, owner boundaries, encoding-v1 constraints and numeric rules. | +| [Bootstrap and directory invariants](#bootstrap-and-directory) | Immutable initial counts, publication proof, directory paging and route validation. | +| [Query, Scan and continuation contract](#point-requests-query-and-pagination) | Logical spans, request-wide budgets, filtering, transaction barriers and segment stability. | +| [Split, replay, index and transaction design](#splits-merges-and-replay) | Fenced copy sequence, ambiguous outcomes, LSI locality, GSI suffixes and immutable participants. | +| [Delivery packages and recorded acceptance evidence](#executable-delivery-sequence) | Original implementation slices, full acceptance matrices, measurements, fixture limits and references. | + +
+Ownership and encoding rationale + Status: ordered routing is implemented in the Cellar-backed executable and passes the local signed SDK/process-restart fixture. The root serving-router fixture puts one PK in three actual cells, queries across them, splits the @@ -97,6 +431,11 @@ encoding is a separate index; it does not redefine equality or the wire shape of `LastEvaluatedKey`. Add differential cases for equivalent numeric spellings before switching production writes. +
+ +
+Bootstrap and directory invariants + ## Bootstrap and directory Replace `TablePlacement::{Account, Single}` and one-cell `auto` on the Cellar @@ -140,6 +479,11 @@ The route cache is bounded and keyed by table generation plus ordered interval. A stale-route response causes a bounded directory refresh. No HTTP client may choose a raw Cellar binding or submit internal invocation envelopes. +
+ +
+Query, Scan and continuation contract + ## Point requests, Query and pagination Get/Put/Update/Delete encode the complete primary key and resolve its owning @@ -180,6 +524,11 @@ The serving fixture verifies this with numeric-equivalent cursor spellings, then splits one PK collection and exercises signed SDK transaction cancellation and restart across the resulting cells. +
+ +
+Split, replay, index and transaction design + ## Splits, merges and replay Choose a split boundary from observed distinct keys using stored-byte and @@ -251,6 +600,11 @@ their existing local atomicity and coordinator decision rules. Source sealing must coordinate with admission so a transaction cannot prepare between the "no unresolved work" check and the fence. +
+ +
+Delivery packages and recorded acceptance evidence + ## Executable delivery sequence | Slice | Files / implementation | Exit evidence | @@ -399,3 +753,5 @@ or startup-admission readiness qualification. * [DynamoDB Query semantics and pagination](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html) * [Key-condition expressions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.KeyConditionExpressions.html) * [Local secondary indexes and item collections](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LSI.html) + +
diff --git a/docs/performance.md b/docs/performance.md index fdb8c65..1c71162 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -1,43 +1,156 @@ # Measure BeyondDB performance +Use this guide to read the recorded results and choose a benchmark. Follow the +[local setup guide](local-benchmark.md#reproduce) for the complete VM, disk, +server and client preparation steps. + +| I need to… | Start here | +| --- | --- | +| Read the Cellar measurements and their limits | [Recorded result](#current-local-result) | +| Choose a harness and run a comparison | [Run a benchmark](#run-a-benchmark) | +| Understand throughput, tail latency and errors | [Read the report](#read-the-report) | +| Investigate an earlier optimization or release | [Detailed evidence](#detailed-design-and-recorded-evidence) | + ## Current Cellar reference -## Current local result +### Current local result -The [local benchmark](local-benchmark.md) records source `674e377`, Cellar `=0.1.0`, -three nodes at 1 vCPU / 3 GiB / 5 GiB each, and one table with ten data cells per -node. The separate object store and client are outside those node budgets. +**Recorded October 18, 2026, at source `674e377` with Cellar `=0.1.0`.** This is a +local characterization. BeyondDB has no qualified production throughput or +latency target. -| API at eight clients | Successful requests/s | p99 ms | -| --- | ---: | ---: | -| GetItem | 293.8 | 70.9 | -| Query | 150.2 | 335.7 | -| PutItem | 204.0 | 214.0 | -| TransactGetItems | 16.5 | 1193.0 | -| TransactWriteItems | 23.7 | 1002.0 | +![Recorded fixture: one table across three resource-limited nodes, with a separate object store](../diagram/operations/benchmark.svg) -![Measured throughput and p99; failed windows are marked](../diagram/perf-throughput-latency.svg) +| Condition | Recorded fixture | +| --- | --- | +| Database nodes | Three; **1 vCPU / 3 GiB RAM / 5 GiB disk each**; metadata and coordinator Cells share those limits | +| Table placement | **30 ordered data ranges**, ten per node; balancing paused and ownership verified | +| Shared infrastructure | One VM and physical disk; client and **1 CPU / 4 GiB RustFS** outside node budgets | +| Workload | Signed SDK calls, retries disabled, uniform **1 KiB payloads**, strong point/range reads | +| Request size | Get/Put: one item; Query: two items; transactions: two items on different nodes | +| Durability | Writes wait for bucket publication; follower durability disabled | +| Sweep | **1, 8, 30, 60 clients**; 30-second windows after five seconds of separate warmup | + +All five eight-client windows completed without public request errors. Rates and +latencies below count successful calls; samples are successful completions: + +| API | Requests/s | p99 ms | Samples | +| --- | ---: | ---: | ---: | +| `GetItem` | 293.8 | 70.9 | 9,064 | +| `Query` | 150.2 | 335.7 | 4,586 | +| `PutItem` | 204.0 | 214.0 | 6,125 | +| `TransactGetItems` | 16.5 | 1193.0 | 500 | +| `TransactWriteItems` | 23.7 | 1002.0 | 715 | + +![Measured throughput and p99 across the client sweep; failed windows are marked](../diagram/perf-throughput-latency.svg) + +**Higher-concurrency transaction qualification failed.** Across all 20 windows, +the sweep retained **65,276 successes, 101 measured errors and 16 warmup errors**. +Both transaction APIs returned 503s at 30 and 60 clients. Every window covered all +30 data Cells, but coverage alone does not make a failed window healthy capacity. +The eight-client transaction tails have fewer than 1,000 successful samples. + +The shared account Cell also logged **3,723 internal admission refusals**. These +include paced reads and are separate from public API errors. Shared VM/storage, +substantial host swap use and evolving table/coordinator state limit comparisons. +The run does not establish fleet capacity, automatic balancing performance or an +arrival-rate latency SLO. See the [full record](local-benchmark.md#measured-result--october-18-2026) +for resource observations, retained failed attempts and write read-back checks. + +## Run a benchmark + +| Harness | Use it for | Boundary | +| --- | --- | --- | +| [`scripts/perf`](../scripts/perf) | Configurable node/range placement; read, write and cross-Cell transaction sweep | Verifies every range and records ownership, resource counters, errors and samples | +| [`scripts/bench.py`](../scripts/bench.py) | Broader API comparisons against an existing disposable table | Requires string `pk`; optional numeric SK; does not prove traffic covered every Cell | -All 20 windows covered all 30 data cells. The sweep retained 65,276 successes, -101 measured errors and 16 warmup errors; transaction APIs failed at 30/60 clients. -Internal shared-account admission refusals were recorded separately from public -errors. At eight clients, transaction p99 has fewer than 1,000 successful samples. +### Choose the topology once + +Follow [Reproduce](local-benchmark.md#reproduce) to provision a dedicated fixture, +build the release server with `--locked`, install client dependencies and size the +VM for the database nodes **plus** the separate object store. The setup script +owns fixed fixture names; only one topology can occupy that fixture at a time. + +For example, initialize a **fresh private run directory** with four nodes and +eight data Cells per node: + +```sh +python3 scripts/perf/setup.py init --run-dir /path/to/fresh-run \ + --nodes 4 --cells-per-node 8 +``` -These short closed-loop windows share VM/storage and evolving state. They do not -measure independent-machine fleet capacity or an open-loop latency SLO. Earlier -failed trials and raw artifacts remain local; the guide records their limits. +The product `nodes × cells-per-node` must be **2–256**. Later actions load this +saved topology. The recorded result above uses three nodes and ten Cells per node; +other topologies have not been performance-qualified. Never rerun `init` over +retained keys or data. Complete store/node startup before preparing the table. -For new experiments, use [scripts/perf](../scripts/perf) and the -[current reproduction guide](local-benchmark.md). The full optimization rationale, -release comparisons, failure counts, hardware conditions and original -reproduction steps are retained below. No earlier runtime sample -is a current Cellar capacity or SQLite-parity claim. +### Measure and report + +After fixture startup, use the client Python environment and paths from the +[setup guide](local-benchmark.md#reproduce). Run `prepare` once for a fresh table; +it intentionally rejects an existing table. For a repeat measurement, retain the +same manifest/output directory and use a new label without running `prepare` again. + +```sh +"$PERF_PY" scripts/perf/bench.py prepare \ + --run-dir "$PERF_RUN" --output "$PERF_OUT" +"$PERF_PY" scripts/perf/bench.py measure \ + --run-dir "$PERF_RUN" --output "$PERF_OUT" \ + --operations transact_write transact_get put get query \ + --clients 1 8 30 60 --seconds 30 --warmup 5 --label main +"$PERF_PY" scripts/perf/report.py \ + --input "$PERF_OUT/results-main.json" --output "$PERF_OUT" --summary-clients 8 +``` + +Use the same source/dependency pins, durability, resources, payload, key +distribution, client settings and background features for a comparison. Capture +resource observations throughout the run. The [plot and report steps](local-benchmark.md#generate-the-report-and-charts) +retain failed cases; keep raw artifacts and fixture credentials local. + +## Read the report + +| Read together | Interpretation | +| --- | --- | +| Successful requests/s + actual elapsed time | Completed API calls divided by elapsed time, including in-flight drain; failed calls are not successes | +| Items/s + request shape | Two-item Query/transactions perform two item operations per successful API call | +| p50/p95/p99 + sample count | End-to-end **successful-call** latency, including SDK/signing, routing and durability; inspect error latencies separately | +| Errors + warmup errors | Warmup is retained separately; workers stop after three errors, so failed windows cannot establish healthy capacity | +| Range coverage + ownership | Every range needs successful traffic; verify placement before and after the sweep | +| Memory, disk, CPU + metadata inventory | Include system/coordinator Cells; throttled-period percentage is not lost wall time | +| Acknowledged write read-backs | Sampled correctness evidence; a failed mutation can still have an uncertain outcome | + +The two harnesses calculate percentiles differently. [`scripts/perf/bench.py`](../scripts/perf/bench.py) +uses nearest rank; [`scripts/bench.py`](../scripts/bench.py) selects sorted sample +index `floor(p × (n − 1))`. Both use successful calls only. Small transaction +samples do not establish a reliable tail-latency distribution. + +This is a **closed-loop concurrency sweep**: clients wait for replies before +sending more work. It does not correct for coordinated omission or measure an +open-loop arrival-rate SLO. Sustained mixed load, hot keys, background work, +splits and owner recovery need separate qualification before capacity planning. ## Detailed design and recorded evidence -> **Revision scope:** The current Cellar measurement is above. The optimization analyses and release comparisons below retain their original runtime, revision, durability, hardware and failures. In these records, “current”, “latest” and “now” mean current at that recorded revision, not the present server. Do not reuse their old configuration or single-cell creation instructions. +The complete optimization journal follows, grouped by topic. Expand a section +for its original rationale, tables, failures and reproduction steps. + +> **Revision scope:** Earlier “current”, “latest” and “now” describe the recorded +> revision. Historical Cellule results, old configuration and single-Cell fixtures +> are not current Cellar qualification or SQLite-parity claims. Use the setup guide +> above for new experiments. > Snapshot: [original detailed document](https://github.com/crabbuild/beyonddb/blob/a1eb96a0e87d4473f1bec89c0f3b4f2b02a21afb/docs/performance.md). +| Topic | Includes | +| --- | --- | +| Context and owner recovery | Original scope, active-owner lookup, restoration and placement | +| Durability and optimization rationale | SQLite cost boundary, heartbeat writes, prepare batching, replay and reply sizing | +| Release qualification | Failed coalesced-prepare release and earlier revision-scoped comparisons | +| Early fixtures and request costs | Object-publication samples, SQLite comparisons and local-node experiments | +| Diagnostics and reproduction | Runtime observations, counter limits, earlier commands and percentile interpretation | + +
+Original context, owner lookup and recovery + The current Cellar `=0.1.0` three-node characterization is documented in [the local benchmark](local-benchmark.md). It verifies ten table data cells per node under 1 vCPU / 3 GiB / 5 GiB data limits, and records both @@ -82,6 +195,11 @@ Authentication also restores a cataloged credential shard after its owner expire The reviewed Cellule EOF fix keeps observed object streams finished after completion. Both new regressions reproduce the original panic before the fix. With the updated pin, the original follower durability process test passes, including withheld object publication, owner process kill, recovery, and graceful shutdown. This closes that local failure; full fleet recovery and sustained performance still require qualification. +
+ +
+Durability costs and optimization rationale + ## Why writes and transactions cost more than SQLite The pinned ExtendDB SQLite backend uses WAL with `synchronous=NORMAL`. SQLite documents that this mode does not synchronize the WAL after every commit; `FULL` adds a synchronization for each commit. BeyondDB waits for object-store publication or a durable follower receipt before acknowledging a mutation. The benchmark therefore compares different durability paths. [SQLite durability reference](https://www.sqlite.org/pragma.html#pragma_synchronous) @@ -199,6 +317,11 @@ A large image returns an explicit `WideRequired` marker, then uses the existing This reduces reserved reply memory for small items. The HTTP peer receiver also retains its own request buffer, so total in-flight memory exceeds the 64 KiB reply alone. End-to-end throughput, large-item latency and SQLite parity require separate release measurements. Existing query opcodes and persisted transaction/image formats remain available; upgrades of roots created by older application releases still need separate qualification. +
+ +
+Release qualification: failed coalesced prepares and scoped-read releases + ## Recorded release verification @@ -263,6 +386,11 @@ The regression proves stalled log I/O held the heartbeat's shared mutex. The fix The follow-up renewal-phase diagnostic repeats the original four-Cell workload at source `7cba575`, adding operational failure observations with unchanged deadlines and durability. All 24 cases run: 8,502 completions, **six timeouts** (five transaction reads, one transaction write). No renewal warning or terminal Fenced error occurs; the earlier endpoint loss is not reproduced and remains unresolved. Eight-client transaction writes complete eight requests at 0.60 req/s with successful p95 7,613.65 ms. This is a diagnostic, with no fresh SQLite comparison or speedup claim. SDK-window load is 74.40→49.94 on 12 CPUs, with about 37 GiB of swap. Three PIDs and one container are absent without forced cleanup. A retained mailbox refusal and incomplete-resolution warning do not establish the cause of a specific timeout. +
+ +
+Earlier release verification: member expiry through placement + ### Previous Cellule member-expiry release verification The Cellule member-expiry release refresh measures committed source `61a74a4`, pinning reviewed Cellule `0f4ca09`. All **72 cases run**. Four-Cell BeyondDB and SQLite complete 24 cases each with zero request errors; single-Cell BeyondDB records **eight timeouts** in its concurrent transaction-write case. **The all-API SQLite performance goal remains unmet.** @@ -468,6 +596,11 @@ Host load was 17.93→16.75 for single, 17.46→23.30 for partitioned, and 24.15 The placement verification passes three signed model tests, five creation/lifecycle tests, two statistics tests, 27 library tests, formatting and strict Clippy. The original record captured Rust CI success and SDK CI pending. Subsequent production-equivalent SDK CI failed as described above; full recovery and older-root upgrades remain unqualified. The new Single variant requires matching compiled peers. +
+ +
+Earlier release pairs: native volumes through resident routing + ### Previous native-volume release pair The native-volume release pair @@ -788,6 +921,11 @@ process CI passed, but one long peer recovery test failed; full qualification remains open. The approved one-read follower append change is not included in this measured release. +
+ +
+Earlier release pairs: runtime upgrade and instrumented diagnostics + ### Previous Cellule upgrade release pair The Cellule `70bd25f` release rerun @@ -968,6 +1106,11 @@ handle authority reads, refreshes after expiry, and rejects unauthorized or drained-owner requests. That focused proof does not establish end-to-end SQLite parity. +
+ +
+Early fixtures, refreshed samples and pinned SQLite comparison + ## Earlier object-publication fixture The release repeat after the peer read fix @@ -1044,6 +1187,11 @@ path before expecting local SQLite write latency from this RustFS fixture. The [follower durability design](follower-durability.md) lists the BeyondDB integration and recovery gates required before that mode can be benchmarked. +
+ +
+Request stages and local-node release experiments + ## What a request waits for ```text @@ -1137,6 +1285,11 @@ The pinned ExtendDB `BatchWriteItem` handler awaits each item mutation in a requ Temporary stage timings on a separate instrumented fixture put typical credential, IAM, table record, and data Cell reads around 3–4 ms each, while each route traversal took around 6–7 ms. The requests perform several of these operations in sequence. The instrumented write fixture differed materially from the clean fixture, so its write timings are not a publication-cost estimate. A temporary S3 proxy disrupted publication and its counts were discarded. +
+ +
+Diagnostic counters, timing scopes and durability acknowledgements + ## Observe runtime costs and durability acknowledgements For a diagnostic run, add an absolute path to the server's JSON configuration: @@ -1200,6 +1353,11 @@ change in `total_us` by the change in `count` to estimate a phase's mean duratio Phases can overlap and cover different events; do not sum them into SDK latency. Use the signed SDK harness for end-to-end latency and foreground errors. +
+ +
+Earlier reproduction commands and result interpretation + ## Run a repeatable point-operation sample > The commands in this recorded section reproduce the earlier experiment. @@ -1248,3 +1406,5 @@ The current [harness](../scripts/bench.py) sorts **successful** request duration Short, closed-loop runs are useful for comparing code changes under the *same* fixture. They are not a peak TPS rating. Report at least the build profile and Git revision; server and client CPU; object-store latency and request counts; table partitions; item size and key distribution; Streams/index settings; client concurrency; errors; p99; and recovery after owner loss. Test hot keys separately from evenly spread keys. Run sustained mixed read/write load, table creation, splits, transactions, and restarts before using a result for capacity planning. A healthy `/health` response alone does not prove that background work is keeping up. Do not use a debug build or a node reporting mailbox exhaustion to set a capacity target. Fix readiness and background-work errors before comparing throughput. + +
diff --git a/docs/scaling.md b/docs/scaling.md index fd80a04..48b5b62 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -1,22 +1,153 @@ -# BeyondDB elastic Cell topology +# Scale BeyondDB with ordered range Cells + + + +A table grows through multiple ordered data ranges, each owned by a Cell. +Cellar places those Cells on nodes; BeyondDB maintains the table's logical +ranges and routes. Capacity remains bounded by the Cells, hosts and shared +services doing the work. + +| I need to… | Start here | +| --- | --- | +| Understand tables, ranges and node owners | [Current Cellar reference](#current-cellar-reference) | +| Choose initial ranges and residency limits | [Creation and capacity](#creation-and-capacity) | +| Understand moves, splits and hot keys | [Moving and splitting](#moving-and-splitting) | +| Plan growth and judge qualification | [Growth decisions](#growth-decisions) | +| Inspect detailed designs and failure evidence | [Detailed record](#detailed-design-and-recorded-evidence) | ## Current Cellar reference -The current server uses ordered ranges, defaulting to four initial cells per -table (2–256 supported). `auto` and `partitioned` use the configured multiple -ranges; `single` is rejected. Native Cellar owns physical placement and balancing. +![One table spans ordered range Cells; native Cellar selects their physical node owners](../diagram/operations/ordered-ranges.svg) + +| Responsibility | Current contract | +| --- | --- | +| Logical partitioning | BeyondDB routes the complete ordered `(PK, SK)` into adjacent half-open ranges | +| Physical ownership | Cellar selects and fences the node owning each Cell; fleet mode enables native rebalancing | +| Point requests | Resolve one range using the full primary key | +| Query | Requires PK equality; visits intersecting SK ranges in order | +| Local indexes | LSIs stay with their base items and preserve PK-collection locality | +| Global indexes | GSIs have independent ordered ranges and asynchronous projection | +| Transaction recovery | Resolve the admitted original participants; preserve identities through topology changes | + +Multiple ranges on one node share its failure domain. More ranges do not +guarantee more hosts, balanced traffic or higher write throughput. See +[ordered partitioning](ordered-cell-partitioning.md) for encoding and pagination, +and [server boundaries](cellar-server.md#server-layout) for component ownership. + +## Creation and capacity + +Merge these fields into a complete [node configuration](cellar-server.md#configuration). +The values below are the current defaults: + +```json +{ + "initial_partitions": 4, + "max_active_cells": 128, + "split_threshold_bytes": 268435456 +} +``` + +| Setting / budget | Meaning | +| --- | --- | +| `initial_partitions` | **2–256** initial base/GSI ranges per table/index; applies to creation, not existing-table repartitioning | +| `beyonddb:cell-model` | Omitted, `auto` or `partitioned` use the configured multi-range policy; `single` is rejected | +| Creation policy | Persisted with the table generation; changing tags later does not convert placement | +| `max_active_cells` | Native per-node residency cap for **all Cell types**, including metadata, inventory and coordinators | +| `split_threshold_bytes` | Range-maintenance threshold; **268435456 = 256 MiB** by default; not usable item capacity | +| Compiled data/GSI budget | **512 MiB database / 64 MiB capture** per Cell; includes more than item payloads | + +![Data/GSI Cell budgets, node resource and residency budgets, and shared service contention must all be considered](../diagram/operations/scaling-capacity.svg) + +Size for local indexes, transaction reservations, journals, stream records, +retained history, recovery and split scratch space. Increasing residency does +not reserve RAM or disk, raise a Cell's compiled budget, or remove a shared +metadata bottleneck. The parser rejects old resource fields such as +`disk_budget_bytes`, `node_retained_bytes` and `sql_workers`. + +## Moving and splitting + +![Moving preserves one Cell and its interval; splitting creates child Cells and replacement range routes](../diagram/operations/scaling-move-split.svg) + +| Operation | Safety boundary | +| --- | --- | +| Move a Cell | Native ownership transfer/recovery keeps the logical Cell identity; physical-owner fencing still applies | +| Choose a split | Select an observed key boundary; without LSIs, a PK's SK siblings can span Cells | +| Seal the source | Unresolved participant work and pending GSI projection entries must drain before sealing | +| Publish children | Retain the durable plan, copy and verify children, publish replacement routes, then open children | +| Recover or retry | Continue the original durable protocol; a lost mutation reply does not establish nonexecution | + +For tables with LSIs, split only between PK collections. One indivisible hot +item still has one serialized write authority. Deterministic initial fences do +not predict workload balance, and the observed-median split policy is not a +byte/load-weighted placement guarantee. + +Source sealing can interrupt access. Online delta catch-up, merges and broad +split-during-pagination qualification remain open. See the +[split safety reference](ordered-cell-partitioning.md#split-safety-and-what-remains-open), +[GSI cutover](global-indexes.md#split-and-recovery-reference) and +[transaction recovery](cross-cell-transactions.md#recovery-and-stable-identities). + +## Growth decisions + +| Observed pressure | Review before changing capacity | +| --- | --- | +| Resident-Cell admission failures | Count every Cell type and retained work; compare memory/disk headroom with residency limits | +| One range fills or queues writes | Inspect key distribution, LSI locality and pending work; verify that a legal split can make progress | +| One node is busy | Check eligible fleet peers, placement/recovery health and shared-service pressure before adding capacity | +| Many ranges, limited throughput | Inspect account/catalog contention, coordinator work, projection lag and object-store latency | +| Recovery or maintenance falls behind | Measure backlog age, retained bytes and restore cost alongside foreground throughput | + +The [local benchmark](local-benchmark.md) exercised **30 data Cells on three +nodes**, including transaction failures at higher concurrency. It is a shared-VM +characterization, not a production sizing rule. Use the +[performance guide](performance.md) to compare throughput, latency, samples, +errors, resource use and range coverage together. + +### Qualification still required + +**10,000 active Cells and multi-TB storage are design targets.** Passing a focused +split or restart test does not establish that capacity. The retained delivery +plan requires the following evidence: + +| Area | Required proof | +| --- | --- | +| Fleet growth and recovery | Add/remove nodes under SDK load; preserve acknowledged data and reject stale owners through interrupted movement | +| Metadata growth | Bounded route lookup, directory updates and table-name ownership as one account/table grows | +| Large collections and indexes | Same-PK SK growth where eligible, correct Query/Scan continuation, and version/tombstone-preserving GSI cutover | +| Coordinator scale | Stable token lookup during expansion, distributed recovery and safe bounded history retirement | +| Operational capacity | Runs at **1,000 then 10,000 active Cells**, real multi-TB data, declared hardware/durability and measured latency/recovery limits | -The original capacity model, controller boundaries, split/recovery mechanisms, -acceptance gates and measured regressions remain below. Metadata/coordinator -cells count against `max_active_cells`; old JSON resource knobs are not accepted. -See [server configuration](cellar-server.md#configuration) and the -[current local result](local-benchmark.md) before applying any sizing advice. +Use [Rust CI](../.github/workflows/rust.yml) and +[signed SDK qualification](../.github/workflows/qualification.yml) as the current +check inventory. Fleet-scale recovery, retention and composed capacity remain +unqualified. Historical timing and test results below apply to their recorded +revisions, fixtures and runtime. + +## Current source map + +| Contract | Source / reference | +| --- | --- | +| JSON defaults, residency and native balancing | [Executable](../src/bin/beyonddb.rs), [server configuration](cellar-server.md#application-policy-and-capacity) | +| Creation selector and ordered bootstrap | [Creation policy](../src/backend/table_creation.rs), [provisioning](../src/cellar/provision.rs) | +| Split selection and orchestration | [Observed split point](../src/partition/split_point.rs), [split controller](../src/split.rs) | +| Compiled database/capture limits | [Data application](../src/lib.rs), [GSI application](../src/global_index.rs) | +| Maintenance and recovery | [Server composition](../src/cellar/server.rs), [transaction protocol](cross-cell-transactions.md) | ## Detailed design and recorded evidence -> **Revision scope:** The current server uses ordered multi-cell tables and native Cellar placement. Hash layouts, one-cell creation, application-owned placement loops and old resource settings in the recorded sections describe earlier revisions, not supported deployment options. The target capacity and acceptance criteria remain design goals. +The complete design and qualification journal follows in expandable topics. +It retains capacity assumptions, fault cases, acceptance gates and measurements. + +> **Revision scope:** Earlier “current” and “now” refer to the recorded revision. +> Hash routing, one-Cell creation, application-owned placement loops and retired +> resource knobs are historical. Target controllers, merges, backup protocols and +> capacity criteria are design material unless separately qualified. Use the +> current reference above for supported configuration and ordered routing. > Snapshot: [original detailed document](https://github.com/crabbuild/beyonddb/blob/a1eb96a0e87d4473f1bec89c0f3b4f2b02a21afb/docs/scaling.md). +
+Original overview, recorded status and historical table selectors + The Cellar contract requires ordered `(PK, SK)` ranges and multiple data cells for every table. See [ordered cell partitioning](ordered-cell-partitioning.md) for the new contract and delivery gates. The HASH routing and optional single-cell @@ -111,6 +242,11 @@ determine placement. Current tables always start with multiple ordered ranges. Small same-Cell transaction reads can use one atomic query. Tokenized transaction writes still require the account-scoped coordinator, and cross-table transactions can span Cells. Placement therefore reduces some transaction work without establishing SQLite write parity. See [transaction execution paths](cross-cell-transactions.md#one-cell-and-multiple-cell-requests) and [measured performance](performance.md). +
+ +
+Capacity target, controller boundaries and delivery gates + ## Horizontal scaling delivery gates The agreed target is 10,000 active Cells and multi-TB stored data. The following @@ -287,6 +423,11 @@ Scheduling and storage latency reduce the achieved rate. This is a source-derived workload estimate, not measured fleet throughput; extending the interval or removing renewals requires proving the runtime fencing contract. +
+ +
+Recorded routing, maintenance, peer recovery and finite bounds + ## Scope and present boundary "Unlimited" is not a literal capacity promise. DynamoDB requests, items, one @@ -496,6 +637,11 @@ can still encounter release or sustained pressure and retry from durable state. data/account/credential activation, distributed placement, retained-history collection, and measured overload/recovery behavior remain scale gates. +
+ +
+Original ownership, split/merge, transaction and completion design + ## Target ownership ```text @@ -690,6 +836,11 @@ used by an AWS SDK and establish all of the following: Until these gates pass, BeyondDB is a bounded prototype, regardless of the declared database budget or the number of Cells the framework can address. +
+ +
+Deleted-table residency and reclamation evidence + ## Table deletion and active residency The independent ExtendDB Python item suite exposed a serving limit after 31 @@ -786,6 +937,11 @@ Verification on 2026-09-26: - Strict all-target Clippy, compiled server build, Rust formatting, Python Ruff, and diff checks passed. No fleet-scale or multi-TB qualification was performed. +
+ +
+Concurrent admission, lease timing and shutdown evidence + ## Concurrent request admission HTTP composition opts into bounded Cell-client waiting. Clones used by storage, @@ -958,6 +1114,11 @@ The draft remains unqualified for sustained load and production scale. The large-transaction smoke failure needs underlying invocation evidence before assigning a cause or changing capacity, deadlines, or retry policy. +
+ +
+Large transaction payload and transfer qualification + ### Large transaction transfer qualification Temporary probes reproduced the compiled-server failure in 515.27 seconds at @@ -1103,6 +1264,11 @@ fences an untouched Cell; this is a candidate mechanism, not yet proof of where the full workload spent its deadline. SQL/page I/O and lifecycle timing still need distinction if that reproduction does not explain the observed path. +
+ +
+SQL queue expiry, collateral fencing and diagnostic follow-ups + ### SQL queue expiry and collateral fencing The focused regression failed before the runtime change in 5.11 seconds: @@ -1535,6 +1701,11 @@ This is a fresh hard-kill scenario with the unchanged 45-second gate. These single-run timings are not a controlled performance comparison or fleet recovery qualification. +
+ +
+Measured placement, cold restoration and initial creation recovery + ### Measured node placement inputs The binary now uses the signed capacity path described above instead of fixed @@ -1727,6 +1898,11 @@ Production Rust grows by 213 net lines for the shared peer composition, scoped provisioning capability, and common range admission/recovery boundary. No wire shape, storage schema, dependency, or configuration setting changed. +
+ +
+Independent splits, unresolved source work and index growth + ## Independent source-range splits Pending data splits are keyed by `(table_id, source_partition_id)`. Each source @@ -1888,6 +2064,10 @@ after route publication before either child opens. These are bounded fixtures; cutover latency qualification remain outstanding. An indivisible HASH group can still exhaust one Cell; sort-key subranges remain necessary. +
+ +
+Sealed-source residency and child-opening recovery ## Sealed-source residency after splits @@ -2098,6 +2278,11 @@ without production fault injection or weaker assertions/timeouts. Production code grows by 38 net lines to share restoration and verify discovery readiness. These are focused local results; the broader process qualification remains in CI. +
+ +
+Usage/statistics, coordinator images and retired-range reclamation + ### Bounded base-range usage measurement `PartitionUsage` previously counted every item row and summed its stored BLOB @@ -2310,6 +2495,10 @@ Clippy, the standalone server build, formatting, diff, Cell/LTX layout and policy entry-point checks pass. Full SDK/process qualification remains a CI gate for the pushed head. +
+ +
+Creation resumption, persisted placement and generation-scoped deletion ## Resume incomplete table creation @@ -2471,6 +2660,11 @@ That intermittent failure is not resolved by this deletion change. Large routed/GSI deletion and process-level recovery still require their broader qualification gates. +
+ +
+Session retirement, peer admission and cold-restoration diagnostics + ## Graceful node-session retirement CI run `36313605590` passed all 20 peer-network cases but failed the standalone @@ -2578,6 +2772,11 @@ this did not reproduce or explain the original failure. The delay was removed. CI retains server warnings so a recurrence can distinguish admission, ownership and routing failures. The intermittent Scan failure remains open. +
+ +
+Discovery freshness and historical settled-range movement + ## Placement freshness after fleet discovery The SDK discovery regression on `21a8c0cf763` fails in 4.41s with @@ -2697,6 +2896,11 @@ the graceful node-session retirement fix on that head. It predates the codec, discovery-timing and range-rebalance increments and does not qualify those changes or establish the cause of the intermittent Scan failure. +
+ +
+Bounded activation replay, restart and final fixture qualification + ## Bounded activation replay Metadata review found two full-directory comparisons in the native activation @@ -2868,3 +3072,5 @@ policy was changed. Final BeyondDB/runtime all-target Clippy with runtime test-support passes in 6.42s after these fixture changes. Format, layout, policy-entry and documentation checks also pass; production Rust remains the same bounded retirement change. + +
diff --git a/docs/user-guide.md b/docs/user-guide.md index f3dd567..2566dd1 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -189,5 +189,5 @@ A retry can outlive the request that caused it. Use the response and current tab control-plane request was rolled back. The [API matrix](api.md) lists explicit unsupported cases. The -[implementation status](implementation-status.md) records focused SDK and -restart evidence, including known load failures. +[server guide](cellar-server.md#detailed-qualification-record) records focused SDK +and restart evidence; [performance](performance.md) retains measured load failures.