Skip to content

feat(TC-887): the S3 block store seals KV blocks - #520

Draft
samgbafa wants to merge 1 commit into
skgbafa/tc-887-r5-fs-sealedfrom
skgbafa/tc-887-r6-s3-sealed
Draft

samgbafa wants to merge 1 commit into
skgbafa/tc-887-r5-fs-sealedfrom
skgbafa/tc-887-r6-s3-sealed

Conversation

@samgbafa

@samgbafa samgbafa commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

TC-887, part 6 of the refreshed at-rest stack (base: #519). Replaces #336 (skgbafa/tc-887-s3c-s3-sealed), ported onto current main (aws-sdk-s3 1.x, TC-735 generations, pending/deleting quota, BlobOperationLocks).

What

  • S3 store seals KV blocks under v1/{space_tag}/{object}[@N], inside main's put_if_absent (per-generation lock, HEAD dedup on the target name, pending reservation charged in plaintext bytes). File staging seals into a temp file and uploads from it; memory staging seals in memory. Only the sealed name dedups.
  • Reads (kept from feat(TC-887): seal KV blocks in the S3 store (S3c, stacked on #333) #336's review): whole reads open the sealed object and serve authenticated plaintext; range reads GET the header, then GET the covering segments pinned to its ETag (If-Match), and require the returned Content-Range to equal the computed ciphertext span; a replaced/removed object restarts once, then 503, never falls back to legacy. strict probes the legacy name only to answer "legacy refused". Legacy reads are main's code paths.
  • remove deletes the sealed copy first, then the legacy copy, each with its own lost-DELETE charge map (deleting stays the legacy, boot-listing one; deleting_sealed is new), and refunds an unlanded upload's reservation only when neither copy nor charge existed. With keys set up but locked it deletes nothing (503) and the orphan collector retries. HeadObject sizing throughout.
  • contains checks the name a write would use; prepares/prepare_read/prepare_write as in the FileSystem store; sizes join the space tag.
  • storage::attach_at_rest now attaches and declares the blocks seal path for both stores.
  • tests/at_rest_s3_store.rs (MinIO, --ignored), added to the at-rest-harness job in rust.yml.
  • Docs: INTERFACES §7 (S3 range pinning, deletion-charge residual); CHANGELOG.

Open risk (documented in §7)

The S3 boot listing can't name sealed objects (keys arrive after storage opens), so for a sealed object whose DELETE lands after a restart's listing, the charge stays counted until the next restart: an over-count, never an under-count.

Verification (run locally)

  • cargo fmt --all; RUSTFLAGS=-Dwarnings cargo clippy --locked -p tinycloud-node --all-targets: clean
  • cargo test --locked -p tinycloud-node --lib storage::: 72 passed
  • cargo test --locked -p tinycloud-node --test at_rest_fs_store --test did_resolution_single_entry: 3 + 10 passed
  • Harness (scripts/at-rest-harness/harness.sh up, env loaded): cargo test --locked -p tinycloud-node --test at_rest_s3_store --test s3_block_store -- --ignored: 8 + 1 passed (round trip and ranges, generation names, legacy/locked/strict/tamper, removal of both copies and refusal while locked, temp-file staging, contains, replaced-object ETag pinning)

@samgbafa
samgbafa force-pushed the skgbafa/tc-887-r6-s3-sealed branch from 67679b2 to fce1bfb Compare November 8, 2026 00:41
@samgbafa
samgbafa force-pushed the skgbafa/tc-887-r5-fs-sealed branch from 11728ac to 203ea34 Compare November 8, 2026 00:41
Sealed writes inside put_if_absent (plaintext reservations), whole reads,
range reads pinned by ETag with a checked Content-Range, removal of the
sealed copy then the legacy copy with per-copy deletion charges, and
refusal while keys are locked. Both block stores declare the blocks seal
path. MinIO tests run in the at-rest harness job.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant