Skip to content

feat(TC-887): the FileSystem block store seals KV blocks - #519

Draft
samgbafa wants to merge 1 commit into
skgbafa/tc-887-r4-kv-sealedfrom
skgbafa/tc-887-r5-fs-sealed
Draft

samgbafa wants to merge 1 commit into
skgbafa/tc-887-r4-kv-sealedfrom
skgbafa/tc-887-r5-fs-sealed

Conversation

@samgbafa

@samgbafa samgbafa commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

TC-887, part 5 of the refreshed at-rest stack (base: #518). Replaces #333 (skgbafa/tc-887-s3b-fs-sealed), ported onto current main.

What

  • FileSystem store seals KV blocks. In encrypt/strict with the gate open, a block is sealed (TCE1) into a .tc-stage- file beside its sealed name {blocks}/v1/{space_tag}/{object}[@N] and renamed into place, under main's per-generation operation lock and with main's reserve/settle/refund quota flow (charged in plaintext bytes). Reads and ranges serve only authenticated plaintext, with plaintext totals; legacy is read in off/encrypt, refused in strict; locked keys answer 503 for anything that may be sealed. Mode off takes main's legacy path unchanged.
  • Core hooks. ImmutableReadStore::{prepares, prepare_read} and ImmutableWriteStore::prepare_write (defaults: no-op), forwarded by Box and Either.
  • db.rs ordering (TC-735 aware). The prepare step runs right after the preupload transaction returns (it authorized the invocation and found its spaces hosted; nothing is locked) and before any upload, sequence/writer lock or transaction: prepare_write for written spaces (creates the data key), prepare_read for spaces a kv/get reads inside the transaction (restores a lost data key from the mirror). Work under the writer lock runs in at_rest::write_locked: the protected legacy upload (which holds the locks across persist) and get_kv in the transaction. Ordinary uploads stay lock-free and outside it.
  • feat(TC-887): seal KV blocks in the FileSystem store; prepare writes before the invoke lock #333's extra pre-authorization (review B1) is dropped: main's preupload transaction already authorizes and checks hosting before any key work, so a refused put creates no key. Kept as tests: tests/at_rest_fs_store.rs::a_refused_put_creates_no_data_key_or_mirror_object, and the core ordering test below.
  • No prepare in invoke_read_only (deviation from the plan, on purpose): read-only invocations and the KV GET routes hold no writer lock, so the keyring restores a lost data key on first use there; preparing would add a database query per read. Covered by a_lost_data_key_row_is_restored_before_the_writer_lock (a read after the row is lost restores it).
  • Orphan collection (remove, cleanup_stages) handles both names: it removes the sealed copy and the legacy copy of the generation, and refuses (503, the collector retries) while keys are set up but locked, so a sealed object is never left unnamed.
  • Boot scan: .tc-stage- files under v1/ are charged their stored bytes and tracked like main's stages in space directories.
  • storage::attach_at_rest runs in lib.rs before orphan_store = block_store.clone(); for the FileSystem store it attaches the runtime and calls declare_seal_path(SealPath::Blocks). S3 declares nothing yet (next PR). encrypt/strict still boot only once SQL artifacts declare their seal path too.
  • Docs: docs/at-rest/INTERFACES.md §7 (prepare ordering, removal, accepted residuals); CHANGELOG.

Tests

  • storage::file_system::sealed_tests (ported + new: generation names, removal of both copies and refusal while locked, crashed sealed stage cleanup/refund, failed persist leaves no files and refunds).
  • tests/at_rest_fs_store.rs (new, runs in CI): SpaceDatabase + FileSystem store + a real AtRestKeyring on SQLite with the FS key mirror: only TCE1 under v1/ names, plaintext and range reads; a refused put creates no key row and no mirror object; a lost data-key row is restored before the writer lock for a protected legacy write (would be NotPrepared 503 otherwise).
  • tinycloud-core db::test::storage_prepares_spaces_after_authorization_and_before_the_writer_lock: hook order and lock state for an unhosted put, an ordinary put, a protected legacy put, and a get beside a put.
  • tests/at_rest_fs_node.rs (ported, app level): the four encrypt tests are #[ignore = "needs the SQL artifact seal path too (TC-887)"] because the app refuses encrypt until both seal paths exist: an_encrypting_filesystem_node_seals_kv_values_and_serves_plaintext, a_refused_put_creates_no_data_key_or_mirror_object, an_unauthorized_put_to_a_hosted_space_creates_no_data_key, a_mixed_read_and_delete_after_losing_the_data_key_row_completes. an_off_node_still_stores_plaintext_under_legacy_names runs.

Verification (run locally)

  • cargo fmt --all; RUSTFLAGS=-Dwarnings cargo clippy --locked -p tinycloud-node -p tinycloud-core --all-targets: clean
  • cargo test --locked -p tinycloud-node --lib: 1024/1025 passed; the one failure was egress::tests::tc1132_tls_connect_stall_is_bounded_before_total_timeout (a wall-clock bound, on a loaded host; an earlier run of the same suite failed three policy_v3::account_alias_tests instead, which pass alone and with policy_v3:: as a module). Neither touches storage.
  • cargo test --locked -p tinycloud-node --lib storage::: 72 passed
  • cargo test --locked -p tinycloud-node --test at_rest_fs_store --test at_rest_fs_node --test at_rest_ceremony --test did_resolution_single_entry --test space_size_restart --test kv_invoke_replay --test kv_batch_read: all passed (4 ignored as listed)
  • cargo test --locked -p tinycloud-core --lib: 1087 passed

Sealed names v1/{space_tag}/{object}[@n] with same-directory .tc-stage-
publish, plaintext quota reservations, and reads/ranges that serve only
authenticated plaintext. Core storage traits get prepares/prepare_read/
prepare_write; a writing invocation prepares its spaces after the TC-735
preupload transaction authorized it, before any upload or write lock, and
work under the writer lock runs in at_rest::write_locked. Orphan removal
deletes both copies and waits while keys are locked. The FileSystem store
declares the blocks seal path.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant