feat(TC-887): the S3 block store seals KV blocks

.github/workflows/rust.yml

@@ -426,6 +426,12 @@ jobs:
         (set -a; . "$TC887_RUN_DIR/harness.env"; set +a
          cargo test --locked -p tinycloud-node --test s3_block_store -- --ignored)
 
+    - name: S3 block store with encryption at rest (against the harness MinIO)
+      timeout-minutes: 10
+      run: |
+        (set -a; . "$TC887_RUN_DIR/harness.env"; set +a
+         cargo test --locked -p tinycloud-node --test at_rest_s3_store -- --ignored)
+
     - name: SQL object store (against the harness MinIO)
       timeout-minutes: 10
       run: |

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## [Unreleased]
 ### Changes
+- Encryption at rest for KV blocks, S3 store (TC-887, off by default). The S3 block store seals new KV blocks under `v1/{space_tag}/…` names like the FileSystem store, with the same mode rules; a sealed range read fetches the header, then only the covering segments, pinned to the same object version (`If-Match`), and answers `503` if the object changes twice during the read. Orphan collection deletes both copies of a block and waits while keys are locked. Both block stores now declare the blocks seal path. Mode `off` is unchanged.
 - Encryption at rest for KV blocks, FileSystem store (TC-887, off by default). With `at_rest.mode = "encrypt"`/`"strict"` and keys set up, new KV blocks on FileSystem block storage are sealed (TCE1) under `{blocks}/v1/{space_tag}/…` and served back as plaintext, ranges included; legacy blocks stay readable in `encrypt`, are refused in `strict`, and locked keys answer `503` for sealed data. A writing invocation prepares its spaces' data keys after its preupload authorization and before any upload or write lock, so a refused request creates no key. Orphan collection deletes both copies of a block and waits while keys are locked. The FileSystem store declares the blocks seal path; `encrypt`/`strict` still boot only once SQL artifacts declare theirs. Mode `off` is unchanged. See `docs/at-rest/INTERFACES.md` §7.
 - Encryption at rest groundwork for KV blocks (TC-887, nothing is sealed yet): the S3 and FileSystem block stores count sealed objects (`v1/{space_tag}/…`) by plaintext size at boot, per space tag, and a space's quota joins them once the at-rest keys can name the tag. The S3 store makes one more `ListObjectsV2` listing at boot, under `v1/`. At-rest storage failures answer `503` (keys locked or key store unavailable) or `500` (integrity, unsupported format, legacy refused in `strict`) on `/invoke` and the KV GET routes. Sealed names carry the same `@N` generation suffix as legacy names (`docs/at-rest/INTERFACES.md` §5).
 - Deploys: the workflow supports an explicit, reason-recorded rollback to an earlier released dstack image on the unchanged N0 compose. It checks image labels, release tag, ancestry, and identical migration sources while preserving the normal forward-release guard.

docs/at-rest/INTERFACES.md

@@ -304,7 +304,14 @@ redacted.
 - **Accepted residual (header trust):** `Content-Length`/`Content-Range`/416 totals come from the
   unverified header. A forged header can only produce a wrong total or a body that aborts with
   no unauthenticated bytes (test `storage::sealed::tests::forged_header_length_aborts_without_extra_bytes`),
-  never wrong bytes.
+  never wrong bytes. On S3 a range read's segment GET is pinned to the header GET's ETag
+  (`If-Match`) and its `Content-Range` must equal the computed ciphertext span: if the object is
+  replaced or removed in between, the read starts over once, then answers 503. It never falls back
+  to the legacy name once a sealed header was read.
+- **S3 deletion charges:** a lost DELETE response is settled per copy (legacy and sealed are
+  tracked apart). The boot listing can't name sealed objects (keys arrive after storage opens), so
+  a sealed object whose DELETE lands after a restart's listing stays counted until the next
+  restart (a conservative over-count, never an under-count).
 - **Quota:** plaintext bytes. Sealed objects are scanned by `space_tag` and joined to a space when
   the object-name key is unlocked (lazily, per lookup), so the first quota check after an unlock
   counts legacy + sealed. While locked, a space reports legacy bytes only (writes are 503 then).

tinycloud-node-server/src/storage/mod.rs

@@ -19,20 +19,16 @@ use crate::{
     BlockStores,
 };
 
-/// Links the opened block store to the node's at-rest state (TC-887).
-///
-/// The FileSystem store seals KV blocks when the mode and keys allow, so it
-/// declares the blocks seal path. The S3 store does not seal in this build
-/// and declares none, so `at_rest::setup` refuses a non-`off` mode with S3
-/// block storage rather than write plaintext.
+/// Links the opened block store to the node's at-rest state (TC-887). Both
+/// stores seal KV blocks when the mode and keys allow, so attaching either
+/// declares the blocks seal path.
 pub fn attach_at_rest(stores: &mut BlockStores, runtime: &Arc<AtRestRuntime>) {
+    let handle = sealed::AtRestHandle::new(runtime.clone());
     match stores {
-        Either::B(fs) => {
-            runtime.declare_seal_path(SealPath::Blocks);
-            fs.attach_at_rest(sealed::AtRestHandle::new(runtime.clone()));
-        }
-        Either::A(_) => {}
+        Either::A(s3) => s3.attach_at_rest(handle),
+        Either::B(fs) => fs.attach_at_rest(handle),
     }
+    runtime.declare_seal_path(SealPath::Blocks);
 }
 
 /// One physical blob lifetime; a later generation must never share its