feat(TC-887): seal SQL and DuckDB artifacts at rest

CHANGELOG.md

@@ -3,9 +3,10 @@
 ## [Unreleased]
 ### Changes
 - Encryption at rest for KV blocks, S3 store (TC-887, off by default). The S3 block store seals new KV blocks under `v1/{space_tag}/…` names like the FileSystem store, with the same mode rules; a sealed range read fetches the header, then only the covering segments, pinned to the same object version (`If-Match`), and answers `503` if the object changes twice during the read. Orphan collection deletes both copies of a block and waits while keys are locked. Both block stores now declare the blocks seal path. Mode `off` is unchanged.
-- Encryption at rest for KV blocks, FileSystem store (TC-887, off by default). With `at_rest.mode = "encrypt"`/`"strict"` and keys set up, new KV blocks on FileSystem block storage are sealed (TCE1) under `{blocks}/v1/{space_tag}/…` and served back as plaintext, ranges included; legacy blocks stay readable in `encrypt`, are refused in `strict`, and locked keys answer `503` for sealed data. A writing invocation prepares its spaces' data keys after its preupload authorization and before any upload or write lock, so a refused request creates no key. Orphan collection deletes both copies of a block and waits while keys are locked. The FileSystem store declares the blocks seal path; `encrypt`/`strict` still boot only once SQL artifacts declare theirs. Mode `off` is unchanged. See `docs/at-rest/INTERFACES.md` §7.
+- Encryption at rest for KV blocks, FileSystem store (TC-887, off by default). With `at_rest.mode = "encrypt"`/`"strict"` and keys set up, new KV blocks on FileSystem block storage are sealed (TCE1) under `{blocks}/v1/{space_tag}/…` and served back as plaintext, ranges included; legacy blocks stay readable in `encrypt`, are refused in `strict`, and locked keys answer `503` for sealed data. A writing invocation prepares its spaces' data keys after its preupload authorization and before any upload or write lock, so a request refused by its authorization creates no key (a later precondition or replay refusal can leave the key the space's first write would create). Orphan collection deletes both copies of a block and waits while keys are locked. The FileSystem store declares the blocks seal path; `encrypt`/`strict` still boot only once SQL artifacts declare theirs. Mode `off` is unchanged. See `docs/at-rest/INTERFACES.md` §7.
 - Encryption at rest groundwork for KV blocks (TC-887, nothing is sealed yet): the S3 and FileSystem block stores count sealed objects (`v1/{space_tag}/…`) by plaintext size at boot, per space tag, and a space's quota joins them once the at-rest keys can name the tag. The S3 store makes one more `ListObjectsV2` listing at boot, under `v1/`. At-rest storage failures answer `503` (keys locked or key store unavailable) or `500` (integrity, unsupported format, legacy refused in `strict`) on `/invoke` and the KV GET routes. Sealed names carry the same `@N` generation suffix as legacy names (`docs/at-rest/INTERFACES.md` §5).
 - Deploys: the workflow supports an explicit, reason-recorded rollback to an earlier released dstack image on the unchanged N0 compose. It checks image labels, release tag, ancestry, and identical migration sources while preserving the normal forward-release guard.
+- Encryption at rest for SQL and DuckDB artifacts: with `at_rest.mode` `encrypt` or `strict`, each `database_artifact` checkpoint and WAL delta is stored as TCE1 ciphertext bound to its service, space, name and content (and, for a delta, its revision). Each sealed save also stores a tag over the row's manifest (revision, which parts it holds, their plaintext hashes, sizes and encodings) under the space's key, and every read checks it first: a dropped or substituted part, or a changed revision, fails as an integrity error instead of reading as a smaller database. A sealed part must be exactly its sealed length. Migration `m20261107_010000_artifact_payload_encoding` adds the nullable `payload_encoding`, `delta_encoding` and `manifest` columns (`NULL` = plaintext and no tag, so existing rows read as before). Sizes and content hashes stay those of the plaintext, so storage usage and quotas are unchanged. Mode `off` (the default) stores exactly what it did before and clears the tag. A write that cannot be sealed, or a sealed read while the keys are locked, answers 503; a part or manifest that fails its integrity check, uses an unsupported format, or is plaintext under `strict` answers 500. Not covered: replacing a row with an older, self-consistent copy of itself, and local working files (SQL and DuckDB working and cache files, upload staging, exports), which rely on the CVM's encrypted storage; operators without a TEE must protect local paths, swap and file-level backups (`docs/at-rest/INTERFACES.md` §8). `tinycloud-sql-identity offline-fingerprint` refuses sealed artifacts. With KV blocks and SQL artifacts both sealing, the node accepts `encrypt` and `strict` once the at-rest gate passes. Details: TC-887
 
 ## [1.25.17] - 2026-11-03
 

docs/at-rest/INTERFACES.md

@@ -1,9 +1,9 @@
 # Encryption at rest: interfaces (TC-887)
 
 > **Status (TC-887):** the envelope, key types, `DataKeyProvider`, the keyring
-> (`at_rest::keyring`), the node wiring (`docs/at-rest-escrow.md`) and the storage view
-> (`at_rest::{AtRestMode, AtRestView, AtRestSource}`) are in. Sealed KV storage (§7) and sealed
-> SQL artifacts follow; until both seal, the node refuses `encrypt` and `strict`.
+> (`at_rest::keyring`), the node wiring (`docs/at-rest-escrow.md`), the storage view
+> (`at_rest::{AtRestMode, AtRestView, AtRestSource}`), sealed KV storage (§7) and sealed SQL
+> artifacts (§8) are in. Both declare their seal path, so `encrypt` and `strict` can be configured.
 
 Normative interface and format for `tinycloud_core::at_rest` and the node's sealed KV block
 storage. Plan: TC-887 (encryption at rest inside the TEE). Key continuity and escrow:
@@ -109,7 +109,12 @@ pub enum BlobIdentity<'a> {                      // what the ciphertext is bound
                     checkpoint_content_hash: &'a str, checkpoint_size_bytes: u64 },
     SqlWal        { service: &'a str, space: &'a str, name: &'a str, revision: i64,
                     checkpoint_content_hash: &'a str, delta_content_hash: &'a str },
+    SqlManifest   { service: &'a str, space: &'a str, name: &'a str, revision: i64,
+                    content_hash: &'a str, size_bytes: i64,
+                    checkpoint: SqlManifestPart<'a>, delta: Option<SqlManifestPart<'a>> },
 }
+pub struct SqlManifestPart<'a> { pub content_hash: &'a str, pub size_bytes: i64,
+                                 pub encoding: Option<i16> }   // §8
 
 /// Parsed, validated 64-byte header (unknown version/alg/flags/padding refused).
 pub struct Header { /* raw bytes, dek_id, plaintext_len */ }
@@ -228,10 +233,13 @@ Header, 64 bytes, all integers big-endian:
 Context (identity) bytes, every variable field `u32 BE len ‖ bytes`:
 ```
 ctx = lp("tinycloud/at-rest/v1") ‖ lp(purpose) ‖ lp(field_1) ‖ … ‖ lp(field_n)
-purpose ∈ {"kv-block","sql-checkpoint","sql-wal"}
+purpose ∈ {"kv-block","sql-checkpoint","sql-wal","sql-manifest"}
 kv-block:       space, content_hash (multihash bytes)
 sql-checkpoint: service, space, name, checkpoint_content_hash, u64 BE checkpoint_size_bytes
 sql-wal:        service, space, name, i64 BE revision, checkpoint_content_hash, delta_content_hash
+sql-manifest:   service, space, name, i64 BE revision, content_hash, i64 BE size_bytes,
+                part(checkpoint), then [0x00] (no delta) or [0x01], part(delta)
+part(p):        p.content_hash, i64 BE p.size_bytes, p.encoding (empty = NULL, else i16 BE)
 ```
 Keys and nonces:
 ```
@@ -330,14 +338,22 @@ redacted.
   preupload transaction (TC-735) authorized it and checked its spaces are hosted, and before any
   upload, sequence or writer lock, or transaction: written spaces get their current data key
   (created if new), spaces a `kv/get` reads in the transaction get any data key the database lost
-  restored from its authenticated mirror (never created). A refused request therefore creates no
-  data key or mirror object; no separate pre-authorization is needed (the preupload transaction
-  already authorizes before any key work). Work under the writer lock (a `kv/get` in a writing
-  invocation, the upload of a protected legacy meeting path) runs in `at_rest::write_locked`,
-  where the provider only looks keys up and a key that would need creating or restoring is
-  `NotPrepared` (503), never a key-store write that would wait on the lock the request holds.
-  Read-only invocations and the KV GET routes hold no writer lock, so a key the database lost is
-  restored on first use there.
+  restored from its authenticated mirror (never created). A request refused by its authorization
+  or hosting checks therefore creates no data key or mirror object; no separate pre-authorization
+  is needed (the preupload transaction already authorizes before any key work). Work under the
+  writer lock (a `kv/get` in a writing invocation, the upload of a protected legacy meeting path)
+  runs in `at_rest::write_locked`, where the provider only looks keys up and a key that would
+  need creating or restoring is `NotPrepared` (503), never a key-store write that would wait on
+  the lock the request holds. Read-only invocations and the KV GET routes hold no writer lock, so
+  a key the database lost is restored on first use there.
+- **Accepted residual (late refusals prepare keys):** a refusal that comes after the spaces were
+  prepared, in the write's own transaction (a failed `If-Match`/`If-None-Match` precondition, 412,
+  or a replayed invocation, 409), can leave a new data key and its mirror object for a space it
+  wrote nothing to. It is the key that space's first write would create; nothing is sealed under it.
+- **Accepted residual (generation not in the AEAD identity):** a block's AAD binds its space and
+  content hash but not its `@N` generation (§5), so a sealed object of one generation opens in
+  place of another generation of the same block. Every generation holds the same plaintext (the
+  content hash), so only same-content replay is possible.
 - **Accepted residual (linkability during the transition):** in `encrypt`, a read of a
   legacy-only block asks for the sealed name first, then the legacy name (`strict` probes the legacy
   name to answer "legacy refused"). A storage provider's access logs can therefore link a
@@ -354,3 +370,84 @@ redacted.
   stored bytes at boot like a stage in a space directory.
 - **Reserved names:** The key-continuity key mirror lives under `_tc/` in the block bucket (`datadir/at-rest` on
   FS). The block boot scan skips it.
+
+## 8. SQL artifacts
+External authoritative payloads are sealed: the durable copy of each SQL and DuckDB database is a
+`database_artifact` row, `payload` (the checkpoint) and `delta_payload` (the WAL delta).
+`SeaOrmDatabaseArtifactRepository` seals and opens them; it reads a fresh `AtRestView` per
+operation (`with_at_rest`), and the node's repository declares `SealPath::SqlArtifacts`.
+- **Local working files are not sealed (known limitation, a later phase).** Local working files
+  (SQL and DuckDB working and cache files, upload staging, exports) rely on the CVM's encrypted
+  storage; operators without a TEE must protect the node's local paths, swap and file-level
+  backups themselves (for example, `node.sh`'s `data.tar.gz` is not encrypted). They are: KV
+  ingress staging (`TempFileSystemStage`, a `NamedTempFile` in the system temp directory,
+  `tinycloud-node-server/src/storage/file_system.rs`); the SQL and DuckDB working databases and
+  WAL files under `{datadir}/sql` and `{datadir}/duckdb`, and the hydration staging files a cold
+  read writes beside them (`CacheFileSink` in `database_artifacts.rs`, `sql/service.rs`,
+  `sql/database.rs`, `duckdb/database.rs`).
+- **Columns:** `payload_encoding` and `delta_encoding` (SMALLINT, `NULL` = legacy plaintext; §5)
+  and `manifest` (BLOB/BYTEA, `NULL` = no tag), migration
+  `m20261107_010000_artifact_payload_encoding`. Each encoding is written with its part: a full
+  save sets `payload_encoding` and clears the delta with its `delta_encoding`; a delta save sets
+  `delta_encoding` only, so a legacy checkpoint can carry a sealed delta. Every save rewrites
+  `manifest`.
+- **Plaintext metadata:** `size_bytes`, `checkpoint_size_bytes`, `delta_size_bytes` and the three
+  content hashes stay those of the plaintext database file. They are part of the identity (§4),
+  and quota (`SqlSizes`) and the SQL growth check read them unchanged. Accepted residual: as
+  before, anyone who can read the table sees each database's size and content hash.
+- **Identity:** checkpoint `SqlCheckpoint { service, space, name, checkpoint_content_hash,
+  checkpoint_size_bytes }`; delta `SqlWal { service, space, name, revision, checkpoint_content_hash,
+  delta_content_hash }`, where `revision` is the row revision the delta was written at (always the
+  row's current revision). A part moved to another row, space, name, revision or checkpoint is
+  an integrity failure.
+- **Manifest (what the row holds, authenticated):** the part identities alone don't say which
+  parts a row must have. A checkpoint opens whatever the revision and whether or not a delta
+  belongs with it, so someone who can write the table could drop a sealed delta (clear the delta
+  columns, reset the unkeyed `content_hash` and `size_bytes`) and a cold read would serve the
+  checkpoint alone, silently losing committed rows. So every sealed save also stores a tag in
+  `manifest`: a TCE1 seal of an empty plaintext under the space's current data key with
+  `SqlManifest { service, space, name, revision, content_hash, size_bytes, checkpoint:
+  (checkpoint_content_hash, checkpoint_size_bytes, payload_encoding), delta: None | Some(
+  (delta_content_hash, delta_size_bytes, delta_encoding)) }` (§4). That is a keyed MAC over the
+  row's manifest, 80 bytes, naming its data key in its header like any sealed part, so it survives
+  key rotation. A full save writes it with `delta: None`, a delta save with the delta; both
+  rewrite it in the same `UPDATE` as the parts. Every read (`read_chunks`, and `open_row` for the
+  offline tool) checks it before any part reaches the sink: a dropped or added delta, a part or
+  metadata from another revision, a changed revision, or a changed encoding fails as integrity
+  (`AtRestIntegrity { part: "manifest" }`, 500), never a partial read. Unknown or invalid
+  encoding columns keep their own errors. Mode `off` writes no tag and clears it (its writes stay
+  plaintext and need no key), so a row without a tag is accepted only if it has no sealed part, or
+  if its only sealed part is a checkpoint under a delta mode `off` wrote; a row with any other
+  sealed part and no tag fails as integrity.
+  **Not protected:** a whole-row rollback, i.e. replacing the row with an older, valid,
+  self-consistent snapshot of itself (parts, metadata and tag all from the same earlier
+  revision), opens. So do rows with no tag: legacy parts are unauthenticated, as in §5, including
+  a delta written in mode `off` over a sealed checkpoint (`strict` refuses that legacy delta).
+- **Stored length and reads:** a TCE1 part stores exactly `ciphertext_len(size)` bytes. The
+  streamed read slices the stored bytes (`substr`, 4 MiB) and opens them one 64 KiB segment at a
+  time (`SegmentOpener`); the last slice of a sealed part asks for one byte past that length, so
+  bytes appended to the part, or a part cut short, fail as integrity. The content hashes are
+  checked over the plaintext, and sinks receive plaintext, so the cache sink still checks the
+  SQLite/DuckDB file header. At most one segment is buffered beyond what the sink keeps.
+- **Writes:** mode `off` stores plaintext exactly as before. Any other mode seals with
+  `provider.current_key(space)` or fails (`AtRestUnavailable`), never plaintext. `current_key`
+  runs before the SQLite writer lock, since it may create the space's key; sealing runs in
+  `spawn_blocking`.
+- **Errors:** keys locked, missing or the encrypt gate closed → 503 (`SqlError::Unavailable`,
+  `DuckDbError::Unavailable`). `UnknownKey`, a malformed header, a header length that disagrees
+  with the row, a stored length that disagrees with it, a manifest tag that doesn't match the row
+  (or is missing where one is required), and any AEAD failure → 500 integrity; `Unsupported`
+  (header or column `2..=9`) → 500 "upgrade the node"; a legacy part in `strict` → 500. A sealed
+  part whose column says `NULL` fails the manifest check; it is never served.
+- **Accepted residual (legacy injection in `encrypt`):** someone who can write the table can
+  replace a row with legacy plaintext parts and no tag (any content, with matching hashes), and
+  `encrypt` serves it, as it serves any legacy row. `strict` (after the backfill) refuses it.
+- **Offline tool:** `tinycloud-sql-identity offline-fingerprint` has no keys and refuses sealed
+  rows (`database_migration::offline_inventory` takes the provider). `open_row` checks the
+  manifest tag first, like a streamed read.
+- **Rollback:** a build without this change ignores the columns. It cannot hydrate a sealed part
+  (the file-header check fails), and a save it makes over a sealed row without reading it
+  (DuckDB `import_db`) leaves `payload_encoding = 1` on plaintext and a `manifest` tag for the
+  previous contents, which this build then refuses as integrity. Roll back below this change only
+  with no sealed rows; a row an older build overwrote that way needs its encoding columns and
+  `manifest` set back to `NULL`.

tinycloud-core/src/at_rest/envelope.rs

@@ -89,6 +89,41 @@ pub enum BlobIdentity<'a> {
         checkpoint_content_hash: &'a str,
         delta_content_hash: &'a str,
     },
+    /// A SQL artifact row's manifest: what the row says it holds. Sealed over
+    /// an empty plaintext, so the blob is a key-bound tag over these fields
+    /// (`docs/at-rest/INTERFACES.md` §8).
+    SqlManifest {
+        service: &'a str,
+        space: &'a str,
+        name: &'a str,
+        revision: i64,
+        content_hash: &'a str,
+        size_bytes: i64,
+        checkpoint: SqlManifestPart<'a>,
+        delta: Option<SqlManifestPart<'a>>,
+    },
+}
+
+/// One stored part of a SQL artifact as its manifest records it.
+#[derive(Clone, Copy, Debug)]
+pub struct SqlManifestPart<'a> {
+    /// Of the plaintext.
+    pub content_hash: &'a str,
+    /// Of the plaintext.
+    pub size_bytes: i64,
+    /// The part's encoding column, `None` = legacy plaintext.
+    pub encoding: Option<i16>,
+}
+
+impl SqlManifestPart<'_> {
+    fn push(&self, out: &mut Vec<u8>) {
+        push_lp(out, self.content_hash.as_bytes());
+        push_lp(out, &self.size_bytes.to_be_bytes());
+        match self.encoding {
+            None => push_lp(out, &[]),
+            Some(encoding) => push_lp(out, &encoding.to_be_bytes()),
+        }
+    }
 }
 
 pub(crate) fn push_lp(out: &mut Vec<u8>, field: &[u8]) {
@@ -140,6 +175,32 @@ impl BlobIdentity<'_> {
                 push_lp(&mut c, checkpoint_content_hash.as_bytes());
                 push_lp(&mut c, delta_content_hash.as_bytes());
             }
+            Self::SqlManifest {
+                service,
+                space,
+                name,
+                revision,
+                content_hash,
+                size_bytes,
+                checkpoint,
+                delta,
+            } => {
+                push_lp(&mut c, b"sql-manifest");
+                push_lp(&mut c, service.as_bytes());
+                push_lp(&mut c, space.as_bytes());
+                push_lp(&mut c, name.as_bytes());
+                push_lp(&mut c, &revision.to_be_bytes());
+                push_lp(&mut c, content_hash.as_bytes());
+                push_lp(&mut c, &size_bytes.to_be_bytes());
+                checkpoint.push(&mut c);
+                match delta {
+                    None => push_lp(&mut c, &[0]),
+                    Some(delta) => {
+                        push_lp(&mut c, &[1]);
+                        delta.push(&mut c);
+                    }
+                }
+            }
         }
         c
     }