Skip to content

feat(TC-887): seal SQL and DuckDB artifacts at rest - #521

Draft
samgbafa wants to merge 4 commits into
skgbafa/tc-887-r6-s3-sealedfrom
skgbafa/tc-887-r7-sql-sealed
Draft

samgbafa wants to merge 4 commits into
skgbafa/tc-887-r6-s3-sealedfrom
skgbafa/tc-887-r7-sql-sealed

Conversation

@samgbafa

@samgbafa samgbafa commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

TC-887: seal SQL and DuckDB artifacts at rest.

What and why

Every durable SQL and DuckDB byte is a database_artifact row: payload (checkpoint) and delta_payload (WAL delta). This change stores both as TCE1 ciphertext whenever the node runs at_rest.mode = "encrypt" or "strict", and declares the SQL seal path the node needs before it accepts those modes. The node still refuses them until KV block storage seals too (separate stack).

  • Migration m20261107_010000_artifact_payload_encoding: nullable SMALLINT payload_encoding and delta_encoding (SQLite and PostgreSQL). NULL = legacy plaintext, so existing rows read exactly as before. ArtifactMeta and load_meta carry them.
  • Plaintext metadata: sizes and content hashes stay those of the plaintext database, so SqlSizes, quota and the growth check are unchanged. They are part of the AEAD identity (SqlCheckpoint, SqlWal with the revision the delta is written at).
  • Writes (save, save_delta): mode off stores plaintext exactly as before. Other modes seal with provider.current_key(space) or fail with DatabaseArtifactError::AtRestUnavailable, never plaintext. current_key runs before the SQLite writer lock (it may create the key, which writes the key store); sealing runs in spawn_blocking. Each part's encoding column is written with that part.
  • Reads: legacy vs sealed comes from the column, never from the bytes. read_part slices the stored length (ciphertext_len(size)) in the existing 4 MiB substr slices and opens it one 64 KiB segment at a time (new push-style at_rest::SegmentOpener). The blake3 hash check runs on the plaintext, and sinks get plaintext, so CacheFileSink still checks the SQLite/DuckDB header. Memory stays bounded to one segment beyond what the sink keeps.
  • Errors: keys locked/missing or the gate closed → SqlError::Unavailable / new DuckDbError::Unavailable (503). Integrity failures, unknown keys, Unsupported (header or column 2..=9) and legacy-in-strict → 500. Never a plaintext fallback, never re-sealed or skipped.
  • Node: node_artifact_repository takes the Arc<AtRestRuntime> (AtRestSource), wires it in with with_at_rest, and declares SealPath::SqlArtifacts.
  • Offline tool: database_migration::offline_inventory takes an optional provider. tinycloud-sql-identity offline-fingerprint passes none, so it refuses sealed rows instead of fingerprinting ciphertext. A snapshot taken before the encoding columns existed (the tool never migrates) is read as plaintext.
  • Docs: docs/at-rest/INTERFACES.md §8 (columns, plaintext metadata, identity, stored length, errors, rollback). CHANGELOG [Unreleased].

This is new work; it replaces no earlier PR. The KV block-store files are untouched, so this can be rebased onto the KV stack. Out of scope: KV blocks, the harness RED→GREEN flip, and the inert sql_store.

Notes for review

  • at_rest/keyring/tests.rs::migration_down_refuses_to_drop_keys assumed the key-table migration was the newest one. It now rolls back down to and including that migration, however many come after it.

  • metadata_and_absence_never_select_payload_columns now checks the quoted "payload" / "delta_payload" identifiers, since payload_encoding is a metadata column.

  • Rollback: a build without this change ignores the encoding columns. It cannot hydrate a sealed part (the header check fails), and a blind full save it makes over a sealed row (DuckDB import_db) would leave payload_encoding = 1 on plaintext. Covered in INTERFACES §8.

  • tests/sql_identity_cli.rs inserts its pre-migration fixture row without the new columns (exec_without_returning), as a snapshot of that age has none.

Verification (run locally)

PostgreSQL tests ran against a private local PostgreSQL 14 (TINYCLOUD_TEST_POSTGRES_URL, TINYCLOUD_ALLOW_LOCAL_PG=1).

cargo fmt --all
RUSTFLAGS=-Dwarnings cargo clippy --locked -p tinycloud-node -p tinycloud-core --all-targets
RUSTFLAGS=-Dwarnings cargo clippy --locked -p tinycloud-node -p tinycloud-core --all-targets --features tinycloud-node/duckdb,tinycloud-core/duckdb
cargo test --locked -p tinycloud-core --lib -- database_artifacts database_migration sql at_rest migrations
cargo test --locked -p tinycloud-core --features duckdb --lib -- duckdb database_artifacts database_migration sql_sizes
cargo test --locked -p tinycloud-node --lib -- sql at_rest artifact invocation_replay did_resolution
cargo test --locked -p tinycloud-node --features duckdb --test duckdb_offline_policy --test sql_identity_cli

Results: all pass. routes::tests::tc1799_two_proof_sql_invocation_is_refused failed once in the filtered node run (a process-wide counter moved by a concurrent test) and passes on its own; it is unrelated to this change.

New tests:

  • database_artifacts::tests::sealed_parts_hold_no_plaintext_and_hydrate_exactly (SQLite): TCE1 bytes with no SQLite/WAL header or marker, correct columns, plaintext sizes and hashes, an 8 MiB+ checkpoint read in 4 MiB slices across segment boundaries back to the exact bytes through a header-checking cache sink, sealed parts still read in off and strict, empty database.
  • database_artifacts::tests::postgres_sealed_parts_round_trip_and_refuse_a_moved_part (PostgreSQL): the same round trip, plus a checkpoint moved from another space refused as integrity.
  • database_artifacts::tests::off_writes_plaintext_and_legacy_parts_read_unless_strict: off stores the bytes as given, legacy reads in encrypt, a sealed delta on a legacy checkpoint, strict refuses legacy, and a full save seals or un-seals by mode.
  • database_artifacts::tests::without_unlocked_keys_writes_fail_and_change_nothing: gate closed, no keyring or locked keys → AtRestUnavailable and the row is unchanged; sealed reads with locked or missing keys refuse.
  • database_artifacts::tests::tampered_or_moved_sealed_parts_are_integrity_errors: flipped body or header byte, a part moved from another space or name, an older revision's WAL, encoding 2 (unsupported), 0 (invalid), and NULL on sealed bytes. All are errors and nothing stays staged.
  • sql::service::tests::sealed_sql_database_hydrates_a_cold_cache and duckdb::service::tests::sealed_duckdb_artifact_hydrates_a_cold_cache: end-to-end through the services with a cold cache; locked keys → Unavailable.
  • database_migration::tests::offline_fingerprint_opens_sealed_rows_only_with_keys, at_rest::tests::pushed_slices_open_like_the_whole_buffer.

Restacked onto the KV stack (base now #520). With both seal paths in, the node boots encrypt and strict:

  • the four app-level FS-node encrypt tests in tests/at_rest_fs_node.rs run again (they were ignored in feat(TC-887): the FileSystem block store seals KV blocks #519);
  • at_rest_ceremony now checks that a pinned, verified escrow turns encryption on in encrypt and strict, and that encrypt without escrow or escrow_none is refused by the gate;
  • INTERFACES status updated.

Local, after the restack: cargo test --locked -p tinycloud-node --test at_rest_ceremony --test at_rest_fs_node --test at_rest_fs_store --test at_rest_env_root --test at_rest_env --test did_resolution_single_entry all pass. The at-rest harness with TC887_AT_REST_MODE=encrypt TC887_AT_REST_ROOT_SOURCE=env reports KV and SQL sealed (see #522).

With at_rest.mode encrypt or strict, each database_artifact checkpoint
and WAL delta is stored as TCE1 ciphertext bound to its service, space,
name and plaintext content (and, for a delta, its revision). Mode off
stores exactly what it did before.

- Migration m20261107_010000_artifact_payload_encoding adds nullable
  payload_encoding and delta_encoding (NULL = legacy plaintext).
- Sizes and content hashes stay those of the plaintext, so quota and the
  growth check are unchanged.
- Writes seal with the space's current key, prepared before the SQLite
  writer lock, or fail (503); never a plaintext fallback.
- Reads decide legacy vs sealed from the column, open the stored bytes
  one 64 KiB segment at a time (at_rest::SegmentOpener) and hash the
  plaintext. Integrity, unsupported format and legacy-in-strict are 500.
- The node's artifact repository reads the at-rest runtime and declares
  the SQL seal path. DuckDB gains an Unavailable (503) error.
- The offline fingerprint tool refuses sealed rows without keys.
…oding columns

A local snapshot from before m20261107_010000_artifact_payload_encoding
has no payload_encoding/delta_encoding columns; read its parts as
plaintext instead of failing the query.
… both seal

The filesystem-node encrypt tests run again, the ceremony test checks that a
verified escrow turns encryption on in encrypt and strict, and INTERFACES
records that both seal paths are in.
@samgbafa
samgbafa force-pushed the skgbafa/tc-887-r7-sql-sealed branch from 4172c08 to b69f29a Compare November 8, 2026 01:06
@samgbafa
samgbafa changed the base branch from skgbafa/tc-887-r3-node-wiring to skgbafa/tc-887-r6-s3-sealed November 8, 2026 01:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant