fix(encryption): v2 column format with key ids and per-purpose keys

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## [Unreleased]
 ### Changes
+- Encryption: v2 column format with key ids, per-purpose keys and recorded format metadata. Optional strict reads are off by default. Details: TC-2516, TC-2517.
 - Deploys: the workflow supports an explicit, reason-recorded rollback to an earlier released dstack image on the unchanged N0 compose. It checks image labels, release tag, ancestry, and identical migration sources while preserving the normal forward-release guard.
 - A new Policy/v3 registration records the share key its content source names (`share/<shareId>` on the owner's `default` network) in the registration's transaction, so publishing a share needs no `tinycloud.encryption/key.create` grant: the publisher encrypts to the key's public key and registers. A missing key is created at `keyVersion` 1 and counts toward the network's key limit; at the limit the registration answers 403 `key_limit_exceeded` and stores nothing. An existing key keeps its state. The registration now also requires the source's network to exist and be active. Docs: `docs/encryption-named-keys.md`. Details: TC-2525
 - Share-email issuer trust bundles can pin one next key for the same issuer DID during an explicit UTC overlap window. The legacy key stops at `issuerNotAfter`; malformed, duplicate, unbounded or expired overlaps fail startup and `--validate-config`. Existing single-key bundles remain unchanged. This is node-side trust support only: the OpenCredentials witness and js-sdk policy contract must also support key rotation before activation. Older node images reject overlap fields; restore the previous single-key bundle before rollback. See `deploy/share-email/production-trust-bundle-contract.md`.

README.md

@@ -270,6 +270,7 @@ The following common options are available:
 | keys.type           | TINYCLOUD_KEYS_TYPE           | Set the type of host key store, options are "Static"                       |
 | spaces.allowlist    | TINYCLOUD_SPACES_ALLOWLIST    | Set the URL of an allowlist service for gating the creation of Space Peers |
 | telemetry.enabled   | TINYCLOUD_TELEMETRY__ENABLED  | Enable Prometheus latency metrics on the configured Prometheus port        |
+| column_encryption.strict | TINYCLOUD_COLUMN_ENCRYPTION__STRICT | Require authenticated envelopes for recorded legacy columns; default `false`. Recorded v2 columns always require v2. |
 | (n/a)               | ROCKET_MAX_BLOCKING           | Set Rocket's blocking-thread-pool cap (default 512). Note the `ROCKET_` prefix, not `TINYCLOUD_` — Rocket builds its tokio runtime from its own Figment before TinyCloud's config is loaded, so `TINYCLOUD_MAX_BLOCKING` is silently ignored for this setting |
 | (n/a)               | NODE_STATS_READ_TOKEN         | Bearer token for the read-only `GET /admin/stats` (TC-1482). Unset, blank, shorter than 32 bytes, or equal to `TINYCLOUD_ADMIN_SECRET` disables the route (`404`). Generate it with `openssl rand -hex 32` (64 hex characters). Separate from `TINYCLOUD_ADMIN_SECRET`, which it does not replace and which does not open it. Secret: set it in the deployment secret store, never in `tinycloud.toml` or a committed env file. See [Admin stats](#admin-stats-get-adminstats) |
 
@@ -323,6 +324,15 @@ The keepalive options are set by a vendored, patched sqlx-core 0.8.6 (`vendor/sq
 
 TinyCloud Protocol will automatically apply the relevant migrations to your chosen SQL database. Use caution if you are sharing this database with another application.
 
+New encrypted delegation, invocation (including read-audit), share-policy and
+webhook-secret writes use the v2 column format with key ids and row-bound
+authentication. Columns and webhook secrets use separate derived keys. Format
+metadata is stored alongside each value; existing rows retain format `0`.
+Existing v1 values remain readable. Set `strict = true` in
+`[global.column_encryption]` to require authenticated legacy reads as well.
+Startup adds format metadata without rewriting existing values. Details:
+TC-2516, TC-2517.
+
 ### Staging Config
 
 TinyCloud Protocol will temporarily stage files it receives before writing them. It can do this in memory or in temporary files. This can be configured by setting `storage.staging` to `Memory` or `FileSystem`. Default is `Memory`.

test/w5-policy-runtime-node-e2e/tests/sql_isolation.rs

@@ -700,6 +700,7 @@ async fn persist_grant(
         not_before: Set(None),
         facts: Set(None),
         serialization: Set(format!("tc780-n1-test-row:{ability}").into_bytes()),
+        serialization_format: Set(0),
     }
     .insert(conn)
     .await?;

test/w5-policy-runtime-node-e2e/tests/tc119_registry_wire_paths.rs

@@ -91,6 +91,7 @@ async fn persist_grant(
         not_before: Set(None),
         facts: Set(None),
         serialization: Set(format!("tc119-test-row:{ability}").into_bytes()),
+        serialization_format: Set(0),
     }
     .insert(conn)
     .await?;

tinycloud-core/src/auth_graph.rs

@@ -211,7 +211,7 @@ async fn load_closure<C: ConnectionTrait>(
 /// Each branch projects the same columns, with a distinct slot for each
 /// underlying SQL type. In particular, JSON and text never share a UNION
 /// slot, and the legacy model decoders still parse every field in Rust.
-fn snapshot_branch(kind: i32, fields: [SimpleExpr; 12]) -> SelectStatement {
+fn snapshot_branch(kind: i32, fields: [SimpleExpr; 13]) -> SelectStatement {
     let mut query = Query::select();
     query.expr_as(Expr::val(kind), Alias::new("kind"));
     for (name, expr) in [
@@ -224,6 +224,7 @@ fn snapshot_branch(kind: i32, fields: [SimpleExpr; 12]) -> SelectStatement {
         "not_before",
         "facts",
         "serialization",
+        "serialization_format",
         "resource",
         "ability",
         "caveats",
@@ -386,6 +387,7 @@ fn snapshot_query(roots: &[Hash]) -> WithQuery {
             Expr::col((delegation::Entity, delegation::Column::NotBefore)).into(),
             Expr::col((delegation::Entity, delegation::Column::Facts)).into(),
             Expr::col((delegation::Entity, delegation::Column::Serialization)).into(),
+            Expr::col((delegation::Entity, delegation::Column::SerializationFormat)).into(),
             Expr::col((abilities::Entity, abilities::Column::Resource)).into(),
             Expr::col((abilities::Entity, abilities::Column::Ability)).into(),
             Expr::col((abilities::Entity, abilities::Column::Caveats)).into(),
@@ -411,6 +413,7 @@ fn snapshot_query(roots: &[Hash]) -> WithQuery {
             null(),
             null(),
             null(),
+            null(),
         ],
     );
     edge_rows.from(bounded_name.clone());
@@ -427,6 +430,7 @@ fn snapshot_query(roots: &[Hash]) -> WithQuery {
             Expr::col((delegation::Entity, delegation::Column::NotBefore)).into(),
             Expr::col((delegation::Entity, delegation::Column::Facts)).into(),
             Expr::col((delegation::Entity, delegation::Column::Serialization)).into(),
+            Expr::col((delegation::Entity, delegation::Column::SerializationFormat)).into(),
             null(),
             null(),
             null(),
@@ -454,6 +458,7 @@ fn snapshot_query(roots: &[Hash]) -> WithQuery {
             null(),
             null(),
             null(),
+            null(),
             Expr::col((abilities::Entity, abilities::Column::Resource)).into(),
             Expr::col((abilities::Entity, abilities::Column::Ability)).into(),
             Expr::col((abilities::Entity, abilities::Column::Caveats)).into(),
@@ -470,7 +475,7 @@ fn snapshot_query(roots: &[Hash]) -> WithQuery {
         .and_where(within_budget());
     query.union(UnionType::All, abilities_query);
     let revocation_fields = || {
-        let mut fields: [SimpleExpr; 12] = std::array::from_fn(|_| null());
+        let mut fields: [SimpleExpr; 13] = std::array::from_fn(|_| null());
         fields[0] = node(lookup_name.clone());
         fields
     };
@@ -602,6 +607,7 @@ impl AuthGraphSnapshot {
                             not_before: row.try_get("", "not_before")?,
                             facts: row.try_get("", "facts")?,
                             serialization: row.try_get("", "serialization")?,
+                            serialization_format: row.try_get("", "serialization_format")?,
                         },
                     );
                 }
@@ -935,6 +941,7 @@ mod tests {
             not_before: Set(None),
             facts: Set(None),
             serialization: Set(id.as_ref().to_vec()),
+            serialization_format: Set(0),
         }
         .insert(db)
         .await
@@ -1136,6 +1143,7 @@ mod tests {
                     not_before: Set(None),
                     facts: Set(None),
                     serialization: Set(id.as_ref().to_vec()),
+                    serialization_format: Set(0),
                 }
                 .insert(db)
                 .await
@@ -1322,7 +1330,7 @@ mod tests {
         // rather than changing unrelated migrations for a query equivalence test.
         for ddl in [
             "CREATE TABLE actor (id VARCHAR(191) PRIMARY KEY)",
-            "CREATE TABLE delegation (id VARBINARY(34) PRIMARY KEY, delegator VARCHAR(191) NOT NULL, delegatee VARCHAR(191) NOT NULL, expiry DATETIME NULL, issued_at DATETIME NULL, not_before DATETIME NULL, facts JSON NULL, serialization BLOB NOT NULL)",
+            "CREATE TABLE delegation (id VARBINARY(34) PRIMARY KEY, delegator VARCHAR(191) NOT NULL, delegatee VARCHAR(191) NOT NULL, expiry DATETIME NULL, issued_at DATETIME NULL, not_before DATETIME NULL, facts JSON NULL, serialization BLOB NOT NULL, serialization_format SMALLINT NOT NULL DEFAULT 0)",
             "CREATE TABLE parent_delegation (child VARBINARY(34) NOT NULL, parent VARBINARY(34) NOT NULL, PRIMARY KEY (parent, child))",
             "CREATE TABLE ability (resource VARCHAR(255) NOT NULL, ability VARCHAR(255) NOT NULL, delegation VARBINARY(34) NOT NULL, caveats JSON NOT NULL, PRIMARY KEY (resource, ability, delegation))",
             "CREATE TABLE revocation (id VARBINARY(34) PRIMARY KEY, revoker VARCHAR(191) NOT NULL, revoked VARBINARY(34) NOT NULL, serialization BLOB NOT NULL, revoked_at DATETIME NULL)",
@@ -1351,6 +1359,7 @@ mod tests {
             not_before: Set(None),
             facts: Set(None),
             serialization: Set(id.as_ref().to_vec()),
+            serialization_format: Set(0),
         }
         .insert(db)
         .await