Add resumable storage re-encryption

docs/rekey-manifest.md

@@ -0,0 +1,121 @@
+# TC-2713 snapshot identity manifest
+
+One manifest covers network and named-key public identities. Its authority is
+the separately pinned SHA-256 in the rotation owner's authenticated custody
+record, not a hash generated from the target database during migration.
+
+## Source and custody
+
+The source is a dated, trusted PlanetScale backup snapshot of
+`tinycloud-node-production-usw`. The custodian records the backup ID, RFC3339
+snapshot timestamp, archive SHA-256, exporter identity, storage location and
+custody record reference. Naomi designates the rotation custodian, who holds the
+separately pinned manifest digest. Before the offline window, that custodian must
+record the archive and manifest SHA-256 values, backup ID, snapshot timestamp,
+exporter, storage location and custody reference in an authenticated private
+TC-2713 Linear comment, and record the same manifest digest in a TinyCloud
+Secrets entry named `TC2713_REKEY_MANIFEST_SHA256`. These records are operational
+prerequisites; the exporter does not create them.
+
+The custodian verifies the archive digest against that custody record, then
+restores **that exact archive** into a newly created, isolated local database or
+private PostgreSQL cluster with no other writers. Record the restore command,
+local cluster/database identity and checked archive digest in the custody record.
+For PostgreSQL, use a different local database name. Tunnels, proxies, port
+forwards, `pscale connect` and SSH forwarding are prohibited during export:
+loopback alone does not prove the source is a local restore. The exporter rejects
+remote and multi-host destinations, query overrides and the live database name,
+but cannot establish that a supplied archive produced the connected PostgreSQL
+database. The custodian's checked restore and custody record provide that binding;
+hashing an unrelated `--snapshot-file` is insufficient. No production endpoint
+may be queried by the exporter.
+
+The runner operator independently authenticates to Linear and TinyCloud Secrets,
+retrieves the digest from both custody records, and compares them before supplying
+the expected digest to the runner. Stop if either record is unavailable or they
+differ. Do not obtain the expected digest from the target database, the manifest
+file, a sidecar delivered with it, or a fresh hash of the received file. The runner
+takes the file and this independently obtained pin; it never chooses its own pin.
+
+Networks or seeds created after the snapshot are not covered. Take a fresh
+trusted snapshot immediately before the migration window. Missing identities
+are blocking categories, never exemptions. Client-held authenticated key pins
+are a possible later second source, not an implemented fallback.
+Coverage also checks the reverse direction: every snapshot network must still
+exist at its pinned key version with the same public key and a stored seal.
+Deleting it, changing its version or clearing it to pending blocks migration
+before the writer fence.
+Every snapshot seed identity must also remain present at its pinned network,
+key path and key version with the same public key. Removing the row blocks
+coverage even if its parent network is intact. Existing deterministic or deleted
+key rows may retain their schema-valid NULL seed; their public identity is still
+checked.
+
+## Canonical version 1 encoding
+
+The file is compact UTF-8 JSON without a BOM, spaces between tokens or trailing
+newline. Strings use normal JSON escaping with non-ASCII characters encoded as
+UTF-8. Field order is fixed:
+
+- Top level: `version`, `provenance`, `rows`.
+- Provenance: `source_database`, `backup_id`, `snapshot_timestamp`,
+  `snapshot_sha256`, `exported_by`, `storage_location`, `custody_record`.
+- Each row: `kind`, `network_id`, `key_path`, `key_version`, `public_key`.
+
+`version` is the integer 1. `source_database` is exactly
+`tinycloud-node-production-usw`. Custody fields must be nonempty. The snapshot
+timestamp must be valid RFC3339 and cannot be in the future. `snapshot_sha256`
+is the lowercase 64-character SHA-256 hex digest of the archive bytes.
+
+Rows use `kind` equal to `network` or `seed`. A network row has `key_path: null`;
+a seed row has its canonical key path. Network IDs are canonical TinyCloud
+encryption-network URNs. Key versions are positive signed 64-bit integers.
+Public keys are 32-byte X25519 public keys encoded as 64 lowercase hex characters.
+No private keys, sealed values or plaintext secrets belong in this file.
+
+Sort rows by kind (`network` before `seed`), network ID in UTF-8 byte order,
+key path (`null` before text, then UTF-8 byte order), then numeric key version.
+Duplicate identities are rejected even if their public keys agree. Unknown or
+duplicate JSON fields, malformed values and noncanonical serialization are
+rejected. SHA-256 covers every byte of the final canonical file.
+
+## Read-only exporter
+
+Run `python3 tools/export-rekey-manifest.py` in the checkout. It requires Python 3
+and, for a restored PostgreSQL snapshot, `psql`. Supply these explicit arguments:
+
+| Argument | Meaning |
+| --- | --- |
+| `--restored-postgres-url` | Loopback-only restored PostgreSQL database; lowercase `postgres://` or `postgresql://` prefix required; URL query options are refused. |
+| `--schema` | Restored schema, default `public`. |
+| `--sqlite-snapshot` | Alternative read-only SQLite image; must also be `--snapshot-file`. |
+| `--snapshot-file` | Actual checked backup archive whose bytes the exporter hashes. |
+| `--backup-id` | PlanetScale backup identifier from the custody record. |
+| `--snapshot-timestamp` | Snapshot's RFC3339 timestamp. |
+| `--exported-by` | Authenticated custodian/exporter identity. |
+| `--storage-location` | Snapshot archive's recorded storage location. |
+| `--custody-record` | Reference to the separate authenticated custody record. |
+| `--output` | New manifest file; existing files are never overwritten. |
+
+The exporter selects only public-key metadata in a read-only snapshot
+transaction. It does not select sealed private keys or seeds. Its terminal output
+contains only the manifest hash and network/seed counts; failures are generic.
+No key values, row identifiers, connection strings or custody metadata are logged.
+
+## Runner contract
+
+The rotation integration calls `reencrypt::manifest::run_offline` with the
+manifest file, its separately pinned digest, injected overlap rings and batch
+size. It verifies the file before any database call or fence activation. A valid
+file is converted into private network/seed maps inside `VerifiedManifest`; there
+is no raw-map conversion into that type. Coverage runs in bounded pages and
+reports only missing/mismatched network and seed counts. Any nonzero category
+blocks activation. Per-row migration checks remain mandatory after coverage.
+The activation and page-step helpers are crate-private; integrations use
+`run_offline` so they cannot bypass the coverage preflight.
+
+This entry point does not provide a production root source or stop serving.
+Do not deploy #541 standalone on nodes with v1 rows. It ships only within the
+TC-2713 rotation event once the production-integrated runner, verified manifest
+and offline migration window exist; serving resumes only after migration and
+authenticated final verification complete.

docs/root-reencryption.md

@@ -0,0 +1,146 @@
+# Resumable storage re-encryption
+
+The node storage migration API in `tinycloud-core::reencrypt` moves existing
+storage ciphertext to generation 2. It has no production root provider, CLI
+activation flag, or change to the current ROOT acquisition path. The injectable
+`NewRootSource` supplies independent new material; `Rings::from_source` retains
+the existing generation 1 ROOT and the exact legacy column and seal keys.
+Existing serving configuration continues selecting its existing keys.
+
+## Deployment restriction
+
+This PR must ship only as part of the TC-2713 rotation event. Do not deploy it
+standalone to a node holding v1 network-key or seed rows. Before deployment,
+the rotation integration must have a production-wired migration runner, a
+verified snapshot manifest and custody record, and an offline migration window
+with serving stopped. Migrate and verify all existing material before resuming
+serving. Unmigrated v1 requests return HTTP 503 with `migration_required`.
+
+The library's `manifest::run_offline` entry verifies a file and its separately
+supplied digest before any database access, checks manifest coverage, and drives
+the fenced job. It does not acquire production roots or install new serving
+rings. Those integration steps remain prerequisites for the rotation event.
+
+## Storage behavior
+
+Column and webhook keys retain #535's HKDF salt and purpose labels. A new
+network-seal purpose separates the next seal key from both. Column readers retain
+legacy v1 keys and generation 1 purpose keys; migration retains legacy seal keys.
+Serving accepts PK-bound v2 seals with generation 0 or 2. Injected rings initially
+keep the existing writers and add generation 2 read keys. After activation commits,
+the returned rings select generation 2 writers. Client wrap-v1 and wrap-v2 envelopes
+are unchanged: the job opens and re-seals server private keys and seeds, without
+changing the underlying X25519 keys or any client ciphertext.
+
+The inventory covers delegation and invocation serialization (including durable
+read-audit invocations), hook secrets, both owner-share-policy byte fields, network
+private keys, and named-key seeds. CID-alias rows hold hashes and an encryption
+marker, not ciphertext. Delegation hashes and canonical aliases remain stable;
+the job updates the alias encryption marker when converting plaintext storage.
+
+Column rewrites retain the exact table, column and ordered PK binding from #535.
+Network private-key bindings include `network_id`; seed bindings include
+`network_id` and `key_path` in database PK order. Serving opens and migration
+verify the network private key against the row's public key. Named-key opens
+also check the derived public key against the key row. Serving rejects v1 private-key
+and seed seals. Only the migration may decode legacy seals after checking the
+verified external manifest described below; rewritten seals use PK-bound v2.
+Every migrated value is authenticated and checked for byte-for-byte plaintext
+equality after it is written back.
+
+## Legacy seal migration prerequisites
+
+Both networks and seeds require one externally pinned manifest from a dated,
+trusted PlanetScale backup of `tinycloud-node-production-usw`, exported from a
+local restored snapshot. The [manifest format and export procedure](rekey-manifest.md)
+define its canonical encoding, snapshot checksum, custodian and custody record.
+The expected manifest SHA-256 must be recorded separately before the window.
+The target database must never supply its own trusted identities or expected pin.
+
+`VerifiedManifest` can be constructed only by checking the bytes against that
+separate pin, the required provenance metadata, canonical encoding and identities.
+Every `step` accepts this verified input, with no raw-map constructor or fallback.
+For v1 network seals, the manifest entry `(network_id, key_version)` must equal
+the stored public key and the public key derived from the private key. For v1
+seeds, `(network_id, key_path, key_version)` must equal the derived named-key
+public key and the stored public key.
+
+Network ceremonies are additional database consistency checks, not independent
+provenance. Exactly one completed `initial` ceremony, a completion record, no
+failure and `transcript_hash = hash_hex(public_key)` are required in addition to
+the external manifest. A missing or mismatching manifest entry or inconsistent
+ceremony aborts the page, preserves ciphertext/checkpoint, and prevents completion
+and old-root retirement.
+
+The runner reports missing or mismatched identities as counts-only blocking
+coverage categories; it never skips them. Networks or seeds created after the
+snapshot require a fresh trusted snapshot immediately before the migration
+window. Authenticated client-held public-key pins may become a later second
+source. Client-wrapped key v1 and v2 protocols remain unchanged.
+
+## Activation and recovery
+
+The rollout order is read overlap, database fence, then writer switch. First,
+install the injected rings on every serving instance with old writer kids
+(columns/webhooks 1 and seals 0), retaining all required old and new read keys.
+The schema's overlap guards keep kid 2 read-only before activation.
+`activate` validates that overlap, installs SQLite or PostgreSQL triggers, and
+advances the durable singleton in one transaction. Only after that transaction
+commits does it return rings selecting writer kid 2. Switch serving writers to
+those returned rings; stale writes are rejected by the database fence.
+
+`SpaceDatabase::new_with_storage_rings` checks the column, webhook and seal rings
+against one locked activation snapshot, holding the lock through alias backfill.
+Before activation, overlap rings must retain their old writers. After activation,
+all three rings must select the target writer and retain the required read
+generations through authenticated verification. Key retirement remains a separate
+integration step. The migration does not change any production root configuration
+itself.
+
+Each `step` locks the singleton, processes at most the requested page size,
+authenticates each value with its recorded kid and format, checks available
+content identities, and compares-and-swaps ciphertext and format. It reads the
+stored replacement back and verifies exact bytes, decryption and identity before
+committing ciphertext, alias markers, cursor and counts together. An error stops
+the page; dropping an interrupted transaction rolls it back. A committed page is
+resumed by calling `step` again with the same rings. Multiple workers serialize
+on the singleton.
+
+After rewriting, a durable verification phase starts from the beginning. Each
+step reads at most one bounded page, validates required seal presence and row
+state, requires the target format and kid, decrypts every value, checks content
+identity, and checks delegation aliases. Missing network parents or required
+sealed values stop the job. NULL values are accepted only for valid lifecycle
+states that intentionally hold no material. Verification cursors and counts
+commit with each checked page and resume after interruption.
+
+Database mutation triggers invalidate the verification revision. A write during
+verification restarts that bounded pass, so completion requires a complete
+authenticated pass at one unchanged revision. Repeated calls after completion
+are idempotent while that revision remains current.
+
+`census` returns only table/column counts by kid plus legacy, plaintext and null
+counts. `progress` returns only rewrite/verification counts, phase and completion. Errors disclose
+no row identities, key material, plaintext or ciphertext. The database cursor
+contains PKs and must be treated as database data, never logged. Database tracing
+must not be configured to emit bound values during an operational migration.
+The fencing migration refuses downgrade after activation; keep old roots until
+verification and a separately approved retirement plan are complete.
+
+## Identity and invitation rollout plan
+
+Identity and invitation keys require a separate protocol rollout. No rotation
+code for them is included here. First inventory client pins, cached node DIDs,
+invitation verification kids, trust bundles and outstanding invitation expiry.
+Publish the new node DID and invitation kid alongside the old entries before
+changing either signing key. Distribute overlapping trust bundles, allow clients
+to refresh caches, and verify that both identities are accepted across supported
+clients and existing sessions. Keep issuance on the old key during publication.
+
+Switch new signing and invitation issuance only after overlap adoption is
+verified. Continue validating old signatures and outstanding invitations through
+their supported lifetime; retain old public verification entries and explicit
+kid selection throughout the overlap. Remove old signing access and later retire
+old public entries only after client adoption, invitation expiry and rollback
+requirements are satisfied. Storage re-encryption alone does not update clients,
+rewrite signed delegation history or authorize identity retirement.

tinycloud-core/src/db.rs

@@ -594,6 +594,35 @@ where
         storage: B,
         secrets: K,
         encryption: Option<ColumnEncryption>,
+    ) -> Result<Self, DbErr> {
+        Self::new_with_ring_options(conn, storage, secrets, encryption, None, None).await
+    }
+
+    /// Boot with all three serving storage rings validated in the same transaction.
+    pub async fn new_with_storage_rings(
+        conn: C,
+        storage: B,
+        secrets: K,
+        rings: crate::reencrypt::Rings,
+    ) -> Result<Self, DbErr> {
+        Self::new_with_ring_options(
+            conn,
+            storage,
+            secrets,
+            Some(rings.columns),
+            Some(rings.webhooks),
+            Some(rings.seals),
+        )
+        .await
+    }
+
+    async fn new_with_ring_options(
+        conn: C,
+        storage: B,
+        secrets: K,
+        encryption: Option<ColumnEncryption>,
+        webhooks: Option<ColumnEncryption>,
+        seals: Option<ColumnEncryption>,
     ) -> Result<Self, DbErr> {
         let pool: RecoveringConnection = conn.clone().into();
         // sea-orm-migration's schema manager accepts only the raw sea-orm
@@ -604,6 +633,9 @@ where
             tx.execute_unprepared(crate::db_pool::EXEMPT_STATEMENT_TIMEOUT)
                 .await?;
         }
+        crate::reencrypt::check_writer(&tx, encryption.as_ref(), webhooks.as_ref(), seals.as_ref())
+            .await
+            .map_err(|_| DbErr::Custom("storage generation fence".into()))?;
         delegation_cid_alias::backfill(&tx, encryption.as_ref()).await?;
         tx.commit().await?;
         let writer_lock = (pool.get_database_backend() == sea_orm::DatabaseBackend::Sqlite)