[joelteply]

How this serves the minds: a citizen's state (her KV pages, genes, memories, identity, projects) has to survive restarts, move between nodes and carry its provenance. Today each layer re-solves encoding, atomic writes, versioning and moving, and each solution is partial (docs/architecture/PACKAGES.md, "The problem, measured"). This PR adds the one base-layer crate those layers will load and store through. Plan lines: 3b (one live engine, with minds' memory moved by us: KV pages become packages in slice 3), goal 8 (genes as one kind, slice 4) and line 1 (a new leaf crate that checks in seconds). It implements slice 1 of the Packages design; the design PR is on fable/packages-design, and the MODULE-ARCHITECTURE link to PACKAGES.md resolves once that PR merges.

New crates only. No caller migrates (that's slice 2).

What it is

• core/continuum-pack is a base-layer leaf crate. It depends on continuum-base plus bytes, serde, serde_json, sha2, thiserror and tracing, all already in the lock.
  • ContentId([u8;32]) is rendered sha256:<hex> only at the edges: Display, FromStr and serde. Lowercase hex is the only spelling it accepts.
  • Kind, Packable and Migration. The migration chain runs on load, before requirements are checked and before any part is read.
  • Packer verbs: field, blob (bytes, reader or writer closure, hashed in flight), blob_value, child/children (eager), handle/handles (lazy), requires, and secret. secret returns SealingUnavailable until slice 5, so plaintext is never stored. Unpacker mirrors these verbs.
  • The manifest is canonical JSON: keys sorted at every depth by its own encoder, independent of serde_json's preserve_order feature. Parts use OCI-shaped descriptors {mediaType, digest, size}. A newer envelope format is refused.
  • Store has five verbs. has and get_ref return Result, see Design feedback 2.
    • DiskStore lays out blobs/sha256/<hex> and refs/<scope>/<name> and has one write path: a temp file under tmp/, fsync, publish (blob: create-if-absent through fs_portable::durable_publish_new; ref: durable_rename), then a directory fsync. Every error is returned. The ref CAS is serialized across processes by an exclusive File::lock on a lock file that is never deleted.
    • RamStore is bounded and returns a typed Full.
  • Handle<T> loads its child on first use; clones share both the load and the packed id. Blob stays lazy after unpack, read_all is verified, and read_range_unverified says in its name that it is not.
  • transport::verify / copy walk the DAG children-first and re-hash every byte on arrival.
• core/continuum-pack-derive provides #[derive(Packable)] with #[pack(kind, schema, migrate)] and field attributes blob = "media", child (Handle<T> / Vec<Handle<T>>) and requires (Required). It is re-exported by continuum-pack, so one dependency gives the trait, the derive and the stores.
• conformance::<T>() + FaultStore sit behind the conformance feature, so production binaries never link them. FaultStore logs every operation, fails the Nth one either before or after delegating it, or flips a byte in reads of one id.
• Placement and wiring:
  • [workspace.dependencies] continuum-pack: any crate adds continuum-pack.workspace = true.
  • source-hygiene audits both crates as base layer (SPLIT_CRATES).
  • MODULE-ARCHITECTURE §9.2 names the base layer.
  • CI test step: cargo test plus cargo clippy -D warnings on both crates.
  • ci-change-scope.cjs has no per-crate TESTED_CRATES list on main (it selects only rust / installer / publication), so there was nothing to add there.
  • Two existing continuum-base clippy lints are fixed (rfind, needless Ok(?)) because clippy -D warnings lints workspace dependencies too.
• lib.rs carries #![cfg_attr(not(test), deny(clippy::unwrap_used, clippy::expect_used, clippy::panic))]. Production code has no unwrap, expect, unwrap_or* or panic!.

Conformance (generic, every Packable inherits it; run on RamStore and DiskStore)

1. Round trip: load, then re-pack in place and into another store; the id never moves, and the moved tree verifies.
2. Same value gives the same id, and a different value gives a different id.
3. One ref flip, and it is the last operation of a commit.
4. A fault at every operation, in both modes (Before: the op never runs; After: the op completes, then reports failure):
   • the old root stays current and loadable;
   • the new root is current only when the faulted op was the flip itself, and it then loads too.
5. A byte flipped in any node (root manifest, child manifest, blob) is caught by verify, and by load when the node is the root.
6. An unmet requires refuses after exactly one store read: ops == [Get(root)].
7. A newer schema is refused.
8. Every recorded old schema migrates. The schema-1 manifest is stored byte for byte as a recorded literal, and this build re-encodes those exact bytes.
9. Shared nodes are stored once: blob count equals the number of distinct nodes.
10. Lazy load: the number of reads a load performs is measured.

Outlier types, in tests only:

• Tiny is manifest-only, on schema 2, with a typed requirement and a recorded schema-1 manifest. Its tree has 1 node, a load reads 1 time, and a commit takes 2 operations.
• Root -> Mid -> Leaf has a 3 MB and a 1 MB blob, two parents that share one Mid, two Mids that share one Leaf, and a hand-written Leaf. It is 7 distinct nodes behind 9 references, a load reads only the root, and the whole 3-level tree commits by one root flip.
• Type-specific tests:
  • handle and blob laziness end to end (one read per child, none on the second load);
  • the Merkle property (a leaf note change moves the root id);
  • a byte flipped in a real file on disk is caught at the read that uses it, while the rest of the tree loads;
  • typed refusals: secret, kind mismatch, absent ref, stale CAS;
  • the DiskStore layout, a range read and a corrupt ref file.

Lines

3,113 added, 7 removed. That breaks down as:

• Production: about 2,150, docs included.
• Tests: tests.rs 390 plus about 115 in in-file test modules.
• conformance feature: conformance.rs 287 and fault.rs 130.
• Wiring: Cargo.toml/lock 31, CI 4, hygiene 5, docs 5.

Nothing is deleted yet; slices 2 to 4 delete the 44 hand-written temp+rename sites, the four engine map stores and the three gene manifests. Every new file is under 400 lines.

Measured

All on Joel's MacBook Pro (Mac17,9, Apple M5 Pro, 18 cores), tree = fda062f, 2026-10-24.

• SHA-256 throughput: 603 MB/s (512 MiB blob, best of 3, --release, sha2 0.10.9).
• DiskStore::put of the same blob: 556 MB/s, hash, write and fsync included.
• How to reproduce: cargo test -p continuum-pack --release -- --ignored --nocapture throughput.
• This is the software backend. sha2 0.10 enables the ARMv8 SHA extensions on aarch64 only with its asm feature (which adds sha2-asm, not in the lock). So the design's "~2 GB/s, ~1 s per 2.1 GB page" is ~3.5x optimistic today: a 2.1 GB KV page hashes in ~3.5 s serially. See Design feedback 1.
• cargo test -p continuum-pack -p continuum-pack-derive: 9 passed, 1 ignored (the throughput measurement), in 8.03 s.
• cargo clippy -p continuum-pack -p continuum-pack-derive --all-targets --features continuum-pack/conformance -- -D warnings: clean.
• Clean check of continuum-pack from an empty target dir, all cores: 4 s.
• cargo test -p source-hygiene: 47 passed with both crates audited, in window 1. In window 2 a concurrent cargo on the shared target dir left a foreign source-hygiene artifact behind, so CI's Source Hygiene run is the confirmation.

Design feedback (found while building)

1. Hashing cost is 3.5x the design's estimate (603 MB/s measured, not ~2 GB/s). Before slice 3, either enable sha2/asm (one new crate, re-measure) or move to sha2 0.11 / a hardware-backed hasher. The design's own validator (spill at 35k tokens with and without the digest) should run with the faster backend.
2. has -> bool and get_ref -> Option would be silent fallbacks. An I/O error would read as "absent", so a copy would re-send and a load would report no ref. Both now return Result.
3. set_ref(expect: Option) was ambiguous. I defined None as "must not exist" and left no blind overwrite, because that is the lost-update defect.
4. Range reads cannot be verified against a whole-blob digest. That contradicts "never trusted for integrity" for lazy and spill extent reads. Verified range reads need chunk digests inside the blob (a chunk list or a bao-style tree). For now the API says read_range_unverified.
5. The manifest is not an OCI manifest "unchanged". OCI requires schemaVersion, mediaType, config and layers, and annotations are string-only, so inline structured fields don't fit. OciStore must wrap: our manifest as the config blob, parts as layers. Descriptors are already OCI-shaped.
6. requires(req) needs a name so the derive can rebuild the field. Requirements are keyed by part name now.
   • KvFormat{..} is unspecified. I placed {model, format: String} as a placeholder for slice 3's owner.
   • Offers use exact membership; Key(audience) needs any-of semantics in slice 5.
7. A migrated value re-packs to a new id, so a provenance signature over the old id does not cover the upgraded package. The design should say whether migration re-signs or whether lineage records the pre-migration id.
8. Adding even an optional field needs a schema bump and a migration. I chose strict: a missing field is a typed error, never a default. Option<Handle<T>> is not supported, because "absent" and "lost" would be indistinguishable.
9. copy relies on "has implies closure" (manifests are written after their children). GC must keep that invariant (root reachability, never per-blob LRU on a primary store). A put of an existing id does not re-verify on-disk bytes, so a corrupt blob is not healed by a re-put; a scrub/repair verb is needed.
10. RamStore as "a ResourcePool" cannot be done from the base layer, because ResourcePool lives in continuum-kernel. Eviction belongs to the Tiered store above, or the pool trait moves down to base.
11. Sealed parts break "same value, same id": random nonces mean the same plaintext gives a different ciphertext id. Slice 5 needs deterministic sealing or a documented exception.
12. Not in slice 1:
    • parallel per-part hashing threads (packing is sequential);
    • ts-rs exports (no client consumes them yet);
    • a context = .. derive attribute: derived kinds get Context = (), and lazy handles take their context at load, which is what kept laziness clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

[joelteply]

Ported onto canary (the branch was based on main, so a rebase showed ~600 unrelated conflicts). New head: 227ea59, base is now canary.

Copied whole: core/continuum-pack, core/continuum-pack-derive.

Hunks re-applied on canary's files:

• Cargo.toml: the two crates join members; continuum-pack goes in [workspace.dependencies]. The PR's SPLIT_CRATES edit in source-hygiene lib.rs no longer applies, because canary moved that list into the root manifest. So both crates are added to [workspace.metadata.ci] split_prefixed instead.
• .github/workflows/continuum-rust-tests.yml: canary tests every member through the crate matrix, so the explicit cargo test -p continuum-pack line is not needed. The clippy -D warnings lock is now a step of the continuum-pack matrix job.
• docs/architecture/MODULE-ARCHITECTURE.md: the base-layer paragraph, which now also names continuum-host. Canary's base has no fs_portable any more, so "portable fs" was dropped from base's description.
• Skipped: the clippy cleanups in continuum-base/src/config_env.rs and shell_portable.rs. The pack's clippy step does not lint base, and touching config_env.rs would pull its existing DEFAULT_HEADROOM and File::create sites into the touched-file budget gate.

Adapted to canary:

• DiskStore imported continuum_base::fs_portable::{durable_publish_new, durable_rename, sync_publication}, which canary moved into continuum-host (The OS seam becomes continuum-host: one module per capability, logic once, adapters only the diff #5375). It now uses continuum_host::durable::{publish_new, replace, fence}, and continuum-pack depends on continuum-host.
• ensure_dir's #[cfg(unix)] parent sync moved into the seam as durable::create_dir. The logic lives once in mod.rs, and os::sync_entries is the only OS-specific part. It has a test in durable/tests.rs. The seam's parent() now maps a bare relative name to ".".
• Two serde_json::to_value sites (packer.rs field, manifest.rs encode) now name their boundary on the same line: the manifest bytes are the stored and wire format.
• Judgment call for review: writes_outside_a_class flagged DiskStore's 3 OpenOptions sites. A new crate counts as touched, so the touched-file gate requires zero sites. I named core/continuum-pack/src/store/ as an owner of persistent bytes beside the ORM (budget_discipline/writes.rs), on the grounds that it writes only under the root its caller hands it. The class claim and eviction belong to the first caller that mounts a DiskStore on a real directory; slice 1 has no production caller. If review prefers that the store take a class handle instead, that is a follow-up.

Cargo.lock: running the ratchets let cargo resolve the manifest offline, and that added only the two new packages (purely additive, no version changes). I committed it, so --locked should pass. No build, check or test of the crates was run here (live serving node), so the first real compile and the pack and host tests run in CI.

Ratchets (cargo run -p source-hygiene --bin ratchets -- origin/canary): head 73/73 pass and every count is unchanged versus base (e.g. unjustified_serialization 219 to 219, os_branches_in_logic 391 to 391). Exit 0. The base-tree pass reports one crate_wiring failure. That is expected: the head's policy names crates the base tree does not have yet. No file is over 1,000 lines; the largest is tests.rs at 396.

The design doc (PACKAGES.md, #5370) is untouched here and lands separately. Until it does, the doc links point at a file that does not exist yet.

[joelteply]

Fable on the port (227ea59): one call I won't take as is. Listing continuum-pack/src/store/ in writes.rs OWNERS exempts a consumer. The pattern it shares with ClassWriter (a staging file streamed into, removed unless placed; re-fencing a name a racing publisher made; a ref lock file) now has one owner in #5651: continuum_host::durable::{Staging, fence_existing, lock}, and the ratchet names that primitive. After #5651 merges, this PR rebases onto it: DiskStore::temp + Temp become Staging::new_in(root/tmp) (write through staging.file() into the HashingWriter, then staging.publish_new(&dest)), the AlreadyExists arm becomes durable::fence_existing(&dest), set_ref's lock becomes durable::lock(&lock_path, true), and the OWNERS entry for the pack store is dropped. Holding this PR until then.

[joelteply]

Closed for the WIP limit, not abandoned: continuum-pack slice 1 must rebase onto continuum_host::durable (#5651) and is not a row in today's plan; the branch stays, reopen when it is rebased.