Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/637-logbook-family-actions.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add append-only family actions for withdrawing a build or recording one compatible family member as the direct replacement for another.
1 change: 1 addition & 0 deletions changelog.d/637-logbook-family-discovery.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add Logbook families and two-column family membership records so builds made from the same verified prepared input can be discovered without treating build order as a release relationship.
1 change: 1 addition & 0 deletions changelog.d/641-logbook-build-cardinality.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add versioned Logbook build records with typed requested and realized dataset cardinality and a normalized record unit. Exact-k builds use a null sampling-fraction field, while every legacy archive record and checksum remains unchanged.
120 changes: 120 additions & 0 deletions logbook/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ public git archives of those rows.
```
logbook/us.jsonl # grandfathered mixed US chain
logbook/<country>/<dataset>.jsonl # one new hash chain per dataset line
logbook/families/<scope>.jsonl # family identity and source checksum
logbook/family_members/<scope>.jsonl # family-to-build associations
logbook/family_actions/<scope>.jsonl # revocations and replacements
logbook-spool/<country>/<dataset>/ # committed spool mirror, same shape
```

Expand All @@ -28,6 +31,46 @@ base-data name. Different base data need different scopes, and scope is also
where builds serialize: if two lines must append concurrently, they need
separate chains.

The three family-data directories are not build sequences. Their files use
the same scope paths for validation and selection, but their records have no
predecessor checksum. Build validation and rendering explicitly skip these
directories.

## Build record versions and dataset families

Committed build records created before typed cardinality support retain their
original JSON fields and checksum. Current writers use
`row_format_version = 2`. That representation adds `requested_k`,
`realized_k`, and `record_unit` to the checksum input. A full-size request such
as `N` is resolved before recording, so the stored values are positive numbers;
for example, a full input with 100,000 households records 100000 in both
cardinality columns and `household` as the unit. A failed build may retain a
known request and leave the realized value null.

The existing `rung` field stores a sampling-fraction category such as `f010`.
An exact-k request specifies an absolute household count, not a fraction, so
the exact-k launcher writes SQL/JSON null in `rung`. Version-2 validation
allows that null value and includes it in the build checksum. Legacy rows and
current fraction-based writers continue to require one of the established
fraction values.

The relational model has three parts:

- `families` assigns a caller-created UUID to one Logbook scope and one
verified prepared-input manifest checksum. The UUID and checksum are
separate: the UUID is the database identifier, while the checksum describes
the input used by the family.
- `family_members` contains only `family_id` and `build_id`. It states that the
build used the family's prepared input. Dataset properties such as size,
random seed, file location, and build status remain on the build record.
- `family_actions` records either `revokes`, which withdraws one family member,
or `supersedes`, which says one same-size family member directly replaces
another. Revocation and replacement remain separate facts.

A build may belong to at most one family. A family may contain any number of
builds. These associations do not change the per-scope predecessor sequence,
and there is no additional release entity or general relationship graph.

## The vocabulary is closed-world

The ratified scopes are exactly `us` and `uk/frs` — deliberately minimal: a
Expand Down Expand Up @@ -89,6 +132,69 @@ python tools/logbook.py render --archive logbook/us.jsonl
`render` is the public-safe projection: it shows `artifact_location` only
for `published` and `certified` rows.

Family records are exported from the durable spool into all three files for
one scope:

```bash
python tools/logbook.py family-export --scope us --source <run-dir>/logbook-spool
python tools/logbook.py family-export --scope uk/frs --remote
```

To restore archives, first copy the scope's build archive and three family
archives into a local spool, then send the spool. The import rejects a member
whose archived build belongs to another scope. Reconciliation sends queued
builds first, families second, memberships third, and actions last. If a
request fails, its file and every dependent file remain available for the same
command to retry.

```bash
python tools/logbook.py family-import --scope us --spool logbook-spool
python tools/logbook.py reconcile --spool logbook-spool
```

The exact-k launcher uses configuration format version 2 and requires the
caller to create the family UUID:

```json
{
"schema_version": 2,
"family": {"id": "12345678-1234-4234-9234-123456789abc"},
"pool": {
"release_id": "prepared-pool-release",
"manifest_sha256": "<verified lowercase SHA-256>"
}
}
```

Use the same `family.id` for every exact-k build that uses that prepared input.
The launcher verifies `pool.manifest_sha256`, stores that same checksum as the
family's `source_pool_sha256`, resolves `N` to a numeric household count, and
writes the build, family, and membership under `<out>/logbook-spool/`. It then
attempts remote insertion in dependency order. If credentials or database
access are unavailable, all files remain local; run `tools/logbook.py
reconcile --spool <out>/logbook-spool` later. A family UUID already associated
with another source checksum is rejected, and the conflicting local files are
retained for inspection.

Pass the current US predecessor checksum with
`--logbook-prev-row-digest` or `POPULACE_LOGBOOK_PREV_ROW_DIGEST`. The exact-k
launcher still only writes a package and a manual publication command; it does
not publish, certify, or update a published-release pointer.

The archive queries emit compact JSON, suitable for reading directly or
piping to `jq`:

```bash
python tools/logbook.py list-families
python tools/logbook.py list-family-builds --family-id <uuid>
python tools/logbook.py show-family-history --family-id <uuid>
```

This implementation completes the typed exact-count fields requested by
issue #641. It supplies the relational family, membership, and action storage
needed by issue #637; workflow-specific discovery and user interfaces remain
separate work.

## The live store

The best-effort Supabase insert (`POPULACE_LEDGER_URL` +
Expand All @@ -111,6 +217,20 @@ only the grandfathered US pipelines; for `logbook/<country>/<dataset>.jsonl`,
it requests matching `<country>-<dataset>-*` pipelines and then verifies the
scope again client-side before ordering the chain.

Family export also reads one scope at a time. It filters `families`, the public
member-build view, and the public action view by the stored family scope, then
validates all family and member references before changing an archive.

## Database migration boundary

The family migration must be applied only after the base Logbook, prediction,
fraction-category, and `20260818000000_logbook_chain_scopes.sql` migrations.
Before deployment, identify the intended Supabase project and owning
organization, record its project reference, and compare its migration history
with this order. Do not link this repository to a project based only on a name
match. This repository change does not deploy the migration; deployment is a
separate, explicitly authorized operation.

## UK M1 receipt

The campaign's M1 row remains a local receipt only. It is structurally
Expand Down
Loading
Loading