The database leaves git: derived artifact, built in CI, published to Supabase storage

[MaxGhenis]

The rebuild experiment behind this: the entire 58,468-claim database regenerates
from committed inputs in ~4 seconds, deterministically (two builds, identical
logical content hash) — and diffing the rebuild against the committed blob
showed the blob silently lagging the code (solo exhibits never re-committed,
placeholder timestamps, unstamped legacy baseline provenance). A committed
derived artifact drifts; a built one cannot.

New: scorecard_db/build_db.py (the one dependency-ordered build entrypoint +
content hash). data/scorecard.db untracked. CI builds fresh, asserts
determinism, runs the suite against the build, uploads the artifact, and on
main publishes .db.gz + latest.db.gz (~6 MB) to the scorecard-artifacts
bucket on the PE Supabase project (bucket created, repo secrets set).
Committed JSON feeds are byte-identical from the fresh build — the app layer
is insulated. Postgres-as-database-of-record was considered and rejected (it
breaks reviewable data diffs and atomic code+data merges). History slimming of
the old blobs is deliberately deferred to just before the repo-public flip.

Dual gate: fable review done; sol pass next.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review — verified end-to-end locally; sound, three should-mentions

Ran the whole contract on this branch (macOS): two from-scratch builds in ~4s each, content hashes match (0c63dff1…) and the artifacts are in fact byte-identical here — stronger than the logical-hash gate CI asserts. git diff --exit-code clean before and after the suite (the no-drift gate catches nothing on a fresh build, as it should), ruff format --check clean, 245 passed / 2 skipped against the built DB. PR CI: both jobs green, publish correctly skipping on a PR.

As the UK-lane owner, the finding I came to check: the UK chain is fully in the build — uk_externals lands 15,851 claims + 1,073 Ledger-routed rows (= the 16,924 of #48), uk_deductions 7, produce_uk/campaign_uk attach the 14 reckoner joins. Nothing drops out of the published DB. The build-step summary's blocked ledger (free_joins/obr_measures/two_child/uprating with reasons) is a nice honesty artifact.

Also verified: the (claim_id, computed_at) tie refusal in build_db.py:100-110 covers exactly the two ORDER BY computed_at, id consumers (export_populations.py:125, db.py:194) — the id-exclusion from the content hash is sound while that gate holds. And the publish job is push+main-gated with skip-on-missing-secrets, so fork PRs never touch credentials.

Should mention

1. No integrity manifest published. <sha>.db.gz + latest.db.gz go up, but the content_hash CI just computed doesn't — a consumer fetching latest.db.gz can't verify a complete download or learn which commit it reflects without trusting the transport. Cheap: upload a latest.json {sha, content_hash, gzip_bytes} alongside, written after both objects (also makes the two-object upsert sequence detectably atomic).
2. conftest.py uses a stale local DB silently — if not db.exists() means a developer with last week's data/scorecard.db runs today's suite against old data (CI is immune: fresh checkout). A one-line staleness warning (or rebuild when the DB predates scorecard_db/'s newest mtime) would close the local footgun.
3. Free-plan 50 MB object cap is fine at 6 MB gzipped, but the DB grew ~40% this week from the UK lanes alone; worth a ::warning in the publish step when the gzip crosses, say, 35 MB rather than discovering the cap as a hard 4xx.

None of these block. The vendoring of populace-sotsn-comparison (repo now buildable from a bare clone — confirmed, that's what my build ran on) and the refuse-to-overwrite build are both the right calls.