Replacement scorecard: head-to-head yardstick and the incumbent's score - #748
Merged
Conversation
…uildp sparse The earlier audit read policyengine.py 4.15.0 (2026-06-10). PyPI latest is 5.0.3 (2026-08-21): its bundle manifest's US default dataset is populace_us_2024 = the buildp sparse artifact (57,240 households) on policyengine/populace-us, sha256 48b9d479..., certified for the same policyengine-us 1.764.6 this workspace locks. Battery re-classified on observed evidence (provenance columns present, zero acs-channel rows), and the CD-provenance legacy-waiver need recorded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…and candidate The owner's publish bar for the next US artifact is a same-yardstick head-to-head against the live incumbent, decided on evidence. This adds that yardstick: tools/score_us_release_head_to_head.py compiles the single post-#741 fiscal target registry once, scores each artifact through the canonical repair/materialize/score path with the production sqrt-concept-budget 50/50 loss basis, refuses dropped or skipped targets, pins an identical scored-column contract across sides, and reports the terminal by-origin battery from an authenticated pool manifest receipt or as observed inapplicability (never synthesized). Entity H5s that predate the CD vintage provenance attrs get one recorded legacy waiver; candidate pools stay provenance-strict. Tests pin the no-membership-switch signature, the battery contract against the production registries (131 + 1 joint over person/tax_unit/spm_unit), observed-not-asserted inapplicability, loud dropped-target failure, byte-deterministic output, and the live-incumbent identity annotation (policyengine.py 5.0.3 -> policyengine/populace-us buildp sparse, sha256 48b9d479). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hugging Face cache snapshots are .h5-named symlinks to extensionless blob files; resolving the artifact path stripped the suffix policyengine-us's USSingleYearDataset validates and would have recorded the blob hash as the filename identity. The scorer now keeps the snapshot path, with a test that pins the symlink shape. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The production materializer emits one full-length float64 household column per registry spec (32,842 distinct measures on the post-#741 surface — measured: registry compile itself peaks at 2.4 GiB, while the whole-registry materialization of the 57,240-household incumbent peaked near 29 GiB, ~2 x n_specs x n_households x 8 bytes, before it was killed). Both sides now materialize and score fixed contiguous registry chunks through the identical canonical calls (release._materialize_target_frame + score_targets), freeing each chunk's columns before the next. Recombination stays canonical: matrix rows and estimates are row-independent, so per-target numbers are bitwise what a one-shot run would produce, and the aggregate is one relative_error_loss evaluation — the single canonical loss definition — over the full combined vectors, with per-target attribution rows reproducing _target_loss_attribution.py's formulas under whole-registry weight normalization. A new test pins chunked-vs-one-shot bitwise equality (aggregate, fraction within 10%, and every attribution field) across chunk boundaries, including a zero-valued and a signed negative target. Boundary RSS receipts now print to stderr; JSON/Markdown bytes stay deterministic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Journal-only conflicts (FINAL_REPORT.md, PROGRESS.md, _LANE-NOTES.md) resolved by keeping this lane's journals; no code conflicts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replacement scorecard: one yardstick for the incumbent and the candidate
Owner ruling (Max): the publish bar for the next US artifact is a head-to-head against the current live artifact on the same yardstick, not a fully green battery. This PR lands the yardstick and scores the incumbent side so the bar exists before the candidate does.
tools/score_us_release_head_to_head.py: scores any artifact (incumbent.h5or microcosm pool/release) on the full compiled fiscal target registry (post-Compile one target surface unconditionally; remove membership flags #741: the one unified surface, 32,842 targets) plus every battery leg computable on a finished artifact; same code path both sides; chunked to a 20 GiB scoring budget; deterministic, with column-contract tests so a missing scored column fails loudly.policyengine/populace-us@populace-us-2024-buildp-sparse-rmloss100-cae8640-20260728T011454Z,populace_us_2024.h5, sha25648b9d479….experiments/replacement_scorecard/incumbent_48b9d479.{json,md}): weighted loss 0.11462; 26.69% of targets within 10%. All 132 battery comparisons and 369 scalar legs are recorded as inapplicable for the incumbent (zero scoped ACS rows) rather than faked._LANE-NOTES.mdrecords the exact dense and sparse commands for the 25% candidate and states what "better" means on this yardstick without inventing thresholds; the flip stays an owner decision.Suite: 7,028 passed, 76 skipped; ruff clean. No build, publication, or tuning.
Authored by sol (codex-6) with fable groundwork; reviewed by Claude.
🤖 Generated with Claude Code