Add the work_experience_inputs stage: industry recodes and worked-last-year on every US person record - #745
Draft
MaxGhenis wants to merge 15 commits into
Draft
Add the work_experience_inputs stage: industry recodes and worked-last-year on every US person record#745MaxGhenis wants to merge 15 commits into
MaxGhenis wants to merge 15 commits into
Conversation
Restores the work-experience industry recodes for every pooled income year from the three SHA-pinned official ASEC public-use archives via exact per-income-year PERIDNUM joins, mirroring the education-assistance sidecar. The loader enforces the official universe identity against each archive's own WKSWORK/WORKYN (WEIND in 1..22 iff WKSWORK > 0; detailed and major recodes zero together; WORKYN = 1 never without positive weeks) and pins per-archive worked/recode row counts and weighted shares measured from the verified members. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Carries WEIND/WEMIND as detailed_industry_recode/major_industry_recode (int16, mirroring the POCCU2 occupation carry) and derives worked_last_year as WKSWORK > 0 — the official universe condition of the work-experience recode block, enforced as a zero-tolerance identity in the derive handler and signal gate alongside the detailed/major zero-row identity. Pool-level plausibility bands anchor to the measured archive shares (worked 0.520-0.523, nonzero recode 0.822-0.824). Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Manifest-defined stage entry with the three pinned archive identities, the official-dictionary citation, and the measured universe-identity receipts; derive_work_experience_inputs joins the allowed operation kinds. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The raw_source_mapping artifact now restores WEIND/WEMIND alongside ED_VAL/LKWEEKS/PAW_TYP (schema version 4; older industry-blind artifacts fail loudly), the shared --asec-education-source archives feed the new loader, and both builder paths run with_us_work_experience_inputs with its signal gate after education inputs. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Post-clone operator contract/order/lambda, late-producer inventory, stacked-spine battery metrics (categorical_tvd industry recodes, boolean_incidence worked_last_year) and stage/spec/sidecar bindings, L0-refit export requirement, POST_REFERENCE release coverage (166 required), and the plan-closure sidecar set. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The work_experience_inputs operator adds one bounded late-transfer group (19 -> 20) with three ordered targets (70 -> 73: two numeric industry recodes, one boolean worked_last_year), and the stacked authority surface grows accordingly (transfer 118 -> 121, terminal 131 -> 134, ASEC-source producer targets 29 -> 32). Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ations Checkpoint fixtures carry WEIND/WEMIND (with zero-coherence and range cases), multispine fixtures carry coherent work-experience recodes, and the pinned order tuples, transfer/terminal counts, receipt counts, and target-name digest move with the new operator. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sources.yaml stage_manifest projection now carries work_experience_inputs (the packaged source_stages.json is its attested copy) and sources.schema.json admits the derive_work_experience_inputs kind. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sidecar: pinned-vintage mapping, loader audit and tamper/unpinned/drift rejection, the official universe identities (WEIND worker codes iff WKSWORK > 0, detailed/major zero together, WORKYN = 1 never without weeks), and the identity-verified two-column fill with its refusal to overwrite measured data. Stage: manifest declaration and artifact pins, derive handler identities and ranges, operator materialization, frozen WKSWORK requirement, sidecar restore equivalence, idempotent passthrough, identical support clones, and the signal gate's bands and coherence failures. The fill now mirrors the education sidecar's per-year redundant identity checks. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The seed-protocol attestation digests the source text of us_runtime/source_runtime.py; registering the derive handler there moves every country's spec hash. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dules The three work-experience inputs join the reviewed non-engine consumer allowlist as CLIENT_SURFACE (the Living Wage Institute county-file schema), the operator and sidecar modules are classified for the spine-blindness scan (runtime graph 65 -> 67), and the builder, raw checkpoint, and L0-export fixtures carry the new columns and loader. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Registry 38 -> 40 producers, 20 transfer groups, 73 targets, 75 edges; dependency kinds partition 53 numeric / 18 boolean / 2 string; raw-input scope counters and the finite-numeric inventory cover with_us_work_experience_inputs. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The canonical post-PUF completion receipt is now 20 groups / 73 targets (source pins), and the stacked-spine test inventories follow: 134-target registry, metric family counts (two categorical recodes, one boolean indicator), boundary dtype inventory, sidecar binding, and producer counts. Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rgets Part of #719. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #719.
What
A new post-clone source stage,
work_experience_inputs, puts three measured CPS work-experience attributes on every US person record:detailed_industry_recodeWEINDmajor_industry_recodeWEMINDworked_last_yearWKSWORK > 0These are the reference-period-coherent siblings of
detailed_occupation_recode(POCCU2, occupation of longest job): the survey-weekA_DTIND/A_MJINDrecodes would describe a different job than the occupation the model already carries. Names and universes are verified against the official ASEC public-use data dictionary (asec2024_ddl_pub_full.pdf, person entries at positions 326/329).The frozen census_cps person inputs never carried
WEIND/WEMIND, so a new sidecar (work_experience_source.py) restores them for every pooled income year from the three SHA-pinned official ASEC archives via exactPERIDNUMjoins — the same archives and pattern as the education-assistance/PAW_TYP sidecars, with the stage's own measured audit pins. Nothing is imputed; both support clones of a source person inherit identical values, as the occupation recode does. The outputs also join the ACS-spine transfer surface automatically (one bounded group, three targets), so ACS-spine persons receive QRF-transferred values like every other pool operator output.Why
WKSWORK > 0rather thanWORKYN == 1Measured on the pinned archives:
WEINDholds a worker code (1–22) iffWKSWORK > 0with zero violations in all three years (worked rows 73,186 / 73,471 / 72,460), whileWORKYN = 1under-covers that universe by 642 / 572 / 594 allocation rows per year that carry positive weeks and a real industry (WORKYN = 1never appears without positiveWKSWORK). The loader enforces these identities at load time and the derive handler re-enforces them on the pool surface with zero tolerance.Wiring
source_stages.json+ authoritativespec/sources.yaml(+sources.schema.jsonkind),US_DONORS,US_STAGE_NAMES, allowed operation kinds, central handler registry.operator_boundaryoutput family, stacked-spine battery metrics (categorical_tvdrecodes,boolean_incidenceindicator) and stage/spec/sidecar bindings; pinned partition counts move 19→20 groups, 70→73 late targets, 118→121 transfer / 131→134 terminal surface.asec_raw_stageschema version 4 restoresWEIND/WEMINDalongsideED_VAL/LKWEEKS/PAW_TYP(industry-blind artifacts fail loudly);build_us_puf_support_base.pyloads the sidecar from the shared--asec-education-sourcearchives and runs the stage with its gate on both builder paths.POST_REFERENCE_ECPS_REQUIRED_INPUTS+ regenerated coverage manifest (163 → 166 required); L0-refit export requirement.Dependency — why this is a draft
The engine-side input variables land in PolicyEngine/policyengine-us#9311 (31/31 checks green, mergeable). Microcosm checks release columns and transfer targets against the locked PolicyEngine-US (
uv.lockpins 1.764.6, unchanged since July), andtools/generate_us_bundle_from_constants.pyprojectsus/spec/*.yaml+engine_abi.lock.jsonfrom engine metadata, so with the current lock exactly two categories stay red:test_us_multispine_pool.py::test_every_pool_transfer_target_is_an_installed_engine_input_leaf,test_release_input_coverage.py::TestShippedManifest::test_manifest_is_current, and the whole oftest_us_pool_input_consumers.py(validating the three new CLIENT_SURFACE allowlist entries looks the variables up in the locked engine).test_us_spec_bundle.py,test_imputation_lineage_spec.py,test_us_bundle_core_contracts.py(catalog entries),test_spec_engine_*(battery/imputation/inventory/field-usage/identity/loader/legacy-adapter/stacked-authority projections and the reviewed digest vector ininventory_coverage.py, includingsource_stages == 37), andtest_us_multispine_pool_tool.py(the constants-vs-bundle battery contract and the stacked goldens that fold bundle identities). The generator cannot run until the installed engine knows the variables.Everything else passes locally: the new-stage suites, checkpoint/builder/L0-export fixtures, the late-DAG and stacked-authority pins (20 groups / 73 late targets / 134-target terminal surface), spine-blindness classification (runtime graph 65 → 67), the CLIENT_SURFACE consumer review for the three columns, and the BE/UK spec digests (re-pinned: the seed-protocol attestation digests the source of
us_runtime/source_runtime.py, where the derive handler registers).test_us_trade_imdb_bulk.py::test_build_cli_success_replaces_prior_publication_completelyfailed once in a parallel full run and passes in isolation — unrelated.Moving the pin to ≥1.817 is itself a procedure — tracked as #746 (generated-variable audit re-pin, take-up-contract totality for four new engine flags, and pe-us's rename of
would_claim_wic→takes_up_wic_if_eligible, which migrates the existingwic_claimstage). Sequence: merge #9311 → #746 → rebase this branch, run the generator, re-cut the attestation vector and goldens, un-draft.No frozen census_cps artifacts, HF destinations, or release publication are touched.
🤖 Generated with Claude Code