Skip to content

Add the work_experience_inputs stage: industry recodes and worked-last-year on every US person record - #745

Draft
MaxGhenis wants to merge 15 commits into
mainfrom
work-experience-inputs-719
Draft

Add the work_experience_inputs stage: industry recodes and worked-last-year on every US person record#745
MaxGhenis wants to merge 15 commits into
mainfrom
work-experience-inputs-719

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Fixes #719.

What

A new post-clone source stage, work_experience_inputs, puts three measured CPS work-experience attributes on every US person record:

column source semantics
detailed_industry_recode ASEC WEIND industry of longest job by detailed groups (0:23; 22 = Military, 23 = Never worked; 0 = NIU)
major_industry_recode ASEC WEMIND industry of longest job by major industry groups (0:15)
worked_last_year ASEC WKSWORK > 0 the official universe condition of the work-experience longest-job recode block

These are the reference-period-coherent siblings of detailed_occupation_recode (POCCU2, occupation of longest job): the survey-week A_DTIND/A_MJIND recodes would describe a different job than the occupation the model already carries. Names and universes are verified against the official ASEC public-use data dictionary (asec2024_ddl_pub_full.pdf, person entries at positions 326/329).

The frozen census_cps person inputs never carried WEIND/WEMIND, so a new sidecar (work_experience_source.py) restores them for every pooled income year from the three SHA-pinned official ASEC archives via exact PERIDNUM joins — the same archives and pattern as the education-assistance/PAW_TYP sidecars, with the stage's own measured audit pins. Nothing is imputed; both support clones of a source person inherit identical values, as the occupation recode does. The outputs also join the ACS-spine transfer surface automatically (one bounded group, three targets), so ACS-spine persons receive QRF-transferred values like every other pool operator output.

Why WKSWORK > 0 rather than WORKYN == 1

Measured on the pinned archives: WEIND holds a worker code (1–22) iff WKSWORK > 0 with zero violations in all three years (worked rows 73,186 / 73,471 / 72,460), while WORKYN = 1 under-covers that universe by 642 / 572 / 594 allocation rows per year that carry positive weeks and a real industry (WORKYN = 1 never appears without positive WKSWORK). The loader enforces these identities at load time and the derive handler re-enforces them on the pool surface with zero tolerance.

Wiring

  • Manifest: source_stages.json + authoritative spec/sources.yaml (+ sources.schema.json kind), US_DONORS, US_STAGE_NAMES, allowed operation kinds, central handler registry.
  • Pool: operator contract/order/lambda, late-producer inventory, operator_boundary output family, stacked-spine battery metrics (categorical_tvd recodes, boolean_incidence indicator) and stage/spec/sidecar bindings; pinned partition counts move 19→20 groups, 70→73 late targets, 118→121 transfer / 131→134 terminal surface.
  • Raw checkpoint: asec_raw_stage schema version 4 restores WEIND/WEMIND alongside ED_VAL/LKWEEKS/PAW_TYP (industry-blind artifacts fail loudly); build_us_puf_support_base.py loads the sidecar from the shared --asec-education-source archives and runs the stage with its gate on both builder paths.
  • Release: POST_REFERENCE_ECPS_REQUIRED_INPUTS + regenerated coverage manifest (163 → 166 required); L0-refit export requirement.
  • Tests: sidecar (16), stage (24), plus fixture/enumeration updates across the checkpoint, plan, multispine, and country-spec suites.

Dependency — why this is a draft

The engine-side input variables land in PolicyEngine/policyengine-us#9311 (31/31 checks green, mergeable). Microcosm checks release columns and transfer targets against the locked PolicyEngine-US (uv.lock pins 1.764.6, unchanged since July), and tools/generate_us_bundle_from_constants.py projects us/spec/*.yaml + engine_abi.lock.json from engine metadata, so with the current lock exactly two categories stay red:

  1. Engine-leaf checks: test_us_multispine_pool.py::test_every_pool_transfer_target_is_an_installed_engine_input_leaf, test_release_input_coverage.py::TestShippedManifest::test_manifest_is_current, and the whole of test_us_pool_input_consumers.py (validating the three new CLIENT_SURFACE allowlist entries looks the variables up in the locked engine).
  2. Bundle projection + attestation (one regeneration pass): test_us_spec_bundle.py, test_imputation_lineage_spec.py, test_us_bundle_core_contracts.py (catalog entries), test_spec_engine_* (battery/imputation/inventory/field-usage/identity/loader/legacy-adapter/stacked-authority projections and the reviewed digest vector in inventory_coverage.py, including source_stages == 37), and test_us_multispine_pool_tool.py (the constants-vs-bundle battery contract and the stacked goldens that fold bundle identities). The generator cannot run until the installed engine knows the variables.

Everything else passes locally: the new-stage suites, checkpoint/builder/L0-export fixtures, the late-DAG and stacked-authority pins (20 groups / 73 late targets / 134-target terminal surface), spine-blindness classification (runtime graph 65 → 67), the CLIENT_SURFACE consumer review for the three columns, and the BE/UK spec digests (re-pinned: the seed-protocol attestation digests the source of us_runtime/source_runtime.py, where the derive handler registers). test_us_trade_imdb_bulk.py::test_build_cli_success_replaces_prior_publication_completely failed once in a parallel full run and passes in isolation — unrelated.

Moving the pin to ≥1.817 is itself a procedure — tracked as #746 (generated-variable audit re-pin, take-up-contract totality for four new engine flags, and pe-us's rename of would_claim_wictakes_up_wic_if_eligible, which migrates the existing wic_claim stage). Sequence: merge #9311 → #746 → rebase this branch, run the generator, re-cut the attestation vector and goldens, un-draft.

No frozen census_cps artifacts, HF destinations, or release publication are touched.

🤖 Generated with Claude Code

MaxGhenis and others added 10 commits August 22, 2026 17:16
Restores the work-experience industry recodes for every pooled income
year from the three SHA-pinned official ASEC public-use archives via
exact per-income-year PERIDNUM joins, mirroring the education-assistance
sidecar. The loader enforces the official universe identity against each
archive's own WKSWORK/WORKYN (WEIND in 1..22 iff WKSWORK > 0; detailed
and major recodes zero together; WORKYN = 1 never without positive
weeks) and pins per-archive worked/recode row counts and weighted
shares measured from the verified members.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Carries WEIND/WEMIND as detailed_industry_recode/major_industry_recode
(int16, mirroring the POCCU2 occupation carry) and derives
worked_last_year as WKSWORK > 0 — the official universe condition of the
work-experience recode block, enforced as a zero-tolerance identity in
the derive handler and signal gate alongside the detailed/major zero-row
identity. Pool-level plausibility bands anchor to the measured archive
shares (worked 0.520-0.523, nonzero recode 0.822-0.824).

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Manifest-defined stage entry with the three pinned archive identities,
the official-dictionary citation, and the measured universe-identity
receipts; derive_work_experience_inputs joins the allowed operation
kinds.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The raw_source_mapping artifact now restores WEIND/WEMIND alongside
ED_VAL/LKWEEKS/PAW_TYP (schema version 4; older industry-blind artifacts
fail loudly), the shared --asec-education-source archives feed the new
loader, and both builder paths run with_us_work_experience_inputs with
its signal gate after education inputs.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Post-clone operator contract/order/lambda, late-producer inventory,
stacked-spine battery metrics (categorical_tvd industry recodes,
boolean_incidence worked_last_year) and stage/spec/sidecar bindings,
L0-refit export requirement, POST_REFERENCE release coverage (166
required), and the plan-closure sidecar set.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The work_experience_inputs operator adds one bounded late-transfer group
(19 -> 20) with three ordered targets (70 -> 73: two numeric industry
recodes, one boolean worked_last_year), and the stacked authority surface
grows accordingly (transfer 118 -> 121, terminal 131 -> 134, ASEC-source
producer targets 29 -> 32).

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ations

Checkpoint fixtures carry WEIND/WEMIND (with zero-coherence and range
cases), multispine fixtures carry coherent work-experience recodes, and
the pinned order tuples, transfer/terminal counts, receipt counts, and
target-name digest move with the new operator.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sources.yaml stage_manifest projection now carries work_experience_inputs
(the packaged source_stages.json is its attested copy) and
sources.schema.json admits the derive_work_experience_inputs kind.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sidecar: pinned-vintage mapping, loader audit and tamper/unpinned/drift
rejection, the official universe identities (WEIND worker codes iff
WKSWORK > 0, detailed/major zero together, WORKYN = 1 never without
weeks), and the identity-verified two-column fill with its refusal to
overwrite measured data. Stage: manifest declaration and artifact pins,
derive handler identities and ranges, operator materialization, frozen
WKSWORK requirement, sidecar restore equivalence, idempotent passthrough,
identical support clones, and the signal gate's bands and coherence
failures. The fill now mirrors the education sidecar's per-year
redundant identity checks.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaxGhenis and others added 5 commits August 22, 2026 23:45
The seed-protocol attestation digests the source text of
us_runtime/source_runtime.py; registering the derive handler there moves
every country's spec hash.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dules

The three work-experience inputs join the reviewed non-engine consumer
allowlist as CLIENT_SURFACE (the Living Wage Institute county-file
schema), the operator and sidecar modules are classified for the
spine-blindness scan (runtime graph 65 -> 67), and the builder, raw
checkpoint, and L0-export fixtures carry the new columns and loader.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Registry 38 -> 40 producers, 20 transfer groups, 73 targets, 75 edges;
dependency kinds partition 53 numeric / 18 boolean / 2 string; raw-input
scope counters and the finite-numeric inventory cover
with_us_work_experience_inputs.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The canonical post-PUF completion receipt is now 20 groups / 73 targets
(source pins), and the stacked-spine test inventories follow: 134-target
registry, metric family counts (two categorical recodes, one boolean
indicator), boundary dtype inventory, sidecar binding, and producer
counts.

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rgets

Part of #719.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Surface work-experience industry and worked-last-year person inputs (LWI MVP schema)

1 participant