Add the work_experience_inputs stage: industry recodes and worked-last-year on every US person record

changelog.d/719-work-experience-inputs.added.md

@@ -0,0 +1 @@
+US `work_experience_inputs` source stage: `detailed_industry_recode` (ASEC WEIND, industry of longest job by detailed groups), `major_industry_recode` (WEMIND), and `worked_last_year` (WKSWORK > 0, the work-experience recode universe) on every person record, restored from the three SHA-pinned official ASEC archives via exact PERIDNUM joins; raw ASEC checkpoint schema version 4 (#719).

packages/microcosm-build/src/microcosm/build/source_manifest.py

@@ -87,6 +87,7 @@
         "derive_workers_compensation",
         "derive_weeks_unemployed",
         "derive_wic_claim",
+        "derive_work_experience_inputs",
         "disaggregate_aggregate_records",
         "fit_labor_market_models",
         "fit_tip_income_model",

packages/microcosm-build/src/microcosm/build/spec_engine/schema/sources.schema.json

@@ -1417,6 +1417,18 @@
       "kind"
      ]
     },
+    {
+     "type": "object",
+     "additionalProperties": false,
+     "properties": {
+      "kind": {
+       "const": "derive_work_experience_inputs"
+      }
+     },
+     "required": [
+      "kind"
+     ]
+    },
     {
      "type": "object",
      "additionalProperties": false,

packages/microcosm-build/src/microcosm/build/us/release_input_coverage_manifest.json

@@ -56,6 +56,9 @@
   "cps_race": {
    "status": "required"
   },
+  "detailed_industry_recode": {
+   "status": "required"
+  },
   "detailed_occupation_recode": {
    "status": "required"
   },
@@ -268,6 +271,9 @@
   "long_term_capital_gains_on_collectibles": {
    "status": "required"
   },
+  "major_industry_recode": {
+   "status": "required"
+  },
   "meets_ssi_disability_criteria": {
    "status": "required"
   },
@@ -523,6 +529,9 @@
   "weeks_unemployed": {
    "status": "required"
   },
+  "worked_last_year": {
+   "status": "required"
+  },
   "workers_compensation": {
    "status": "required"
   },
@@ -534,9 +543,9 @@
   }
  },
  "counts": {
-  "required": 163,
+  "required": 166,
   "reviewed_exclusion": 7,
-  "total": 170
+  "total": 173
  },
  "derivation": "Required surface = input columns in the pinned, sha-verified ecps_parity_reference.json populated layers, plus the documented post-reference fsla_overtime_premium, qualified_passenger_vehicle_loan_interest, five desired retirement-contribution inputs, meets_ssi_disability_criteria required by shipped validation probes, and the #282 Schedule-D capital-gain-distributions route leg schedule_d_capital_gain_distributions (PolicyEngine/microcosm#462). status='reviewed_exclusion' for ecps_parity_known_gaps.json entries (reason+issue from that register); EXCEPT every primary-source restoration pinned by RESTORED_REFERENCE_ECPS_REQUIRED_INPUTS (including the Section 199A QBI family), and the SSI countable-resource asset inputs (bank_account_assets, stock_assets, bond_assets), which are status='required' with NO exclusion per PolicyEngine/microcosm#368 so the gate fails on today's artifacts and asset restoration (Deliverable 2) turns it green. All other populated layers are 'required'. Regenerate with tools/build_us_release_input_coverage_manifest.py.",
  "description": "Declared full-coverage contract for a US release: every input column the reference eCPS exports must be persisted as a key with non-default signal, or carry a reviewed exclusion. Enforced as a hard release gate (microcosm.build.us_runtime.release_input_coverage) that generalizes assert_required_us_release_source_columns from 5 columns to the full eCPS input surface.",

packages/microcosm-build/src/microcosm/build/us/source_stages.json

@@ -2907,6 +2907,77 @@
         "health_insurance_premiums"
       ],
       "notes": "After ACA and Medicaid take-up inputs are materialized, the retired annual residual is max(PHIP_VAL - chip_premium - marketplace_net_premium - medicaid_premium, 0), with tax-unit modeled premiums assigned to the first person (archived commit 42ed5d45c56df80d754fbe24cce21cfeb8d05cbe, datasets/cps/cps.py lines 828-944). The archived CPS-only output list and exact eight-predictor, at-most-5,000-person joint QRF are extended_cps.py lines 135-194, 234-248, and 639-745, with only the PUF support half replaced at lines 1014-1076. Because the archive predicted the two PUF leaves jointly without clipping one to the other, PUF-only residual-order exceedances remain diagnostics; measured ASEC rows must preserve the exact residual identity. The archive did not explicitly pin tree count or weights; Microcosm pins 100 trees and typed person weights as reproducibility hardening. The final attribution operation (PolicyEngine/microcosm#451 item 2) is deterministic: it copies the finished reported non-Part-B premium onto the person health_insurance_premiums input exactly where combined Schedule C income (self_employment_income_before_lsr + sstb_self_employment_income_before_lsr) is strictly positive, the person is outside the Medicare proxy (age 65+ or any Social Security disability income), and the person is outside measured employer-sponsored coverage (has_esi - the conservative, incomplete proxy for the 26 USC 162(l)(2)(B) subsidized-employer-plan month exclusion; eligibility through a spouse's or dependent's employer is not measured and remains documented residual overbreadth), and marks is_self_employed for every strictly-positive Schedule C person. PolicyEngine-US 1.764.6 then computes self_employed_health_insurance_premiums (a formula-owned adds-aggregation gated by is_self_employed, so it cannot ship as a column) and the section 162(l) ALD as min(total_self_employment_income, premiums). The Medicare proxy keeps the statutory medical-expense premium concept numerically invariant because the direct input equals the decomposed reported premium whenever the modeled Part B add-on is zero. Beyond the federal ALD, PolicyEngine-US 1.764.6 reads the bare health_insurance_premiums input in five state/local formulas (MO qualified-premium subtraction, MI household resources, OH uninsured unreimbursed medical expenses, CA Orange County general relief countable income, MO child-care-subsidy adjusted income): attribution moves those programs from premiums-invisible toward the measured premium structure for the self-employed slice, outside the federal income_tax probe's scope. External anchor class: IRS SOI Pub 1304 Table 1.4 TY2023 self-employed health insurance deduction, 3,595,764 returns / $31.23B (ledger#105, buildn v9.2 feed)."
+    },
+    {
+      "stage": "work_experience_inputs",
+      "survey": "Census CPS ASEC",
+      "source": "https://www.census.gov/programs-surveys/cps.html",
+      "grain": "person",
+      "artifacts": [
+        {
+          "kind": "public_microdata",
+          "format": "zip_csv",
+          "vintage": "2023 ASEC / 2022 income reference year",
+          "locator": "https://www2.census.gov/programs-surveys/cps/datasets/2023/march/asecpub23csv.zip",
+          "sha256": "d2e000250782adfbdd7f29c82b66d866591a30f0d330496698ec19f9c784ce11",
+          "size_bytes": 150165063,
+          "member": "pppub23.csv",
+          "member_size_bytes": 281065733,
+          "member_crc32": "49c09e5f",
+          "member_sha256": "19b56537e50e7663f954361ef2bb5ce9cef8d9d45f156fe1a69a99b654198ffe"
+        },
+        {
+          "kind": "public_microdata",
+          "format": "zip_csv",
+          "vintage": "2024 ASEC / 2023 income reference year",
+          "locator": "https://www2.census.gov/programs-surveys/cps/datasets/2024/march/asecpub24csv.zip",
+          "sha256": "cdb39cdac34bef99dd0940ab28e306f692404c2eea44d85dfd634214872a0a09",
+          "size_bytes": 148664101,
+          "member": "pppub24.csv",
+          "member_size_bytes": 277250415,
+          "member_crc32": "87950ece",
+          "member_sha256": "21a2b9e0e4b08534563578a45acad77868af4ae9a7d46f23776b707d4a559aa7"
+        },
+        {
+          "kind": "public_microdata",
+          "format": "zip_csv",
+          "vintage": "2025 ASEC / 2024 income reference year",
+          "locator": "https://www2.census.gov/programs-surveys/cps/datasets/2025/march/asecpub25csv.zip",
+          "sha256": "318845a2b5e0034eb2973898de1738f4df0025727de38499e7669cb9c0deef0b",
+          "size_bytes": 147271429,
+          "member": "pppub25.csv",
+          "member_size_bytes": 277882549,
+          "member_crc32": "7dc2878f",
+          "member_sha256": "06921fe83fc66c907e6c7b86b82255dc70458ee7d76258fc48297cb34f0c06b5"
+        },
+        {
+          "kind": "official_data_dictionary",
+          "format": "pdf",
+          "vintage": "2024",
+          "locator": "https://www2.census.gov/programs-surveys/cps/datasets/2024/march/asec2024_ddl_pub_full.pdf",
+          "lines": "person WEIND entry, position 326 length 2; WEMIND entry, position 329 length 2; WKSWORK/WORKYN work-experience universe statements"
+        }
+      ],
+      "operations": [
+        {
+          "kind": "read_table",
+          "table": "person",
+          "weight": "person_weight"
+        },
+        {
+          "kind": "derive_work_experience_inputs"
+        }
+      ],
+      "outputs": [
+        "detailed_industry_recode",
+        "major_industry_recode",
+        "worked_last_year"
+      ],
+      "nonnegative_outputs": [
+        "detailed_industry_recode",
+        "major_industry_recode"
+      ],
+      "notes": "Net-new factual-input coverage with no archived derivation to port: the retired eCPS build never surfaced industry or an explicit worked-last-year indicator (its cps.py carried only the POCCU2 occupation recode). WEIND ('IND. OF LONGEST JOB BY DETAILED GROUPS', position 326 length 2; 0 = NIU, 22 = Military, 23 = Never worked) and WEMIND ('IND. OF LONGEST JOB BY MAJOR IND. GROUPS', position 329 length 2), universe all persons 15+, restore from the three SHA-pinned official ASEC person archives via exact per-income-year PERIDNUM joins (the frozen census_cps inputs never carried them; identical zip/member identities to the education-assistance sidecar pins) and carry directly to both support clones through the shared source identity, exactly as the POCCU2 occupation recode carries. worked_last_year derives as WKSWORK > 0, the official universe condition of the work-experience recode block: measured on the pinned archives, WEIND holds a worker code 1-22 iff WKSWORK > 0 with zero violations (worked rows 73,186 / 73,471 / 72,460; A_FNLWGT-weighted worked shares 0.519968 / 0.523171 / 0.522794; nonzero-recode shares 0.821737 / 0.822002 / 0.824376 across income years 2022-2024), while WORKYN = 1 under-covers that universe by 642 / 572 / 594 allocation rows carrying positive weeks and a real industry (WORKYN = 1 never appears without positive WKSWORK), so the recode-universe definition is the indicator and WORKYN is load-time audit evidence only. The loader also enforces recode coherence ((WEIND = 0) iff (WEMIND = 0), zero violations measured) and the derive handler re-enforces both identities on the pool surface with zero tolerance. Downstream demand: PolicyEngine/microcosm#719 (Living Wage Institute MVP person schema: working indicator + industry + occupation on every person record). Industry-conditional modeling stays owned by PolicyEngine-US; this stage persists measured facts only."
     }
   ]
 }

packages/microcosm-build/src/microcosm/build/us/spec/sources.yaml

@@ -2881,3 +2881,72 @@ stages:
     toward the measured premium structure for the self-employed slice, outside the federal income_tax
     probe''s scope. External anchor class: IRS SOI Pub 1304 Table 1.4 TY2023 self-employed health insurance
     deduction, 3,595,764 returns / $31.23B (ledger#105, buildn v9.2 feed).'
+- stage: work_experience_inputs
+  survey: Census CPS ASEC
+  source: https://www.census.gov/programs-surveys/cps.html
+  grain: person
+  artifacts:
+  - kind: public_microdata
+    format: zip_csv
+    vintage: 2023 ASEC / 2022 income reference year
+    locator: https://www2.census.gov/programs-surveys/cps/datasets/2023/march/asecpub23csv.zip
+    sha256: d2e000250782adfbdd7f29c82b66d866591a30f0d330496698ec19f9c784ce11
+    size_bytes: 150165063
+    member: pppub23.csv
+    member_size_bytes: 281065733
+    member_crc32: 49c09e5f
+    member_sha256: 19b56537e50e7663f954361ef2bb5ce9cef8d9d45f156fe1a69a99b654198ffe
+  - kind: public_microdata
+    format: zip_csv
+    vintage: 2024 ASEC / 2023 income reference year
+    locator: https://www2.census.gov/programs-surveys/cps/datasets/2024/march/asecpub24csv.zip
+    sha256: cdb39cdac34bef99dd0940ab28e306f692404c2eea44d85dfd634214872a0a09
+    size_bytes: 148664101
+    member: pppub24.csv
+    member_size_bytes: 277250415
+    member_crc32: 87950ece
+    member_sha256: 21a2b9e0e4b08534563578a45acad77868af4ae9a7d46f23776b707d4a559aa7
+  - kind: public_microdata
+    format: zip_csv
+    vintage: 2025 ASEC / 2024 income reference year
+    locator: https://www2.census.gov/programs-surveys/cps/datasets/2025/march/asecpub25csv.zip
+    sha256: 318845a2b5e0034eb2973898de1738f4df0025727de38499e7669cb9c0deef0b
+    size_bytes: 147271429
+    member: pppub25.csv
+    member_size_bytes: 277882549
+    member_crc32: 7dc2878f
+    member_sha256: 06921fe83fc66c907e6c7b86b82255dc70458ee7d76258fc48297cb34f0c06b5
+  - kind: official_data_dictionary
+    format: pdf
+    vintage: '2024'
+    locator: https://www2.census.gov/programs-surveys/cps/datasets/2024/march/asec2024_ddl_pub_full.pdf
+    lines: person WEIND entry, position 326 length 2; WEMIND entry, position 329 length 2; WKSWORK/WORKYN work-experience
+      universe statements
+  operations:
+  - kind: read_table
+    table: person
+    weight: person_weight
+  - kind: derive_work_experience_inputs
+  outputs:
+  - detailed_industry_recode
+  - major_industry_recode
+  - worked_last_year
+  nonnegative_outputs:
+  - detailed_industry_recode
+  - major_industry_recode
+  notes: 'Net-new factual-input coverage with no archived derivation to port: the retired eCPS build never surfaced industry
+    or an explicit worked-last-year indicator (its cps.py carried only the POCCU2 occupation recode). WEIND (''IND. OF LONGEST
+    JOB BY DETAILED GROUPS'', position 326 length 2; 0 = NIU, 22 = Military, 23 = Never worked) and WEMIND (''IND. OF LONGEST
+    JOB BY MAJOR IND. GROUPS'', position 329 length 2), universe all persons 15+, restore from the three SHA-pinned official
+    ASEC person archives via exact per-income-year PERIDNUM joins (the frozen census_cps inputs never carried them; identical
+    zip/member identities to the education-assistance sidecar pins) and carry directly to both support clones through the
+    shared source identity, exactly as the POCCU2 occupation recode carries. worked_last_year derives as WKSWORK > 0, the
+    official universe condition of the work-experience recode block: measured on the pinned archives, WEIND holds a worker
+    code 1-22 iff WKSWORK > 0 with zero violations (worked rows 73,186 / 73,471 / 72,460; A_FNLWGT-weighted worked shares
+    0.519968 / 0.523171 / 0.522794; nonzero-recode shares 0.821737 / 0.822002 / 0.824376 across income years 2022-2024), while
+    WORKYN = 1 under-covers that universe by 642 / 572 / 594 allocation rows carrying positive weeks and a real industry (WORKYN
+    = 1 never appears without positive WKSWORK), so the recode-universe definition is the indicator and WORKYN is load-time
+    audit evidence only. The loader also enforces recode coherence ((WEIND = 0) iff (WEMIND = 0), zero violations measured)
+    and the derive handler re-enforces both identities on the pool surface with zero tolerance. Downstream demand: PolicyEngine/microcosm#719
+    (Living Wage Institute MVP person schema: working indicator + industry + occupation on every person record). Industry-conditional
+    modeling stays owned by PolicyEngine-US; this stage persists measured facts only.'

packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py

@@ -1061,6 +1061,26 @@
     us_wic_claim_summary,
     with_us_wic_claim_input,
 )
+from microcosm.build.us_runtime.work_experience_inputs import (
+    US_WORK_EXPERIENCE_NONCONSTANT_PERSON_COLUMNS,
+    US_WORK_EXPERIENCE_OUTPUT_COLUMNS,
+    US_WORK_EXPERIENCE_REQUIRED_SOURCE_COLUMNS,
+    US_WORK_EXPERIENCE_STAGE_NAME,
+    derive_us_work_experience_inputs_from_manifest,
+    us_work_experience_signal_gate,
+    us_work_experience_stage_spec,
+    us_work_experience_summary,
+    with_us_work_experience_inputs,
+)
+from microcosm.build.us_runtime.work_experience_source import (
+    ASEC_WORK_EXPERIENCE_ARCHIVES,
+    ASEC_WORK_EXPERIENCE_INCOME_YEARS,
+    ASEC_WORK_EXPERIENCE_SOURCE_COLUMNS,
+    WORK_EXPERIENCE_OFFICIAL_DICTIONARY_URL,
+    fetch_asec_work_experience_source,
+    fill_asec_work_experience_source,
+    load_asec_work_experience_sources,
+)
 from microcosm.build.us_runtime.workers_compensation import (
     US_WORKERS_COMPENSATION_NONCONSTANT_PERSON_COLUMNS,
     US_WORKERS_COMPENSATION_OUTPUT_COLUMNS,
@@ -1403,6 +1423,22 @@
     "us_wic_claim_stage_spec",
     "us_wic_claim_summary",
     "with_us_wic_claim_input",
+    "ASEC_WORK_EXPERIENCE_ARCHIVES",
+    "ASEC_WORK_EXPERIENCE_INCOME_YEARS",
+    "ASEC_WORK_EXPERIENCE_SOURCE_COLUMNS",
+    "US_WORK_EXPERIENCE_NONCONSTANT_PERSON_COLUMNS",
+    "US_WORK_EXPERIENCE_OUTPUT_COLUMNS",
+    "US_WORK_EXPERIENCE_REQUIRED_SOURCE_COLUMNS",
+    "US_WORK_EXPERIENCE_STAGE_NAME",
+    "WORK_EXPERIENCE_OFFICIAL_DICTIONARY_URL",
+    "derive_us_work_experience_inputs_from_manifest",
+    "fetch_asec_work_experience_source",
+    "fill_asec_work_experience_source",
+    "load_asec_work_experience_sources",
+    "us_work_experience_signal_gate",
+    "us_work_experience_stage_spec",
+    "us_work_experience_summary",
+    "with_us_work_experience_inputs",
     "US_MISC_ITEMIZED_NONCONSTANT_PERSON_COLUMNS",
     "US_MISC_ITEMIZED_OUTPUT_COLUMNS",
     "US_MISC_ITEMIZED_STAGE_NAME",
@@ -2355,6 +2391,17 @@ def to_manifest(self) -> dict[str, object]:
             "educational assistance carries directly from ASEC ED_VAL."
         ),
     ),
+    US_WORK_EXPERIENCE_STAGE_NAME: DonorSpec(
+        survey="Census CPS ASEC",
+        source="https://www.census.gov/programs-surveys/cps.html",
+        notes=(
+            "Industry of the longest job (WEIND detailed groups, WEMIND major "
+            "groups) restores from the three SHA-pinned official ASEC person "
+            "archives by exact PERIDNUM join and carries directly; "
+            "worked_last_year is WKSWORK > 0, the work-experience recode "
+            "universe. Nothing is imputed (microcosm#719)."
+        ),
+    ),
     US_RETIREMENT_CONTRIBUTION_STAGE_NAME: DonorSpec(
         survey="Census CPS ASEC + published retirement-contribution shares",
         source="https://www.census.gov/programs-surveys/cps.html",
@@ -2437,6 +2484,7 @@ def to_manifest(self) -> dict[str, object]:
     US_WORKERS_COMPENSATION_STAGE_NAME,
     US_WEEKS_UNEMPLOYED_STAGE_NAME,
     US_EDUCATION_INPUTS_STAGE_NAME,
+    US_WORK_EXPERIENCE_STAGE_NAME,
     "capital_gain_distributions",
     "scf_wealth",
     US_SSI_DISABILITY_CRITERIA_STAGE_NAME,