Add label-verified CPS ASEC firm-size (NOEMP) reader with band-regime maps

[daphnehanse11]

Closes #193 (Workstream A week-1 piece of #192).

What

data/asec_firm_size.py: a label-verified reader for the CPS ASEC firm-size variable (NOEMP) — the employer-firm plan's designated firm-size training label — following the family.py pattern of dictionary-adjudicated per-year maps verified at read time, plus firm_size_tabulation, the weighted evidence artifact the C2 banding decision consumes.

Why regime-aware

NOEMP keeps the identical code domain (0:6) in every year while codes 2–3 silently change meaning. Verified against all fifteen Census public-use data dictionaries (2011–2025, www2.census.gov/programs-surveys/cps/datasets, read 2026-07-14):

Code	2011–2018	2019–2025
2	10–49	10–24
3	50–99	25–99

A year-blind read mis-bands those codes with no error, and the regime break is what hides the ACA 50-FTE threshold post-2019 (C2 discussion on #192). The reader hard-codes the per-year map, refuses unverified years, refuses a pppub{yy} filename that contradicts the requested year, and enforces the WKSWORK > 0 universe and dictionary code domains at read time.

Record contents

One row per person in the longest-job-last-year universe: raw code + regime-labeled band, I_NOEMP allocation flag, LJCW class of worker (labeled — SUSB/QWI calibration universes exclude government and self-employment, see the C1 discussion on #192), longest-job industry (major + detailed), WKSWORK, MARSUPWT.

Staging

Raw microdata stays out of the repo, PSID-style: Census person files as pppub{yy}.csv[.gz] under ~/PolicyEngine/asec-data, POPULACE_DYNAMICS_ASEC_DIR override.

Tests

17 tests (unit tier; tier_counts.json 410 → 427): both regimes' banding, regime boundaries (2011/2018/2019/2025), unverified-year and path/year-mismatch refusal, missing-column / out-of-domain / off-universe failures, allocation and class-of-worker flags, env-var staging, tabulation weighted counts + allocated share, and a skipif-gated real-data pass. Full unit tier: 424 passed, 3 skipped.

Non-goals

No band harmonization across regimes (that is C2's decision, which this feeds), no imputation, no C1 spell emission, no tenure-supplement loader (separate piece).

🤖 Generated with Claude Code

[vercel]

@daphnehanse11 is attempting to deploy a commit to the PolicyEngine Team on Vercel.

A member of the Team first needs to authorize it.

[vahid-ahmadi]

Reviewed against the 194 diff and the C1/C2 discussion on #192. Overall: this is exactly the reader Workstream B needs to consume for the C2 evidence, the regime-break handling is right (the 2011–2018 vs 2019+ code 2/3 swap matches what our independent data audit verified against IPUMS/Census), and the refuse-don't-guess posture (unverified years, path/year mismatch, dictionary domains, universe check) fits the repo's label-verified reader discipline well. Findings, ranked:

1. 🔴 person_id will silently collide on real 2017+ CSVs — read PERIDNUM as a string. PERIDNUM is a 22-digit identifier; pd.read_csv without a dtype will overflow int64 (19 digits) and fall back to float64, whose 53-bit mantissa holds only ~15–16 significant digits — so distinct persons collapse to the same rounded value before astype(str) runs, and the resulting ids also render as 1.100234e+21-style strings. The fixture tests use small ids so nothing catches it, and TestRealData checks len(out) > 10_000 but not uniqueness. Fix: pd.read_csv(..., dtype={"PERIDNUM": "string"}) (and add a uniqueness assertion to TestRealData).

2. 🟡 tests/tier_counts.json will conflict with #195. This PR moves unit 410→427; #195 (Workstream B) moves it 410→464 on the same base. Whichever merges second needs 481. Not a defect — just flagging so the second merge doesn't get a surprise CI failure; happy to rebase #195 if this lands first.

3. 🟡 The error path in _domain_error can itself raise on mixed-type columns. sorted(bad.unique()) — bad holds the original values (pre-to_numeric), so a column containing both a stray string and an out-of-range number gives sorted(['A', 7.0]) → TypeError, masking the intended ValueError. sorted(map(str, bad.unique())) keeps the message and is total.

4. 🟢 LJCW universe asymmetry. The universe check refuses a nonzero NOEMP off WKSWORK > 0, but a row with WKSWORK > 0 and LJCW = 0 passes through with class_of_worker = "niu". If the dictionary states the same universe for LJCW, the symmetric check would catch a malformed file the current one misses; if it doesn't, a comment saying so would prevent someone "fixing" it later.

5. 🟢 Perf/memory nit. pd.read_csv(person_path, usecols=None, low_memory=False) loads all ~800 ASEC columns to then keep 8. Reading with usecols=lambda c: c in set(_REQUIRED_COLUMNS) keeps the friendly missing-column error (check what actually arrived afterward) and cuts peak memory ~100× on the real files.

Also noted: I_NOEMP and MARSUPWT get no domain validation (negative weights would flow into firm_size_tabulation silently) — fine to defer, but worth a line in the docstring if intentional.

Finding 1 is the only one I'd hold the merge for; everything else is polish. The firm_size_tabulation shape (weighted persons + allocated_share per cell) is exactly what the C2 banding memo on the #195 side wants to cite — once both land I'll wire it into the reconciliation table.

🤖 Generated with Claude Code

[daphnehanse11]

Thanks — all five addressed in 56e2bb3:

1. PERIDNUM → string (the blocker): now read with dtype={"PERIDNUM": "string"} and a comment explaining the int64/float64 trap; added a fixture test with two 22-digit ids one final digit apart (they round together under any numeric read) and person_id.is_unique to the real-data pass. Good catch — the fixtures' small ids would never have surfaced this.
2. tier_counts: now 410→432 here (five new tests on top of the original 17). Whichever of this and Employer-firm Workstream B: canonical banding (C2), ADR 0003, target pipeline (#192) #195 lands second takes master to the sum — happy to do the rebase on my side if Employer-firm Workstream B: canonical banding (C2), ADR 0003, target pipeline (#192) #195 goes first.
3. _domain_error totality: sorted(map(str, bad.unique())); added a mixed-type fixture (NOEMP = ["A", 7]) asserting the error stays a ValueError.
4. LJCW symmetry: enforced, and in both directions for both variables — I checked the dictionaries: 2011–2018 state the universe as WORKYN = 1 and 2019+ as WKSWORK > 0 (they coincide), and LJCW shares NOEMP's universe in both regimes. NOEMP or LJCW nonzero exactly where WKSWORK > 0, loud failure otherwise, with the docstring recording the two universe statements.
5. usecols: reads only the eight required columns; the missing-column error now checks what actually arrived, same message.

Also took the deferred suggestion since it was two lines: I_NOEMP validates to 0–9 and MARSUPWT to non-negative, so bad weights fail at read time rather than flowing into the tabulation.

🤖 Generated with Claude Code

[vahid-ahmadi]

All five addressed and verified in the new diff — the 22-digit PERIDNUM regression test is exactly the right pin, and the two-regime universe verification on point 4 goes beyond what I asked. CI green. One merge-order note: with your +22 and #195's +54, whichever lands second sets unit to 486 (410 + 22 + 54).

🤖 Generated with Claude Code

[MaxGhenis]

Automated review pass (full read of the reader, band regimes, and tests; suite run locally since fork PRs don't get pytest/lint CI here — 29 pass / 1 skip, black + ruff clean).

Verdict: mergeable quality. The reader is solid — band-regime maps check out against the NOEMP vintages, and the PERIDNUM-as-string fix is right.

Two things before merge:

1. Stale approval: the ✓ review was given at 56e2bb3; head is now f8b6093 (the round-2 tests, delta +22 → +30). Needs a re-approval of the current head.
2. tests/tier_counts.json is stale vs master: master moved today to unit 420 / artifact 1011 (PR M6 factory: (0,24) projection-coverage pad + pe-us param-dir binding (3e patch: F1/F2/N1 + bridge test) #197 added tests on both axes). This branch's committed 440/1010 will fail the tier-policy test on merge. Recount at merge time by running the full-collection policy test (pytest tests/ -k tier_policy) on the merged tree and committing the actual counts — if this PR merges first, that's unit 450 / artifact 1011; later in the queue, cumulative.

One heads-up for the C2 seam (not blocking this PR): this reader emits raw Census NOEMP codes (0–6) + native labels, while #195's cps_firmsize_to_canonical expects IPUMS FIRMSIZE numbering — same concept, different code space (NOEMP 6 = 1000+; FIRMSIZE 6 = 50–99). Feeding one into the other today silently mis-bands. Flagged on #192 with a proposed adapter + seam test before the C2 freeze.

🤖 Generated with Claude Code

[daphnehanse11]

Rebased onto master (57e0baa, post-#197) and recounted with the full-collection policy test on the merged tree: tier_counts now unit 450 / artifact 1011 (this PR's first-to-merge value — whichever of #199/#201/#195 lands after this resolves cumulatively). Tier policy + module suite verified green on the rebased head. Re-requesting @vahid-ahmadi's review since the earlier ✓ predates the round-2 commits.

On the C2 seam heads-up: agreed and tracked — the adapter + seam test proposal is in the #195 review; this side holds raw NOEMP codes until that contract lands.

🤖 Generated with Claude Code

[daphnehanse11]

Superseded by #204 — same commits on an in-repo branch, so CI can actually report (upstream Actions never received this fork's PR events). All review feedback from this thread is addressed at the new head.