Observed firm frame: real sponsor records calibrated to SUSB (#192) - #390
Open
vahid-ahmadi wants to merge 8 commits into
Open
Observed firm frame: real sponsor records calibrated to SUSB (#192)#390vahid-ahmadi wants to merge 8 commits into
vahid-ahmadi wants to merge 8 commits into
Conversation
Builds the firm population from observed Form 5500 sponsor records instead of generating one. No synthetic firm row is created. Form 5500 alone populates all 97 SUSB sector x canonical-band cells with at least ten records each, so the frame uses a single consistent unit (plan sponsor) and OSHA ITA stays an independent measurement reference. Unioning the two would silently mix a sponsor unit with an establishment unit: they agree on the canonical band for only 66.3% of the 43,001 EINs they share. The calibration took three attempts and the first two are recorded because the failures are informative: 1. post_stratify (firm-count margin only) matches firms exactly and overshoots employment by 2.06x. A single weight per cell treats a 307,086-participant enterprise as representative of twenty ordinary 500+ firms. 2. Finer stratification on SUSB's 22 detail size classes makes it worse (2.52x), because SUSB's top class is also unbounded. 3. calibrate_dual_margin lets weights vary within a cell via a maximum-entropy tilt, matching firm and employment margins at once. Employment ratio 0.973 across 92 of 97 cells. The five remaining cells fail closed rather than being forced, and they are a SUSB data-quality artifact: noise infusion distorts thin published cells enough to make them arithmetically impossible for their own size class. NAICS 11's 2,000-2,499 class reports 5 firms and 292 employees (flag H); its 5,000+ class reports 28 firms and 6,778 (flag J). Where the implied cell mean falls outside the band it belongs to, no reweighting can reach it. Post-stratification is retained alongside the working method because its employment_coverage diagnostic is what falsified the naive design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to the frame's first commit, resolving the two defects it
left open. Both choices are measured rather than assumed.
## Weight bound: 140, from the sensitivity curve
The unbounded exponential tilt put a weight of 3,866 on one NAICS 99
record in a 35-record cell, beside another at 1.3e-08. Bounded
calibration is the Deville-Sarndal (1992) remedy the project already
cites. The bound is not a guess — p99 is 76.5, so bounds above ~140
never bind and give identical margins, while tighter ones degrade the
firm margin sharply:
bound firm ratio emp ratio
50 0.7023 0.9766
100 0.9558 1.0151
130 0.9960 1.0226
140 0.9988 1.0237
250 0.9988 1.0237
1000 0.9990 1.0237
140 is the tightest bound that costs nothing on either margin. It is a
referee parameter, like the OSHA employment cap, not a default to
inherit silently.
Fixed along the way: the first bounded implementation clipped and then
rescaled to hit the firm total, which pushes weights straight back over
the bound — it left two records at 4,138 while the bound was nominally
250. The final clip no longer rescales, and the firm-margin residual
that leaves is reported instead.
## NAICS 55 employment is excluded, firms are kept
NAICS 55 (Management of Companies) is a holding-company sector: one
Form 5500 covers the whole enterprise's workforce, while SUSB
attributes those workers to the operating sectors and records only
head-office staff under 55. Counting them again double-counts, and the
effect is not marginal: 42,676,524 weighted employees against SUSB's
3,661,977 (11.7x), which is 24% of all weighted employment. It is
systematic across every band (1.0x, 2.6x, 4.2x, 6.0x, 12.4x), not a
thin-cell artifact.
These sponsors stay in the firm margin — SUSB does count 25,413 of
them — but carry employment_in_scope=False. The flag travels on the
frame rather than only in attrs, so a downstream consumer cannot sum
double-counted employment by accident.
Also corrected: the five impossible cells were earlier attributed to
SUSB noise infusion. That was wrong — 507 of 532 rows carry the
low-noise G flag. They are a documented SUSB scope rule: NAICS 55 as
above, and NAICS 11 because SUSB excludes crop and animal production.
## Result, and a held-out check
firms 6,453,598 / 6,461,497 = 0.9988
employment (in-scope) 1.0237
weights min 0.100 median 2.99 max 140.0
Per band, in-scope: 0.9993 / 1.0000 / 1.0000 / 1.0024 / 1.0440.
Validated against BDS 2022, which is never used in the calibration,
with SUSB-vs-BDS as a control to separate our error from source
disagreement. Firm counts: our/BDS 1.176, 1.270, 1.892 against
SUSB/BDS 1.177, 1.270, 1.888 — our own error is 0.999, 1.000, 1.002.
The whole gap is SUSB and BDS disagreeing, not the calibration.
No random draw, sampling or generation anywhere in the firm path:
every row traces to a real DOL filing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…ms (#192) The seam where the two sides meet. Both sides are observed microdata — people and job spells from SIPP/CPS, firms from Form 5500 — and nothing on either side is generated. The link between them is not observed and cannot be from public data. This builds a simulated roster: a reproducible, capacity-constrained allocation of real workers to real firm records inside pre-registered compatibility cells. assign_workers returns firm_instance_id, an artificial key, never an EIN or sponsor name, and sets observed_link=False on the result. The #282 claims boundary — no identified firm effects, coworker sorting, spillovers or AKM decomposition — is unchanged by having observed firm records. Matching uses only the registered cell keys (NAICS sector x canonical band). Earnings, tenure, geography and demographics are deliberately excluded: adding one would turn a capacity allocation into an unregistered imputation. Two design choices worth a referee's eye: - Weighted-to-discrete expansion. A roster needs discrete employers, so each calibrated record is replicated into integer instances. Residual weight is allocated by a seeded Bernoulli draw rather than rounding, so the expected count equals the weight; most calibrated weights sit between 1 and 4, where rounding would bias the firm margin systematically. Measured drift on the real frame: -0.005% across 6,453,306 instances. The consequence stated plainly in the docstring: replicates of one sponsor are not distinct real firms, so any statistic treating them as independent employers measures the replication, not the economy. Those are exactly the E12 statistics phase 2 does not certify. - Workers in a cell with no firm instance are returned unassigned rather than relocated to a neighbouring cell, which would fabricate cross-cell mobility the data does not support. Both steps require an explicit seed and are exactly reproducible; the tests pin reproducibility and seed-sensitivity in both directions. Verified end to end on the real calibrated frame: 789,640 sponsor records expand to 6,453,306 instances carrying 177,275,280 slots; assignment respects every capacity, reports unassigned cells by reason, and is identical under a repeated seed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…192) ADR 0003 froze IC1 as "one tidy table, written by workstream A, read by workstream B", but nothing in the repository enforced it. This adds the schema, its validator, and the adapter from the SIPP spell reader, so the two sides meet at a checked contract instead of a convention. firms/assignment.py consumes IC1; a mis-shaped frame now fails here rather than producing a plausible-looking roster. Three contract rules are enforced rather than documented: - The column set is exact, not a minimum. An hours column is rejected by name with its own message, because IC1's hours deferral is live (the registered consumer is SNAP ABAWD compliance, whose 80-hours-per-month test needs month-resolved hours) and a consumer finding an hours column would reasonably assume it was ratified. Adding one is the first scheduled amendment, by joint PR. - person_id must be an opaque string. The ASEC PERIDNUM is 22 digits, so int64 overflows and float64 rounds distinct persons together (#194 review). A numeric key silently merges people. - Self-employed and unpaid-family spells carry no firm-size band. That is a category error, not a missing value, and from_sipp_spells clears any band the raw SIPP slot carried through. calibration_universe applies ADR 0003's universe rule explicitly: private-sector spells only, because SUSB excludes government establishments, NAICS 92, crop/animal production and non-employers, and QWI in-scope jobs are non-federal. Calibrating against jobs the targets never counted would bias every margin by the excluded share. Dropped counts are recorded on attrs so the exclusion is visible in any artifact built from the result. from_sipp_spells performs a named, lossy promotion rather than a rename: SIPP 2014+ measures establishment size at the worker's location while IC2's canonical variable means enterprise size, so the adapter records the proxy on attrs["size_concept"]. That promotion is the most consequential approximation on the person side and it is not hidden behind a column name. Verified end to end against the real calibrated firm frame: 5,000 IC1 spells validate, the calibration universe drops federal and self-employed spells with counts recorded, and the survivors assign to observed firm instances at rate 1.0000 with observed_link=False. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Workstream A's remaining deliverable. Real ASEC persons receive IC1 job spells drawn from real SIPP donor records through a quantile regression forest, the same microimpute recipe ECPS uses for earnings histories. Completes the person side of the seam: imputed spells pass ic1.validate, feed ic1.calibration_universe, and assign to observed firm instances. The ADR 0003 firm-size x tenure bridge is an explicit argument with no default. A run that does not name its bridge cannot be refereed, and the two ratified bridges (SIPP 2008 primary, SIPP 2014+ proxy chain) give different joint structure at the same seed. ## A silent-failure bug found by the seed-sensitivity test The first implementation drew every target "at a quantile". That is wrong for a categorical target and wrong in a way that looks like it works: microimpute returns the same modal class at every quantile — measured, the band-code mean is 1.505 at q=0.05 and at q=0.95 alike. So every host person in a predictor cell received the *same* band, the cross-sectional variance collapsed to zero, and the seed had no effect whatsoever. A deterministic modal assignment was being presented as an imputation. It only surfaced because a test asserted that a *different* seed must change the draw. Reproducibility tests alone would have passed happily, since a constant is trivially reproducible. The draw is now split by target kind: - categorical (the firm-size band) samples each row from its predicted class distribution, via return_probs=True; - continuous (tenure, earnings share) draws by inverse-CDF on a fixed quantile grid. Verified: the band draw now varies across seeds and reproduces the donor's band distribution shape. ## Boundaries enforced, not just documented - class_of_worker is carried from the host, never imputed: it decides the calibration universe and whether a band is even defined, so a drawn value would let the imputation choose which jobs the SUSB/QWI targets count. - Self-employed and unpaid-family hosts never receive a band. - A predictor present only on the donor is rejected rather than dropped, because dropping it changes the joint structure the bridge exists to supply. - Imputing over a column the host already observes is rejected: overwriting measurement with a draw must be deliberate. - attrs record bridge, seed, predictors and band_is_imputed=True, so a consumer cannot mistake a drawn band for a measured one. microimpute added to dependencies; it was used but never declared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # tests/README-tiers.md # tests/tier_counts.json
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #386. Builds the firm population from observed Form 5500 sponsor records instead of generating one. No synthetic firm row is created; every row traces to a real DOL filing.
Why the frame can be observed rather than generated
Form 5500 alone populates all 97 SUSB sector x canonical-band cells with at least ten records each, so the frame uses a single consistent unit and OSHA ITA stays an independent measurement reference. Unioning the two would silently mix a plan-sponsor unit with an establishment unit — they agree on the canonical band for only 66.3% of the 43,001 EINs they share.
The calibration took three attempts; the failures are recorded
post_stratify— firm margin onlycalibrate_dual_margin— bounded max-entropy tiltPost-stratification matches firms exactly and overshoots employment by 2x: a single weight per cell treats a 307,086-participant enterprise as representative of twenty ordinary
500+firms. Finer stratification makes it worse, because SUSB's top class is also unbounded. The fix is to let weights vary within a cell so both margins are met at once.post_stratifyis retained because its diagnostic is what falsified the naive design.Two parameters, both measured
Weight bound 140. p99 is 76.5, so bounds above ~140 never bind and give identical margins; below it the firm margin degrades sharply (0.956 at 100, 0.702 at 50). 140 is the tightest bound that costs nothing. A referee parameter, like the OSHA employment cap.
NAICS 55 excluded from the employment margin, kept in the firm margin. Holding companies file one plan covering the whole enterprise while SUSB attributes those workers to the operating sectors — 42,676,524 weighted employees against SUSB's 3,661,977 (11.7x), 24% of all weighted employment, systematic across every band. Rows carry
employment_in_scope=Falseso a consumer cannot sum double-counted employment by accident.Result
Held-out validation against BDS 2022
BDS is never used in the calibration. SUSB-vs-BDS acts as a control separating our error from source disagreement:
Firm counts are faithful to three decimals; the entire gap vs BDS is the two sources disagreeing.
Corrections to earlier claims on #192
Gflag). They are a documented SUSB scope rule: NAICS 55 as above, NAICS 11 because SUSB excludes crop and animal production.Scope
Does not modify
gates.yaml, IC3 targets, floors, thresholds, partitions, or candidate fitting. The worker-to-firm assignment is unchanged and still constructed — no public source links workers to employers, and the phase-2 claims boundary on #282 stands.Relates to #192, #282, #386.