diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index add8ba9..1120a1d 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -216,6 +216,8 @@ "./skills/data-science/microcalibrate-skill", "./skills/data-science/l0-skill", "./skills/data-science/policyengine-us-data-skill", + "./skills/data-science/policyengine-us-data-pipeline-skill", + "./skills/data-science/policyengine-variable-tracing-skill", "./skills/tools-and-apis/policyengine-simulation-mechanics-skill", "./skills/tools-and-apis/policyengine-microsimulation-skill", "./skills/domain-knowledge/policyengine-us-skill", @@ -405,6 +407,8 @@ "./skills/data-science/microcalibrate-skill", "./skills/data-science/l0-skill", "./skills/data-science/policyengine-us-data-skill", + "./skills/data-science/policyengine-us-data-pipeline-skill", + "./skills/data-science/policyengine-variable-tracing-skill", "./skills/documentation/policyengine-design-skill", "./skills/documentation/policyengine-standards-skill", "./skills/documentation/policyengine-writing-skill", diff --git a/changelog.d/policyengine-variable-tracing-skill.added.md b/changelog.d/policyengine-variable-tracing-skill.added.md new file mode 100644 index 0000000..e3eb3b2 --- /dev/null +++ b/changelog.d/policyengine-variable-tracing-skill.added.md @@ -0,0 +1 @@ +Added policyengine-variable-tracing-skill for data pipeline developers: find variable definitions in country model packages, read variable types (input/formula/adds), trace dependency chains, map H5 values to their data sources, and verify against legal citations. Works for both policyengine-us and policyengine-uk. diff --git a/changelog.d/us-data-pipeline-skill.added.md b/changelog.d/us-data-pipeline-skill.added.md new file mode 100644 index 0000000..6b1b05b --- /dev/null +++ b/changelog.d/us-data-pipeline-skill.added.md @@ -0,0 +1 @@ +Add policyengine-us-data-pipeline skill for tracing variables through the US data pipeline: imputation sources, calibration targets, run ID / HuggingFace artifact linkage, and cross-stage investigation patterns. diff --git a/skills/data-science/policyengine-us-data-pipeline-skill/SKILL.md b/skills/data-science/policyengine-us-data-pipeline-skill/SKILL.md new file mode 100644 index 0000000..acf2740 --- /dev/null +++ b/skills/data-science/policyengine-us-data-pipeline-skill/SKILL.md @@ -0,0 +1,333 @@ +--- +name: policyengine-us-data-pipeline +description: | + Investigates how a variable flows through the policyengine-us-data pipeline: + which step sets its microdata value, what administrative targets constrain its + weighted aggregate, and where in the code each implementation choice lives. + Use this skill when you need to trace a variable from raw survey through + imputation, calibration targets, and L0 weight optimization — or when you need + to locate and fetch diagnostics for a specific pipeline run from HuggingFace. + Triggers: "pipeline trace", "where does this value come from", "imputation source", + "calibration target", "policy_data.db", "target_config.yaml", "build_package", + "fit_weights", "matrix builder", "L0 calibration", "takeup in calibration", + "source impute", "QRF imputation", "run ID", "pipeline diagnostics", "HuggingFace artifacts", + "why is this aggregate wrong", "debug h5", "trace pipeline", "implementation choice", + "how is this calculated in the pipeline", "unified_calibration", "matrix_builder", + "calibration package", "calibration convergence", "domain variable", "stratum", + "aca_ptc calibration", "snap calibration", "employment income imputation" +--- + +# PolicyEngine US Data Pipeline — Investigator Guide + +This skill helps navigate the policyengine-us-data pipeline to locate where +implementation choices live and how pipeline stages relate to each other. +It is a starting point for investigation, not a reference manual. + +## CRITICAL: Verify Against Code + +The descriptions here reflect the pipeline as understood when written. The pipeline +evolves. Use this skill to identify *which file or function* is likely responsible for +a behavior, then **read that file** and confirm. Report what the code actually does — +not what this skill predicted. If something has changed, note the discrepancy. + +The skill's job is to cut the time spent finding the right place. The explanation of +how something actually works must come from reading the code directly. + +--- + +## 1. What the Pipeline Does — Conceptual Overview + +Understanding the pipeline's purpose prevents misattributing problems to the wrong stage. + +### Microdata and weights are separate concerns + +The pipeline produces two things: **microdata values** (what each record says) and +**household weights** (how much each record counts). These are set at different stages +and must be reasoned about separately. Calibration never edits individual record values — +it only changes weights. So if a variable is wrong at the record level, calibration +cannot fix it. + +### What imputation is + +The CPS survey is the primary microdata source. It covers demographics, income, and +program participation, but lacks detailed tax information and some financial variables. +**Imputation** fills these gaps by statistically matching CPS records to donor surveys +that do contain them (IRS PUF, ACS, SIPP, SCF). The matching is done via Quantile +Random Forest (QRF): a model trained on the donor survey that predicts the target +variable given a set of predictors available in both surveys. The predictor set is the +key implementation choice — it determines what variation the imputed values can capture. +If state is not a predictor, the imputation cannot be state-specific. + +### What calibration does + +Even with good microdata, survey weights don't match administrative totals — the CPS +doesn't perfectly represent every geography or income group. **Calibration** corrects this +by solving an optimization problem: find a new set of household weights such that, when +you multiply each record's variable values by its weight and sum, the result matches +published administrative benchmarks (e.g., USDA SNAP totals by state, IRS income totals +by AGI bracket). + +This is implemented via a **calibration matrix** (Stage 2) and an **L0-regularized +optimizer** (Stage 3). The matrix encodes, for each target, how much each cloned record +would contribute to that target at weight 1. The optimizer adjusts weights to minimize +squared relative errors across all targets simultaneously, with a sparsity penalty that +drives most weights to zero. The result is a sparse weight array — most records are +dropped; the retained ones have adjusted weights. + +A critical implication: calibration can only match targets that are representable as +a weighted sum over records. If the variable whose aggregate you care about is a formula +variable (computed by PolicyEngine at simulation time), the matrix builder must run +PolicyEngine to compute it before building the matrix row. This is why the matrix builder +(`calibration/unified_matrix_builder.py`) is one of the most important files to read +when investigating calibration behavior — it is where the connection between +administrative targets and PolicyEngine formulas is made concrete. + +### What the H5 files are + +The final output of the pipeline is a set of H5 files — one per geographic area. Each +stores only **input variables** (values from Stage 1 microdata) and their calibrated +weights. Formula variables are not stored; they are recomputed at simulation time by +the PolicyEngine engine. This means that when you load an H5 and run a simulation, the +simulation recalculates formula variables using the stored inputs and the country model's +current formula definitions. The H5 value and the simulated value for a formula variable +will only agree if the country model version matches the one used during pipeline build. + +--- + +## 2. Pipeline Shape + +The pipeline has 6 stages. The first builds the calibration database; the next 5 run +on Modal and are tracked together under a single **run ID**. + +``` +Stage 0 make database Builds policy_data.db — calibration targets from admin sources + Starting point: db/ + +Stage 1 build_datasets Sets all microdata values. Raw CPS → cloned + imputed dataset. + Starting point: modal_app/data_build.py, datasets/cps/, calibration/ + +Stage 2 build_package Runs PolicyEngine on cloned records to build the calibration matrix. + Starting point: modal_app/remote_calibration_runner.py, + calibration/unified_matrix_builder.py + +Stage 3 fit_weights L0-regularized optimization. Finds weights that match targets. + Starting point: calibration/unified_calibration.py, utils/l0.py + +Stage 4 publish_and_stage Applies weights, writes H5 files, validates, stages to HuggingFace. + Starting point: modal_app/local_area.py, + calibration/publish_local_area.py + +Stage 5 promote Moves staged H5s to production on HuggingFace. No new computation. + Starting point: modal_app/pipeline.py::promote_run() +``` + +**Separation of concerns:** Stages 1 and 2–3 are cleanly separated — microdata values are +frozen after Stage 1; Stages 2–3 only touch weights. A wrong aggregate can come from a +wrong microdata value (Stage 1 bug) or from miscalibration (Stage 2–3 issue). Knowing +which it is determines where to look. + +--- + +## 2. Run ID and HuggingFace Artifact Linkage + +**Run ID format:** `{version}_{sha[:8]}_{timestamp}` + +Every artifact from a run is associated with this ID. When reviewing an H5 file, find its +run ID to locate all associated diagnostics and base datasets on HuggingFace. + +**HF repo:** `policyengine/policyengine-us-data` (model repo) + +The run metadata (`meta.json`) lives in the pipeline volume at `runs/{run_id}/meta.json` +and records which steps completed, their timings, and the git SHA. + +**Diagnostics path on HF:** +``` +calibration/runs/{run_id}/diagnostics/ + calibration_log.csv ← per-epoch per-target errors (Stage 3) + unified_diagnostics.csv ← full optimization log (Stage 3) + unified_run_config.json ← hyperparameters used (Stage 3) + validation_results.csv ← per-area H5 validation (Stage 4) + national_validation.txt ← national validation output (Stage 4) +``` + +**Base dataset paths on HF** (per run, not under run ID — verify current paths in +`modal_app/pipeline.py::stage_base_datasets()`): +``` +calibration/source_imputed_*.h5 ← microdata input to calibration +calibration/policy_data.db ← targets used in this run +datasets/*.h5 ← earlier pipeline intermediates +``` + +To fetch: use `huggingface_hub.hf_hub_download()` with +`repo_id="policyengine/policyengine-us-data"`, `repo_type="model"`. + +--- + +## 3. Investigating a Variable — Navigation by Question + +### "Where does the microdata value come from?" (Stage 1) + +There are four possible origins. Check them in order — the first match is the answer: + +1. **Direct CPS column** — `datasets/cps/cps.py` +2. **PUF QRF imputation** (tax variables) — `calibration/puf_impute.py` +3. **CPS-only second-stage QRF** (non-tax variables) — `datasets/cps/extended_cps.py` +4. **ACS/SIPP/SCF source imputation** (rent, assets, tips) — `calibration/source_impute.py` + +For imputed variables, the most important implementation detail is the **predictor set** — +it determines what demographic and geographic variation the model captures. Read the +`*_PREDICTORS` constants in the relevant file. Note that some donor surveys lack state +identifiers, making those imputations state-blind at the microdata level. + +Takeup booleans (`takes_up_*`) are a fifth case — see below. + +### "Is there a calibration target?" (Stage 0 + Stage 2) + +Two places to check, in order: + +1. `calibration/target_config.yaml` — include/exclude rules that gate which DB targets + enter the calibration matrix. If a variable is excluded here, calibration ignores it + regardless of what is in the database. + +2. `storage/calibration/policy_data.db` — the `target_overview` view is the primary + query interface. Check `active`, `geo_level`, `domain_variable`, and `source`. + The `source` column points to the ETL script (`db/etl_*.py`) that populated the target; + read that script to understand what population the target represents. + +If a target exists but the aggregate still looks wrong, the question becomes whether it +is a Stage 1 (microdata) or Stage 3 (optimization convergence) problem. Check Stage 1 +first — if individual record values are systematically wrong, calibration cannot fix them +by weight adjustment alone. + +### "How does feature X interact across stages?" — Cross-Stage Tracing + +Some implementation choices touch multiple stages. When investigating these, the key is +to trace forward: where is it first set, what does each subsequent stage do with it? + +**Example: takeup** + +Takeup is not confined to one stage — it appears in three: + +- **Stage 1** (`utils/takeup.py`, called from `datasets/cps/cps.py`): boolean takeup + values are assigned to each cloned record using seeded Bernoulli draws at parametric + rates. These values are frozen into the source-imputed dataset. + +- **Stage 2** (`calibration/unified_matrix_builder.py`): when building the calibration + matrix, takeup-affected targets are handled specially. Read the matrix builder to + understand exactly how — the implementation here directly determines what the optimizer + is trying to match for any variable whose aggregate depends on takeup. + +- **Stage 4** (`calibration/publish_local_area.py`): when H5 files are assembled per area, + takeup may be re-evaluated. Read this file to confirm whether the takeup in the final H5 + reflects Stage 1 values, or whether it is recomputed. + +This forward trace is the pattern for any cross-cutting concern. Start at Stage 1 (where +is it first set?), check Stage 2 (does the matrix builder treat it specially?), then +Stage 4 (does the H5 builder change it?). + +**Example: stratum domain constraints** + +Domain constraints (e.g., `tax_unit_is_filer = 1` for IRS SOI targets, `snap > 0` for +SNAP-recipient targets) are defined in Stage 0 (`db/`) and consumed in Stage 2 +(`calibration/unified_matrix_builder.py`). They determine which records contribute to +which calibration targets. A target with a domain constraint calibrates only the subpopulation +satisfying that constraint — weights are adjusted to match admin data for that subgroup, +not the full population. This is why comparing a full-population aggregate against an +IRS-sourced target will always show a gap: the target was never meant to cover non-filers. + +To investigate a specific domain constraint: read how the matrix builder filters records +when it encounters a constraint variable, then trace back to `db/create_initial_strata.py` +to see how the stratum was constructed. + +--- + +## 4. Key Entry Points + +These are starting files for each major question. Read them — don't assume what they +contain matches what is described here. + +| Starting question | Where to start | +|---|---| +| How is variable X extracted from CPS? | `datasets/cps/cps.py` | +| What variables are QRF-imputed and how? | `calibration/puf_impute.py`, `datasets/cps/extended_cps.py`, `calibration/source_impute.py` | +| How are takeup booleans assigned? | `utils/takeup.py` | +| Which targets are active and at what granularity? | `calibration/target_config.yaml`, then `policy_data.db` | +| Where do target values come from? | `db/etl_*.py` — one file per data source; check `source` column in DB | +| How is the calibration matrix constructed? | `calibration/unified_matrix_builder.py` | +| How does the L0 optimizer work? | `calibration/unified_calibration.py`, `utils/l0.py` | +| How are H5 files assembled per area? | `calibration/publish_local_area.py` | +| How is an H5 validated post-build? | `calibration/validate_h5_quality.py`, `calibration/sanity_checks.py` | +| What artifacts does a run produce and where? | `modal_app/pipeline.py::run_pipeline()` | +| How are diagnostics uploaded to HF? | `modal_app/pipeline.py::upload_run_diagnostics()`, `stage_base_datasets()` | + +--- + +## 5. Investigation Best Practices + +### Establish version and run identity first + +Before interpreting any artifact or diagnostic, confirm which run produced it. The run ID +encodes the package version and git SHA — check that the SHA matches the code you are +reading. A diagnostic from one run interpreted against a different version of the code +will be misleading. When in doubt, fetch `meta.json` from the run to confirm the SHA, then +check out that commit before reading implementation files. + +### Locate the stage before reading implementation + +Ask: at which pipeline stage does this behavior occur? An imputed value is a Stage 1 +question; a calibrated aggregate is a Stage 2–3 question; a value in a final H5 is a +Stage 4 question. Reading the wrong stage's code wastes time and may produce a +plausible-sounding but wrong explanation. The stage taxonomy in Section 2 is a starting +point — verify it by reading the pipeline orchestrator (`modal_app/pipeline.py`) to +confirm what each step actually does. + +### Trace cross-stage relationships, not just the target stage + +Most interesting behaviors involve more than one stage. When you find the relevant +file, always ask: +- What does this stage receive as input, and where did that input come from? +- What does this stage produce, and what does the next stage do with it? + +For example, a variable's calibrated aggregate depends on: how it is imputed (Stage 1), +how the matrix row is constructed (Stage 2), whether it has an active target with the +right constraints (Stage 0 + Stage 2), and whether calibration converged for it (Stage 3). +Explaining the aggregate correctly requires tracing all four. + +### Use the variable-tracing skill for the formula side + +When a calibration target involves a variable that is computed by PolicyEngine (rather +than stored directly in the H5), you need to understand both the data pipeline side and +the formula side. Load the **policyengine-variable-tracing** skill to trace the variable's +formula, entity, definition period, and dependencies in the country model. This matters +because: +- The matrix builder runs PolicyEngine to compute the variable for each record; if the + formula has changed between pipeline runs, the matrix rows and the H5 simulation output + may disagree +- A variable calibrated against an admin total may have a formula that only activates + under certain conditions (`defined_for`, parameter switches, eligibility gates) — + these conditions determine which records actually contribute to the calibration target +- Input variables that feed a formula variable's calibration target have their own + imputation path in Stage 1; tracing the formula's dependencies reveals which imputed + inputs matter most for the calibrated aggregate + +### When something looks wrong, distinguish microdata from weights + +The first diagnostic split is always: is the individual record value wrong, or is the +weighted aggregate wrong despite correct individual values? + +- Read a sample of individual records from the source-imputed H5 to assess microdata quality +- Compare the weighted aggregate against the target in `policy_data.db` to assess calibration +- If the microdata looks correct but the aggregate is off, the issue is in Stage 2–3 +- If individual records look wrong, the issue is in Stage 1 — no amount of calibration can fix it + +--- + +## Related Skills + +- **policyengine-variable-tracing** — Variable definitions, formula types, and dependency + tracing in the country model (policyengine-us). Load alongside this skill when you need + to understand both the data pipeline value and how PolicyEngine computes the variable at + simulation time. +- **policyengine-us-data** — Cross-repo workflow for adding new variables to the pipeline + (`FINANCIAL_SUBSET`, version bumps, `IMPUTED_VARIABLES`). +- **microcalibrate** — L0, entropy balancing, and QRF imputation methods in general. diff --git a/skills/data-science/policyengine-variable-tracing-skill/SKILL.md b/skills/data-science/policyengine-variable-tracing-skill/SKILL.md new file mode 100644 index 0000000..3476341 --- /dev/null +++ b/skills/data-science/policyengine-variable-tracing-skill/SKILL.md @@ -0,0 +1,470 @@ +--- +name: policyengine-variable-tracing +description: | + Tracing PolicyEngine variable definitions, formulas, and data lineage in country model packages. + Use this skill when working in policyengine-us-data or policyengine-uk-data to understand where + variable values come from, how to find variable source code, read formulas, trace dependencies, + and map H5 dataset values back to their origins. + Triggers: "trace variable", "find variable", "where is this variable defined", "variable formula", + "why does this value", "data lineage", "H5 value", "variable source", "navigate policyengine-us", + "navigate policyengine-uk", "variable dependency", "how is this calculated", "input variable", + "formula variable", "adds variable", "entity level", "cross-entity" +--- + +# PolicyEngine Variable Tracing + +Practical guide for finding, reading, and tracing variable definitions in PolicyEngine country +model packages (policyengine-us, policyengine-uk) when working on data pipelines. + +**Before using this skill:** Load the relevant country domain skill (`policyengine-us` or +`policyengine-uk`) for country-specific variable semantics, entity structures, and program details. + +## Tracing workflow and narration + +When tracing a variable, **surface findings progressively** — do not accumulate all results +silently and report at the end. After each meaningful discovery, output a short inline summary +before continuing. This keeps the user informed and lets them redirect if needed. + +After finding the variable definition, immediately output a summary block like: + +``` +**Variable: `snap_reported`** +- Entity: SPMUnit +- Type: Input (no formula) +- Period: YEAR +- Legal reference: [URL if present] +- Next: trace back into the data pipeline +``` + +After identifying the pipeline source, output: + +``` +**Pipeline source: `snap_reported`** +- Origin: Direct from CPS survey — column `SPM_SNAPSUB` +- Calibration target: yes — `snap` targeted at state + national level (USDA FNS data) +- Next: check for known nuances or pitfalls +``` + +Use your judgment on granularity — one summary per logical step is enough. The goal is that +the user can follow the reasoning in real time without waiting for a final report. + +## 1. Finding a Variable Definition + +Every variable is a Python class in a `.py` file. The class name IS the variable name. + +### Search strategies (fastest first) + +**Grep for the class definition:** +```bash +# In the country model repo +grep -r "class snap" policyengine_us/variables/ --include="*.py" -l +``` + +**Infer from directory structure:** +``` +policyengine_{country}/variables/ +├── input/ # Input variables (no formula, data-driven) +├── gov/ # Government programs +│ ├── {agency}/ # Federal agencies (irs, usda, ssa, hhs, etc.) +│ └── states/ # State-specific (us) or regions (uk) +│ └── {code}/ # e.g., ca/, ny/, scotland/ +├── household/ # Household-level aggregations +│ └── income/ # Income variables by entity level +└── contrib/ # Contributed/experimental programs +``` + +Variable names often encode their location: +- `snap_*` -> `gov/usda/snap/` +- `ca_*` -> `gov/states/ca/` +- `income_tax*` -> `gov/irs/` or `gov/hmrc/` +- `is_*` or `has_*` -> boolean eligibility, search within the program directory +- `*_reported` -> typically input variables + +**Use the tax-benefit system at runtime:** +```python +from policyengine_us import CountryTaxBenefitSystem +system = CountryTaxBenefitSystem() +var = system.variables["snap_normal_allotment"] +print(var.entity.key) # Which entity (person, household, etc.) +print(var.value_type) # float, int, bool, Enum +print(var.definition_period) # YEAR, MONTH, ETERNITY +print(var.reference) # List of legal citation URLs +``` + +Note: there is no `source_file_path` attribute on variable objects. To find the source +file, use grep — the class name always matches the variable name. + +## 2. Variable Types + +Look at the class body to determine the type. This matters for data pipelines because it tells +you whether the variable's value comes from data or from computation. + +### Input variables (data-driven, no formula) + +```python +class employment_income_before_lsr(Variable): + value_type = float + entity = Person + label = "employment income before labor supply responses" + unit = USD + definition_period = YEAR + uprating = "calibration.gov.irs.soi.employment_income" +``` + +**How to identify:** No `formula` method, no `adds`/`subtracts`. Often has `uprating`. +**Data pipeline meaning:** This variable's value comes directly from the dataset (H5 file). +If missing from the dataset, it defaults to the `value_type` zero value (0.0, 0, False). + +### Aggregating variables (adds/subtracts) + +```python +class household_net_income(Variable): + value_type = float + entity = Household + unit = USD + definition_period = YEAR + adds = [ + "household_market_income", + "household_benefits", + "household_refundable_tax_credits", + ] + subtracts = ["household_tax_before_refundable_credits"] +``` + +**How to identify:** Has `adds` and/or `subtracts` lists instead of a `formula`. +**What it means:** The framework auto-generates `formula = sum(adds) - sum(subtracts)`. +**Tracing:** Follow each variable name in the lists to find sub-components. The `adds` field +can also be a string referencing a parameter path (e.g., `"gov.household.market_income_sources"`) +which resolves to a list of variable names defined in parameter YAML. + +### Calculated variables (explicit formula) + +```python +class snap_emergency_allotment(Variable): + value_type = float + entity = SPMUnit + definition_period = MONTH + unit = USD + + def formula(spm_unit, period, parameters): + p = parameters(period).gov.usda.snap.emergency_allotment + eligible = spm_unit("is_snap_eligible", period) + max_allotment = spm_unit("snap_max_allotment", period) + state = spm_unit.household("state_code", period) + ... +``` + +**How to identify:** Has a `def formula(entity, period, parameters)` method. +**Tracing:** Read the formula body. Every `entity("variable_name", period)` call is a +dependency. Every `parameters(period).path.to.param` is a parameter lookup. + +### Conditional variables (defined_for) + +```python +class ca_care(Variable): + entity = Household + defined_for = StateCode.CA + ... +``` + +**How to identify:** Has a `defined_for` attribute. +**What it means:** The formula only runs when the condition is met. For all other cases, +the variable returns its zero value. `defined_for` can be a state code enum or a string +referencing a boolean variable name. + +## 3. Entity Level + +Every variable belongs to exactly one entity. The entity determines which records in the +dataset the variable applies to and how it interacts with other variables. + +**Always check the entity:** Read `entity = ...` on the variable class. Do not assume — +the same program may have variables at different entity levels. + +### Discovering entities + +Each country model defines its entities in an `entities.py` file: +```python +# Check what entities exist +grep -n "class.*Entity" policyengine_{country}/entities.py +``` + +### Entity hierarchy and cross-references + +Entities form a containment hierarchy. The exact hierarchy varies by country — inspect +`entities.py` and the `containing_entities` attribute on each entity definition. + +**Upward reference (child to containing entity):** +```python +# A variable on SPMUnit accessing its containing Household +state = spm_unit.household("state_code", period) +``` + +**Downward reference (parent to contained members):** +```python +# A variable on SPMUnit accessing Person members +adult_count = spm_unit.sum(spm_unit.members("is_adult", period)) +``` + +**Key patterns in formulas:** +- `entity("var", period)` — same-entity variable reference +- `entity.parent_entity("var", period)` — look up to containing entity +- `entity.members("var", period)` — look down to member-level, returns array +- `entity.sum(...)`, `entity.max(...)`, `entity.any(...)` — aggregate members +- `entity.nb_persons()` — count of members + +**Why this matters for data pipelines:** When you see a variable value in an H5 file, you +must know its entity to understand which rows it maps to. A `household_weight` applies to +household-level rows, not person-level rows. Cross-entity aggregations (e.g., summing +person-level income to household level) happen at simulation time, not in the H5 file. + +## 4. Reading Formulas + +### The formula signature + +```python +def formula(entity, period, parameters): +``` + +- `entity` — the entity instance (e.g., `person`, `tax_unit`). Used to reference other variables. +- `period` — the time period being calculated. Passed through to dependencies. +- `parameters` — the parameter tree. Access with `parameters(period).path.to.param`. + +### Common formula patterns + +**Parameter lookups:** +```python +p = parameters(period).gov.usda.snap +rate = p.some_rate # Scalar parameter +threshold = p.threshold[filing_status] # Keyed by enum +``` + +Parameter paths match the YAML file hierarchy in `policyengine_{country}/parameters/`. +File `parameters/gov/usda/snap/some_rate.yaml` -> `parameters(period).gov.usda.snap.some_rate`. + +**Vectorized operations (all PolicyEngine formulas operate on arrays, never scalars):** +```python +where(condition, value_if_true, value_if_false) # Vectorized if/else +select([cond1, cond2, ...], [val1, val2, ...], default=0) # Multi-branch +max_(a, b) # Element-wise max (not Python's max()) +min_(a, b) # Element-wise min +not_(array) # Boolean negation +``` + +**The `add()` helper inside formulas:** +```python +# Sums multiple variables on the same entity +total = add(tax_unit, period, ["var_a", "var_b", "var_c"]) +``` + +### Dated formulas + +Some variables have multiple formula versions for different time periods: +```python +class some_variable(Variable): + def formula_2020(person, period, parameters): + ... # Applies for 2020 onwards + + def formula_2017(person, period, parameters): + ... # Applies for 2017-2019 +``` + +The engine picks the most recent formula that does not exceed the requested period. + +## 5. Dependency Tracing + +To trace why a variable has a specific value, walk the dependency tree. + +### Manual trace (reading code) + +1. Find the variable's `.py` file (Section 1) +2. Check the type (Section 2): + - **Input**: value comes from dataset. Stop — trace into the data pipeline. + - **Adds/subtracts**: follow each listed variable recursively. + - **Formula**: read the body, note every `entity("var_name", period)` call. +3. For each dependency, repeat from step 1. +4. The leaves of the tree are always input variables or parameters. + +### Runtime trace (using the simulation engine) + +```python +from policyengine_us import Microsimulation + +sim = Microsimulation() +# or with a specific dataset: +# sim = Microsimulation(dataset=path_to_h5) + +sim.trace = True +sim.calculate("snap_normal_allotment", 2024) + +# Print the dependency tree +sim.tracer.print_computation_log() +``` + +This shows every variable computed, its value, and its dependencies — the definitive +way to verify how a value was derived. + +### Verifying a formula against legal sources + +Every variable should have a `reference` field linking to the authoritative legal text. +When tracing, compare the formula logic against the cited law to confirm correctness: +```python +var = system.variables["snap_normal_allotment"] +print(var.reference) # URL to legal citation +``` + +The variable name itself often encodes what legal concept it represents — use it as a +starting point, but read the cited legal text directly to verify the formula captures the +correct conditions, thresholds, and scope. If you want to read the full implementation, +find the source file using grep (the class name always matches the variable name): +```bash +grep -r "class snap_normal_allotment" policyengine_us/variables/ --include="*.py" -l +``` + +## 6. Data Lineage: From H5 Value to Source + +When you see a variable's value in an H5 file produced by a data pipeline (e.g., +policyengine-us-data or policyengine-uk-data), the value came from one of these sources. + +### Source determination workflow + +1. **Is the variable an input variable in the country model?** + - Check: no `formula`, no `adds`/`subtracts` in its class definition. + - If yes: the value was placed into the H5 by the data pipeline. Continue to step 2. + - If no: it should NOT be in the H5 — the simulation engine recomputes it at runtime. + (Data pipelines drop formula variables before writing H5 files.) + +2. **Where did the data pipeline get the input value?** + + Search the pipeline codebase for the variable name as a starting point: + ```bash + grep -rn "\"variable_name\"" policyengine_{country}_data/ --include="*.py" + ``` + + Then trace through the following stages in order: + + **Stage A — Direct from survey microdata:** + Check the survey processing file (e.g., `cps.py`, `frs.py`) for a direct column mapping. + These are the most transparent: a raw survey column is renamed to a PolicyEngine variable. + ```bash + grep -n "variable_name" policyengine_{country}_data/datasets/cps/cps.py + ``` + + **Stage B — Imputed from a primary or secondary source:** + If not directly from the survey, it may be statistically imputed. Check: + - `IMPUTED_VARIABLES` list (controls what the extended dataset imputes) + - `FINANCIAL_SUBSET` list (controls what is extracted from tax records like PUF) + - Source imputation files (e.g., `source_impute.py`) for variables drawn from + alternative surveys (ACS, SIPP, SCF for US) + Imputed values are model predictions — their accuracy depends on the training data + and predictors used. Check which predictors the model uses; variables not in the + predictor set will produce state- or demographic-blind estimates. + + **Stage C — Stochastic takeup assignment:** + For boolean participation variables (`takes_up_*`), values are drawn from Bernoulli + distributions using program-specific administrative takeup rates. These are not + deterministic across pipeline runs. + + **Stage D — Check calibration targets:** + Even after the microdata value is set (stages A–C), the calibration step adjusts + household weights to match administrative totals. If the variable (or a formula + variable that depends on it) is a calibration target, the weighted aggregate will + be forced toward the target — but the individual microdata values are unchanged. + + Check `target_config.yaml` for the variable name: + ```bash + grep -n "variable_name" policyengine_{country}_data/policyengine_{country}_data/calibration/target_config.yaml + ``` + If it appears, note the `geo_level` (national, state, district) — this tells you + the granularity at which the weighted total is constrained. A variable calibrated + at state level will have correct state-level weighted sums, but individual record + values may still be noisy or biased. + + Also note the **source of the administrative target itself**. Check the ETL script + that populates the target (e.g., `db/etl_snap.py`, `db/etl_national_targets.py`): + - Targets sourced from **USDA, CMS, SSA** cover the full population regardless of + tax filing status. + - Targets sourced from **IRS SOI** (Statistics of Income) cover only tax filers. + Variables calibrated against IRS SOI targets — such as `employment_income`, + `adjusted_gross_income`, or `capital_gains` — will have correctly weighted totals + only when aggregated over the filer population. Comparing an unfiltered population + aggregate against an IRS SOI target is a common source of apparent discrepancy: the + target was never meant to represent everyone, only those who filed a return. + + For deeper inspection, the calibration targets are also stored in the policy database: + ```bash + # Check the SQLite target database + sqlite3 policyengine_{country}_data/db/policy_data.db \ + "SELECT * FROM targets WHERE variable = 'variable_name' LIMIT 10;" + ``` + +3. **Reconcile microdata vs. weighted totals:** + If the individual H5 values look reasonable but weighted aggregates are off (or + vice versa), the distinction between stages A–C (microdata) and stage D (weights) + is the key. Calibration fixes the aggregate but cannot correct a systematically + wrong imputation at the record level. + +See `examples/snap-trace.md` for a walkthrough of this workflow +applied to the `snap` variable. + +### Important: formula variables are NOT stored + +Data pipelines store only leaf input variables. Variables with formulas or `adds`/`subtracts` +are recomputed by the simulation engine. If you find a formula variable in an H5 file, +something unexpected happened — investigate the pipeline code. + +The pipeline typically has an explicit step that drops formula variables: +```python +# Example pattern — look for this in the data pipeline +formula_vars = [v for v in system.variables if system.variables[v].formulas] +# These get excluded from the H5 output +``` + +## 7. Quick Reference + +### Checklist: tracing a variable end-to-end + +- [ ] Find the variable class in the country model (`grep -r "class {name}" ...`) +- [ ] Note the **entity** (`entity = ...`) — which dataset rows does this apply to? +- [ ] Note the **type**: input (no formula) / adds / subtracts / formula / defined_for +- [ ] Note the **definition_period**: YEAR, MONTH, or ETERNITY +- [ ] If input: search the data pipeline for how the value enters the H5 +- [ ] If formula: read dependencies, trace each one recursively +- [ ] If adds/subtracts: follow each listed variable +- [ ] Check `reference` field for the authoritative legal source +- [ ] Use `sim.trace = True` to verify at runtime if needed + +### Common pitfalls + +- **Wrong entity assumption**: A variable named `household_*` might actually be defined on + `SPMUnit` or `TaxUnit`. Always check `entity = ...`. +- **Missing from H5**: If an input variable has no value in the H5, it defaults to zero. + This is silent — the simulation won't error, it just uses 0. +- **Stale country model version**: The data pipeline's pinned version of the country model + may not include a recently added variable. Check version constraints in `pyproject.toml`. +- **`adds` referencing a parameter path**: When `adds` is a string like + `"gov.household.market_income_sources"`, it resolves to a list defined in a parameter + YAML file, not a single variable. Read the parameter file to find the actual components. +- **For US variables IRS SOI calibration targets are filer-only**: Variables whose administrative targets + come from IRS Statistics of Income data (e.g., `employment_income`, `adjusted_gross_income`, + capital gains variables) are calibrated against totals that only cover tax filers. If you + aggregate these variables over the full population and compare against the target, the numbers + will not match — not because the data is wrong, but because the target was never meant to + represent non-filers. Always check the ETL source (`db/etl_*.py`) to know what population + a calibration target represents before drawing conclusions from aggregate comparisons. +- **Variable definitions may not perfectly match legal reality**: A variable's name, label, + or formula is an implementation choice — it reflects the developer's interpretation at + the time of writing. The legal code may have conditions, exclusions, or edge cases that + aren't captured, or the variable may aggregate concepts that the law treats separately. + Before assuming a variable represents exactly what its name suggests, read its `reference` + citation and verify the formula against the actual legal text. Pay attention to what + conditions the formula applies (via `defined_for`, parameter switches, or `where()` guards) + and what it silently omits. When the data pipeline uses a variable as a calibration target + or imputation output, these assumptions propagate into the dataset. + +## Related Skills + +- **policyengine-us** / **policyengine-uk** — Country-specific variable semantics and programs +- **policyengine-variable-patterns** — How to *write* new variables (this skill is for *reading* them) +- **policyengine-core** — Simulation engine internals (formula resolution, period handling) +- **policyengine-us-data** / **policyengine-uk-data** — Data pipeline specifics +- **policyengine-simulation-mechanics** — Running simulations and inspecting output \ No newline at end of file diff --git a/skills/data-science/policyengine-variable-tracing-skill/examples/snap-trace.md b/skills/data-science/policyengine-variable-tracing-skill/examples/snap-trace.md new file mode 100644 index 0000000..91fb585 --- /dev/null +++ b/skills/data-science/policyengine-variable-tracing-skill/examples/snap-trace.md @@ -0,0 +1,189 @@ +# Worked Example: Tracing `snap` end-to-end + +> **Note:** This example is specific to the **policyengine-us-data** pipeline (CPS ASEC, +> IRS PUF, policy_data.db, target_config.yaml). Other country pipelines (e.g., +> policyengine-uk-data using FRS/SPI) have different survey sources, imputation steps, +> and calibration infrastructure. Apply the same reasoning but look for the equivalent +> files in the relevant pipeline — load the country-specific domain skill for guidance. + +**Scenario:** SNAP benefit values in an H5 file look unexpectedly low. Trace the variable +end-to-end to understand what it represents and where its value comes from. + +--- + +**Step 1 — Find the variable definition** +```bash +grep -r "class snap\b" policyengine_us/variables/ --include="*.py" -l +# → policyengine_us/variables/gov/usda/snap/snap.py +``` + +**Step 2 — Check the variable type and entity** + +Opening `snap.py`: +```python +class snap(Variable): + value_type = float + entity = SPMUnit # defined at SPM unit level, not household or person + definition_period = MONTH + reference = "https://www.law.cornell.edu/uscode/text/7/2017#a" + + def formula(spm_unit, period, parameters): + ... +``` + +`snap` has a `formula` → it is a **calculated variable**. It will NOT be stored in the H5 +file. The simulation recomputes it at runtime from its dependencies. + +**Step 3 — Check the legal reference** + +Open `https://www.law.cornell.edu/uscode/text/7/2017#a` (7 U.S.C. § 2017). This defines +the SNAP allotment as the maximum allotment minus 30% of net income, subject to minimum +benefit rules. Verify the formula body reflects these conditions — specifically what +eligibility conditions, income deductions, and phase-out rules it applies or omits. +Note any simplifications or assumptions in the implementation that diverge from the +statutory text. + +**Step 4 — Read the formula dependencies** + +```python +def formula(spm_unit, period, parameters): + takes_up = spm_unit("takes_up_snap_if_eligible", period) + is_in_microsim = hasattr(spm_unit.simulation, "dataset") + if parameters(period).gov.usda.snap.abolish_snap: + return 0 + elif parameters(period).gov.simulation.reported_snap: + return spm_unit("snap_reported", period) # microsim: use reported value + else: + value = add(spm_unit, period, [ + "snap_normal_allotment", + "snap_emergency_allotment", + "dc_snap_temporary_local_benefit", + ]) + if is_in_microsim: + return value * takes_up # microsim: apply take-up factor + else: + return value # policy analysis: full benefit, no take-up +``` + +There are three critically different code paths here — which one runs depends on +parameter flags and runtime context, not on anything visible in the H5 file: + +1. **`reported_snap = True`** (microsimulation mode): returns `snap_reported` directly. + This is the path the data pipeline typically exercises. The formula variables + (`snap_normal_allotment`, etc.) are not used at all. + +2. **`reported_snap = False`, in microsim** (simulation has a `dataset` attribute): + calculates the full allotment from formula variables, then multiplies by + `takes_up_snap_if_eligible`. Take-up is applied automatically — you cannot + disable it from the formula side. + +3. **`reported_snap = False`, not in microsim** (policy analysis mode): + returns the full calculated allotment with no take-up adjustment. + +**Definition period mismatch:** `snap` is `MONTH`-period; `snap_reported` is `YEAR`-period. +When `reported_snap = True`, the engine divides the annual reported value by 12 to fit +the monthly context. If you compare H5 `snap_reported` values (annual) against simulated +`snap` values (monthly), you must account for this scaling. + +In microsimulation mode with `reported_snap = True`, trace `snap_reported` next. + +**Step 5 — Trace the input variable `snap_reported`** +```bash +grep -r "class snap_reported" policyengine_us/variables/ --include="*.py" -l +# → policyengine_us/variables/.../snap_reported.py +``` +`snap_reported` has no formula → **input variable**, defined on `SPMUnit`, `definition_period = YEAR`. +This IS stored in the H5 file. + +**Step 6 — Find it in the data pipeline** +```bash +grep -n "snap_reported" policyengine_us_data/datasets/cps/cps.py +# → snap_reported="SPM_SNAPSUB" +``` +`snap_reported` maps directly to the `SPM_SNAPSUB` column from the CPS ASEC — the +Census-reported SNAP subsidy amount at the SPM unit level. This is a direct survey +extraction, not an imputation. + +**Step 7 — Check calibration targets** + +First check `target_config.yaml` to see if `snap` is listed as a calibration target +and at what geographic granularity: +```bash +grep -n -A2 "variable: snap" policyengine_us_data/policyengine_us_data/calibration/target_config.yaml +# → - variable: snap +# → geo_level: state +# → - variable: snap +# → geo_level: national +``` + +`snap` is targeted at both state and national levels. This means the L0 calibration +optimization adjusts household weights so that `sum(snap * weight)` matches +administrative SNAP totals at each geographic level — but it does this by changing +weights only, not the underlying `snap_reported` microdata values. + +Next, inspect the actual target values the calibration is trying to hit. These are +stored in the policy database: +```bash +sqlite3 policyengine_us_data/policyengine_us_data/db/policy_data.db \ + "SELECT * FROM targets WHERE variable = 'snap' LIMIT 20;" +``` + +This shows the administrative benchmarks (e.g., USDA SNAP program totals by state) +that weights are being optimized against. If a weighted SNAP total deviates from +expectations, compare it against these target values directly: +- A large gap between the H5 weighted total and the target suggests calibration + didn't converge well for that geography (check calibration logs or loss values) +- A target that itself looks wrong points to a data issue in the ETL scripts + that populate the database (look in `db/etl_*.py`) +- A target that no longer exists (`# REMOVED` in target_config.yaml) means the + variable was dropped from calibration and aggregates are now unconstrained + +**Step 8 — Watch for non-obvious eligibility conditions** + +Even if the microdata and calibration look correct, SNAP values can be wrong because +of implicit conditions in the model that are easy to miss: + +- **Unit size excludes ineligible members.** `snap_unit_size` subtracts members who + are ineligible students or immigration-ineligible. Allotments, deductions, and FPG + thresholds all use this reduced unit size. If person-level eligibility flags are + wrong, the entire allotment calculation is off. + +- **`snap_normal_allotment` has `defined_for = "is_snap_eligible"`**, meaning it + returns zero (silently) for ineligible units. Ineligible units in aggregate + statistics count as $0 — they are not excluded from the denominator. + +- **Categorical eligibility via SSI or TANF.** An SPM unit can become eligible + because it receives any SSI or TANF, bypassing income and asset tests entirely. + If those benefit amounts are wrong in your dataset, eligibility will be wrong too. + +- **FPG uses fiscal-year timing**, not calendar-year. January–September calculations + use the previous fiscal year's poverty guidelines. If you compare against + calendar-year published FPG tables, values will appear off for those months. + +**Step 9 — Watch for user-applied filters that the formula doesn't require** + +A common source of aggregate discrepancies is filters that users or analysts apply +when computing totals from the H5 file, which do not reflect how SNAP is actually +defined. For example: + +- **Household vs. SPM unit**: SNAP is defined on `SPMUnit`, not `Household`. If + weights or aggregations are applied at the household level, unit-level values will + be double-counted or missed depending on how SPM units map to households. + +The general rule: when an aggregate from the H5 diverges from an administrative total, +check whether any filter applied to the data is more restrictive than the variable's +own `defined_for` and eligibility conditions. The formula is the specification — any +additional filter is an assumption that needs justification. + +--- + +**Conclusion:** +- Individual `snap_reported` values come from CPS `SPM_SNAPSUB` (direct survey, Stage A) +- Weighted SNAP aggregates are constrained to USDA administrative totals via calibration (Stage D) +- If weighted totals look wrong → compare against `policy_data.db` targets, check + calibration convergence, and inspect `target_config.yaml` for removed targets +- If individual record values look wrong → investigate CPS extraction (`SPM_SNAPSUB` + column), the `takes_up_snap_if_eligible` stochastic assignment in `cps.py`, or + whether the `reported_snap` parameter is set correctly for your simulation context +- If aggregates look wrong even with correct microdata → check whether any user-applied + filters are more restrictive than the variable's own eligibility conditions