Skip to content

Add skills for data work - #131

Open
juaristi22 wants to merge 2 commits into
PolicyEngine:mainfrom
juaristi22:maria/add-us-data-skills
Open

Add skills for data work#131
juaristi22 wants to merge 2 commits into
PolicyEngine:mainfrom
juaristi22:maria/add-us-data-skills

Conversation

@juaristi22

@juaristi22 juaristi22 commented Mar 27, 2026

Copy link
Copy Markdown

Summary

  • policyengine-variable-tracing skill — Guides tracing variable definitions, formulas, and data lineage across country model packages (policyengine-us, policyengine-uk). Helps map H5 dataset values back to source code, navigate dependency trees, and understand cross-entity aggregation. Includes a worked example tracing snap end-to-end.

  • policyengine-us-data-pipeline skill — Guides investigation of how variables flow through the policyengine-us-data pipeline: from raw CPS/PUF survey data through QRF imputation, calibration targets (target_config.yaml, policy_data.db), and L0 weight optimization. Covers locating pipeline diagnostics on HuggingFace and debugging aggregate mismatches.

Both skills are designed as investigative starting points — they point you to the right files and functions, then emphasize reading the actual code to confirm.

Test plan

  • Load plugin locally with claude --plugin-dir and verify both skills trigger on relevant prompts
  • Confirm marketplace.json entries are valid
  • Verify changelog fragments are present

🤖 Generated with Claude Code

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both skills are well aimed — "where does this variable's value come from" and "how does it flow through the pipeline" are the two questions that actually cost people time, and splitting them so the tracing skill owns the formula side while the pipeline skill owns the stage side is the right seam. The ## CRITICAL: Verify Against Code framing up front, and describing the pipeline skill as "a starting point for investigation, not a reference manual", set the right expectation for docs that will drift.

The conceptual section earning its place before the navigation section is a good call too — "microdata values and weights are separate concerns" is precisely the confusion that sends people to the wrong stage.

1. Duplicate section number in the pipeline skill.

policyengine-us-data-pipeline-skill/SKILL.md has two ## 2. headings:

## 2. Pipeline Shape
## 2. Run ID and HuggingFace Artifact Linkage
## 3. Investigating a Variable — Navigation by Question

so the run-ID section is unreferenceable and everything after it is off by one relative to the reader's mental count. Renumber 2→3 onward, or drop the numbers entirely — the headings are descriptive enough to stand without them, and unnumbered sections don't rot when someone inserts one.

2. Worth confirming the two skills' cross-references point at the right names.

The pipeline skill has a ### Use the variable-tracing skill for the formula side section and a ## Related Skills block, which is exactly the right instinct given they're landing together. Just worth a check that the names used there match the frontmatter name: values (policyengine-us-data-pipeline, and whatever the tracing skill declares) rather than the directory names — the directories carry a -skill suffix that the frontmatter doesn't.

3. Two skills, one PR.

Both are additive and both have changelog entries, so this is fine — but they're independently reviewable, and the tracing skill (470 lines + a 189-line example) is the one most likely to attract wording feedback. If you'd rather not have the pipeline skill held up by that, splitting is easy. Your call.

4. Minor

  • Both are registered in the two plugin lists in marketplace.json consistently — good, that's the easy thing to half-do.
  • The tracing skill ships examples/snap-trace.md; the pipeline skill has no example. A single worked trace ("here's snap from raw CPS to calibrated weight") would make the cross-stage section much more concrete, and you already have most of it in the SNAP example next door.
  • ### Verifying a formula against legal sources is a strong section to have included. Given it's the one step with a real correctness cost when skipped, it might deserve promotion nearer the top rather than sitting at the end of section 5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants