test(dml): replicate Chang (2020) §4.2.2 RCS simulation DGP (DML PR-B2) - #805
Conversation
Overall assessment✅ Looks good — no unmitigated P0/P1 findings. Executive summary
Methodology
The generator and assertions correctly establish Code QualityNo findings. The warning gate rejects unexpected learner, degeneracy, and inference warnings. PerformanceNo findings. The expensive Monte Carlo coverage test is marked MaintainabilityNo findings. The fixture helpers isolate generation, fitting, and the comparison contrast cleanly. Tech Debt
SecurityNo findings; no secrets, external inputs, or security-sensitive behavior were introduced. Documentation/TestsNo findings. Documentation consistently states the variance interpretation, learner substitution, overlap caveat, and replication scope. Recovery checks fail closed if SEs or confidence intervals become NaN. Python AST parsing passed for both changed test modules; full execution was unavailable because test dependencies are absent. |
Replaces the "paper's own §4 RCS DGPs are NOT replicated" caveat with the real thing, scoped by five dual-review rounds that executed the candidate designs: - §4.2.2 (kernel-estimation design, arXiv v3 p. 19) replicated verbatim in tests/test_methodology_dml_did.py: an exact-draw-order frame generator; a DGP-shape pin covering the distributions, all three innovation scales (0.1/0.2/0.3), stationary-sampling checks (T independent of (D, X)), the design's built-in confounded unadjusted contrast (-> theta0 + 1 = 4, its covariate-adjustment point), and both correct-specification facts (true propensity sigmoid(X - 1/2) by Bayes; true l20 = 0.25 X); seed-pinned recovery at both paper Ns (sieve + linear at N=500, linear at N=200, each with a 4*se gate, a seed-pin absolute bound, and a discriminating comparison requiring the adjusted estimate to beat the unadjusted contrast); and a slow MC coverage class (the file's third), seeds offset from the incumbent streams. Fits run under a strict warning gate that tolerates exactly the documented panel=False Assumption 2.3 UserWarning and the occasional propensity-trimming UserWarning (rare fitted-tail excursions on the unbounded-X design) and fails on anything else. "N(0, 0.1)" adopted as VARIANCE 0.1 with the ambiguity recorded. - §4.2.1 (ML-estimation design) extracted into the paper review but documented as NOT replicable with the bundled unpenalized learners: order-of-magnitude noise at (N=500, p=100) (SE ~4-10, EPV ~1.4-1.6 vs the solve_logit threshold 10, fitted out-of-fold clipping ~5-29% of rows - estimated propensities; true-propensity tail mass is only ~0.06%), and a fail-closed linear-outcome rank deficiency at (N=200, p=100) even under an oracle propensity (ridge/sieve fit there). TODO row narrowed accordingly (penalized propensity learner + p-robust outcome learner); the review gains the full §4.2 parameterizations, a ninth suspected-typo entry (the printed-but-unused beta_0, with the trend-cancellation note), and a Gaps entry for the paper's own regularity-condition quirks (Gaussian kernel vs Assumption 3.3; unbounded-index propensities vs a.s. strict overlap 3.2(a) in BOTH §4.2 designs). - REGISTRY: Case 2 checklist bullet updated + a standalone replication-scope Note (native learners stand in for kernel first stages; overlap quirk; §4.2.1 evidence). dr_scores test comment updated (parameterization now extracted; §4.2.2 replicated). CHANGELOG Added entry. Verified: default lane + both existing suites green on Rust and pure-Python backends; the new MC coverage class green on both (200 reps Rust / 22-rep reduced band pure-Python); the seed-pinned recovery values reproduce the review measurements exactly (att 2.7581/2.9054/2.5580, unadjusted contrast 4.008/3.978/3.617).
bdb4fd0 to
9563898
Compare
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall assessment✅ Looks good — no unmitigated P0/P1 findings. Executive summary
Methodology
The generator, propensity, outcome nuisance, and confounded contrast are correctly encoded at Code QualityNo findings. Unexpected learner, degeneracy, and inference warnings fail the fixtures at PerformanceNo findings. The 200-replication coverage check is marked MaintainabilityNo findings. DGP generation, warning validation, and unadjusted comparison are isolated helpers. Tech Debt
SecurityNo findings. No secrets, external inputs, or security-sensitive behavior were introduced. Documentation/TestsNo findings. Documentation consistently describes the variance interpretation, learner substitution, overlap caveat, and replication scope. Both changed Python files pass AST parsing; runtime execution was unavailable due missing dependencies. |
1 participant
Summary
DMLDiD(panel=False), retiring the long-standing "paper's own §4 RCS DGPs are NOT replicated" caveat. Scope was settled by five dual plan-review rounds that executed the candidate designs.tests/test_methodology_dml_did.py("Chang Sec. 4.2.2" section): an exact-draw-order frame generator; a DGP-shape pin (distributions, all three innovation scales 0.1/0.2/0.3, stationary-sampling checks, the design's built-in confounded unadjusted contrast → θ₀+1 = 4 — its covariate-adjustment point — and both correct-specification facts: true propensitysigmoid(X − ½)by Bayes, true ℓ₂₀ = 0.25·X); seed-pinned recovery at both paper Ns with a 4·se gate, seed-pin absolute bounds, and a discriminating comparison requiring the adjusted estimate to beat the unadjusted contrast; and a slow MC coverage class. Fits run under a strict warning gate (tolerates only the documented A2.3 and propensity-trimmingUserWarnings).solve_logitthreshold, fitted out-of-fold clipping ~5-29% across seeds; true-propensity tail mass only ~0.06%) and a fail-closed linear-outcome rank deficiency at (N=200, p=100) even under an oracle propensity. The TODO row is narrowed accordingly (penalized propensity learner + p-robust outcome learner).- **Note:**recording the replication scope (native learners stand in for the paper's kernel first stages; the fixture is correctly specified on both nuisances, so it adds paper-DGP faithfulness rather than plug-in-bias reproduction; the overlap quirk; the §4.2.1 evidence).test_methodology_dr_scores.pybanner comment synced; CHANGELOG### Addedentry.Methodology references (required if estimator / math changes)
- **Note:**; "N(0, 0.1)" is adopted as variance 0.1 with the printed-notation ambiguity recorded in the paper review.Validation
tests/test_methodology_dml_did.py(+3 classes: shape pin, seed-pinned recovery, slow MC coverage; incumbent RCS banner updated),tests/test_methodology_dr_scores.py(comment sync). Default lane + both suites green on the Rust and pure-Python backends; the new MC class green on both (200 reps Rust / reduced 22-rep band pure-Python).Security / privacy