diff --git a/TODO.md b/TODO.md index b8152d2d..c33da9c9 100644 --- a/TODO.md +++ b/TODO.md @@ -37,7 +37,7 @@ Related tracking surfaces: | `EventStudyResults` inference-provenance fields: the container records no `vcov_type`/`cluster_name`/`n_clusters`/`df_convention`/Conley metadata, so a serialized surface cannot distinguish unit auto-clustering from explicit clustering, survey, Conley, or the one-way carve-out (3(a) R9 review). Adding them is a cross-producer M-092 schema amendment (six builders, to_dict/summary rendering, surface-suite pins) - follow the pre-cut amendment convention (optional fields appended last, ledger note same-diff) rather than bolting onto one producer | `diff_diff/results_base.py` | 3(a) R9 | Mid | Low | | Opt-in singleton-group pruning for TwoWayFixedEffects (static + event-study mode; reghdfe parity): singleton units/periods are currently RETAINED class-wide - the within-demeaned row is zero so points are unchanged, but N/G/residual-df count it and CR1/finite-sample SEs shift (~0.41019 -> 0.40962 measured; REGISTRY "Deviation from R" Note, R5 review) - reghdfe iteratively drops singletons by default while fixest retains them (diff-diff matches fixest); an opt-in knob needs iterative unit+period pruning with consistent cluster/survey/replicate/Conley array subsetting and a default-flip decision protocol (moves published SEs) | `diff_diff/twfe.py`, `diff_diff/estimators.py`, `diff_diff/utils.py` | 3(a) R5 | Mid | Low | | Cohort-timing validation input for the simultaneous-adoption event-study family (TWFE `event_study=True` + MultiPeriodDiD through 3.9): an optional `first_treat=`/`cohort=` column so simultaneous adoption becomes checkable under the contract-valid time-invariant `D_i` indicator - today the staggered-adoption advisory derives timing from within-unit 0->1 transitions, so it can only fire on off-contract time-varying `D_it` input, and with valid `D_i` adoption timing is not observable in the inputs at all (REGISTRY "staggered-adoption detection limit" Notes, both sections); design questions: validate-only vs steering error, and interplay with the M-011 removal | `diff_diff/twfe.py`, `diff_diff/estimators.py` | 3(a) R2 | Mid | Medium | -| DMLDiD replicate-weight survey designs (currently fail closed with `NotImplementedError` pointing here): the IF-reweighting route (`compute_replicate_if_variance` on the augmented scores) is BETTER justified for DMLDiD than for CS — Neyman orthogonality makes the score first-order insensitive to nuisance perturbation — but needs per-cell replicate SE plumbing, the aggregate `_se_from_psi` replicate branch already activates on kit keys, and its own rejection/df test matrix (QR-rank df, `n_valid - 1` tightening) | `diff_diff/dml_did.py` | DML survey PR | Mid | Low | +| CS parity with DMLDiD's replicate conventions: per-cell replicate SEs for `CallawaySantAnna` (CS leaves per-cell SEs on the weighted sqrt-sum under replicate designs; DMLDiD now computes them via `compute_replicate_if_variance`) AND flip the replace-style effective-df relays to min-cap (`min(df_survey, n_valid - 1)`) at all five sites: CS overall (`staggered.py` overall relay), the shared event-study/group aggregation sites (`staggered_aggregation.py` `df_survey_val = min(non_none_dfs)` x2), and the staggered-DDD engine twins (`_staggered_triple_diff_engine.py` overall + event-study) — the replace convention can RAISE df above the QR-rank design df (anti-conservative); latent, fires only when `n_valid < R` (all-zero/non-finite replicate column). DMLDiD's min-cap + the pinned inherited ES/group behavior are the templates (REGISTRY DMLDiD replicate Note) | `diff_diff/staggered.py`, `diff_diff/staggered_aggregation.py`, `diff_diff/_staggered_triple_diff_engine.py` | DML replicate PR | Mid | Low | | ImputationDiD/TwoStageDiD `aggregate()` recompute levels on bootstrapped fits fail closed ('simple' relays since the M-027 per-level convergence; M-021/M-022); ImputationDiD's per-target psi machinery makes seeded replay tractable (the panel-backed kit retains everything the psi precompute reads), TwoStageDiD's per-level GMM scores are function-locals and would need retention | `diff_diff/imputation_results.py`, `diff_diff/two_stage_results.py`, `diff_diff/aggregation.py` | 2(b) PR-3b | Mid | Low | | ContinuousDiD `aggregate('event_study')` on bootstrapped fits fails closed (M-025); a seeded post-fit bootstrap-ES replay is tractable - the multiplier draws are seeded (`np.random.default_rng(self.seed)`) - but needs the FULL per-cell `_bootstrap_info` (bread/ee_treated/Psi_eval/dPsi_*/beta_pred) the pruned kit deliberately drops, so shipping it means a kit-payload change with its own memory contract | `diff_diff/continuous_did_aggregation.py`, `diff_diff/continuous_did_results.py` | 2(b) PR-3c | Mid | Low | | practitioner `step_name="heterogeneity"` producer-side collisions: three OTHER estimators' advice steps reuse the key with non-heterogeneity labels (`:975` ContinuousDiD dose-response, `:1022` Triple placebo-group, `:1413` LPDiD WAS arrays), so DiagnosticReport's heterogeneity completion silently drops that unrelated advice from `next_steps` via `_filter_steps` - the same latent collision fixed for StackedDiD in M-024 (renamed to `sub_experiment_balance`). Renaming these changes those estimators' report output; audit + rename with per-estimator pins. | `diff_diff/practitioner.py` | 2(b) PR-2 review R9 | Quick | Low | diff --git a/changelog.d/20260830-dml-replicate-weights.md b/changelog.d/20260830-dml-replicate-weights.md new file mode 100644 index 00000000..9f01f6e2 --- /dev/null +++ b/changelog.d/20260830-dml-replicate-weights.md @@ -0,0 +1,12 @@ +### Added +- **DMLDiD replicate-weight survey designs**: `DMLDiD` now accepts + replicate-weight `SurveyDesign`s (BRR / Fay / JK1 / JKn / SDR) on both + lanes (panel and repeated cross sections), computing per-cell AND + aggregate variances by IF-reweighting the augmented cross-fitted scores + (`compute_replicate_if_variance`; nuisances are not re-estimated per + replicate). Inference uses `df = rank(replicate matrix) - 1` with + `min(df_survey, n_valid - 1)` capping; degenerate cells (zero or + non-finite replicate variance) fail closed to NaN inference. Replicate + + `cluster=` and replicate + `n_bootstrap > 0` are rejected with targeted + errors (previously all replicate designs failed closed with a blanket + `NotImplementedError`). diff --git a/diff_diff/dml_did.py b/diff_diff/dml_did.py index 1ebff5ef..aa0e15a2 100644 --- a/diff_diff/dml_did.py +++ b/diff_diff/dml_did.py @@ -1364,15 +1364,43 @@ def _compute_dml_gt( "control_inf": inf_full[control_idx], } - # Per-cell SE. PSU designs (declared OR bare cluster=) route through - # the CS per-cell CR1 helper (3-valued contract: float = use it, - # NaN = unidentified clustered variance and MUST propagate, - # None = malformed -> fall back). Non-PSU survey designs use the - # weighted sqrt-sum (CS mirror: the full design enters aggregate SEs - # only); the no-survey branch is verbatim. + # Per-cell SE. Replicate designs use IF-reweighting on the Hajek + # payload (compute_replicate_if_variance; the same psi the aggregate + # _se_from_psi call consumes) — a zero or non-finite replicate + # variance is degenerate and fails closed to NaN (stricter than the + # shared aggregate clamp; REGISTRY DMLDiD Note). PSU designs + # (declared OR bare cluster=) route through the CS per-cell CR1 + # helper (3-valued contract: float = use it, NaN = unidentified + # clustered variance and MUST propagate, None = malformed -> fall + # back). Non-PSU survey designs use the weighted sqrt-sum (CS + # mirror: the full design enters aggregate SEs only); the no-survey + # branch is verbatim. se: float with np.errstate(over="ignore", invalid="ignore"): - if ( + if resolved_survey_unit is not None and resolved_survey_unit.uses_replicate_variance: + from diff_diff.survey import compute_replicate_if_variance + + variance, n_valid_rep = compute_replicate_if_variance( + inf_full, resolved_survey_unit + ) + if not np.isfinite(variance) or variance <= 0.0: + se = float("nan") + else: + se = float(np.sqrt(variance)) + # Per-cell df: min(design df, n_valid - 1) — the dCDH + # _effective_df_survey rule, inlined. n_valid is computed + # over the WHOLE replicate columns, so it equals R for every + # cell in practice (defensive; REGISTRY Note). df_survey is + # a CELL-LOCAL binding — never mutate + # precomputed["df_survey"], which feeds the post-fit + # aggregation kit. + if df_survey is not None: + df_survey = min(int(df_survey), int(n_valid_rep) - 1) + else: + # Undefined replicate df (QR rank <= 1): df=0 sentinel + # -> NaN inference (CS sentinel parity). + df_survey = 0 + elif ( resolved_survey_unit is not None and getattr(resolved_survey_unit, "psu", None) is not None ): @@ -1386,9 +1414,10 @@ def _compute_dml_gt( else: se = float(np.sqrt(np.mean(psi_bar**2) / n_cell)) # NOTE: `se` is deliberately OUTSIDE the non_finite_score gate on the - # design-based branches — a NaN from the CR1 helper is the - # unidentified-variance signal and must flow to safe_inference as a - # NaN-consistent inference tuple on a RETAINED cell. + # design-based branches — a NaN from the CR1 helper (or a degenerate + # replicate variance) is the unidentified-variance signal and must + # flow to safe_inference as a NaN-consistent inference tuple on a + # RETAINED cell. if resolved_survey_unit is None and not np.isfinite(se): diagnostics["skip_reason"] = "non_finite_score" return ( @@ -1760,12 +1789,28 @@ def _compute_dml_rcs_gt( } # Per-cell SE (same dispatch as the panel cell; see the comment - # there): PSU designs -> CS per-cell CR1 helper (NaN propagates as + # there): replicate designs -> IF-reweighting with the degenerate + # fail-closed guard and the cell-local min(df, n_valid - 1) rule; + # PSU designs -> CS per-cell CR1 helper (NaN propagates as # the deliberate unidentified-variance signal on a RETAINED cell); # non-PSU survey -> weighted sqrt-sum; no-survey verbatim. se: float with np.errstate(over="ignore", invalid="ignore"): - if ( + if resolved_survey_unit is not None and resolved_survey_unit.uses_replicate_variance: + from diff_diff.survey import compute_replicate_if_variance + + variance, n_valid_rep = compute_replicate_if_variance( + inf_full, resolved_survey_unit + ) + if not np.isfinite(variance) or variance <= 0.0: + se = float("nan") + else: + se = float(np.sqrt(variance)) + if df_survey is not None: + df_survey = min(int(df_survey), int(n_valid_rep) - 1) + else: + df_survey = 0 + elif ( resolved_survey_unit is not None and getattr(resolved_survey_unit, "psu", None) is not None ): @@ -1831,19 +1876,24 @@ def fit( Parameters ---------- survey_design : SurveyDesign, optional - Complex survey design (pweight-only; full-design TSL — - weights/strata/PSU/FPC). Declared designs weight the moment - kernels (Hajek p-hat/lambda-hat/theta), pass ``sample_weight`` - into the nuisance learners (user learner objects must accept - ``sample_weight`` by keyword — a learner without it is rejected - up front), switch cross-fitting to PSU-cohesive folds when the - PSU is strictly coarser than the sampling unit, and route the - per-cell and aggregate variances through the design-based - kernels with ``df = n_PSU - n_strata`` t-inference. Survey - support is a documented library extension of Chang (2020), - which assumes i.i.d. sampling — Theorem 2's coverage claim - does not carry over (REGISTRY DMLDiD Notes). Replicate-weight - designs are not supported yet (fail closed; TODO.md). + Complex survey design (pweight-only). Declared designs weight + the moment kernels (Hajek p-hat/lambda-hat/theta) and pass + ``sample_weight`` into the nuisance learners (user learner + objects must accept ``sample_weight`` by keyword — a learner + without it is rejected up front). Two variance lanes: + full-design TSL (weights/strata/PSU/FPC) switches cross-fitting + to PSU-cohesive folds when the PSU is strictly coarser than the + sampling unit and routes the per-cell and aggregate variances + through the design-based kernels with ``df = n_PSU - n_strata`` + t-inference; replicate-weight designs (BRR / Fay / JK1 / JKn / + SDR) compute per-cell AND aggregate variances by IF-reweighting + the cross-fitted scores with ``df = rank(replicate matrix) - 1`` + t-inference (nuisances are not re-estimated per replicate; + REGISTRY DMLDiD Note). Replicate designs reject ``cluster=`` + and ``n_bootstrap > 0`` combinations. Survey support is a + documented library extension of Chang (2020), which assumes + i.i.d. sampling — Theorem 2's coverage claim does not carry + over (REGISTRY DMLDiD Notes). """ df, covariates = self._validate_and_prepare( data, outcome, unit, time, first_treat, covariates @@ -1866,13 +1916,18 @@ def fit( survey_metadata, ) = _resolve_survey_for_fit(survey_design, data, "analytical") - # Replicate designs fail closed FIRST (None-guarded): no replicate - # variance path exists for the cross-fitted scores yet. - if resolved_survey is not None and resolved_survey.uses_replicate_variance: + # Replicate + bootstrap rejected FIRST (before any fit work) — + # replicate variance is an analytical alternative, not compatible + # with bootstrap (CS parity, staggered.py). + if ( + self.n_bootstrap > 0 + and resolved_survey is not None + and resolved_survey.uses_replicate_variance + ): raise NotImplementedError( - "DMLDiD does not support replicate-weight survey designs yet " - "(tracked in TODO.md); use a full-design SurveyDesign " - "(weights/strata/psu/fpc) instead." + "DMLDiD bootstrap (n_bootstrap > 0) is not supported " + "with replicate-weight survey designs. Replicate weights provide " + "analytical variance; use n_bootstrap=0 instead." ) # Raw (pre-normalization) per-obs design weights, for metadata @@ -1902,6 +1957,25 @@ def fit( "drop or impute them before fitting" ) cluster_ids_for_check = _cluster_col.to_numpy() + # Reject replicate-weight + cluster= AFTER the column checks (CS + # ordering: a bogus cluster name raises ValueError first). + # Replicate IF variance is computed by replicate reweighting and + # ignores PSU/cluster entirely (replicate_weights are mutually + # exclusive with strata/psu/fpc) — honoring cluster= would + # silently have no effect on the variance, and the inject-as-PSU + # paths below would violate that mutual exclusion. + if resolved_survey is not None and resolved_survey.uses_replicate_variance: + raise NotImplementedError( + f"DMLDiD(cluster={self.cluster!r}) is not " + "supported with replicate-weight survey designs. " + "Replicate-weight variance is computed by replicate " + "reweighting (BRR / Fay / JK1 / JKn / SDR) and ignores " + "PSU/cluster entirely — setting cluster= would silently " + "have no effect on the variance estimate. Either omit " + "cluster= (the replicate weights encode the design " + "structure implicitly) or use a non-replicate survey " + "design (with explicit strata/psu/fpc)." + ) if resolved_survey is None: # Bare cluster=: synthesize a PSU-only design. survey_metadata # stays None DELIBERATELY (it is the declared-survey marker: @@ -2188,15 +2262,31 @@ def fit( # Overall ATT (simple aggregation over post-treatment finite cells). # overall_effective_df is non-None only when replicate variance - # dropped replicates — unreachable while replicate designs are - # rejected, but kept for structural parity with CS. + # dropped replicates (n_valid < R). MIN-CAP, not CS's replace: the + # QR-rank design df stays the ceiling (CS's replace convention can + # RAISE df above the design df — anti-conservative; deliberate + # documented divergence, REGISTRY DMLDiD Note + CS-parity TODO row). overall_att, overall_se, overall_effective_df = self._aggregate_simple( group_time_effects, influence_func_info, df, unit, precomputed ) - if overall_effective_df is not None: - df_survey = overall_effective_df + if overall_effective_df is not None and df_survey is not None: + df_survey = min(int(df_survey), int(overall_effective_df)) + # Propagate to survey_metadata for display consistency (CS + # parity) — the capped value, never the sentinel below. + if survey_metadata is not None: + survey_metadata.df_survey = df_survey + # Replicate design with undefined df (QR rank <= 1): df=0 sentinel + # -> NaN inference, applied to the LOCAL df only (survey_metadata + # keeps None). + df_overall = df_survey + if ( + df_survey is None + and resolved_survey is not None + and resolved_survey.uses_replicate_variance + ): + df_overall = 0 overall_t_stat, overall_p_value, overall_conf_int = safe_inference( - overall_att, overall_se, alpha=self.alpha, df=df_survey + overall_att, overall_se, alpha=self.alpha, df=df_overall ) # Optional multiplier bootstrap (keyword form; aggregate=None is the diff --git a/diff_diff/dml_did_results.py b/diff_diff/dml_did_results.py index 3ed80785..cce685a0 100644 --- a/diff_diff/dml_did_results.py +++ b/diff_diff/dml_did_results.py @@ -15,9 +15,12 @@ and DMLDiD's augmented-score SE ``sqrt(mean(psi_bar**2)/n)`` is exactly that on NO-DESIGN fits — per UNIT on panel fits, per OBSERVATION on repeated-cross-section fits (rows are the sampling units there). Under a -``survey_design=``/``cluster=`` the per-cell SE is the design-based CR1 / -weighted-IF variance instead (the CS clustered-``hc1`` convention: -``SurveyDesign(psu=...)`` routed through the shared stratified-PSU meat). +``survey_design=``/``cluster=`` the per-cell SE is design-based instead: +full-design TSL fits use the CR1 / weighted-IF variance (the CS +clustered-``hc1`` convention: ``SurveyDesign(psu=...)`` routed through the +shared stratified-PSU meat), while replicate-weight fits use IF-reweighting +via ``compute_replicate_if_variance`` on the same per-cell payload +(``df = rank(replicate matrix) - 1``; REGISTRY DMLDiD Note). """ from dataclasses import dataclass, field diff --git a/diff_diff/guides/llms-autonomous.txt b/diff_diff/guides/llms-autonomous.txt index 863870f7..07ac6624 100644 --- a/diff_diff/guides/llms-autonomous.txt +++ b/diff_diff/guides/llms-autonomous.txt @@ -349,7 +349,7 @@ supported / out of scope; `warn` supported but with documented caveats; | `StackedDiD` | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | | `WooldridgeDiD` (ETWFE) | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | | `LWDiD` | ✓ | ✓ | ✗ | ✗ | ✓ | warn | partial | ✗ | ✓ | -| `DMLDiD` | ✓ | ✓ | ✗ | ✗ | partial | ✓ (REQUIRED) | ✗ | ✗ | ✓ (pweight full-design TSL + coarser cluster=; PSU-cohesive folds; replicate designs rejected; panel=False = declared RCS) | +| `DMLDiD` | ✓ | ✓ | ✗ | ✗ | partial | ✓ (REQUIRED) | ✗ | ✗ | ✓ (pweight full-design TSL + replicate weights + coarser cluster=; PSU-cohesive folds; panel=False = declared RCS) | | `EfficientDiD` | ✓ | ✓ | ✗ | ✗ | partial | ✓ | ✗ | ✗ | ✓ | | `SyntheticDiD` | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | partial | | `TROP` | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | partial | @@ -766,10 +766,11 @@ Explicit RCS support in this library: - `DMLDiD(panel=False)` - Chang (2020) Case 2 declared-RCS mode (row-unique unit IDs; level-outcome orthogonal scores with the lambda-corrected variance; REGISTRY.md §DMLDiD). Survey/cluster - support on both designs: `survey_design=` (pweight full-design TSL; - the weighted-lambda plug-in is a library extension - Theorem 2's - coverage claim does not carry over) and coarser-than-unit `cluster=` - (variance/folds only); replicate designs rejected; + support on both designs: `survey_design=` (pweight full-design TSL + or replicate weights — per-cell AND aggregate IF-reweighting, df = + rank-1; replicate + cluster=/bootstrap rejected; the weighted-lambda + plug-in is a library extension - Theorem 2's coverage claim does not + carry over) and coarser-than-unit `cluster=` (variance/folds only); aggregate('total') fails closed on RCS and declared-survey fits. - `TripleDifference` - DDD cross-sectional use cases are documented in `docs/choosing_estimator.rst`; the two-period DDD estimator does diff --git a/diff_diff/guides/llms-full.txt b/diff_diff/guides/llms-full.txt index 0d658bda..ac7bd3a7 100644 --- a/diff_diff/guides/llms-full.txt +++ b/diff_diff/guides/llms-full.txt @@ -1423,7 +1423,7 @@ Key contracts: - HonestDiD / PreTrendsPower consume the aggregate('event_study') container (admitted source); the native `compute_honest_did(results)` route raises with the container instruction. Varying-base containers warn; use base_period='universal' for clean Rambachan-Roth interpretation. - `SieveLearner(k_max=None, criterion="bic")` is the exported configurable learner (adaptive polynomial degree by IC); any sklearn-style estimator object also plugs in (seed stochastic learners yourself — the library seed pins folds, not learner internals). - Per-cell complete cases (one consolidated unbalanced-input warning); degenerate cells (fewer members than folds, singleton treated stratum, fail-closed learner) become NaN cells with machine-readable skip_reason and a consolidated warning; surviving cells still aggregate. -- Survey/cluster support on BOTH designs. `survey_design=` (pweight-only, full-design TSL — weights/strata/PSU/FPC): weighted moment kernels (Hajek p-hat/lambda-hat/theta), sample_weight into the nuisance learners (user learner objects must accept sample_weight by keyword — rejected up front otherwise), PSU-cohesive cross-fitting folds when the PSU is strictly coarser than the sampling unit, design-based per-cell/aggregate variance with df = n_PSU - n_strata t-inference. A documented library EXTENSION of Chang's i.i.d. theory (Theorem 2's coverage claim does not carry over on the weighted-lambda RCS lane — REGISTRY DMLDiD Notes). Bare cluster= keeps the kernels unweighted (variance/folds/df only). Replicate-weight designs are rejected (tracked in TODO.md); aggregate('total') fails closed on declared-survey fits. panel=False requires row-unique unit IDs and assumes stationary cross-sectional sampling (Assumption 2.3, warned, not data-checkable). Reproducibility: set seed (fold draws move point estimates; PSU folds consume the RNG differently than stratified folds). +- Survey/cluster support on BOTH designs. `survey_design=` (pweight-only): weighted moment kernels (Hajek p-hat/lambda-hat/theta), sample_weight into the nuisance learners (user learner objects must accept sample_weight by keyword — rejected up front otherwise); two variance lanes. Full-design TSL (weights/strata/PSU/FPC): PSU-cohesive cross-fitting folds when the PSU is strictly coarser than the sampling unit, design-based per-cell/aggregate variance with df = n_PSU - n_strata t-inference. Replicate weights (BRR/Fay/JK1/JKn/SDR): per-cell AND aggregate IF-reweighting variance on the augmented cross-fitted scores with df = rank(replicate matrix) - 1 t-inference (nuisances not re-estimated per replicate; replicate + cluster= and replicate + bootstrap rejected). A documented library EXTENSION of Chang's i.i.d. theory (Theorem 2's coverage claim does not carry over on the weighted-lambda RCS lane — REGISTRY DMLDiD Notes). Bare cluster= keeps the kernels unweighted (variance/folds/df only). aggregate('total') fails closed on declared-survey fits. panel=False requires row-unique unit IDs and assumes stationary cross-sectional sampling (Assumption 2.3, warned, not data-checkable). Reproducibility: set seed (fold draws move point estimates; PSU folds consume the RNG differently than stratified folds). ### TROP @@ -2692,7 +2692,7 @@ sd_female, data_female = sd.subpopulation(data, mask=lambda df: df['sex'] == 'F' **Key features:** - Taylor Series Linearization (TSL) variance with strata + PSU + FPC -- Replicate weight variance: BRR, Fay's BRR, JK1, JKn, SDR (13 of 24 estimators, including dCDH) +- Replicate weight variance: BRR, Fay's BRR, JK1, JKn, SDR (14 of 24 estimators, including dCDH and DMLDiD) - Survey-aware bootstrap: multiplier at PSU (Hall-Mammen wild; dCDH, staggered) or Rao-Wu rescaled (SunAbraham, SyntheticDiD, TROP). SyntheticDiD bootstrap composes Rao-Wu rescaled per-draw weights with the weighted Frank-Wolfe variant of `_sc_weight_fw` (PR #355): each draw solves `min ||A·diag(rw)·ω - b||² + ζ²·Σ rw_i ω_i²` and composes `ω_eff = rw·ω/Σ(rw·ω)` for the SDID estimator. Pweight-only fits use constant `rw = w_control`; full designs use Rao-Wu. SDID's placebo (stratified permutation + weighted FW) and jackknife (PSU-level LOO with stratum aggregation, Rust & Rao 1996) paths also support pweight-only and full strata/PSU/FPC designs - DEFF diagnostics, subpopulation analysis, weight trimming (`trim_weights`) - Repeated cross-sections: `CallawaySantAnna(panel=False)` diff --git a/diff_diff/guides/llms-practitioner.txt b/diff_diff/guides/llms-practitioner.txt index f715db9d..3ab24ef6 100644 --- a/diff_diff/guides/llms-practitioner.txt +++ b/diff_diff/guides/llms-practitioner.txt @@ -231,8 +231,8 @@ Is treatment adoption staggered (multiple cohorts, different timing)? | | repeated cross sections (Case 2, | | lambda-corrected variance); | | survey_design=/cluster= supported on -| | both designs (pweight TSL; replicate -| | designs rejected) +| | both designs (pweight TSL + replicate +| | weights via IF-reweighting) | \-- WooldridgeDiD (ETWFE) -- nonlinear outcomes (logit/Poisson) or saturated OLS | |-- NO, simple 2x2 design: diff --git a/diff_diff/guides/llms.txt b/diff_diff/guides/llms.txt index c621344b..f9220d90 100644 --- a/diff_diff/guides/llms.txt +++ b/diff_diff/guides/llms.txt @@ -81,7 +81,7 @@ The site is organized into 5 sections, each with a landing page: - [ChangesInChanges](https://diff-diff.readthedocs.io/en/stable/api/changes_in_changes.html): Athey & Imbens (2006) nonlinear/distributional DiD for the 2x2 design: recovers the treated group's full counterfactual outcome distribution and quantile treatment effects (ATT + QTE grid) via the CDF transformation `F_10(F_00^{-1}(F_01(y)))`; invariant to monotone outcome transformations (unconditional fits; the covariate QR branch is not); bootstrap inference (panel or repeated cross-section resampling); point parity with R `qte::CiC()`, including its covariate branch (`covariates=` -> per-cell linear quantile regression, Melly-Santangelo-style conditional CiC). Continuous outcomes, numeric covariates. Alias `CiC`. - [QDiD](https://diff-diff.readthedocs.io/en/stable/api/changes_in_changes.html): **Deprecated 3.9, removed 4.0 - use `ChangesInChanges(method="qdid")`.** Athey & Imbens (2006) quantile DiD comparison estimator (additive quantile-by-quantile DiD, matching R `qte::QDiD()` including its covariate branch via `covariates=`); same bootstrap machinery as ChangesInChanges. The paper recommends CiC over QDiD (scale-dependent model with testable restrictions; a non-monotonicity warning fires when violated - unconditional fits only, the covariate-path counterfactual quantile curve is monotone by construction). - [LWDiD](https://diff-diff.readthedocs.io/en/stable/api/lwdid.html): Lee & Wooldridge (2025, 2026) rolling-transformation DiD — unit-specific demean/detrend converts panel to cross-section, supports staggered adoption with flexible control groups. Signature: `LWDiD(rolling='demean', estimation_method='reg', vcov_type='hc1', cluster=None, control_group='not_yet_treated', alpha=0.05, n_bootstrap=0, seed=None, pscore_trim=0.01, n_neighbors=1, caliper=None, with_replacement=True, n_jobs=1).fit(data, outcome, unit, time, treatment, first_treat=None, covariates=None)`. `estimation_method` values: `reg` (papers' RA), `ipw`, `dr` (papers' IPWRA, doubly robust), `psm`; `vcov_type` values: `classical`/`hc1`/`hc2`/`hc3` for `reg`; `ipw`/`dr` accept `hc1` only (influence-function variance); `psm` accepts `hc1` as configuration only - PSM inference is unavailable (NaN) pending an Abadie-Imbens matching variance; cluster-robust inference via the constructor's `cluster=` column (hc1/CR1 only, not a `vcov_type` value; rejected for `psm`). Per-period effects: post-fit `results.aggregate('event_study')`. -- [DMLDiD](https://diff-diff.readthedocs.io/en/stable/api/dml_did.html): Chang (2020) double/debiased machine learning DiD — staggered ATT(g,t) with cross-fitted ML nuisances (DML2) and Neyman-orthogonal scores; covariates REQUIRED (conditional parallel trends). Signature: `DMLDiD(propensity_learner='logit', outcome_learner='linear', n_folds=5, control_group='never_treated', anticipation=0, alpha=0.05, n_bootstrap=0, bootstrap_weights=None, seed=None, base_period='varying', cband=True, pscore_trim=0.01, panel=True, cluster=None).fit(data, outcome, unit, time, first_treat, covariates, survey_design=None)`. `panel=False` = declared repeated cross sections (Chang Case 2: level outcomes, row-unique unit IDs, lambda-corrected variance). Survey/cluster support on BOTH designs: `survey_design=` (pweight full-design TSL — weighted moments, PSU-cohesive folds, design-based variance with t-inference; a library extension of Chang's i.i.d. theory) and coarser-than-unit `cluster=` (variance/folds only, kernels stay unweighted); replicate-weight designs are not supported yet. Learners: string names (`linear`/`ridge`/`sieve` regressors, `logit` classifier) or any object with fit/predict(_proba) (sklearn-compatible); `SieveLearner(k_max, criterion)` is exported for adaptive polynomial nuisances. Aggregation is POST-FIT: `results.aggregate('event_study'/'group'/'simple')`, plus `'total'` on panel non-survey fits (RCS and declared-survey fits fail 'total' closed); sup-t bands via bootstrap replay. With seed=None point estimates vary across fits (random folds); set seed for reproducibility. +- [DMLDiD](https://diff-diff.readthedocs.io/en/stable/api/dml_did.html): Chang (2020) double/debiased machine learning DiD — staggered ATT(g,t) with cross-fitted ML nuisances (DML2) and Neyman-orthogonal scores; covariates REQUIRED (conditional parallel trends). Signature: `DMLDiD(propensity_learner='logit', outcome_learner='linear', n_folds=5, control_group='never_treated', anticipation=0, alpha=0.05, n_bootstrap=0, bootstrap_weights=None, seed=None, base_period='varying', cband=True, pscore_trim=0.01, panel=True, cluster=None).fit(data, outcome, unit, time, first_treat, covariates, survey_design=None)`. `panel=False` = declared repeated cross sections (Chang Case 2: level outcomes, row-unique unit IDs, lambda-corrected variance). Survey/cluster support on BOTH designs: `survey_design=` (pweight full-design TSL — weighted moments, PSU-cohesive folds, design-based variance with t-inference; a library extension of Chang's i.i.d. theory), replicate-weight designs (BRR/Fay/JK1/JKn/SDR — per-cell AND aggregate IF-reweighting variance, df = rank-1; cluster= and bootstrap combinations rejected), and coarser-than-unit `cluster=` (variance/folds only, kernels stay unweighted). Learners: string names (`linear`/`ridge`/`sieve` regressors, `logit` classifier) or any object with fit/predict(_proba) (sklearn-compatible); `SieveLearner(k_max, criterion)` is exported for adaptive polynomial nuisances. Aggregation is POST-FIT: `results.aggregate('event_study'/'group'/'simple')`, plus `'total'` on panel non-survey fits (RCS and declared-survey fits fail 'total' closed); sup-t bands via bootstrap replay. With seed=None point estimates vary across fits (random folds); set seed for reproducibility. - [BaconDecomposition](https://diff-diff.readthedocs.io/en/stable/api/bacon.html): Goodman-Bacon (2021) decomposition for diagnosing TWFE bias in staggered settings ## Diagnostics and Sensitivity Analysis diff --git a/docs/api/dml_did.rst b/docs/api/dml_did.rst index 69d1ca8a..38c0c61b 100644 --- a/docs/api/dml_did.rst +++ b/docs/api/dml_did.rst @@ -181,21 +181,25 @@ Restrictions CPS) — pass a pweight :class:`~diff_diff.SurveyDesign` via ``survey_design=`` for weighted RCS. - **Survey/cluster support (both lanes)** — ``survey_design=`` - (pweight-only, full-design TSL: weights/strata/PSU/FPC) weights the - moment kernels, passes ``sample_weight`` into the nuisance learners, - switches to PSU-cohesive cross-fitting folds when the PSU is strictly - coarser than the sampling unit, and uses design-based variance with - ``df = n_PSU - n_strata`` t-inference: PSU designs get the - cluster-robust survey kernel per cell, strata/FPC-only designs use the - weighted influence-function per-cell SE with the full design entering - the AGGREGATE variances (the CallawaySantAnna convention). This is a documented LIBRARY - EXTENSION of Chang (2020), which assumes i.i.d. sampling — on the - weighted-λ RCS lane Theorem 2's coverage claim does not carry over - (REGISTRY DMLDiD Notes). Bare ``cluster=`` (constructor) keeps the - kernels unweighted and affects folds, variance and df only. - Replicate-weight designs are not supported yet (fail closed; - ``TODO.md``); ``aggregate('total')`` also fails closed on - declared-survey fits. + (pweight-only) weights the moment kernels and passes ``sample_weight`` + into the nuisance learners; two variance lanes. Full-design TSL + (weights/strata/PSU/FPC) switches to PSU-cohesive cross-fitting folds + when the PSU is strictly coarser than the sampling unit and uses + design-based variance with ``df = n_PSU - n_strata`` t-inference: PSU + designs get the cluster-robust survey kernel per cell, strata/FPC-only + designs use the weighted influence-function per-cell SE with the full + design entering the AGGREGATE variances (the CallawaySantAnna + convention). Replicate-weight designs (BRR / Fay / JK1 / JKn / SDR) + compute per-cell AND aggregate variances by IF-reweighting the + cross-fitted scores with ``df = rank(replicate matrix) - 1`` + t-inference (nuisances are not re-estimated per replicate); replicate + + ``cluster=`` and replicate + ``n_bootstrap > 0`` are rejected. This + is a documented LIBRARY EXTENSION of Chang (2020), which assumes + i.i.d. sampling — on the weighted-λ RCS lane Theorem 2's coverage + claim does not carry over (REGISTRY DMLDiD Notes). Bare ``cluster=`` + (constructor) keeps the kernels unweighted and affects folds, variance + and df only. ``aggregate('total')`` fails closed on declared-survey + fits. - **Propensity clipping, never dropping** — fitted propensities are clipped to ``[pscore_trim, 1 - pscore_trim]`` after an extremeness warning (the paper gives no trimming rule). diff --git a/docs/choosing_estimator.rst b/docs/choosing_estimator.rst index 8506fea0..2bbd7cdf 100644 --- a/docs/choosing_estimator.rst +++ b/docs/choosing_estimator.rst @@ -687,11 +687,12 @@ insensitive to the nuisance learners' regularization bias. **vs Callaway-Sant'Anna**: same cell architecture and aggregation surface; DMLDiD replaces CS's parametric nuisances with cross-fitted ML learners — prefer it when the covariate relationship is nonlinear/high-dimensional, -prefer CS otherwise (fewer moving parts, replicate-weight support). Both +prefer CS otherwise (fewer moving parts). Both handle declared repeated cross sections via ``panel=False``, and both -carry pweight survey designs and ``cluster=`` there (DMLDiD's survey -lane is a documented library extension of Chang's i.i.d. theory; -replicate-weight designs stay CS-only). +carry pweight survey designs, replicate-weight designs, and ``cluster=`` +there (DMLDiD's survey lane is a documented library extension of Chang's +i.i.d. theory; DMLDiD's replicate variance is per-cell AND aggregate, +exceeding CS's per-cell convention). **Example**:: @@ -1017,7 +1018,7 @@ estimation. The depth of support varies by estimator and variance method: * - ``DMLDiD`` - Full (pweight only) - Full (TSL; df = ``n_PSU - n_strata``) - - -- + - Full (IF-reweighting; per-cell + aggregate) - Multiplier (PSU) * - ``ChangesInChanges`` / ``QDiD`` - -- diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 654bd878..4c245f2f 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -3028,15 +3028,16 @@ the finite-dimensional `p_0` is handled by the variance correction below. (Assumption 2.3) is warned at fit (not data-checkable). - **Note:** Survey support (3.11) is a LIBRARY EXTENSION — Chang (2020) assumes i.i.d. sampling and never discusses clustering or weighting. - Declared `survey_design=` (pweight-only, full-design TSL: - weights/strata/PSU/FPC, resolved and validated by the shared CS - machinery) enters the ESTIMATOR, not just the variance: p̂ and (Case 2) + Declared `survey_design=` (pweight-only; full-design TSL — + weights/strata/PSU/FPC — or replicate weights, resolved and validated + by the shared CS machinery) enters the ESTIMATOR, not just the + variance: p̂ and (Case 2) λ̂ become Hájek weighted shares, θ̂ the weighted score mean, Ĝ₂λ the weighted slope mean, the nuisance learners receive `sample_weight` (user learner objects must accept it by keyword — a `TypeError` fires up front otherwise), and the IF payload is the weighted analogue - `w_i·ψ̄_i/Σw`. The per-cell SE is PSU-gated exactly like CS: designs - with a PSU route through `_cluster_robust_se_from_per_gt_if` + `w_i·ψ̄_i/Σw`. The TSL-lane per-cell SE is PSU-gated exactly like CS: + designs with a PSU route through `_cluster_robust_se_from_per_gt_if` (`compute_survey_if_variance`; a NaN return is the deliberate unidentified-variance signal and flows to `safe_inference` on a RETAINED cell), strata/FPC-only designs use the weighted @@ -3044,12 +3045,60 @@ the finite-dimensional `p_0` is handled by the variance correction below. `_se_from_psi`. Analytical inference uses `df = df_survey` (on bootstrapped surfaces the container's `df_survey` SCALAR persists while the per-row `df` column is cleared - the two-channel contract in the CallawaySantAnna - post-fit Note applies to DMLDiD identically) - (`n_PSU − n_strata`) t-statistics; bootstrap overrides keep normal + post-fit Note applies to DMLDiD identically) — TSL designs at + (`n_PSU − n_strata`) t-statistics, replicate designs at + (`rank(replicate matrix) − 1`) (see the replicate Note below); + bootstrap overrides keep normal theory (the CS convention). On the weighted-λ̂ RCS lane Theorem 2's coverage claim does NOT carry over — the weighted plug-in is validated by the survey invariant battery (`tests/test_survey_dml.py`), not by - the paper. Replicate-weight designs fail closed (TODO.md row). + the paper. +- **Note:** Replicate-weight designs (BRR / Fay / JK1 / JKn / SDR) are + supported via IF-reweighting on the AUGMENTED cross-fitted scores, per + cell AND aggregate (`compute_replicate_if_variance` on the same Hájek + payload `w_i·ψ̄_i/Σw` the aggregate `_se_from_psi` route consumes). The + replicate estimate is the LINEARIZED Hájek form with a FIXED full-weight + denominator: `θ_r = Σ_i w_{r,i}·ψ̄_i / Σ_{j∈cell} w_j` + (`combined_weights=True`; under `combined_weights=False` the helper's + ratio is `w_r` alone, i.e. `θ_r = Σ_i w_{r,i}·w_i·ψ̄_i / Σ_{j∈cell} w_j`) + — the denominator does NOT re-normalize per replicate (verified against + the implementation to 1e-15). Justification: the augmented score is + Neyman-orthogonal in the learned nuisances (g, ℓ) — first-order + insensitive to nuisance perturbation — so reweighting the REALIZED + scores without per-replicate re-cross-fitting is better grounded for + DMLDiD than for parametric first-stage estimators; the nuisances are + nonetheless NOT re-estimated per replicate (the same + conditional-on-nuisance convention as every IF-reweighting route; the + refit alternative is the ImputationDiD/TwoStageDiD lane). Because + replicate designs skip mean-1 weight normalization, the RAW weight + scale reaches the learners' `sample_weight`: the `linear` and `sieve` + learners are scale-invariant, but `RidgeLearner` (fixed alpha and + `alpha="loocv"`) is weight-scale-DEPENDENT via `solve_ridge`'s + unnormalized weighted loss against a fixed penalty — documented + behavior, bounded (ridge → OLS as the scale grows), pinned by the + scale-invariance tests. Df conventions: design df = QR rank of the + analysis-weight matrix − 1; per-cell and overall inference use + `min(df_survey, n_valid − 1)` capping (`n_valid` equals R in practice, + so the cap is defensive). The overall MIN-CAP deliberately diverges + from CS's replace convention (`df_survey = overall_effective_df`), + which can RAISE df above the design df — anti-conservative; the CS-side + flip, along with the inherited replace-style event-study/group + aggregation sites in shared code (latent — they fire only when + `n_valid < R`) and the staggered-DDD engine twins, is tracked in the + TODO.md CS-parity row. `df_survey is None` (rank ≤ 1) → NaN inference + (df=0 sentinel local to `safe_inference`; `survey_metadata.df_survey` + keeps `None`). PER-CELL replicate SEs EXCEED CS's per-cell convention + (CS leaves per-cell SEs on the weighted sqrt-sum under replicates) — + a deliberate documented divergence, same TODO row. Degenerate cells + (zero or non-finite replicate variance, e.g. a cell no replicate + column perturbs) fail closed to NaN inference PER-CELL — stricter than + the shared aggregate path, which clamps negatives and can report a 0.0 + aggregate SE (documented asymmetry: new per-cell code holds the + stricter fail-closed contract; the aggregate clamp is inherited + shared code). Rejected combinations: replicate + `n_bootstrap > 0` and + replicate + `cluster=` (both `NotImplementedError`, message-parity + with CS; the cluster check runs AFTER column validation so a bogus + `cluster=` name stays `ValueError`). - **Note:** PSU-cohesive cross-fitting (survey/cluster designs) — when the effective design's PSU is strictly coarser than the sampling unit AND there are at least `n_folds` PSUs globally, fold assignment switches @@ -6951,7 +7000,7 @@ variance from the distribution of replicate estimates. design structure is fixed and dropped replicates contribute zero to the sum without changing the scale. Survey df uses `n_valid - 1` for t-based inference. -- **Note:** Replicate-weight support matrix (13 of 20 public estimators): +- **Note:** Replicate-weight support matrix (14 of 21 public estimators): - **Supported**: CallawaySantAnna (reg/ipw/dr with or without covariates, no bootstrap; IF-based replicate variance is covariate-agnostic), ContinuousDiD (no bootstrap), EfficientDiD (no bootstrap), @@ -6965,13 +7014,17 @@ variance from the distribution of replicate estimates. ImputationDiD (two-stage refit), TwoStageDiD (two-stage refit), ChaisemartinDHaultfoeuille (closed-form cell-collapse replicate ATT, multi-horizon and placebo paths; replicate + `n_bootstrap > 0` rejected - — see the ChaisemartinDHaultfoeuille Notes for the allocator contract) + — see the ChaisemartinDHaultfoeuille Notes for the allocator contract), + DMLDiD (IF-reweighting on the augmented cross-fitted scores, PER-CELL + and aggregate — exceeds CS's per-cell convention; replicate + + `cluster=` and replicate + bootstrap rejected — see the DMLDiD + replicate Note) - **Rejected with NotImplementedError**: SyntheticDiD, TROP (bootstrap-based variance), WooldridgeDiD, LPDiD, SpilloverDiD, HeterogeneousAdoptionDiD (TSL-only survey paths; replicate designs rejected at `fit()`), SyntheticControl (rejects `survey_design` entirely) - - **BaconDecomposition** is diagnostic-only — outside the 20-estimator + - **BaconDecomposition** is diagnostic-only — outside the 21-estimator count — and likewise rejects replicate designs - Estimators with replicate support reject replicate + bootstrap (replicate weights provide analytical variance) diff --git a/docs/methodology/survey-theory.md b/docs/methodology/survey-theory.md index 89f9cbcd..9001b5fd 100644 --- a/docs/methodology/survey-theory.md +++ b/docs/methodology/survey-theory.md @@ -592,8 +592,8 @@ would require R complete re-fits of the estimator, which is computationally expensive. diff-diff avoids this for most IF-based estimators (CallawaySantAnna, -EfficientDiD, ContinuousDiD, TripleDifference, StaggeredTripleDifference) -using weight-ratio rescaling: the replicate estimate is computed by +EfficientDiD, ContinuousDiD, TripleDifference, StaggeredTripleDifference, +DMLDiD) using weight-ratio rescaling: the replicate estimate is computed by reweighting the per-unit IF values rather than re-running the estimator. The `SurveyDesign` parameter `combined_weights` controls the interpretation: @@ -692,7 +692,7 @@ Each estimator uses one of three variance strategies under survey designs: | TwoWayFixedEffects | TSL sandwich | OLS-based, all weight types | | MultiPeriodDiD | TSL sandwich | OLS-based, all weight types | | CallawaySantAnna | TSL on IFs | pweight only | -| DMLDiD | TSL on IFs | pweight only; cross-fitted augmented scores; library extension of Chang (2020) i.i.d. theory | +| DMLDiD | TSL on IFs | pweight only; cross-fitted augmented scores; replicate designs: IF-reweighting on augmented scores, per-cell + aggregate; library extension of Chang (2020) i.i.d. theory | | SunAbraham | TSL sandwich | OLS-based, all weight types | | TripleDifference | TSL on IFs | pweight only | | StaggeredTripleDifference | TSL on IFs | pweight only | diff --git a/docs/methodology/variance-conventions.md b/docs/methodology/variance-conventions.md index ca3c39e2..ee99a8c9 100644 --- a/docs/methodology/variance-conventions.md +++ b/docs/methodology/variance-conventions.md @@ -38,8 +38,8 @@ produced wrong figures three separate times while this inventory was drafted). | `imputation_pretrends_event_study` | unpinned | 157, 157, 157, None, None, None, None, None | **legitimate** | pretrends lead regression: CR1 k = K_reference (D2 fixed); tail df converged in 3.9 (M-127): leads use t(residual df) via the df_convention knob (was silent normal theory); the None calls are the knob-independent BJS aggregates (L3) | | `two_stage_default` | — | None | **legitimate** | L3: Gardner two-stage variance, not the shared CR1 sandwich | | `callaway_santanna_default` | — | None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None | **legitimate** | L3: influence-function variance anchored to Stata csdid | -| `dml_did` | — | None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None | **legitimate** | L3: Chang (2020) Thm 2 augmented-score plug-in variance (per-unit influence function; normal-theory safe_inference on no-design fits; survey_design=/cluster= fits use the design-based CR1/weighted-IF per-cell variance with df=df_survey t-inference outside the shared CR1 sandwich) | -| `dml_did_rcs` | — | None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None | **legitimate** | L3: Chang (2020) Thm 2 lambda-corrected augmented-score plug-in variance (Case 2; per-observation influence function; normal-theory safe_inference on no-design fits; survey_design=/cluster= fits use the design-based CR1/weighted-IF per-cell variance with df=df_survey t-inference outside the shared CR1 sandwich) | +| `dml_did` | — | None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None | **legitimate** | L3: Chang (2020) Thm 2 augmented-score plug-in variance (per-unit influence function; normal-theory safe_inference on no-design fits; survey_design=/cluster= fits use the design-based CR1/weighted-IF per-cell variance — replicate designs the IF-reweighting per-cell variance — with df=df_survey t-inference outside the shared CR1 sandwich) | +| `dml_did_rcs` | — | None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None | **legitimate** | L3: Chang (2020) Thm 2 lambda-corrected augmented-score plug-in variance (Case 2; per-observation influence function; normal-theory safe_inference on no-design fits; survey_design=/cluster= fits use the design-based CR1/weighted-IF per-cell variance — replicate designs the IF-reweighting per-cell variance — with df=df_survey t-inference outside the shared CR1 sandwich) | cr1_k is the sorted multiset of K_reference counts reaching the shared clustered CR1 denominator — visible columns + the signed cluster_k_adjustment (linalg._compute_robust_vcov_numpy with diff --git a/docs/practitioner_decision_tree.rst b/docs/practitioner_decision_tree.rst index aa43c72f..bec4b515 100644 --- a/docs/practitioner_decision_tree.rst +++ b/docs/practitioner_decision_tree.rst @@ -441,8 +441,8 @@ by geography), or probability weights. :class:`~diff_diff.SurveyDesign`. (:class:`~diff_diff.LWDiD` is the exception: it accepts no ``survey_design`` parameter at all — see the :ref:`survey-design-support` matrix. :class:`~diff_diff.DMLDiD` supports -pweight full-design TSL and ``cluster=`` on both lanes; replicate-weight -designs are not supported yet.) +pweight full-design TSL, replicate-weight designs (IF-reweighting), and +``cluster=`` on both lanes.) Ignoring survey weights and clustering makes your confidence intervals too narrow - you will be overconfident about the result. Passing a ``SurveyDesign`` to ``fit()`` @@ -575,9 +575,9 @@ The six scenarios above cover the most common business use cases. set ``seed=`` for reproducible fold draws. Panel data by default; ``panel=False`` runs declared repeated cross sections (Chang Case 2, λ-corrected variance). Survey/cluster support on both lanes: - ``survey_design=`` (pweight full-design TSL — a documented library - extension of Chang's i.i.d. theory) and coarser-than-unit - ``cluster=``; replicate-weight designs are not supported yet. + ``survey_design=`` (pweight full-design TSL or replicate weights via + IF-reweighting — a documented library extension of Chang's i.i.d. + theory) and coarser-than-unit ``cluster=``. For the full academic decision tree with all estimators, see :doc:`choosing_estimator`. diff --git a/docs/survey-roadmap.md b/docs/survey-roadmap.md index f7a3eca4..ffdb16b8 100644 --- a/docs/survey-roadmap.md +++ b/docs/survey-roadmap.md @@ -271,7 +271,6 @@ the limitation and suggested alternative. | Estimator | Limitation | Alternative | |-----------|-----------|-------------| | LWDiD | Any `survey_design` / sampling weights | No weight argument exists on any path, so the failure mode is a bare `TypeError: unexpected keyword argument` rather than a descriptive error (the exception to the preamble above). The LW papers derive the transformation and exact-inference layer for unweighted panels; a weighted counterpart is DEFERRED pending user demand. Use `CallawaySantAnna` (or another survey-capable staggered estimator) when design-based variance is required. | -| DMLDiD | Replicate weights | Full-design pweight TSL (weights/strata/PSU/FPC) + survey multiplier bootstrap + `cluster=` shipped in 3.11 on BOTH lanes (a documented library extension of Chang 2020's i.i.d. theory; REGISTRY DMLDiD Notes). Replicate-weight designs remain rejected (`NotImplementedError`; the IF-reweighting route is tracked as a TODO.md Actionable row) — use a full-design SurveyDesign, or `CallawaySantAnna` when replicate weights are required. | | SyntheticDiD | Replicate weights | Pre-existing limitation: no replicate-weight survey support on SDID. All three variance methods (bootstrap, placebo, jackknife) now support pweight-only and strata/PSU/FPC designs; replicate-weight designs remain rejected. | | TROP | Replicate weights | Use strata/PSU/FPC design with Rao-Wu rescaled bootstrap | | BaconDecomposition | Replicate weights | Diagnostic only, no inference | diff --git a/docs/tutorials/32_dml_did.ipynb b/docs/tutorials/32_dml_did.ipynb index 7ac5de43..8c46a819 100644 --- a/docs/tutorials/32_dml_did.ipynb +++ b/docs/tutorials/32_dml_did.ipynb @@ -92,7 +92,7 @@ "\n", "*When to prefer it over `CallawaySantAnna`*: covariate relationships that are\n", "nonlinear or high-dimensional. When a logit + linear-OLS specification is plausible,\n", - "prefer CS - fewer moving parts, replicate-weight support, and no learner tuning (see\n", + "prefer CS - fewer moving parts and no learner tuning (see\n", "*Choosing an estimator* in the docs). DMLDiD **requires** covariates: without them the\n", "learners have nothing to do, and `fit()` points you back to CS.\n" ] @@ -1246,9 +1246,11 @@ "Repeated cross-sections are where complex survey data usually lives, and DMLDiD\n", "accepts a `SurveyDesign` at `fit()` time (it is a `fit()` keyword, not a constructor\n", "parameter). A declared design weights the moments and the learners (`sample_weight`),\n", - "makes cross-fitting folds **PSU-cohesive** (whole PSUs enter a fold together, so\n", - "within-cluster dependence cannot leak across the train/score split), and switches\n", - "inference to a design-based $t$ with $df = n_{PSU} - n_{strata}$:\n" + "and — for full designs with a PSU — makes cross-fitting folds **PSU-cohesive**\n", + "(whole PSUs enter a fold together, so within-cluster dependence cannot leak across\n", + "the train/score split) with design-based $t$ inference at $df = n_{PSU} - n_{strata}$.\n", + "Replicate-weight designs have no PSU, so they retain the ordinary D×T-stratified\n", + "folds and use $df = \\mathrm{rank}(\\text{replicate matrix}) - 1$:\n" ] }, { @@ -1354,7 +1356,7 @@ "| Situation | Reach for |\n", "|-----------|-----------|\n", "| Covariates with nonlinear / high-dimensional relationships to treatment or trends | `DMLDiD` |\n", - "| A logit + linear-OLS nuisance specification is plausible | `CallawaySantAnna` (fewer moving parts, replicate-weight support) |\n", + "| A logit + linear-OLS nuisance specification is plausible | `CallawaySantAnna` (fewer moving parts) |\n", "| No covariates at all | `CallawaySantAnna` (DMLDiD requires covariates) |\n", "| Repeated cross-sections, optionally with a complex survey design | `DMLDiD(panel=False)` + `survey_design=` |\n", "| Conclusion must survive a learner swap | Refit with `outcome_learner=` alternatives (section 4; this is Baker et al.'s step 8) |\n", diff --git a/tests/test_survey_dml.py b/tests/test_survey_dml.py index 43a13e6b..866fd3ac 100644 --- a/tests/test_survey_dml.py +++ b/tests/test_survey_dml.py @@ -1,11 +1,13 @@ -"""DMLDiD survey-design support (both lanes): TSL + survey bootstrap + cluster=. +"""DMLDiD survey-design support (both lanes): TSL + replicate weights + +survey bootstrap + cluster=. Survey support is a documented library extension of Chang (2020), which assumes i.i.d. sampling (Assumption 2.3) — no external oracle exists (DoubleML has no survey support; R ``did::`` is survey-naive), so the evidence is the library's standard survey invariant battery (mirroring ``tests/test_survey_phase4.py``'s CS coverage) plus direct kernel -cross-checks against ``compute_survey_if_variance``. +cross-checks against ``compute_survey_if_variance`` (TSL lane) and +``compute_replicate_if_variance`` (replicate lane). """ import warnings @@ -16,7 +18,11 @@ from diff_diff import DMLDiD from diff_diff.staggered_aggregation import fixed_cohort_agg_weights -from diff_diff.survey import SurveyDesign, compute_survey_if_variance +from diff_diff.survey import ( + SurveyDesign, + compute_replicate_if_variance, + compute_survey_if_variance, +) from diff_diff.utils import safe_inference from tests.conftest import assert_nan_inference @@ -83,6 +89,66 @@ def _fit(df, *, panel=True, survey=None, cluster=None, seed=42, ignore_warnings= _DESIGN = SurveyDesign(weights="w", strata="stratum", psu="psu") +def _attach_jk1(df, psu_col="psu", w_col="w"): + """Delete-cluster JK1 replicate columns over the frame's PSUs (the + test_survey_phase6.py construction). Returns (df_copy, rep_cols).""" + df = df.copy() + psus = np.unique(df[psu_col]) + n_rep = len(psus) + rep_cols = [] + for r, p in enumerate(psus): + w_r = df[w_col].to_numpy(dtype=np.float64).copy() + mask = (df[psu_col] == p).to_numpy() + w_r[mask] = 0.0 + w_r[~mask] *= n_rep / (n_rep - 1) + col = f"rep_{r}" + df[col] = w_r + rep_cols.append(col) + return df, rep_cols + + +def _attach_brr(df, psu_col="psu", w_col="w", n_rep=8): + """Half-sample BRR replicate columns (double one half, zero the other).""" + df = df.copy() + psu = df[psu_col].to_numpy() + rep_cols = [] + for r in range(n_rep): + half = ((psu + r) % 2) == 0 + w_r = df[w_col].to_numpy(dtype=np.float64).copy() + w_r[half] *= 2.0 + w_r[~half] = 0.0 + col = f"brr_{r}" + df[col] = w_r + rep_cols.append(col) + return df, rep_cols + + +def _jk1_design(rep_cols, **kw): + return SurveyDesign(weights="w", replicate_weights=rep_cols, replicate_method="JK1", **kw) + + +@pytest.fixture(scope="module") +def panel_replicate_df(panel_df): + return _attach_jk1(panel_df) + + +@pytest.fixture(scope="module") +def rcs_replicate_df(rcs_df): + return _attach_jk1(rcs_df) + + +@pytest.fixture(scope="module") +def panel_replicate(panel_replicate_df): + df, rep_cols = panel_replicate_df + return _fit(df, survey=_jk1_design(rep_cols)) + + +@pytest.fixture(scope="module") +def rcs_replicate(rcs_replicate_df): + df, rep_cols = rcs_replicate_df + return _fit(df, panel=False, survey=_jk1_design(rep_cols)) + + @pytest.fixture(scope="module") def panel_df(): return _make_panel() @@ -727,21 +793,22 @@ def test_survey_bootstrap_replay_matches_fit(self, panel_df, ci_params): class TestRejections: - def test_replicate_design_fails_closed(self, panel_df): - df = panel_df.copy() - df["rw1"] = df["w"] * 1.1 - df["rw2"] = df["w"] * 0.9 - design = SurveyDesign(weights="w", replicate_weights=["rw1", "rw2"], replicate_method="JK1") - with pytest.raises(NotImplementedError, match="replicate"): - _fit(panel_df.assign(rw1=df["rw1"], rw2=df["rw2"]), survey=design) - - def test_cluster_plus_replicate_hits_blanket_replicate_message(self, panel_df): - df = panel_df.copy() - df["rw1"] = df["w"] * 1.1 - df["rw2"] = df["w"] * 0.9 - design = SurveyDesign(weights="w", replicate_weights=["rw1", "rw2"], replicate_method="JK1") - with pytest.raises(NotImplementedError, match="replicate"): - _fit(df, survey=design, cluster="psu") + def test_cluster_plus_replicate_hits_targeted_message(self, panel_replicate_df): + df, rep_cols = panel_replicate_df + with pytest.raises(NotImplementedError, match="cluster.*replicate-weight"): + _fit(df, survey=_jk1_design(rep_cols), cluster="psu") + + def test_bogus_cluster_on_replicate_fit_raises_value_error_first(self, panel_replicate_df): + # CS ordering parity: the cluster column-existence check runs BEFORE + # the replicate + cluster= rejection, so a bogus name is ValueError. + df, rep_cols = panel_replicate_df + with pytest.raises(ValueError, match="cluster column 'nope' not found"): + _fit(df, survey=_jk1_design(rep_cols), cluster="nope") + + def test_bootstrap_plus_replicate_rejected(self, panel_replicate_df): + df, rep_cols = panel_replicate_df + with pytest.raises(NotImplementedError, match="bootstrap.*replicate-weight"): + _fit(df, survey=_jk1_design(rep_cols), n_bootstrap=29) def test_non_pweight_rejected(self, panel_df): with pytest.raises(ValueError, match="pweight"): @@ -880,3 +947,283 @@ def test_survey_mass_dict_no_int64_collision(self): masses = fixed_cohort_agg_weights({"survey_weights": sw, "unit_cohorts": unit_cohorts}) assert masses[g1] == 3.0 assert masses[g2] == 5.0 + + +# --------------------------------------------------------------------------- +# 10. Replicate-weight designs (IF-reweighting; per-cell + aggregate) +# --------------------------------------------------------------------------- + + +class TestReplicateDesigns: + @pytest.mark.parametrize("fixture", ["panel_replicate", "rcs_replicate"]) + def test_acceptance_both_lanes(self, fixture, request): + res = request.getfixturevalue(fixture) + assert np.isfinite(res.overall_att) and np.isfinite(res.overall_se) + assert res.survey_metadata is not None + # Full-rank JK1 over 20 PSUs: QR-rank df = R - 1. + assert res.survey_metadata.df_survey == 19 + finite_cells = [ + v + for v in res.group_time_effects.values() + if v.get("skip_reason") is None and not v.get("is_reference") + ] + assert finite_cells + assert all(np.isfinite(v["se"]) and v["se"] > 0 for v in finite_cells) + + def test_brr_method_accepted(self, panel_df): + df, rep_cols = _attach_brr(panel_df) + res = _fit( + df, + survey=SurveyDesign(weights="w", replicate_weights=rep_cols, replicate_method="BRR"), + ) + assert np.isfinite(res.overall_att) and np.isfinite(res.overall_se) + assert res.survey_metadata is not None + + def test_combined_weights_false_branch(self, panel_replicate_df, panel_replicate): + # combined_weights=False takes the ratio = w_r branch (not w_r/w) and + # a different df_survey analysis-weight construction; the SE must + # DIFFER from the combined fit on the same frame (a regression that + # routed it through the combined branch would reproduce it exactly). + df, rep_cols = panel_replicate_df + res = _fit(df, survey=_jk1_design(rep_cols, combined_weights=False)) + assert np.isfinite(res.overall_att) and np.isfinite(res.overall_se) + np.testing.assert_allclose(res.overall_att, panel_replicate.overall_att, rtol=1e-12) + assert not np.isclose(res.overall_se, panel_replicate.overall_se, rtol=1e-6) + + @pytest.mark.parametrize("fixture", ["panel_replicate", "rcs_replicate"]) + def test_per_cell_se_matches_compute_replicate_if_variance(self, fixture, request): + res = request.getfixturevalue(fixture) + kit = res._aggregation_kit + checked = 0 + for gt_key, data in res.group_time_effects.items(): + if data.get("skip_reason") is not None or data.get("is_reference"): + continue + if gt_key not in kit.influence: + continue + psi, resolved = _reconstruct_cell_psi(kit, gt_key) + variance, _n_valid = compute_replicate_if_variance(psi, resolved) + np.testing.assert_allclose(data["se"], np.sqrt(variance), rtol=1e-12) + checked += 1 + assert checked > 0 + + @pytest.mark.parametrize("learner", ["linear", "sieve"]) + def test_scale_invariance_linear_sieve(self, panel_replicate_df, learner): + # Replicate designs skip mean-1 normalization, so the raw weight + # scale reaches the learners' sample_weight verbatim; linear/sieve + # solves and the Hajek moments are scale-invariant (allclose, not + # exact equality: BLAS summation order varies by backend/OS). + df, rep_cols = panel_replicate_df + base = _fit(df, survey=_jk1_design(rep_cols), outcome_learner=learner) + df_s = df.copy() + for c in ["w"] + rep_cols: + df_s[c] = df_s[c] * 100.0 + scaled = _fit(df_s, survey=_jk1_design(rep_cols), outcome_learner=learner) + np.testing.assert_allclose(scaled.overall_att, base.overall_att, rtol=1e-14) + np.testing.assert_allclose(scaled.overall_se, base.overall_se, rtol=1e-14) + + def test_ridge_scale_dependence_bounded(self, panel_replicate_df): + # RidgeLearner is weight-SCALE-sensitive by documented solve_ridge + # behavior (unnormalized weighted loss vs a fixed penalty; REGISTRY + # DMLDiD Note) — no equality pin, only a sanity ceiling on the drift. + from diff_diff._learners import RidgeLearner + + df, rep_cols = panel_replicate_df + base = _fit(df, survey=_jk1_design(rep_cols), outcome_learner=RidgeLearner(alpha=1.0)) + df_s = df.copy() + for c in ["w"] + rep_cols: + df_s[c] = df_s[c] * 100.0 + scaled = _fit(df_s, survey=_jk1_design(rep_cols), outcome_learner=RidgeLearner(alpha=1.0)) + assert np.isfinite(scaled.overall_att) + assert abs(scaled.overall_att - base.overall_att) < 1e-2 + + def test_df_tightening_spy_reaches_all_surfaces(self, panel_replicate_df, monkeypatch): + # dCDH two-pass convention: reduce n_valid on EVERY call; the + # min(df_survey, n_valid - 1) rule must reach per-cell inference, + # the overall, AND survey_metadata.df_survey. + from scipy import stats as _stats + + from diff_diff import survey as _survey_mod + + df, rep_cols = panel_replicate_df + reduced_n_valid = 7 # (7 - 1) < R - 1 = 19 -> the cap binds + original = _survey_mod.compute_replicate_if_variance + + def reduce_n_valid(psi, resolved_arg): + var, _n_valid = original(psi, resolved_arg) + return var, reduced_n_valid + + monkeypatch.setattr(_survey_mod, "compute_replicate_if_variance", reduce_n_valid) + res = _fit(df, survey=_jk1_design(rep_cols)) + expected_df = reduced_n_valid - 1 + assert res.survey_metadata.df_survey == expected_df + # Overall: min-cap picked the reduced value (19 -> 6). + t_overall = res.overall_att / res.overall_se + assert res.overall_p_value == pytest.approx( + 2 * _stats.t.sf(abs(t_overall), df=expected_df), rel=1e-10 + ) + # Per-cell: every finite cell's p-value uses the reduced df. + checked = 0 + for v in res.group_time_effects.values(): + if v.get("skip_reason") is not None or v.get("is_reference"): + continue + if not (np.isfinite(v["se"]) and v["se"] > 0): + continue + t_cell = v["effect"] / v["se"] + assert v["p_value"] == pytest.approx( + 2 * _stats.t.sf(abs(t_cell), df=expected_df), rel=1e-10 + ) + checked += 1 + assert checked > 0 + + def test_overall_df_caps_never_replaces(self, panel_replicate_df, monkeypatch): + # The CAP-vs-REPLACE discriminator: a rank-deficient-but-usable + # design (duplicated replicate columns -> QR rank 5, df_survey = 4) + # plus a spy n_valid with n_valid - 1 = 6 > 4. CS's replace + # convention would report df = 6; DMLDiD's min-cap must keep 4 + # (deliberate documented divergence; REGISTRY DMLDiD Note). + from scipy import stats as _stats + + from diff_diff import survey as _survey_mod + + df, rep_cols = panel_replicate_df + keep = rep_cols[:5] + r8_cols = [] + for i in range(8): + src = keep[min(i, 4)] # cols 5..7 duplicate col 4 -> rank 5 + col = f"r8_{i}" + df = df.assign(**{col: df[src]}) + r8_cols.append(col) + design = _jk1_design(r8_cols) + assert design.resolve(df).df_survey == 4 + + original = _survey_mod.compute_replicate_if_variance + + def reduce_n_valid(psi, resolved_arg): + var, _n_valid = original(psi, resolved_arg) + return var, 7 # < R = 8 -> the overall relay activates with 6 + + monkeypatch.setattr(_survey_mod, "compute_replicate_if_variance", reduce_n_valid) + res = _fit(df, survey=design) + assert res.survey_metadata.df_survey == 4 + t_overall = res.overall_att / res.overall_se + assert res.overall_p_value == pytest.approx( + 2 * _stats.t.sf(abs(t_overall), df=4), rel=1e-10 + ) + + def test_rank_one_replicate_matrix_nan_inference(self, panel_df): + # All replicate columns identical to the full weights: QR rank 1 -> + # df_survey None AND zero replicate contrast -> per-cell degenerate + # guard (se == NaN, not the clamped 0.0) + NaN inference on cell and + # overall surfaces; survey_metadata.df_survey stays None (the df=0 + # sentinel is local to safe_inference, never leaked to metadata). + df = panel_df.copy() + rep_cols = [] + for r in range(6): + col = f"same_{r}" + df[col] = df["w"] + rep_cols.append(col) + res = _fit(df, survey=_jk1_design(rep_cols)) + assert res.survey_metadata.df_survey is None + checked = 0 + for v in res.group_time_effects.values(): + if v.get("skip_reason") is not None or v.get("is_reference"): + continue + assert np.isnan(v["se"]) + assert_nan_inference(v) + checked += 1 + assert checked > 0 + assert_nan_inference( + { + "se": res.overall_se, + "t_stat": res.overall_t_stat, + "p_value": res.overall_p_value, + "conf_int": res.overall_conf_int, + } + ) + + @pytest.mark.parametrize("degenerate", [(0.0, 20), (float("nan"), 1)]) + def test_degenerate_cell_fails_closed_to_nan(self, panel_replicate_df, monkeypatch, degenerate): + # Zero variance (a cell no replicate column perturbs) and the + # n_valid < 2 helper contract (variance NaN) both fail closed + # PER-CELL: se must be NaN EXPLICITLY (assert_nan_inference alone + # cannot discriminate NaN from a clamped 0.0 — safe_inference NaNs + # t/p/CI at se=0 either way). Spy hits the FIRST per-cell call only; + # every other cell stays finite. + from diff_diff import survey as _survey_mod + + df, rep_cols = panel_replicate_df + original = _survey_mod.compute_replicate_if_variance + counter = {"n": 0} + + def degrade_first_cell(psi, resolved_arg): + counter["n"] += 1 + if counter["n"] == 1: + return degenerate + return original(psi, resolved_arg) + + monkeypatch.setattr(_survey_mod, "compute_replicate_if_variance", degrade_first_cell) + res = _fit(df, survey=_jk1_design(rep_cols)) + cells = [ + v + for v in res.group_time_effects.values() + if v.get("skip_reason") is None and not v.get("is_reference") + ] + nan_cells = [v for v in cells if np.isnan(v["se"])] + finite_cells = [v for v in cells if np.isfinite(v["se"]) and v["se"] > 0] + assert len(nan_cells) == 1 + assert_nan_inference(nan_cells[0]) + assert len(finite_cells) == len(cells) - 1 + # The degenerate cell is RETAINED (NaN-consistent inference), never + # silently skipped or given an analytical fallback. + assert nan_cells[0].get("skip_reason") is None + + def test_aggregations_finite_es_vcov_none_total_closed(self, panel_replicate): + es = panel_replicate.aggregate("event_study") + se_vals = np.asarray(es.se, dtype=float) + ref = np.asarray(es.is_reference, dtype=bool) + assert np.all(np.isfinite(se_vals[~ref])) + assert es.vcov is None # deliberate under replicates (HonestDiD diagonal) + gr = panel_replicate.aggregate("group") + assert np.all(np.isfinite(np.asarray(gr.se, dtype=float))) + with pytest.raises(NotImplementedError): + panel_replicate.aggregate("total") + + def test_es_df_pins_inherited_replace_convention(self, panel_replicate_df, monkeypatch): + # The SHARED event-study aggregation path REPLACES the design df + # with min(non_none effective dfs) (staggered_aggregation.py) — + # the inherited CS convention, NOT the fit-level min-cap. To + # discriminate, the design must be rank-deficient (df_survey = 4) + # with spy n_valid - 1 = 6 > 4: replace reports 6, a cap would + # report 4. Pinned at 6 so the follow-up flip to min-cap (TODO.md + # CS-parity row: all five replace-style relay sites) is detectable; + # latent in practice (n_valid == R on well-formed designs). + from diff_diff import survey as _survey_mod + + df, rep_cols = panel_replicate_df + keep = rep_cols[:5] + r8_cols = [] + for i in range(8): + col = f"esr8_{i}" + df = df.assign(**{col: df[keep[min(i, 4)]]}) + r8_cols.append(col) + design = _jk1_design(r8_cols) + assert design.resolve(df).df_survey == 4 + res = _fit(df, survey=design) + original = _survey_mod.compute_replicate_if_variance + + def reduce_n_valid(psi, resolved_arg): + var, _n_valid = original(psi, resolved_arg) + return var, 7 # < R = 8 -> effective df 6 at the shared site + + monkeypatch.setattr(_survey_mod, "compute_replicate_if_variance", reduce_n_valid) + es = res.aggregate("event_study") + df_vals = np.asarray(es.df, dtype=float) + finite = df_vals[np.isfinite(df_vals)] + assert finite.size > 0 + assert np.all(finite == 6.0) # inherited replace; a min-cap would give 4 + + def test_replicate_fit_keeps_stratified_folds(self, panel_replicate): + # Replicate designs have psu None by mutual exclusion -> the + # PSU-cohesive fold gate is structurally inert. + diags = [d for d in panel_replicate.cross_fit_diagnostics.values() if "psu_folds" in d] + assert diags and all(d["psu_folds"] is False for d in diags) diff --git a/tests/test_variance_conventions.py b/tests/test_variance_conventions.py index c9b1a1e1..bc70123e 100644 --- a/tests/test_variance_conventions.py +++ b/tests/test_variance_conventions.py @@ -452,7 +452,8 @@ def snapshot(self): "L3: Chang (2020) Thm 2 augmented-score plug-in variance " "(per-unit influence function; normal-theory safe_inference on " "no-design fits; survey_design=/cluster= fits use the design-based " - "CR1/weighted-IF per-cell variance with df=df_survey t-inference " + "CR1/weighted-IF per-cell variance — replicate designs the " + "IF-reweighting per-cell variance — with df=df_survey t-inference " "outside the shared CR1 sandwich)" ), ), @@ -480,8 +481,9 @@ def snapshot(self): "plug-in variance (Case 2; per-observation influence function; " "normal-theory safe_inference on no-design fits; survey_design=/" "cluster= fits use the design-based CR1/weighted-IF per-cell " - "variance with df=df_survey t-inference outside the shared CR1 " - "sandwich)" + "variance — replicate designs the IF-reweighting per-cell " + "variance — with df=df_survey t-inference outside the shared " + "CR1 sandwich)" ), ), ]