From 5665377362250aa9c19aa58ed305e16ebd626339 Mon Sep 17 00:00:00 2001 From: igerber Date: Thu, 29 Oct 2026 09:33:01 -0400 Subject: [PATCH 1/2] feat(v4): item 7b of the 4.0 cut - SpilloverDiD auto-cluster, uniform cluster=False, cluster metadata (M-080, M-155) SpilloverDiD auto-clusters at unit when cluster= is omitted - the one default standard error that moves at 4.0 - with TwoWayFixedEffects' carve-outs mirrored: an explicit cluster= wins on every lane, the auto-cluster is dropped on the Conley path and is never injected as a survey PSU. cluster=False reproduces the 3.x observation-level HC1 numbers; results report cluster_name="unit" / n_clusters. Inference stays on the residual df until the df_convention knob lands (new planned row M-154, item 7c). cluster=False is the uniform explicit off-switch (M-080): the plain heteroskedasticity-robust sandwich with residual-df t inference on TwoWayFixedEffects (static and event-study; one-way CR2 Bell-McCaffrey under hc2_bm), SunAbraham, WooldridgeDiD (the OLS within-transform lane gains the absorbed-FE full-K rescale so its SE equals HC1 on the explicit full-dummy design; logit/poisson run the plain QMLE sandwich) and LPDiD (per-horizon HC1, an if_robust unclustered IF variance on the regression-adjustment path, implicit per-observation PSUs under a survey design); an accepted bit-identical no-op on the unit-level-by- construction estimators (CallawaySantAnna incl. its RC auto-PSU path, EfficientDiD incl. hausman_pretest, DMLDiD, ChaisemartinDHaultfoeuille, staggered TripleDifference, LWDiD incl. the sensitivity helpers, HeterogeneousAdoptionDiD, ImputationDiD, TwoStageDiD) and on the 2x2 DifferenceInDifferences / 2x2x2 TripleDifference surfaces; still rejected on StackedDiD. The combinations that would otherwise ship an unverified variance reject it: inference="wild_bootstrap" (DiD/TWFE), WooldridgeDiD(n_bootstrap > 0) and vcov_type="hc2_bm" on SunAbraham / WooldridgeDiD (cluster-only Bell-McCaffrey DOF; the one-way port is a TODO.md row). SunAbraham's pairs bootstrap accepts it as a no-op. The shared contract is utils.validate_cluster (new row M-155): cluster is str | None | False on every estimator exposing it, validated at __init__, through set_params and again at fit(); DMLDiD's pre-4.0 str-only check is generalized and False is normalized locally so it never reaches a column lookup. Metadata: DiD/TWFE analytical fits report n_clusters from the regression's effective cluster count (non-survey fits); SunAbrahamResults gains an appended n_clusters field and reports cluster_name on clustered analytical fits, and its summary() renders the variance-family label on analytical non-survey fits; TWFE's survey-lane auto label is None (the regression received no cluster ids). Tests: new tests/test_v4_cluster_policy.py (the two rows' test_ref) pins the contract roster, the Spillover default and its carve-outs, every c1 off-switch lane (incl. the TWFE and Wooldridge full-dummy HC1 parity, the LPDiD singleton-cluster identity, the saturated-Wooldridge NaN contract and the four rejects), the b/c2 no-op, the metadata lanes and the ledger. test_spillover.py pins its Wave D goldens with cluster=False (the HC1 lane they were captured on) and extends the CR1-routing test to the default; the inference-policy and wild-bootstrap pins follow the new DiD message and the analytical n_clusters; the tutorial-23 drift suite pins the default, the CR1 > HC1 > Conley ordering and the four-row table source. Tutorial 23 section 6 compares CR1 at unit, HC1 (cluster=False) and the two Conley fits. Ledger: M-080 and M-155 -> done, M-154 -> planned (matrix gate counts 137); spec section 7/9, the variance-conventions roster, REGISTRY notes (DiD, TWFE, SunAbraham, WooldridgeDiD, LPDiD, SpilloverDiD, the TWFE knob roster, the category-b classes), the migration guide, llms-full.txt, docs/api/spillover.rst and TODO.md updated. --- TODO.md | 5 +- .../20261029-v4-pr7b-m080-cluster-policy.md | 10 + diff_diff/chaisemartin_dhaultfoeuille.py | 39 +- diff_diff/dml_did.py | 45 +- diff_diff/efficient_did.py | 53 +- diff_diff/estimators.py | 45 +- diff_diff/guides/llms-full.txt | 38 +- diff_diff/had.py | 17 +- diff_diff/imputation.py | 35 +- diff_diff/lpdid.py | 101 +- diff_diff/lpdid_results.py | 8 +- diff_diff/lwdid.py | 31 +- diff_diff/lwdid_sensitivity.py | 20 +- diff_diff/spillover.py | 62 +- diff_diff/staggered.py | 51 +- diff_diff/sun_abraham.py | 112 ++- diff_diff/triple_diff.py | 39 +- diff_diff/twfe.py | 67 +- diff_diff/two_stage.py | 37 +- diff_diff/utils.py | 38 +- diff_diff/wooldridge.py | 139 ++- docs/api/spillover.rst | 3 +- docs/methodology/REGISTRY.md | 159 +++- docs/methodology/variance-conventions.md | 39 +- docs/migration-4.0.md | 49 +- docs/tutorials/23_spillover_tva.ipynb | 71 +- docs/v4-deprecations.yaml | 30 +- docs/v4-design.md | 42 +- tests/test_spillover.py | 21 +- tests/test_t23_spillover_tva_drift.py | 74 +- tests/test_v4_cluster_policy.py | 900 ++++++++++++++++++ tests/test_v4_inference_policy.py | 17 +- tests/test_v4_matrix.py | 20 +- tests/test_wild_bootstrap.py | 7 +- 34 files changed, 2053 insertions(+), 371 deletions(-) create mode 100644 changelog.d/20261029-v4-pr7b-m080-cluster-policy.md create mode 100644 tests/test_v4_cluster_policy.py diff --git a/TODO.md b/TODO.md index 39ffc6ab8..cec98268d 100644 --- a/TODO.md +++ b/TODO.md @@ -21,6 +21,7 @@ Related tracking surfaces: | Issue | Location | Origin | Effort | Priority | |-------|----------|--------|--------|----------| +| One-way (unclustered) Bell-McCaffrey DOF for `SunAbraham` / `WooldridgeDiD` `vcov_type="hc2_bm"` + `cluster=False`: both reject the combination at 4.0 (item 7b) because their BM contrast helper `_compute_cr2_bm_contrast_dof` is cluster-only; port the one-way lane through `linalg._compute_bm_dof_from_contrasts` (the TWFE event-study core's helper) with a clubSandwich `vcovCR(cluster=1:n, type="CR2")` parity pin, then lift the two rejects and their tests | `diff_diff/sun_abraham.py`, `diff_diff/wooldridge.py` | #v4-7b | Mid | Low | | `decompose_twfe_weights(method="aipw")`: port upstream `implicit_aipw_weights` (covariate-adjusted only; `twfe_weights.py` rejects the value today and lists `"fwl"`). The R goldens already carry the `decompose.aipw` / `balance.aipw` / `two_period.*` blocks under `meta.reserved_blocks`, so no R re-run is needed | `diff_diff/twfe_weights.py` | #812 | Mid | Medium | | Consolidate the remaining estimator-entangled DR/logit score variants (`staggered.py::_doubly_robust` + RC twins, `triple_diff.py`, `lwdid.py`, `wooldridge.py`) onto the shared `_dr_scores.py` module, each migration with its own committed oracle capture (the ContinuousDiD lift's two-tier pattern in `tests/test_dr_scores.py`); and add a ridge vcov path to `solve_ridge` if an estimator ever needs analytical ridge inference | `diff_diff/_dr_scores.py` | dml-b0 | Mid | Low | | Numeric between-period cohorts (e.g. `first_treat=4.5` with integer times) are rejected by LWDiD while CallawaySantAnna estimates them and LWDiD's own datetime/Period cohorts map to the next observed period — close the dtype asymmetry by adopting the next-observed-period mapping for numeric cohorts too (contract documented in REGISTRY cohort-encodings Note + `docs/api/lwdid.rst` Input Contract). Lands only after PR #588 merges | `diff_diff/lwdid.py` | #588 | Quick | Low | @@ -85,8 +86,8 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m | Issue | Location | Origin | Effort | Priority | |-------|----------|--------|--------|----------| -| Re-execute the nine tutorials whose committed outputs still show the 3.x residual-df p-values/CIs after the `df_convention` default moved to `"cluster"` (t(G-1)): `01_basic_did`, `02_staggered_did`, `04_parallel_trends`, `05_honest_did`, `07_pretrends_power`, `09_real_world_examples`, `11_imputation_did`, `13_stacked_did`, `16_wooldridge_etwfe` (not `16_survey_did`: the survey df wins). Run under `DIFF_DIFF_BACKEND=rust` with the per-worktree kernel, reconcile any prose that restates a moved number, and `--nbmake` each; scheduled for the 4.0 release PR (`docs/v4-design.md` section 9 item 10) together with any item-7b/8/9 movers (24 was refreshed in #879; 23 belongs to 7b). | `docs/tutorials/*.ipynb` | #879 | Mid | Medium | -| TWFE event-study one-way HC2-BM absorbed-FE R parity against the `mpd_absorbed_fe_did` golden (`clubSandwich::vcovCR(cluster=1:n, type="CR2")`): the within spec with hc2_bm auto-clusters at unit, so the one-way lane needs an explicit singleton-cluster column (`cluster=` + hc2_bm = CR2 with G=n, PT2018 §3.3) once the G=n CR2 path is verified; the hc2 lane is pinned by `TestTWFEEventStudyAbsorbedFERParity` and the clustered hc2_bm per-period lane by `test_twfe_event_study_cluster_hc2_bm_per_period_df_matches_clubsandwich`; the MultiPeriodDiD lanes died at the 4.0 cut (PR 4c-ii) | `tests/test_estimators_vcov_type.py` | #v4-4c-i | Mid | Medium | +| Re-execute the nine tutorials whose committed outputs still show the 3.x residual-df p-values/CIs after the `df_convention` default moved to `"cluster"` (t(G-1)): `01_basic_did`, `02_staggered_did`, `04_parallel_trends`, `05_honest_did`, `07_pretrends_power`, `09_real_world_examples`, `11_imputation_did`, `13_stacked_did`, `16_wooldridge_etwfe` (not `16_survey_did`: the survey df wins). Run under `DIFF_DIFF_BACKEND=rust` with the per-worktree kernel, reconcile any prose that restates a moved number, and `--nbmake` each; scheduled for the 4.0 release PR (`docs/v4-design.md` section 9 item 10) together with any item-7c/8/9 movers (24 was refreshed in #879; 23 in 7b and again in 7c). 7b's label changes also drift any committed analytical DiD/TWFE summary (", G=" on the variance line) and every SunAbraham summary (the new "Std. errors:" line) - covered by the same re-execution. | `docs/tutorials/*.ipynb` | #879 | Mid | Medium | +| TWFE event-study one-way HC2-BM absorbed-FE R parity against the `mpd_absorbed_fe_did` golden (`clubSandwich::vcovCR(cluster=1:n, type="CR2")`): since 4.0 (item 7b) `cluster=False` + hc2_bm reaches the one-way lane directly (no singleton-cluster column needed; `tests/test_v4_cluster_policy.py` pins finiteness and the static-surface LinearRegression parity only), so the open item is the R pin itself; the hc2 lane is pinned by `TestTWFEEventStudyAbsorbedFERParity` and the clustered hc2_bm per-period lane by `test_twfe_event_study_cluster_hc2_bm_per_period_df_matches_clubsandwich`; the MultiPeriodDiD lanes died at the 4.0 cut (PR 4c-ii) | `tests/test_estimators_vcov_type.py` | #v4-4c-i | Mid | Medium | | TWFE event-study power profile: `simulate_power`/`simulate_mde`/`simulate_sample_size` have no `_EstimatorProfile` for `TwoWayFixedEffects.fit(event_study=True)` since the MultiPeriodDiD registry entry went with the class (PR 4c-ii); `EventStudyResults` carries no scalar headline, so the profile needs a design (post-period average via the `tests/helpers/event_study_average.py` arithmetic, or a named period) plus its `_SURVEY_FIT_BUILDERS` twin | `diff_diff/power.py` | #v4-4c-ii | Mid | Low | | Clustered hc2_bm Bell-McCaffrey DOF guard is batch-relative (`linalg._cr2_bm_dof_inner`: a contrast is NaN'd when its scaled max\|B\| falls below 1e-10 of the batch maximum), so the event-study core keeps the otherwise-unused post-period-average contrast column as the anchor to hold the 3.x per-period NaN masks bit-identical (PR 4c-ii, user decision 2026-10-25). Reproduction: `x_nc = treated * 1[period >= 4] + 3e-6 * default_rng(seed).normal(size=n)` as a covariate on the oracle panel flips the mask between `[eye(k), avg]` and `eye(k)` on 20/20 seeds (cond(X) ~ 9e5; 1e-7 never flips, <= 3e-8 rank-drops) - too BLAS-sensitive for a committed oracle, hence the structural pin `test_hc2_bm_contrast_batch_keeps_post_average_anchor`. Decide whether the guard should use a batch-independent anchor (the treatment contrast, or the absolute floor only) and then drop the column | `diff_diff/estimators.py`, `diff_diff/linalg.py` | #v4-4c-ii | Mid | Medium | | Re-validate the plan-review dual engine on `gpt-6.1-sol` with `tools/plan-review-eval`: re-pin every arm's composite `cli_version` to the live Claude Code + codex CLI pair, add a gpt-6.1-sol arm, run the harness against the gpt-5.6-sol baseline (the 2026-10 swap carried the model over from the code-review campaign 2 verdict without a plan-review campaign). | `tools/plan-review-eval/config/configs.json` | reviewer-eval gpt61sol PR-2 | Heavy | Low | diff --git a/changelog.d/20261029-v4-pr7b-m080-cluster-policy.md b/changelog.d/20261029-v4-pr7b-m080-cluster-policy.md new file mode 100644 index 000000000..765610108 --- /dev/null +++ b/changelog.d/20261029-v4-pr7b-m080-cluster-policy.md @@ -0,0 +1,10 @@ +### Breaking Changes + +- `SpilloverDiD` auto-clusters at `unit` when `cluster=` is omitted ([M-080], item 7b of the 4.0 cut), mirroring `TwoWayFixedEffects`' carve-outs (explicit `cluster=` wins; the auto-cluster is dropped on the Conley path and never injected as a survey PSU). Its default standard errors, p-values and CIs move; `cluster=False` reproduces the 3.x observation-level HC1 numbers. Results report `cluster_name="unit"` / `n_clusters`. Inference stays on the residual df until the `df_convention` knob lands ([M-154], the next PR). +- `cluster=False` is the uniform explicit off-switch ([M-080]): the plain heteroskedasticity-robust sandwich with residual-df t inference on `TwoWayFixedEffects` (static + event-study; one-way CR2 Bell-McCaffrey under `hc2_bm`), `SunAbraham`, `WooldridgeDiD` (OLS within-transform lane with the absorbed-FE full-K rescale; logit/poisson plain QMLE sandwich) and `LPDiD` (per-horizon HC1; `if_robust` unclustered IF variance on the regression-adjustment path; implicit per-observation PSUs under a survey design); an accepted bit-identical no-op on the unit-level-by-construction estimators (`CallawaySantAnna` incl. its RC auto-PSU path, `EfficientDiD` incl. `hausman_pretest`, `DMLDiD`, `ChaisemartinDHaultfoeuille`, staggered `TripleDifference`, `LWDiD` incl. the sensitivity helpers, `HeterogeneousAdoptionDiD`, `ImputationDiD`, `TwoStageDiD`) and on the 2x2 `DifferenceInDifferences` / 2x2x2 `TripleDifference` surfaces; still rejected on `StackedDiD`. Rejected with `inference="wild_bootstrap"` (DiD/TWFE), `WooldridgeDiD(n_bootstrap > 0)` and `vcov_type="hc2_bm"` on `SunAbraham`/`WooldridgeDiD` (cluster-only Bell-McCaffrey DOF; tracked in TODO.md). +- `cluster` accepts `str | None | False` only ([M-155], `diff_diff.utils.validate_cluster`): `True` and non-`str` column labels raise one shared `ValueError` at construction, through `set_params` and at `fit()` on every estimator exposing the parameter (DMLDiD's existing rule generalized; integer-labelled cluster columns must be renamed). +- Cluster metadata: `DifferenceInDifferences` / `TwoWayFixedEffects` report `n_clusters` from the regression's effective cluster count on non-survey analytical fits (previously `None` unless the wild bootstrap ran); `SunAbrahamResults` gains an appended `n_clusters` field and reports `cluster_name` on clustered analytical fits (previously `None` off the Conley path), and its `summary()` renders the variance-family label on analytical non-survey fits; `TwoWayFixedEffects` under a survey design with no explicit `cluster=` reports `cluster_name=None` (the auto-cluster is never injected as a PSU). + +### Internal + +- Ledger: M-080 and the new M-155 (shared cluster contract) -> `done`; new M-154 (SpilloverDiD `df_convention`, item 7c) -> `planned`; pinned by `tests/test_v4_cluster_policy.py`. Tutorial 23 section 6 shows the four-row SE table (CR1 at unit, HC1 via `cluster=False`, Conley lag 0 / lag 1) and was re-executed; `tests/test_t23_spillover_tva_drift.py` pins the new default and the table's source. diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index 3436f392e..9b43f5f14 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -45,7 +45,7 @@ DCDHBootstrapResults, ) from diff_diff.linalg import solve_ols -from diff_diff.utils import safe_inference, validate_n_bootstrap +from diff_diff.utils import ClusterArg, safe_inference, validate_cluster, validate_n_bootstrap __all__ = [ "ChaisemartinDHaultfoeuille", @@ -407,12 +407,14 @@ class ChaisemartinDHaultfoeuille(ChaisemartinDHaultfoeuilleBootstrapMixin, BaseE ---------- alpha : float, default=0.05 Significance level for confidence intervals. - cluster : str, optional, default=None - Must be ``None`` (the default). User-specified clustering via - this kwarg is not supported — passing any non-``None`` value - raises ``NotImplementedError`` at construction time (and the - same gate fires from ``set_params``). The effective clustering - depends on how you call ``fit()``: + cluster : str, None or False, optional, default=None + ``None`` (the default) and ``False`` (4.0, M-080: an accepted explicit + no-op - the variance is group-level by construction) both select the + default path. User-specified clustering via this kwarg is not + supported — a column name raises ``NotImplementedError`` at + construction time (and the same gate fires from ``set_params``); + other values raise the shared ``ValueError`` (labels must be ``str``). + The effective clustering depends on how you call ``fit()``: - **Default (no survey_design)**: clustered at the group level via the cohort-recentered influence-function plug-in @@ -731,7 +733,7 @@ class ChaisemartinDHaultfoeuille(ChaisemartinDHaultfoeuilleBootstrapMixin, BaseE def __init__( self, alpha: float = 0.05, - cluster: Optional[str] = None, + cluster: ClusterArg = None, n_bootstrap: int = 0, bootstrap_weights: str = "rademacher", seed: Optional[int] = None, @@ -778,7 +780,12 @@ def __init__( "paths_of_interest=[(...), ...] for explicit user-" "specified paths. Set one and leave the other as None." ) - if cluster is not None: + # Shared cluster contract first ([M-155]): True / non-str labels raise the + # shared ValueError; False is an accepted explicit no-op ([M-080] category + # b: the variance is unit-level by construction); a column name is + # the unsupported custom clustering below. + cluster = validate_cluster(cluster) + if cluster is not None and cluster is not False: raise NotImplementedError( f"cluster={cluster!r}: user-specified clustering is not " f"supported in ChaisemartinDHaultfoeuille. dCDH clusters at " @@ -787,8 +794,9 @@ def __init__( f"multiplier bootstrap. Under survey_design with strictly-" f"coarser PSUs, bootstrap clustering automatically upgrades " f"to PSU-level Hall-Mammen wild. To use the default path, " - f"pass cluster=None (the " - f"default). Custom clustering is reserved for a future " + f"pass cluster=None (the default) or cluster=False (an accepted " + f"no-op: the variance is unit-level by construction). Custom " + f"clustering is reserved for a future " f"phase. See REGISTRY.md ChaisemartinDHaultfoeuille section " f"for the full contract." ) @@ -853,14 +861,16 @@ def _validate_invariants(self) -> None: "paths_of_interest=[(...), ...] for explicit user-" "specified paths. Set one and leave the other as None." ) - if self.cluster is not None: + validate_cluster(self.cluster) + if self.cluster is not None and self.cluster is not False: raise NotImplementedError( f"cluster={self.cluster!r}: user-specified clustering is " f"not supported in ChaisemartinDHaultfoeuille. dCDH clusters " f"at the group level by default; under survey_design with " f"strictly-coarser PSUs the bootstrap automatically upgrades " f"to PSU-level Hall-Mammen wild clustering. Pass cluster=None " - f"(the default) to use this path. User-specified custom " + f"(the default) or cluster=False (accepted no-op) to use this " + f"path. User-specified custom " f"clustering is reserved for a future phase. See REGISTRY.md " f"ChaisemartinDHaultfoeuille section for the full contract." ) @@ -1041,6 +1051,9 @@ def fit( # public parameters are unit/covariates. group = unit controls = covariates + # Fit-time re-check of the shared cluster contract ([M-155]; the + # custom-clustering NotImplementedError lives in __init__). + validate_cluster(self.cluster) # ------------------------------------------------------------------ # Step 1: Column validation # ------------------------------------------------------------------ diff --git a/diff_diff/dml_did.py b/diff_diff/dml_did.py index e9f2df734..291faf784 100644 --- a/diff_diff/dml_did.py +++ b/diff_diff/dml_did.py @@ -81,8 +81,10 @@ from diff_diff.staggered_aggregation import CallawaySantAnnaAggregationMixin from diff_diff.staggered_bootstrap import CallawaySantAnnaBootstrapMixin from diff_diff.utils import ( + ClusterArg, safe_inference, validate_anticipation, + validate_cluster, validate_covariate_names, validate_n_bootstrap, validate_pscore_trim, @@ -353,7 +355,10 @@ class DMLDiD(CallawaySantAnnaBootstrapMixin, CallawaySantAnnaAggregationMixin, B and treatment effects are expected, not violations), which is not data-checkable — ``fit()`` warns. ``aggregate('total')`` is unavailable on RCS fits (fails closed, the library-wide RC convention). - cluster : str, optional + cluster : str, None or False, optional + ``False`` (4.0, M-080) is an accepted explicit no-op: the variance is + unit-level by construction, so there is nothing to switch off. Column + labels must be ``str``. Column name for COARSER-than-unit clustering. The column is synthesized into a ``SurveyDesign(psu=cluster)`` (or injected as the PSU of a declared PSU-less design): it drives PSU-cohesive @@ -382,7 +387,7 @@ def __init__( cband: bool = True, pscore_trim: float = 0.01, panel: bool = True, - cluster: Optional[str] = None, + cluster: ClusterArg = None, ) -> None: # Raw assignment, then ONE shared validator (also re-run at the top # of fit() as the direct-mutation defense) validates and normalizes @@ -474,11 +479,9 @@ def _revalidate_config(self, *, require_sample_weight: bool = False) -> None: "would silently select the panel lane" ) self.panel = bool(self.panel) - if self.cluster is not None and not isinstance(self.cluster, str): - raise ValueError( - f"cluster must be a column name (str) or None, got " - f"{self.cluster!r} (type {type(self.cluster).__name__})" - ) + # Shared cluster contract (str | None | False, [M-155]) - the pre-4.0 + # DMLDiD-local str check generalized to every estimator. + validate_cluster(self.cluster) # ------------------------------------------------------------------ # Input validation @@ -2794,6 +2797,12 @@ def fit( """ self.results_ = None self.is_fitted_ = False + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]): __init__ and set_params validate eagerly, so this only + # catches DIRECT attribute mutation. `False` is normalized to None + # locally so it never reaches a column lookup ([M-080]). + validate_cluster(self.cluster) + user_cluster = None if self.cluster is False else self.cluster df, covariates, w_names = self._validate_and_prepare( data, outcome, @@ -2855,13 +2864,13 @@ def fit( effective_survey_design = survey_design cluster_ids_for_check: Optional[np.ndarray] = None - if self.cluster is not None: - if self.cluster not in data.columns: - raise ValueError(f"cluster column '{self.cluster}' not found in data") - _cluster_col = data[self.cluster] + if user_cluster is not None: + if user_cluster not in data.columns: + raise ValueError(f"cluster column '{user_cluster}' not found in data") + _cluster_col = data[user_cluster] if _cluster_col.isna().any(): raise ValueError( - f"cluster column '{self.cluster}' contains missing values; " + f"cluster column '{user_cluster}' contains missing values; " "drop or impute them before fitting" ) cluster_ids_for_check = _cluster_col.to_numpy() @@ -2874,7 +2883,7 @@ def fit( # paths below would violate that mutual exclusion. if resolved_survey is not None and resolved_survey.uses_replicate_variance: raise NotImplementedError( - f"DMLDiD(cluster={self.cluster!r}) is not " + f"DMLDiD(cluster={user_cluster!r}) is not " "supported with replicate-weight survey designs. " "Replicate-weight variance is computed by replicate " "reweighting (BRR / Fay / JK1 / JKn / SDR) and ignores " @@ -2889,7 +2898,7 @@ def fit( # stays None DELIBERATELY (it is the declared-survey marker: # the aggregate('total') gate and the weighted-kernel gate # both key on it); df carried on Results.df_inference instead. - effective_survey_design = SurveyDesign(psu=self.cluster, weight_type="pweight") + effective_survey_design = SurveyDesign(psu=user_cluster, weight_type="pweight") ( resolved_survey, _survey_weights_resolved, @@ -2903,14 +2912,14 @@ def fit( assert survey_design is not None from dataclasses import replace as _dc_replace - effective_survey_design = _dc_replace(survey_design, psu=self.cluster) + effective_survey_design = _dc_replace(survey_design, psu=user_cluster) resolved_survey = _inject_cluster_as_psu(resolved_survey, cluster_ids_for_check) assert raw_obs_weights is not None survey_metadata = compute_survey_metadata(resolved_survey, raw_obs_weights) else: # Both supplied: the design's PSU wins (warn on differing # partitions); the return value is intentionally unused. - _resolve_effective_cluster(resolved_survey, cluster_ids_for_check, self.cluster) + _resolve_effective_cluster(resolved_survey, cluster_ids_for_check, user_cluster) if resolved_survey is not None: if self.panel: @@ -3271,8 +3280,8 @@ def fit( if resolved_survey_unit is not None: if survey_design is not None and survey_design.psu is not None: cluster_name = survey_design.psu - elif self.cluster is not None: - cluster_name = self.cluster + elif user_cluster is not None: + cluster_name = user_cluster if resolved_survey_unit.psu is not None: n_clusters = int(np.unique(resolved_survey_unit.psu).size) if survey_metadata is None and df_survey is not None: diff --git a/diff_diff/efficient_did.py b/diff_diff/efficient_did.py index de28cfae7..3a727aeb2 100644 --- a/diff_diff/efficient_did.py +++ b/diff_diff/efficient_did.py @@ -63,7 +63,13 @@ compute_omega_star_nocov, enumerate_valid_triples, ) -from diff_diff.utils import safe_inference, validate_anticipation, validate_n_bootstrap +from diff_diff.utils import ( + ClusterArg, + safe_inference, + validate_anticipation, + validate_cluster, + validate_n_bootstrap, +) # Re-export for convenience __all__ = ["EfficientDiD", "EfficientDiDResults", "EDiDBootstrapResults"] @@ -273,7 +279,10 @@ class EfficientDiD(EfficientDiDBootstrapMixin, _EfficientAggregationMixin, BaseE (just-identified, single baseline, equivalent to CS). alpha : float, default 0.05 Significance level. - cluster : str or None + cluster : str, None or False + ``False`` (4.0, M-080) is an accepted explicit no-op: the variance is + unit-level by construction, so there is nothing to switch off. Column + labels must be ``str``. Column name for cluster-robust SEs. When set, analytical SEs use the Liang-Zeger clustered sandwich estimator on EIF values. With ``n_bootstrap > 0``, bootstrap weights are generated at the @@ -357,7 +366,7 @@ def __init__( self, pt_assumption: str = "all", alpha: float = 0.05, - cluster: Optional[str] = None, + cluster: ClusterArg = None, vcov_type: str = "hc1", control_group: str = "never_treated", n_bootstrap: int = 0, @@ -372,7 +381,7 @@ def __init__( ): self.pt_assumption = pt_assumption self.alpha = alpha - self.cluster = cluster + self.cluster = validate_cluster(cluster) self.vcov_type = vcov_type self.control_group = control_group validate_n_bootstrap(n_bootstrap) @@ -549,8 +558,14 @@ def fit( or PT-Post without a never-treated group. """ self._validate_params() - - if self.cluster is not None and survey_design is not None: + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]): __init__ and set_params validate eagerly, so this only + # catches DIRECT attribute mutation. `False` is normalized to None + # locally so it never reaches a column lookup ([M-080]). + validate_cluster(self.cluster) + user_cluster = None if self.cluster is False else self.cluster + + if user_cluster is not None and survey_design is not None: raise NotImplementedError( "cluster and survey_design cannot both be set. " "Use survey_design with PSU/strata for cluster-robust inference." @@ -709,9 +724,9 @@ def fit( self._unit_resolved_survey = None # Build cluster mapping if cluster-robust SEs requested - if self.cluster is not None: + if user_cluster is not None: unit_cluster_indices, n_clusters = _validate_and_build_cluster_mapping( - df, unit, self.cluster, all_units + df, unit, user_cluster, all_units ) if n_clusters < 50: warnings.warn( @@ -1356,12 +1371,12 @@ def _finalize_cell(g: Any, att_gt: float, eif_vals: np.ndarray) -> Dict[str, Any cluster_name=( None if resolved_survey is not None - else (self.cluster if self.cluster is not None else None) + else (user_cluster if user_cluster is not None else None) ), n_clusters=( None if resolved_survey is not None - else (n_clusters if self.cluster is not None else None) + else (n_clusters if user_cluster is not None else None) ), vcov_type=self.vcov_type, # Independent COPY, never the kit's arrays: the aggregation kit @@ -1461,7 +1476,7 @@ def hausman_pretest( time: str, first_treat: str, covariates: Optional[List[str]] = None, - cluster: Optional[str] = None, + cluster: ClusterArg = None, anticipation: int = 0, control_group: str = "never_treated", alpha: float = 0.05, @@ -1478,8 +1493,11 @@ def hausman_pretest( ---------- data, outcome, unit, time, first_treat, covariates Same as :meth:`fit`. - cluster : str, optional + cluster : str, None or False, optional Cluster column for cluster-robust covariance. + ``False`` (4.0, M-080) is an accepted explicit no-op: the variance is + unit-level by construction, so there is nothing to switch off. Column + labels must be ``str``. anticipation : int Anticipation periods. Must be a non-negative integer; ``bool`` is rejected. @@ -1500,6 +1518,11 @@ def hausman_pretest( # constructed estimators — validate and normalize it here so an # unsigned numpy scalar cannot wrap the comparison. anticipation = validate_anticipation(anticipation) + # Same shared contract for the classmethod's own cluster argument + # (str | None | False, [M-155]); the normalized column feeds the + # covariance branch below, the raw value the two constructors. + cluster = validate_cluster(cluster) + cluster_col = None if cluster is False else cluster # Fit under both assumptions (analytical SEs only, no bootstrap) common_kwargs = dict( @@ -1647,8 +1670,10 @@ def _aggregate_es( ) cl_idx: Optional[np.ndarray] = None n_cl: Optional[int] = None - if cluster is not None: - cl_idx, n_cl = _validate_and_build_cluster_mapping(data, unit, cluster, all_units_list) + if cluster_col is not None: + cl_idx, n_cl = _validate_and_build_cluster_mapping( + data, unit, cluster_col, all_units_list + ) if not np.all(row_finite): eif_all_mat = eif_all_mat[row_finite] eif_post_mat = eif_post_mat[row_finite] diff --git a/diff_diff/estimators.py b/diff_diff/estimators.py index 4f7ff4e0f..4ca149441 100644 --- a/diff_diff/estimators.py +++ b/diff_diff/estimators.py @@ -30,6 +30,7 @@ from diff_diff.results import DiDResults from diff_diff.results_base import _EventStudyCorePayload from diff_diff.utils import ( + ClusterArg, WildBootstrapResults, absorbed_fe_cr1_k_increment, absorbed_fe_rank, @@ -41,6 +42,7 @@ safe_inference, snap_absorbed_regressors, validate_binary, + validate_cluster, validate_covariate_names, validate_design_term_names, validate_df_convention, @@ -67,7 +69,11 @@ class DifferenceInDifferences(BaseEstimator): formula : str, optional R-style formula for the model (e.g., "outcome ~ treated * post"). If provided, overrides column name parameters. - cluster : str, optional + cluster : str, None or False, optional + ``False`` (4.0, M-080) is the uniform explicit off-switch and a no-op here: + the 2x2 fit is HC-robust unless ``cluster=`` is given. + ``inference="wild_bootstrap"`` rejects it (the resampler needs a cluster). + Column labels must be ``str``. Column name for cluster-robust standard errors. Combined with ``vcov_type``: with ``"hc1"`` dispatches to CR1 (Liang-Zeger); with ``"hc2_bm"`` dispatches to CR2 Bell-McCaffrey (Pustejovsky-Tipton 2018 @@ -226,7 +232,7 @@ class DifferenceInDifferences(BaseEstimator): def __init__( self, - cluster: Optional[str] = None, + cluster: ClusterArg = None, vcov_type: Optional[str] = None, alpha: float = 0.05, inference: str = "analytical", @@ -255,7 +261,7 @@ def __init__( if not isinstance(inference, str) or inference not in _INFERENCE_METHODS: raise ValueError(f"inference must be one of {_INFERENCE_METHODS}, got {inference!r}") - self.cluster = cluster + self.cluster = validate_cluster(cluster) self.vcov_type = resolve_vcov_type(vcov_type) # Preserve the raw constructor arg (possibly None) alongside the # resolved `vcov_type`. `get_params()` returns the raw arg so @@ -379,6 +385,12 @@ def fit( >>> did.fit(data, outcome='sales', treatment='treated', post='post', ... absorb=['firm_id']) """ + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]): __init__ and set_params validate eagerly, so this only + # catches DIRECT attribute mutation. `False` is normalized to None + # locally so it never reaches a column lookup ([M-080]). + validate_cluster(self.cluster) + user_cluster = None if self.cluster is False else self.cluster # Body-local name for the post-period indicator column; the public # parameter is `post`. time = post @@ -517,7 +529,7 @@ def fit( conley_lag_cutoff=self.conley_lag_cutoff, survey_design=survey_design, inference=self.inference, - cluster=self.cluster, + cluster=user_cluster, ) # Fail-closed wild-bootstrap coherence (M-096). Placed AFTER the @@ -526,11 +538,14 @@ def fit( # wild+Conley fit would be contradictory guidance — Conley rejects # the combination regardless of cluster). if self.inference == "wild_bootstrap": - if self.cluster is None: + if user_cluster is None: + # cluster=None and cluster=False both leave this fit unclustered + # (DiD has no auto-cluster), so neither can feed the resampler. raise ValueError( - "inference='wild_bootstrap' requires cluster=. The wild cluster " - "bootstrap resamples at the cluster level; pass cluster= or use " - "inference='analytical'." + "inference='wild_bootstrap' requires cluster=. The wild cluster " + "bootstrap resamples at the cluster level; cluster=None and " + "cluster=False both leave a DifferenceInDifferences fit unclustered, " + "so pass a cluster column or use inference='analytical'." ) if self.n_bootstrap < 2: raise ValueError( @@ -651,11 +666,11 @@ def fit( # Always use LinearRegression for initial fit (unified code path) # For wild bootstrap, we don't need cluster SEs from the initial fit - cluster_ids = data[self.cluster].values if self.cluster is not None else None + cluster_ids = data[user_cluster].values if user_cluster is not None else None # When survey PSU is present, it overrides cluster for variance estimation effective_cluster_ids = _resolve_effective_cluster( - resolved_survey, cluster_ids, self.cluster + resolved_survey, cluster_ids, user_cluster ) # Inject cluster as effective PSU for survey variance estimation @@ -851,7 +866,7 @@ def _refit_did_absorb(w_r): survey_metadata.df_survey = _df_rep if _df_rep > 0 else None t_stat, p_value, conf_int = safe_inference(att, se, alpha=self.alpha, df=_df_rep) _inference_df_used = float(_df_rep) if _df_rep is not None and _df_rep > 0 else None - elif self.inference == "wild_bootstrap" and self.cluster is not None: + elif self.inference == "wild_bootstrap" and user_cluster is not None: # Override with wild cluster bootstrap inference (bootstrap # test-inversion based; no reference t-distribution, so no # effective inference df). @@ -918,6 +933,12 @@ def _refit_did_absorb(w_r): n_bootstrap_used = self._bootstrap_results.n_bootstrap n_clusters_used = self._bootstrap_results.n_clusters p_val_type_used = self._bootstrap_results.p_val_type + elif resolved_survey is None: + # 4.0 (M-080): the analytical regression's own effective cluster + # count (None when it ran unclustered). Survey fits keep None - + # the survey metadata owns PSU reporting, and under a PSU design + # the regression received PSU labels, not the user's cluster. + n_clusters_used = reg.n_clusters_ # Store results self.results_ = DiDResults( @@ -944,7 +965,7 @@ def _refit_did_absorb(w_r): # the remapped "hc1" under a replicate-weight survey design, not # the stored `self.vcov_type`. vcov_type=_fit_vcov_type, - cluster_name=self.cluster, + cluster_name=user_cluster, conley_lag_cutoff=(self.conley_lag_cutoff if _fit_vcov_type == "conley" else None), df_convention=self.df_convention, inference_df=_inference_df_used, diff --git a/diff_diff/guides/llms-full.txt b/diff_diff/guides/llms-full.txt index f27de7bec..b3b41c946 100644 --- a/diff_diff/guides/llms-full.txt +++ b/diff_diff/guides/llms-full.txt @@ -53,7 +53,7 @@ Basic 2x2 Difference-in-Differences estimator. ```python DifferenceInDifferences( vcov_type: str | None = None, # Variance family: "hc1" (default), "classical", "hc2", "hc2_bm", "hc3", "conley" - cluster: str | None = None, # Column for cluster-robust SEs + cluster: str | None | False = None, # Column for cluster-robust SEs; False = explicitly unclustered (no-op here: 2x2 DiD is HC-robust unless cluster= is given) alpha: float = 0.05, # Significance level inference: str = "analytical", # "analytical" or "wild_bootstrap" (wild_bootstrap requires cluster=) n_bootstrap: int = 999, # Bootstrap replications; >= 2 under inference="wild_bootstrap" @@ -106,7 +106,7 @@ Two-Way Fixed Effects estimator for panel data. Inherits from DifferenceInDiffer ```python TwoWayFixedEffects( - cluster: str | None = None, # Auto-clusters at unit level if None + cluster: str | None | False = None, # Auto-clusters at unit level if None; False = plain HC sandwich (4.0 off-switch; rejected under wild bootstrap) alpha: float = 0.05, df_convention: str = "cluster", # Analytical t/p/CI df: "cluster" (G-1; default since 4.0), "residual" (n-K; 3.x default), or "normal" (z) ) @@ -143,7 +143,7 @@ results.print_summary() # Event-study mode (the MultiPeriodDiD class was removed at the 4.0 cut; # spec="pooled" runs its design; wild bootstrap raises in this mode; the -# unit auto-cluster applies with the static carve-outs) +# unit auto-cluster applies with the static carve-outs; cluster=False switches it off) es = twfe.fit(data, outcome='y', treatment='treated', unit='unit_id', event_study=True, time='period', post_periods=[4, 5, 6, 7]) print(es.summary()) @@ -189,7 +189,7 @@ CallawaySantAnna( anticipation: int = 0, # Anticipation periods (non-negative integer; bool rejected) estimation_method: str = "dr", # "dr", "ipw", or "reg" alpha: float = 0.05, - cluster: str | None = None, # Cluster col; activates CR1 on the IF via synthesized SurveyDesign(psu=col). None → per-unit IF. + cluster: str | None | False = None, # Cluster col; activates CR1 on the IF via synthesized SurveyDesign(psu=col). None → per-unit IF; False = accepted no-op (unit-level by construction) n_bootstrap: int = 0, # 0 = analytical SEs, 999+ recommended bootstrap_weights: str | None = None, # "rademacher", "mammen", or "webb" seed: int | None = None, @@ -267,7 +267,7 @@ de Chaisemartin & D'Haultfœuille (2020/2022) estimator for **non-absorbing (rev ```python ChaisemartinDHaultfoeuille( alpha: float = 0.05, - cluster: str | None = None, # Must be None; non-None raises NotImplementedError + cluster: str | None | False = None, # None or False (both the default group-level IF path; False is an accepted no-op); a column name raises NotImplementedError n_bootstrap: int = 0, # 0 = analytical SE only bootstrap_weights: str = "rademacher", # "rademacher", "mammen", or "webb" seed: int | None = None, @@ -366,7 +366,7 @@ SunAbraham( control_group: str = "never_treated", # "never_treated" or "not_yet_treated" anticipation: int = 0, # non-negative integer; bool rejected alpha: float = 0.05, - cluster: str | None = None, # Defaults to unit-level clustering (dropped on explicit vcov_type='hc2' / 'classical') + cluster: str | None | False = None, # Defaults to unit-level clustering (dropped on explicit vcov_type='hc2' / 'classical'); False = plain HC sandwich (4.0 off-switch; rejected with hc2_bm) n_bootstrap: int = 0, # 0 = analytical cluster-robust SEs seed: int | None = None, rank_deficient_action: str = "warn", @@ -414,7 +414,7 @@ Borusyak-Jaravel-Spiess (2024) imputation DiD estimator. Efficient estimator pro ImputationDiD( anticipation: int = 0, # non-negative integer; bool rejected alpha: float = 0.05, - cluster: str | None = None, # Defaults to unit-level clustering + cluster: str | None | False = None, # Defaults to unit-level clustering; False = accepted no-op (unit-level IF variance) vcov_type: str = "hc1", # {"hc1"} only — IF-based variance per Borusyak et al. (2024) Theorem 3 n_bootstrap: int = 0, # 0 = analytical (Theorem 3 variance) bootstrap_weights: str = "rademacher", # "rademacher", "mammen", or "webb" @@ -467,7 +467,7 @@ Gardner (2022) two-stage DiD estimator. Point estimates match ImputationDiD; use TwoStageDiD( anticipation: int = 0, # non-negative integer; bool rejected alpha: float = 0.05, - cluster: str | None = None, + cluster: str | None | False = None, # Defaults to unit-level clustering; False = accepted no-op (unit-level IF variance) n_bootstrap: int = 0, bootstrap_weights: str = "rademacher", seed: int | None = None, @@ -514,12 +514,12 @@ Butts (2021) ring-indicator spillover-aware DiD. Augments two-stage Gardner with SpilloverDiD( rings: list[float], # K+1 sorted breakpoints; K rings d_bar: float | None = None, # Far-away cutoff (defaults to max(rings)) - vcov_type: str = "hc1", # "hc1", "conley", or default cluster + vcov_type: str = "hc1", # "hc1" (CR1 at unit by default since 4.0) or "conley" conley_coords: tuple[str, str] | None = None, # (lat_col, lon_col), required conley_metric: str = "haversine", # or "euclidean" / callable conley_cutoff_km: float | None = None, conley_lag_cutoff: int | None = None, - cluster: str | None = None, + cluster: str | None | False = None, # None = auto-cluster at unit (4.0; dropped on Conley, never a survey PSU); False = observation-level HC1 (the 3.x default); str = that column alpha: float = 0.05, anticipation: int = 0, # non-negative integer; bool rejected event_study: bool = False, # Wave C: per-event-time × ring decomposition (Butts Table 2) @@ -680,7 +680,7 @@ Triple Difference (DDD) estimator following Ortiz-Villavicencio & Sant'Anna (202 ```python TripleDifference( estimation_method: str = "dr", # "dr", "reg", or "ipw" - cluster: str | None = None, + cluster: str | None | False = None, # 2x2x2: CR1 on that column (False = plain HC, a no-op); staggered mode: a column raises, False is a no-op vcov_type: str = "hc1", # {"hc1"} only — IF-based variance per Ortiz-Villavicencio & Sant'Anna (2025). Analytical-sandwich {classical, hc2, hc2_bm} and conley REJECTED at __init__ (see REGISTRY.md IF-vs-sandwich subsection). alpha: float = 0.05, pscore_trim: float = 0.01, @@ -828,7 +828,7 @@ HeterogeneousAdoptionDiD( kernel: str = "epanechnikov", # Local-linear kernel alpha: float = 0.05, vcov_type: str | None = None, # Mass-point only: "classical" (default) or "hc1" - cluster: str | None = None, # Cluster column for cluster-robust SEs (all designs; event-study adds a cluster-robust sup-t band). cluster= + survey_design= rejected + cluster: str | None | False = None, # Cluster column for cluster-robust SEs (all designs; event-study adds a cluster-robust sup-t band); False = accepted no-op. cluster= + survey_design= rejected n_bootstrap: int = 999, # Multiplier-bootstrap iterations for sup-t bands (event-study, when weighted/survey OR clustered) seed: int | None = None, ) @@ -1146,7 +1146,7 @@ Efficient DiD estimator (Chen, Sant'Anna & Xie 2025). Achieves the semiparametri EfficientDiD( pt_assumption: str = "all", # "all" (overidentified) or "post" (just-identified) alpha: float = 0.05, - cluster: str | None = None, # Column name for cluster-robust SEs (Liang-Zeger on EIF values); cluster-level multiplier bootstrap when n_bootstrap > 0 + cluster: str | None | False = None, # Column name for cluster-robust SEs (Liang-Zeger on EIF values); cluster-level multiplier bootstrap when n_bootstrap > 0; False = accepted no-op vcov_type: str = "hc1", # {"hc1"} only — IF-based variance per Chen-Sant'Anna-Xie (2025); analytical-sandwich families {classical, hc2, hc2_bm} and conley rejected at __init__/set_params n_bootstrap: int = 0, # Multiplier bootstrap iterations bootstrap_weights: str = "rademacher", # "rademacher", "mammen", or "webb" @@ -1200,7 +1200,7 @@ LPDiD( no_composition: bool = False, # Hold the post-treatment composition fixed across post horizons pmd: str | int | None = None, # Base period: None=first-lag (t-1), "max"=premean over all pretreatment periods, int=last-k premean alpha: float = 0.05, - cluster: str | None = None, # Cluster column for cluster-robust SEs; defaults to the unit identifier + cluster: str | None | False = None, # Cluster column for cluster-robust SEs; defaults to the unit identifier; False = unclustered per-horizon regressions (4.0 off-switch) rank_deficient_action: str = "warn", # "warn", "error", or "silent" non_absorbing: str | None = None, # None=absorbing; "first_entry" (Eq. 12); "effect_stabilization" (Eq. 13) stabilization_window: int | None = None, # The paper's L; required when non_absorbing="effect_stabilization" @@ -1312,7 +1312,7 @@ LWDiD( rolling: str = "demean", # "demean", "detrend", "demeanq", "detrendq" (q = seasonal variants). Demeaning needs >= 1 pre period per cohort; detrending >= 2. Detrending stays consistent under unit-specific linear trends (CHT) where demeaning is inconsistent. estimation_method: str = "reg", # "reg" (papers' RA), "ipw", "dr" (papers' IPWRA, doubly robust), "psm" (LWDiD-only extra; POINT ONLY - inference is NaN, see below) vcov_type: str = "hc1", # reg: {"classical", "hc1", "hc2", "hc3"} - "classical" enables the exact small-N t; ipw/dr: {"hc1"} only (influence-function variance); psm: {"hc1"} as configuration only. NO "cluster" value - clustering activates via cluster= below. - cluster: str | None = None, # Unit-constant cluster column; CR1, composes with vcov_type="hc1" only; rejected for psm. + cluster: str | None | False = None, # Unit-constant cluster column; CR1, composes with vcov_type="hc1" only; rejected for psm; False = accepted no-op control_group: str = "not_yet_treated", # "not_yet_treated" (OVLS default) or "never_treated". All-eventually-treated designs raise under BOTH. alpha: float = 0.05, n_bootstrap: int = 0, # 0 = analytical. Common timing: unit/cluster bootstrap for the HEADLINE inference AND a Rademacher multiplier bootstrap with sup-t bands for the post-period event-study surface. Staggered: multiplier bootstrap for the EVENT-STUDY surface only (overall/cohort stay analytical). Rejected (>0) for psm - the standard bootstrap is invalid for matching (Abadie & Imbens 2008). @@ -1397,7 +1397,7 @@ DMLDiD( cband=True, # sup-t bands on the event-study replay pscore_trim=0.01, # clip fitted propensities (never drop) panel=True, # False = declared repeated cross sections (Chang Case 2) - cluster=None, # coarser-than-unit clustering (variance/folds only) + cluster=None, # str | None | False: coarser-than-unit clustering (variance/folds only); False = accepted no-op ).fit(data, outcome, unit, time, first_treat, covariates, survey_design=None, bad_control=None, # Caetano et al. (2026) bad-control lane: a covariate treatment can affect (NOT in covariates) bad_control_covariates=None, # W for the bad control's unconfoundedness model; [outcome] = base-period outcome (Remark 5) @@ -1591,7 +1591,7 @@ WooldridgeDiD( anticipation: int = 0, # Number of anticipation periods (non-negative integer; bool rejected) demean_covariates: bool = True, # Demean covariates within cohort*period cells alpha: float = 0.05, - cluster: str | None = None, + cluster: str | None | False = None, # Defaults to unit-level clustering; False = plain HC / QMLE sandwich (4.0 off-switch; rejected with n_bootstrap > 0 and hc2_bm) n_bootstrap: int = 0, bootstrap_weights: str = "rademacher", seed: int | None = None, @@ -1768,7 +1768,7 @@ Returned by `DifferenceInDifferences.fit()` and `TwoWayFixedEffects.fit()`. | `r_squared` | `float` | R-squared | | `inference_method` | `str` | "analytical" or "wild_bootstrap" | | `n_bootstrap` | `int` | Number of bootstrap replications | -| `n_clusters` | `int` | Number of clusters | +| `n_clusters` | `int | None` | Number of clusters (the regression's effective count on non-survey analytical fits and the bootstrap count under wild; None on unclustered and survey fits) | | `bootstrap_distribution` | `np.ndarray` | Bootstrap ATT distribution | **Methods:** `summary(alpha=None)`, `print_summary()`, `to_dict()`, `to_dataframe()` @@ -1834,6 +1834,8 @@ Returned by `SunAbraham.fit()`. | `cohort_effects` | `dict` | Cohort-level effects | | `df_convention` | `str | None` | The configured df convention echoed onto the results ("residual" | "cluster" | "normal"; 3.9) | | `inference_df` | `float | None` | The df the overall-ATT p/CI actually used (BM contrast df / survey df / knob-resolved fallback; None under bootstrap or "normal") | +| `cluster_name` | `str | None` | Cluster ids the analytical regression received (`"unit"` on the default, the explicit column; 4.0); None on unclustered, survey and pairs-bootstrap fits | +| `n_clusters` | `int | None` | Effective count of those clusters (4.0, appended field); None when `cluster_name` is None | | `event_study_df` | `dict[int, float] | None` | Per-event-time df provenance (G-1 on plain clustered analytic fits under the 4.0 default; the finite residual df under `df_convention="residual"`; None under bootstrap) | **Methods:** `summary()`, `print_summary()`, `to_dataframe(level="event_study"|"cohort")` diff --git a/diff_diff/had.py b/diff_diff/had.py index 8e513fdb4..d82a71b47 100644 --- a/diff_diff/had.py +++ b/diff_diff/had.py @@ -89,7 +89,7 @@ SurveyMetadata, compute_survey_metadata, ) -from diff_diff.utils import safe_inference, validate_n_bootstrap +from diff_diff.utils import ClusterArg, safe_inference, validate_cluster, validate_n_bootstrap __all__ = [ "HeterogeneousAdoptionDiD", @@ -2847,7 +2847,10 @@ class HeterogeneousAdoptionDiD(BaseEstimator): the CCT-2014 robust SE from Phase 1c); passing a non-default ``vcov_type`` on a continuous path emits a ``UserWarning`` per fit call. - cluster : str or None + cluster : str, None or False + ``False`` (4.0, M-080) is an accepted explicit no-op: the variance is + unit-level by construction, so there is nothing to switch off. Column + labels must be ``str``. Column name for cluster-robust SE. On the mass-point path this is the 2SLS CR1 sandwich; on the continuous (``continuous_at_zero`` / ``continuous_near_d_lower``) paths (Phase 2a) it threads the cluster @@ -2958,7 +2961,7 @@ def __init__( kernel: str = "epanechnikov", alpha: float = 0.05, vcov_type: Optional[str] = None, - cluster: Optional[str] = None, + cluster: ClusterArg = None, n_bootstrap: int = 999, seed: Optional[int] = None, ) -> None: @@ -2967,7 +2970,7 @@ def __init__( self.kernel = kernel self.alpha = alpha self.vcov_type = vcov_type - self.cluster = cluster + self.cluster = validate_cluster(cluster) # Event-study sup-t simultaneous-CI support. ``n_bootstrap`` = # number of multiplier-bootstrap replicates for the sup-t band; # ``seed`` = reproducibility seed for the multiplier draws. Both are @@ -3297,7 +3300,8 @@ def fit( design_arg = self.design d_lower_arg = self.d_lower vcov_type_arg = self.vcov_type - cluster_arg = self.cluster + validate_cluster(self.cluster) # shared contract re-check ([M-155]) + cluster_arg = None if self.cluster is False else self.cluster # False = no-op # ---- Validate panel contract ---- t_pre, t_post = _validate_had_panel( @@ -4124,7 +4128,8 @@ def _fit_event_study( design_arg = self.design d_lower_arg = self.d_lower vcov_type_arg = self.vcov_type - cluster_arg = self.cluster + validate_cluster(self.cluster) # shared contract re-check ([M-155]) + cluster_arg = None if self.cluster is False else self.cluster # False = no-op n_bootstrap_eff = int(self.n_bootstrap) seed_eff = None if self.seed is None else int(self.seed) diff --git a/diff_diff/imputation.py b/diff_diff/imputation.py index 6a0cb435b..5a96f0a6c 100644 --- a/diff_diff/imputation.py +++ b/diff_diff/imputation.py @@ -46,8 +46,10 @@ ImputationDiDResults, ) from diff_diff.utils import ( + ClusterArg, safe_inference, validate_anticipation, + validate_cluster, validate_df_convention, validate_n_bootstrap, ) @@ -85,7 +87,10 @@ class ImputationDiD(ImputationDiDBootstrapMixin, _ImputationAggregationMixin, Ba Must be a non-negative integer; ``bool`` is rejected. alpha : float, default=0.05 Significance level for confidence intervals. - cluster : str, optional + cluster : str, None or False, optional + ``False`` (4.0, M-080) is an accepted explicit no-op: the default + ``cluster_var = unit`` lives inside the unit-level influence-function + variance, so no unclustered variance exists. Column labels must be ``str``. Column name for cluster-robust standard errors. If None, clusters at the unit level by default. vcov_type : str, default="hc1" @@ -210,7 +215,7 @@ def __init__( self, anticipation: int = 0, alpha: float = 0.05, - cluster: Optional[str] = None, + cluster: ClusterArg = None, vcov_type: str = "hc1", n_bootstrap: int = 0, bootstrap_weights: str = "rademacher", @@ -243,7 +248,7 @@ def __init__( self.anticipation = validate_anticipation(anticipation) self.alpha = alpha - self.cluster = cluster + self.cluster = validate_cluster(cluster) self.vcov_type = vcov_type validate_n_bootstrap(n_bootstrap) self.n_bootstrap = n_bootstrap @@ -316,6 +321,12 @@ def fit( # out-of-domain value silently changes the ESTIMAND); the assignment # also re-normalizes a mutated numpy scalar to a Python int. self.anticipation = validate_anticipation(self.anticipation) + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]): __init__ and set_params validate eagerly, so this only + # catches DIRECT attribute mutation. `False` is normalized to None + # locally so it never reaches a column lookup ([M-080]). + validate_cluster(self.cluster) + user_cluster = None if self.cluster is False else self.cluster # Validate inputs required_cols = [outcome, unit, time, first_treat] @@ -387,12 +398,12 @@ def fit( # variance while populating cluster_name/n_clusters on Results # dishonestly. Fail-closed mirroring CallawaySantAnna. if ( - self.cluster is not None + user_cluster is not None and survey_design is not None and getattr(survey_design, "replicate_weights", None) is not None ): raise NotImplementedError( - f"ImputationDiD(cluster={self.cluster!r}) is not supported " + f"ImputationDiD(cluster={user_cluster!r}) is not supported " "with replicate-weight survey designs. Replicate-weight " "variance is computed by replicate reweighting (BRR / Fay / " "JK1 / JKn / SDR) and ignores PSU/cluster entirely — setting " @@ -518,10 +529,10 @@ def fit( n_control_units = len(units_in_omega_0) # Cluster variable - cluster_var = self.cluster if self.cluster is not None else unit - if self.cluster is not None and self.cluster not in df.columns: + cluster_var = user_cluster if user_cluster is not None else unit + if user_cluster is not None and user_cluster not in df.columns: raise ValueError( - f"Cluster column '{self.cluster}' not found in data. " + f"Cluster column '{user_cluster}' not found in data. " f"Available columns: {list(df.columns)}" ) @@ -531,7 +542,7 @@ def fit( effective_cluster_ids = _resolve_effective_cluster( resolved_survey, cluster_ids_raw, - cluster_var if self.cluster is not None else None, + cluster_var if user_cluster is not None else None, ) resolved_survey = _inject_cluster_as_psu(resolved_survey, effective_cluster_ids) # When survey PSU is present, use it as the effective cluster for @@ -999,9 +1010,9 @@ def _precompute_psi( if resolved_survey is not None: _cluster_name_for_results: Optional[str] = None _n_clusters_for_results: Optional[int] = None - elif self.cluster is not None: - _cluster_name_for_results = self.cluster - _n_clusters_for_results = int(data[self.cluster].nunique()) + elif user_cluster is not None: + _cluster_name_for_results = user_cluster + _n_clusters_for_results = int(data[user_cluster].nunique()) else: _cluster_name_for_results = unit _n_clusters_for_results = int(data[unit].nunique()) diff --git a/diff_diff/lpdid.py b/diff_diff/lpdid.py index c09ea372c..d2fd7ee37 100644 --- a/diff_diff/lpdid.py +++ b/diff_diff/lpdid.py @@ -8,10 +8,12 @@ from diff_diff.linalg import InvalidClusterKAdjustment, _rank_guarded_inv, solve_ols from diff_diff.lpdid_results import LPDiDResults from diff_diff.utils import ( + ClusterArg, absorbed_fe_rank, cluster_nested_fe_dims, resolve_tail_df, safe_inference, + validate_cluster, validate_df_convention, ) @@ -28,7 +30,7 @@ def __init__( no_composition: bool = False, pmd: Optional[Union[str, int]] = None, alpha: float = 0.05, - cluster: Optional[str] = None, + cluster: ClusterArg = None, rank_deficient_action: str = "warn", non_absorbing: Optional[str] = None, stabilization_window: Optional[int] = None, @@ -41,7 +43,7 @@ def __init__( self.no_composition = no_composition self.pmd = pmd self.alpha = alpha - self.cluster = cluster + self.cluster = validate_cluster(cluster) self.rank_deficient_action = rank_deficient_action self.non_absorbing = non_absorbing self.stabilization_window = stabilization_window @@ -710,12 +712,13 @@ def _estimate_regression_adjustment_sample( effect = float(treated_residual.mean()) se = np.nan cluster_ids = sample["_cluster"].to_numpy() - n_clusters = len(pd.unique(cluster_ids)) + unclustered = self.cluster is False + n_clusters: Optional[int] = None if unclustered else len(pd.unique(cluster_ids)) # Hoisted above the branch: the "residual" df resolution below needs # n_total on every lane (including G>=2 with a non-finite effect, # where today's G-1 df is still reported alongside the NaN se). n_total = len(sample) - if n_clusters >= 2 and np.isfinite(effect): + if (unclustered or (n_clusters is not None and n_clusters >= 2)) and np.isfinite(effect): n_treated = len(treated) q0_inv, _, _ = _rank_guarded_inv(control_design.T @ control_design) mu_treated = treated_design.mean(axis=0) @@ -729,8 +732,14 @@ def _estimate_regression_adjustment_sample( ) phi[~treated_mask] = -n_total * control_projection * control_residual - cluster_scores = pd.Series(phi).groupby(cluster_ids).sum().to_numpy(dtype=float) - vcov_scalar = float(cluster_scores @ cluster_scores) / float(n_total**2) + if unclustered: + # 4.0 (M-080): cluster=False - the unclustered IF variance + # sum(phi_i^2) / n^2 (no small-sample factor, like the clustered + # lane; REGISTRY LPDiD Note 2). + vcov_scalar = float(phi @ phi) / float(n_total**2) + else: + cluster_scores = pd.Series(phi).groupby(cluster_ids).sum().to_numpy(dtype=float) + vcov_scalar = float(cluster_scores @ cluster_scores) / float(n_total**2) if np.isfinite(vcov_scalar) and vcov_scalar >= 0: se = float(np.sqrt(vcov_scalar)) @@ -745,11 +754,11 @@ def _estimate_regression_adjustment_sample( # kept nuisance coefficients PLUS the ATT (library convention, no # external anchor - Stata teffects ra reports z; see the REGISTRY # LPDiD note for the derivation). - if n_clusters >= 2: + if unclustered or (n_clusters is not None and n_clusters >= 2): df = resolve_tail_df( self.df_convention, residual_df=float(n_total - k0_kept - 1), - n_clusters=n_clusters, + n_clusters=None if unclustered else n_clusters, ) else: df = None @@ -863,6 +872,7 @@ def _estimate_sample( design = np.column_stack(design_columns) response = sample[response_column].to_numpy(dtype=float) cluster_ids = sample["_cluster"].to_numpy() + unclustered = self.cluster is False weights = None if weight_column is None else sample[weight_column].to_numpy(dtype=float) if survey_design is not None: # Complex-survey path: WLS point estimate weighted by the survey design, @@ -889,13 +899,25 @@ def _estimate_sample( "conf_low": np.nan, "conf_high": np.nan, "n_obs": n_obs, - "n_clusters": len(pd.unique(cluster_ids)), + "n_clusters": None if unclustered else len(pd.unique(cluster_ids)), "df": np.nan, } - use_cluster_vcov = len(pd.unique(cluster_ids)) >= 2 + use_cluster_vcov = (not unclustered) and len(pd.unique(cluster_ids)) >= 2 vcov = None - if use_cluster_vcov: + if unclustered: + # 4.0 (M-080): cluster=False - the plain HC1 sandwich on this explicit- + # dummy design (no absorbed FE here, so no full-K rescale is needed); + # the tail df is the residual df below. + coef, _, vcov = solve_ols( + design, + response, + return_vcov=True, + rank_deficient_action=self.rank_deficient_action, + column_names=column_names, + weights=weights, + ) + elif use_cluster_vcov: # Clustered-CR1 K_reference adjustment (variance-conventions.md # D1 family): LPDiD's built-in `_event_time` dummies ARE its time # FE (the authors' reference recipe absorbs time FE), and inline @@ -954,7 +976,7 @@ def _estimate_sample( if vcov is not None and vcov.shape[0] > 1 and np.isfinite(vcov[1, 1]) and vcov[1, 1] >= 0: se = float(np.sqrt(vcov[1, 1])) - n_clusters = len(pd.unique(cluster_ids)) + n_clusters: Optional[int] = None if unclustered else len(pd.unique(cluster_ids)) # Tail df: the resolver runs only on the healthy clustered lane — # matching today's expression `n_clusters - 1 if vcov is not None # and n_clusters > 1 else None` — so the default "cluster" @@ -965,7 +987,7 @@ def _estimate_sample( # above, or G<=1) keep the literal df=None (normal theory + NaN # provenance) under ALL conventions and never touch the resolver. # "residual" = n_eff - k_kept of this per-horizon design. - if vcov is not None and n_clusters >= 2: + if vcov is not None and (unclustered or (n_clusters is not None and n_clusters >= 2)): _n_eff_lp = ( int(np.count_nonzero(np.asarray(weights, dtype=float) > 0)) if weights is not None @@ -974,7 +996,7 @@ def _estimate_sample( df = resolve_tail_df( self.df_convention, residual_df=float(_n_eff_lp - int(np.count_nonzero(np.isfinite(coef)))), - n_clusters=n_clusters, + n_clusters=None if unclustered else n_clusters, ) else: df = None @@ -1015,17 +1037,21 @@ def _estimate_survey_sample(self, sample, design, response, column_names, n_obs, "ignore", message="pweight weights normalized", category=UserWarning ) resolved = survey_design.resolve(sample) - if resolved.psu is None: + unclustered = self.cluster is False + if resolved.psu is None and not unclustered: # No explicit PSU: cluster LP-DiD's unit (the default cluster) as the # effective PSU. The rebind is load-bearing -- _inject_cluster_as_psu # returns a NEW object; dropping it would leave each observation its own - # PSU and silently deflate the SE. + # PSU and silently deflate the SE. Under cluster=False (4.0, M-080) + # that implicit per-observation PSU IS the requested off-switch. resolved = _inject_cluster_as_psu(resolved, cluster_ids) - else: + elif resolved.psu is not None: # Explicit survey PSU wins; warn only if the user ALSO set an explicit - # cluster that partitions differently (self.cluster is None under the - # default unit clustering, which suppresses the warning). - _resolve_effective_cluster(resolved, cluster_ids, cluster_name=self.cluster) + # cluster that partitions differently (None under the default unit + # clustering and under cluster=False, which suppresses the warning). + _resolve_effective_cluster( + resolved, cluster_ids, cluster_name=None if unclustered else self.cluster + ) weights = resolved.weights # solve_ols returns the original-scale residuals computed on the IDENTIFIED @@ -1075,7 +1101,7 @@ def _estimate_survey_sample(self, sample, design, response, column_names, n_obs, "conf_low": conf_int[0], "conf_high": conf_int[1], "n_obs": n_obs, - "n_clusters": int(resolved.n_psu), + "n_clusters": None if unclustered else int(resolved.n_psu), # df provenance for the unified surface: recorded iff the value # governed a t-reference (finite, > 0; the df<=0 sentinel and # None both yield non-t inference). @@ -1404,7 +1430,7 @@ def fit( required.extend(covariates) if absorb: required.extend(absorb) - if self.cluster: + if isinstance(self.cluster, str): required.append(self.cluster) missing = [col for col in required if col not in data.columns] if missing: @@ -1449,12 +1475,19 @@ def fit( stacklevel=2, ) - cluster = self.cluster or unit + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]). cluster=False (4.0, M-080) is the explicit off-switch: every + # per-horizon regression runs unclustered (plain HC1 / unclustered IF + # variance / implicit per-observation survey PSUs); `panel["_cluster"]` + # keeps the unit for bookkeeping only and is never read on that lane. + validate_cluster(self.cluster) + unclustered = self.cluster is False + cluster = unit if (self.cluster is None or self.cluster is False) else self.cluster survey_metadata = None survey_n_strata = None survey_n_psu = None - survey_cluster_name = cluster + survey_cluster_name: Optional[str] = None if unclustered else cluster if survey_design is not None: from diff_diff.survey import ( SurveyDesign, @@ -1497,7 +1530,7 @@ def fit( # per-observation PSUs). The per-horizon SE uses each realized sample's own # resolved design (see _estimate_survey_sample); the headline G in the # SE-family line is the realized pooled-post PSU count. - if resolved_panel.psu is None: + if resolved_panel.psu is None and not unclustered: resolved_panel = _inject_cluster_as_psu(resolved_panel, data[cluster].to_numpy()) raw_weights = ( data[survey_design.weights].to_numpy(dtype=float) @@ -1506,8 +1539,15 @@ def fit( ) survey_metadata = compute_survey_metadata(resolved_panel, raw_weights) survey_n_strata = int(resolved_panel.n_strata) - survey_n_psu = int(resolved_panel.n_psu) - survey_cluster_name = survey_design.psu or cluster + # Under cluster=False no PSU was injected: follow the metadata's + # implicit per-observation-PSU convention (n_psu = n_obs; the + # resolved design itself carries n_psu=0 when psu is None). + survey_n_psu = ( + int(survey_metadata.n_psu) + if (unclustered and survey_metadata.n_psu is not None) + else int(resolved_panel.n_psu) + ) + survey_cluster_name = survey_design.psu or (None if unclustered else cluster) pre_horizons = self._resolve_pooled_horizons(pre_pooled, kind="pre") post_horizons = self._resolve_pooled_horizons(post_pooled, kind="post") @@ -1639,7 +1679,7 @@ def fit( # Headline G = realized clusters in the pooled-post (headline ATT) sample # when computed; otherwise the panel-level cluster count. Per-horizon rows # carry their own realized n_clusters in the event_study/pooled tables. - headline_n_clusters = int(panel["_cluster"].nunique()) + headline_n_clusters: Optional[int] = int(panel["_cluster"].nunique()) if survey_n_psu is not None: # Under a survey design the effective variance cluster is the PSU, and # cluster_name reports the PSU column. Use the panel-level PSU count so the @@ -1651,6 +1691,9 @@ def fit( _post = pooled.loc[pooled["window"] == "post", "n_clusters"] if not _post.empty and pd.notna(_post.iloc[0]): headline_n_clusters = int(_post.iloc[0]) + if unclustered: + # 4.0 (M-080): an unclustered fit reports no cluster count. + headline_n_clusters = None self.results_ = LPDiDResults( event_study=event_study, @@ -1674,7 +1717,7 @@ def fit( "survey_tsl" if survey_design is not None else ( - "if_cluster" + ("if_robust" if unclustered else "if_cluster") if (self.reweight and bool(covariates or ylags or dylags or absorb)) else "hc1" ) diff --git a/diff_diff/lpdid_results.py b/diff_diff/lpdid_results.py index 4c95d6fa9..36c5c0a88 100644 --- a/diff_diff/lpdid_results.py +++ b/diff_diff/lpdid_results.py @@ -199,7 +199,9 @@ def to_dict(self) -> Dict[str, Any]: if self.df_convention is not None: result["df_convention"] = self.df_convention result["inference_method"] = ( - "survey_tsl" if self.vcov_type == "survey_tsl" else "cluster_robust" + "survey_tsl" + if self.vcov_type == "survey_tsl" + else ("heteroskedasticity_robust" if self.cluster_name is None else "cluster_robust") ) return result @@ -262,6 +264,10 @@ def _fmt(x: Any, nd: int = 4) -> str: # (ImputationDiD/BJS family), not an OLS CR1 sandwich. g = f", G={self.n_clusters}" if self.n_clusters else "" vcov_label = f"Influence-function cluster-robust at {self.cluster_name}{g}" + elif self.vcov_type == "if_robust": + # Regression-adjustment path under cluster=False (4.0, M-080): the + # unclustered influence-function variance sum(phi_i^2)/n^2. + vcov_label = "Influence-function heteroskedasticity-robust (unclustered)" elif self.vcov_type == "survey_tsl": # Complex-survey path: stratified-PSU Taylor-linearization (Binder TSL) # sandwich. _format_vcov_label does not know "survey_tsl", so build the diff --git a/diff_diff/lwdid.py b/diff_diff/lwdid.py index a78eb684a..11ed837aa 100644 --- a/diff_diff/lwdid.py +++ b/diff_diff/lwdid.py @@ -27,7 +27,13 @@ from diff_diff._base import BaseEstimator from diff_diff.linalg import _detect_rank_deficiency, solve_logit, solve_ols from diff_diff.lwdid_results import LWDiDResults -from diff_diff.utils import safe_inference, validate_binary, validate_pscore_trim +from diff_diff.utils import ( + ClusterArg, + safe_inference, + validate_binary, + validate_cluster, + validate_pscore_trim, +) _VALID_ROLLING = ("demean", "detrend", "demeanq", "detrendq") _VALID_ESTIMATION_METHODS = ("reg", "ipw", "dr", "psm") @@ -444,7 +450,10 @@ class LWDiD(BaseEstimator): Cluster-robust (CR1) inference activates via the ``cluster=`` parameter, not through a ``vcov_type`` value, composes only with 'hc1', and is rejected for 'psm'. - cluster : str or None, default None + cluster : str, None or False, default None + ``False`` (4.0, M-080) is an accepted explicit no-op: the variance is + unit-level by construction, so there is nothing to switch off. Column + labels must be ``str``. Column name for cluster-robust (CR1) standard errors. When set, clustered inference is active for the whole fit. control_group : {'never_treated', 'not_yet_treated'}, default 'not_yet_treated' @@ -558,7 +567,7 @@ def __init__( rolling: str = "demean", estimation_method: str = "reg", vcov_type: str = "hc1", - cluster: Optional[str] = None, + cluster: ClusterArg = None, control_group: str = "not_yet_treated", alpha: float = 0.05, n_bootstrap: int = 0, @@ -588,7 +597,10 @@ def __init__( ) if vcov_type not in _VALID_VCOV_TYPES: raise ValueError(f"vcov_type must be one of {_VALID_VCOV_TYPES}, got '{vcov_type}'") - self._validate_vcov_config(vcov_type, estimation_method, cluster) + cluster = validate_cluster(cluster) + self._validate_vcov_config( + vcov_type, estimation_method, None if cluster is False else cluster + ) # Validate control_group if control_group not in _VALID_CONTROL_GROUPS: raise ValueError( @@ -699,7 +711,10 @@ def fit( """ # --- Input validation --- df = data.copy() - cluster = self.cluster + # Fit-time re-check of the shared cluster contract ([M-155]); False is + # an accepted no-op here (category b) and is normalized to None. + validate_cluster(self.cluster) + cluster = None if self.cluster is False else self.cluster # Re-check the vcov configuration at fit time (set_params probe # re-init covers most mutations; this closes direct-attribute edits). self._validate_vcov_config(self.vcov_type, self.estimation_method, cluster) @@ -1010,7 +1025,7 @@ def _validate_inputs( required_cols = [outcome, unit, time, treatment] if cohort is not None: required_cols.append(cohort) - if cluster is not None: + if isinstance(cluster, str): required_cols.append(cluster) if controls: required_cols.extend(controls) @@ -1167,7 +1182,7 @@ def _validate_inputs( "column to one value per unit (e.g. its pre-treatment " "value) before fitting." ) - if cluster is not None and cluster != unit: + if isinstance(cluster, str) and cluster != unit: n_missing = int(df[cluster].isna().sum()) if n_missing > 0: raise ValueError( @@ -3980,7 +3995,7 @@ def _resolve_vcov_type(self) -> str: # Post fix-wave WS6, the config validator guarantees cluster only # composes with hc1, so the requested family IS the resolved family # on every path (no silent remap can occur). - if self.cluster is not None: + if isinstance(self.cluster, str): assert self.vcov_type == "hc1", "validator invariant violated" return self.vcov_type diff --git a/diff_diff/lwdid_sensitivity.py b/diff_diff/lwdid_sensitivity.py index 2f64da114..6768c359b 100644 --- a/diff_diff/lwdid_sensitivity.py +++ b/diff_diff/lwdid_sensitivity.py @@ -34,6 +34,8 @@ import numpy as np import pandas as pd +from diff_diff.utils import ClusterArg, validate_cluster + # ============================================================================= # Constants # ============================================================================= @@ -318,7 +320,7 @@ def _fit_single_spec( } if cohort is not None: required["cohort"] = cohort - if cluster is not None: + if isinstance(cluster, str): required["cluster"] = cluster missing = [f"{role}={name!r}" for role, name in required.items() if name not in data.columns] if controls is not None: @@ -451,7 +453,7 @@ def robustness_pre_periods( rolling: str = "demean", estimation_method: str = "reg", vcov_type: str = "hc1", - cluster: Optional[str] = None, + cluster: ClusterArg = None, controls: Optional[List[str]] = None, control_group: str = "not_yet_treated", k_min: int = 2, @@ -494,8 +496,11 @@ def robustness_pre_periods( Estimation method. vcov_type : str, default 'hc1' Variance-covariance family. - cluster : str, optional + cluster : str, None or False, optional Cluster variable for standard errors. + ``False`` (4.0, M-080) is an accepted explicit no-op: the variance is + unit-level by construction, so there is nothing to switch off. Column + labels must be ``str``. controls : list of str, optional Control variable column names. k_min : int, default 2 @@ -535,6 +540,7 @@ def robustness_pre_periods( # RAISE here instead of being swallowed as per-spec "failed fits" # inside _fit_single_spec (campaign finding: a string covariate's # ValueError became a silent NaN spec). + cluster = validate_cluster(cluster) # str | None | False ([M-155]); False = no-op _prevalidate_frame(data, outcome, unit, time, treatment, cohort, cluster, controls) for name, value in (("k_min", k_min), ("k_max", k_max)): @@ -707,7 +713,7 @@ def sensitivity_no_anticipation( rolling: str = "demean", estimation_method: str = "reg", vcov_type: str = "hc1", - cluster: Optional[str] = None, + cluster: ClusterArg = None, controls: Optional[List[str]] = None, control_group: str = "not_yet_treated", # lwdid-py compatible aliases @@ -752,8 +758,11 @@ def sensitivity_no_anticipation( Estimation method. vcov_type : str, default 'hc1' Variance-covariance family. - cluster : str, optional + cluster : str, None or False, optional Cluster variable for standard errors. + ``False`` (4.0, M-080) is an accepted explicit no-op: the variance is + unit-level by construction, so there is nothing to switch off. Column + labels must be ``str``. controls : list of str, optional Control variable column names. @@ -785,6 +794,7 @@ def sensitivity_no_anticipation( if treatment is None: raise ValueError("'treatment' (or 'd') parameter is required") + cluster = validate_cluster(cluster) # str | None | False ([M-155]); False = no-op _prevalidate_frame(data, outcome, unit, time, treatment, cohort, cluster, controls) if exclude_periods is None: diff --git a/diff_diff/spillover.py b/diff_diff/spillover.py index 2d736d266..d526733dd 100644 --- a/diff_diff/spillover.py +++ b/diff_diff/spillover.py @@ -42,7 +42,13 @@ from diff_diff.linalg import _rank_guarded_inv, solve_ols from diff_diff.results import SpilloverDiDResults from diff_diff.two_stage import _compute_gmm_corrected_meat, _LSMRUnconvergedError -from diff_diff.utils import _iterative_fe_solve, safe_inference, validate_anticipation +from diff_diff.utils import ( + ClusterArg, + _iterative_fe_solve, + safe_inference, + validate_anticipation, + validate_cluster, +) if TYPE_CHECKING: from diff_diff.survey import SurveyDesign @@ -1686,9 +1692,17 @@ class SpilloverDiD(BaseEstimator): conley_lag_cutoff : int, optional Within-unit Bartlett max lag. Required when ``vcov_type="conley"``. Use ``0`` to suppress the serial-component sandwich. - cluster : str, optional - Column name for cluster-robust variance, or the combined Conley - cluster product kernel when paired with ``vcov_type="conley"``. + cluster : str, None or False, optional + ``None`` (default; 4.0, M-080) clusters at ``unit`` - CR1 at the unit + level, mirroring ``TwoWayFixedEffects``: the auto-cluster is dropped + on the Conley path and is never injected as a survey PSU. ``False`` + switches clustering off explicitly (the observation-level HC1 + sandwich - the 3.x default). A column name clusters on that column + on every lane: CR1, the combined Conley cluster product kernel when + paired with ``vcov_type="conley"``, or the effective PSU under a + ``survey_design`` without its own PSU. Column labels must be ``str``. + Inference uses the residual df on every lane (``df_convention`` + arrives with [M-154]). alpha : float, default=0.05 Significance level for confidence intervals. anticipation : int, default=0 @@ -1748,7 +1762,7 @@ def __init__( conley_metric: SpilloverMetric = "haversine", conley_cutoff_km: Optional[float] = None, conley_lag_cutoff: Optional[int] = None, - cluster: Optional[str] = None, + cluster: ClusterArg = None, alpha: float = 0.05, anticipation: int = 0, event_study: bool = False, @@ -1790,7 +1804,7 @@ def __init__( self.conley_metric = conley_metric self.conley_cutoff_km = conley_cutoff_km self.conley_lag_cutoff = conley_lag_cutoff - self.cluster = cluster + self.cluster = validate_cluster(cluster) self.alpha = alpha self.anticipation = validate_anticipation(anticipation) self.event_study = event_study @@ -2054,7 +2068,7 @@ def _validate_spillover_inputs( # path, not just conley. Missing cluster ids would produce wrong # SEs (NaN counted as its own cluster by np.unique() but dropped # by pandas groupby() in the cluster meat). - if self.cluster is not None: + if isinstance(self.cluster, str): if self.cluster not in data.columns: raise ValueError(f"cluster column '{self.cluster}' not in data.") cluster_nan_mask = data[self.cluster].isna() @@ -2219,6 +2233,9 @@ def fit( # `-1 - self.anticipation` and would otherwise raise a raw TypeError # on non-numeric input (PR #456 R2 fix). self.anticipation = validate_anticipation(self.anticipation) + # Same direct-mutation defense for the shared cluster contract + # (str | None | False, [M-155]); the lane resolution lives below. + validate_cluster(self.cluster) # Wave C: event-study path is now supported. Validate horizon_max # up front (fail-fast before any stage-1 work). if self.horizon_max is not None: @@ -2362,6 +2379,25 @@ def fit( # coords, panel structure — all on COERCED time/first_treat labels). self._validate_spillover_inputs(data, treatment, first_treat, time, unit, outcome) + # Step 1a (4.0, M-080): resolve the ONE cluster column every variance + # lane below reads. Explicit `cluster=` wins on every lane (incl. the + # Conley product kernel and survey PSU injection); `False` is the + # uniform explicit off-switch (observation-level HC - the 3.x default); + # `None` auto-clusters at `unit` with TWFE's carve-outs mirrored: the + # auto-cluster is dropped on the Conley path (a unit mask would zero + # every between-unit spatial pair) and is never injected as a survey + # PSU (the documented implicit-per-observation-PSU rule). + if self.cluster is False: + cluster_col: Optional[str] = None + elif self.cluster is not None: + cluster_col = self.cluster + elif self.vcov_type == "conley": + cluster_col = None + elif survey_design is not None: + cluster_col = None + else: + cluster_col = unit + # Step 1b (Wave E.1): survey-design resolution + validation. # # Mirrors TwoStageDiD's resolution block at `two_stage.py:485-511`. @@ -2879,9 +2915,7 @@ def fit( # `imputation.py:2175-2183` precedent). Under no-survey, keep the # historic finite_mask subset so downstream CR1 sample-size matches # X_2_fit. - cluster_ids_full = ( - np.asarray(data[self.cluster].values) if self.cluster is not None else None - ) + cluster_ids_full = np.asarray(data[cluster_col].values) if cluster_col is not None else None if n_nan > 0: X_2_fit = X_2[finite_mask] y_tilde_fit = y_tilde[finite_mask] @@ -2963,7 +2997,7 @@ def fit( effective_cluster_ids = _resolve_effective_cluster( resolved_survey_fit, cluster_ids_fit, - self.cluster if self.cluster is not None else None, + cluster_col, ) if effective_cluster_ids is not None: # Wave E.1 R11 fix: when `cluster=` becomes the effective @@ -2977,7 +3011,7 @@ def fit( # via `survey_design.psu=` would be rejected by the panel- # survey validator at `survey.py:1015`. if ( - self.cluster is not None + cluster_col is not None and resolved_survey is not None and resolved_survey.psu is None ): @@ -3000,7 +3034,7 @@ def fit( if len(nonconstant) > 0: bad_units = list(nonconstant.index[:5]) raise ValueError( - f"`cluster='{self.cluster}'` is being used as the " + f"`cluster='{cluster_col}'` is being used as the " f"effective PSU under survey_design= (no explicit " f"survey_design.psu provided), but the cluster " f"column varies within unit for " @@ -3610,7 +3644,7 @@ def fit( and resolved_survey_fit is not None and resolved_survey_fit.psu is not None ) - else self.cluster + else cluster_col ), conley_lag_cutoff=(self.conley_lag_cutoff if self.vcov_type == "conley" else None), spillover_effects=spillover_df, diff --git a/diff_diff/staggered.py b/diff_diff/staggered.py index e18c8ef69..41d19a392 100644 --- a/diff_diff/staggered.py +++ b/diff_diff/staggered.py @@ -40,9 +40,11 @@ GroupTimeEffect, ) from diff_diff.utils import ( + ClusterArg, safe_inference, safe_inference_batch, validate_anticipation, + validate_cluster, validate_n_bootstrap, validate_pscore_trim, ) @@ -349,7 +351,10 @@ class CallawaySantAnna( - "reg": Outcome regression alpha : float, default=0.05 Significance level for confidence intervals. - cluster : str, optional + cluster : str, None or False, optional + ``False`` (4.0, M-080) is an accepted explicit no-op: the variance is + unit-level by construction, so there is nothing to switch off. Column + labels must be ``str``. Column name for cluster-robust standard errors. When set, the influence-function aggregator clusters at the named level via a synthesized ``SurveyDesign(psu=cluster_col)`` threaded through the @@ -567,7 +572,7 @@ def __init__( anticipation: int = 0, estimation_method: str = "dr", alpha: float = 0.05, - cluster: Optional[str] = None, + cluster: ClusterArg = None, n_bootstrap: int = 0, bootstrap_weights: Optional[str] = None, seed: Optional[int] = None, @@ -635,7 +640,7 @@ def __init__( self.anticipation = validate_anticipation(anticipation) self.estimation_method = estimation_method self.alpha = alpha - self.cluster = cluster + self.cluster = validate_cluster(cluster) self.vcov_type = vcov_type # Track whether vcov_type was explicitly set (for future symmetry # with SA / StackedDiD / WooldridgeDiD set_params patterns; the @@ -1914,6 +1919,12 @@ def fit( # out-of-domain value silently changes the ESTIMAND); the assignment # also re-normalizes a mutated numpy scalar to a Python int. self.anticipation = validate_anticipation(self.anticipation) + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]): __init__ and set_params validate eagerly, so this only + # catches DIRECT attribute mutation. `False` is normalized to None + # locally so it never reaches a column lookup ([M-080]). + validate_cluster(self.cluster) + user_cluster = None if self.cluster is False else self.cluster # --- allow_unbalanced_panel routing (RC-on-panel = R's allow_unbalanced_panel) --- # Detect an unbalanced panel (some units unobserved in some periods). @@ -2029,12 +2040,12 @@ def fit( f"cohort ('{first_treat}') per unit; unit(s) {_bad} have a changing " f"first_treat." ) - if self.cluster is not None and self.cluster in data.columns: - _cl_counts = data.groupby(unit)[self.cluster].nunique() + if user_cluster is not None and user_cluster in data.columns: + _cl_counts = data.groupby(unit)[user_cluster].nunique() if (_cl_counts > 1).any(): _bad = list(_cl_counts.index[_cl_counts > 1][:5]) raise ValueError( - f"allow_unbalanced_panel=True with cluster='{self.cluster}' " + f"allow_unbalanced_panel=True with cluster='{user_cluster}' " f"requires a time-invariant cluster per unit; unit(s) {_bad} " f"have a changing cluster value." ) @@ -2086,16 +2097,16 @@ def fit( # transparently activates the same code paths used when the user # passes SurveyDesign(psu=X) explicitly. effective_survey_design = survey_design # refreshed below if synthesized - if self.cluster is not None: - if self.cluster not in data.columns: - raise ValueError(f"cluster column '{self.cluster}' not found in data") + if user_cluster is not None: + if user_cluster not in data.columns: + raise ValueError(f"cluster column '{user_cluster}' not found in data") # Pre-validate cluster NaN with a cluster-domain error message. # Without this, SurveyDesign.resolve() raises "PSU column ... # contains missing values" — wrong domain for the user-facing # cluster= API. - if data[self.cluster].isna().any(): + if data[user_cluster].isna().any(): raise ValueError( - f"cluster column '{self.cluster}' contains missing " + f"cluster column '{user_cluster}' contains missing " "values. All observations must have valid cluster " "identifiers." ) @@ -2111,7 +2122,7 @@ def fit( and getattr(survey_design, "replicate_weights", None) is not None ): raise NotImplementedError( - f"CallawaySantAnna(cluster={self.cluster!r}) is not " + f"CallawaySantAnna(cluster={user_cluster!r}) is not " "supported with replicate-weight survey designs. " "Replicate-weight variance is computed by replicate " "reweighting (BRR / Fay / JK1 / JKn / SDR) and ignores " @@ -2121,14 +2132,14 @@ def fit( "structure implicitly) or use a non-replicate survey " "design (with explicit strata/psu/fpc)." ) - cluster_ids = data[self.cluster].values + cluster_ids = data[user_cluster].values if resolved_survey is None: # Bare cluster=, no survey: synthesize minimal # SurveyDesign(psu=...). Activates the same survey-PSU # aggregator + bootstrap machinery already used when the # user passes SurveyDesign(psu=X) explicitly. - synthetic_design = SurveyDesign(psu=self.cluster, weight_type="pweight") + synthetic_design = SurveyDesign(psu=user_cluster, weight_type="pweight") ( resolved_survey, survey_weights, @@ -2149,7 +2160,7 @@ def fit( # User provided survey_design (weights/strata/etc.) without # PSU AND cluster=X. Inject cluster as PSU into the resolved # survey. Construct effective_survey_design with - # psu=self.cluster (via dataclasses.replace) so the + # psu=user_cluster (via dataclasses.replace) so the # downstream _validate_unit_constant_survey catches movers # on panel data (otherwise the validator runs on the user- # provided design which has no PSU column). @@ -2162,7 +2173,7 @@ def fit( # This branch is only reached with a survey design present. assert survey_design is not None - effective_survey_design = replace(survey_design, psu=self.cluster) + effective_survey_design = replace(survey_design, psu=user_cluster) resolved_survey = _inject_cluster_as_psu(resolved_survey, cluster_ids) # Recompute survey_metadata to reflect the new effective PSU # (so n_psu / df_survey on Results reflect the injected @@ -2183,7 +2194,7 @@ def fit( # is already resolved_survey.psu (no rewrite needed). from diff_diff.survey import _resolve_effective_cluster - _resolve_effective_cluster(resolved_survey, cluster_ids, self.cluster) + _resolve_effective_cluster(resolved_survey, cluster_ids, user_cluster) # allow_unbalanced_panel (RC-routing) with NO user cluster=: cluster the # per-observation RC influence function by the ORIGINAL unit so the SE @@ -2192,7 +2203,7 @@ def fit( # → the same PSU-meat aggregator + PSU multiplier bootstrap used for an # explicit cluster=. When the user set cluster=X the block above already # synthesized psu=X (their choice); do NOT double-cluster here. - if _route_as_rc and self.cluster is None and resolved_survey is None: + if _route_as_rc and user_cluster is None and resolved_survey is None: _unit_psu_design = SurveyDesign(psu=unit, weight_type="pweight") ( resolved_survey, @@ -2837,8 +2848,8 @@ def fit( # set (a UserWarning would have fired above if partitions differed). if survey_design is not None and getattr(survey_design, "psu", None) is not None: cluster_name_for_results: Optional[str] = survey_design.psu - elif self.cluster is not None: - cluster_name_for_results = self.cluster + elif user_cluster is not None: + cluster_name_for_results = user_cluster elif _route_as_rc: # RC-on-panel auto-synthesizes SurveyDesign(psu=unit); surface the # effective unit-level clustering so introspection / summaries don't diff --git a/diff_diff/sun_abraham.py b/diff_diff/sun_abraham.py index 3a982b051..a25d5d9cc 100644 --- a/diff_diff/sun_abraham.py +++ b/diff_diff/sun_abraham.py @@ -41,6 +41,7 @@ from diff_diff.results import _format_survey_block, _get_significance_stars from diff_diff.results_base import BaseResults, _coverage_pct, _require_fit_alpha from diff_diff.utils import ( + ClusterArg, absorbed_fe_cr1_k_increment, absorbed_fe_rank, pre_demean_norms, @@ -48,6 +49,7 @@ safe_inference, snap_absorbed_regressors, validate_anticipation, + validate_cluster, validate_df_convention, validate_n_bootstrap, ) @@ -171,10 +173,11 @@ class SunAbrahamResults(BaseResults): # in ``event_study_vcov_index``. event_study_vcov: Optional["np.ndarray"] = field(default=None, repr=False) event_study_vcov_index: Optional[list] = field(default=None, repr=False) - # Conley spatial-HAC metadata (populated only when vcov_type == "conley"). - # ``conley_lag_cutoff`` carries the within-unit Bartlett max lag; ``cluster_name`` - # records an explicit cluster= column (enables the spatial+cluster product-kernel - # summary label). Both None on non-conley fits. + # ``conley_lag_cutoff`` carries the within-unit Bartlett max lag (None off the + # Conley path). ``cluster_name`` names the cluster ids the analytical regression + # received (4.0, M-080): the unit auto-cluster or an explicit cluster= column + # (on Conley: the explicit product-kernel column); None on unclustered, + # survey and pairs-bootstrap fits. ``n_clusters`` (appended below) is its count. conley_lag_cutoff: Optional[int] = None cluster_name: Optional[str] = None # The normalization reference relative time (e = -1 - anticipation, the @@ -221,6 +224,13 @@ class SunAbrahamResults(BaseResults): fallback (None under "normal" = normal theory). None on bootstrap fits, whose overall p/CI are percentile-based and never used a df.""" + n_clusters: Optional[int] = None + """Effective cluster count of the ids the analytical regression + clustered on (4.0, M-080): G under the unit auto-cluster or an explicit + ``cluster=`` (incl. the Conley product-kernel column); None on + unclustered (``cluster=False`` / one-way) fits, under a survey design + (the survey block owns PSU reporting) and on pairs-bootstrap fits.""" + # --- Deprecated 3.x storage names (ledger row M-051) --- # FutureWarning from 4.0, removed at 5.0. Plain class attributes (no # annotation) so they stay descriptors, never dataclass fields - the @@ -314,15 +324,20 @@ def summary(self, alpha: Optional[float] = None) -> str: sm = self.survey_metadata lines.extend(_format_survey_block(sm, 85)) - # Conley spatial-HAC variance label (rendered only on the conley path; - # a full vcov-family label for all families is a separate follow-up). - if self.vcov_type == "conley": + # Variance-family label (4.0, M-080): rendered on the Conley path and on + # every analytical non-survey fit (CR1 at unit with G, the plain HC + # families, CR2-BM). Survey fits keep the survey block alone and + # bootstrap fits their bootstrap line alone - `vcov_type` still names + # the configured analytical family there, which would mislabel the SE. + if self.vcov_type == "conley" or ( + self.bootstrap_results is None and self.survey_metadata is None + ): from diff_diff.results import _format_vcov_label _vlabel = _format_vcov_label( self.vcov_type, cluster_name=self.cluster_name, - n_clusters=None, + n_clusters=self.n_clusters, n_obs=self.n_obs, conley_lag_cutoff=self.conley_lag_cutoff, ) @@ -417,6 +432,8 @@ def to_dict(self) -> Dict[str, Any]: } if self.cluster_name is not None: result["cluster_name"] = self.cluster_name + if self.n_clusters is not None: + result["n_clusters"] = self.n_clusters if self.conley_lag_cutoff is not None: result["conley_lag_cutoff"] = self.conley_lag_cutoff if self.df_convention is not None: @@ -556,8 +573,15 @@ class SunAbraham(BaseEstimator): Must be a non-negative integer; ``bool`` is rejected. alpha : float, default=0.05 Significance level for confidence intervals. - cluster : str, optional - Column name for cluster-robust standard errors. + cluster : str, None or False, optional + Column name for cluster-robust standard errors. ``False`` (4.0, + M-080) switches clustering off explicitly: the plain + heteroskedasticity-robust sandwich on the saturated regression + (residual-df t inference; rejected with ``vcov_type="hc2_bm"``, + whose Bell-McCaffrey DOF lane is cluster-only; an accepted no-op + for the pairs bootstrap, which resamples units regardless; under a + survey design it means no unit PSU is injected). Column labels must + be ``str``. If None, clusters at the unit level by default — UNLESS ``vcov_type`` is explicitly set to ``"hc2"`` or ``"classical"``, in which case the unit auto-cluster is dropped (both are @@ -720,7 +744,7 @@ def __init__( control_group: str = "never_treated", anticipation: int = 0, alpha: float = 0.05, - cluster: Optional[str] = None, + cluster: ClusterArg = None, n_bootstrap: int = 0, seed: Optional[int] = None, rank_deficient_action: str = "warn", @@ -755,7 +779,7 @@ def __init__( self.control_group = control_group self.anticipation = validate_anticipation(anticipation) self.alpha = alpha - self.cluster = cluster + self.cluster = validate_cluster(cluster) validate_n_bootstrap(n_bootstrap) self.n_bootstrap = n_bootstrap self.seed = seed @@ -828,6 +852,12 @@ def fit( # — an out-of-domain value silently changes the ESTIMAND. The # assignment also re-normalizes a mutated numpy scalar to int. self.anticipation = validate_anticipation(self.anticipation) + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]): __init__ and set_params validate eagerly, so this only + # catches DIRECT attribute mutation. `False` is normalized to None + # locally so it never reaches a column lookup ([M-080]). + validate_cluster(self.cluster) + user_cluster = None if self.cluster is False else self.cluster # Validate inputs required_cols = [outcome, unit, time, first_treat] @@ -843,10 +873,10 @@ def fit( # and silently downgrade clustered inference to one-way (HC1 → # heteroskedasticity-only; HC2-BM → singleton CR2-BM). Explicit # user input must error, not silently weaken the SE convention. - if self.cluster is not None: - if self.cluster not in data.columns: + if user_cluster is not None: + if user_cluster not in data.columns: raise ValueError( - f"cluster column {self.cluster!r} not found in data; " + f"cluster column {user_cluster!r} not found in data; " f"available columns: {list(data.columns)}" ) # NA cluster labels are silently dropped by the meat-side @@ -854,10 +884,10 @@ def fit( # in `n_clusters`, producing malformed cluster-robust SEs. Reject # explicitly so the user fixes the cluster column rather than # consuming silently-wrong inference. - if data[self.cluster].isna().any(): - n_na = int(data[self.cluster].isna().sum()) + if data[user_cluster].isna().any(): + n_na = int(data[user_cluster].isna().sum()) raise ValueError( - f"cluster column {self.cluster!r} contains {n_na} " + f"cluster column {user_cluster!r} contains {n_na} " "NA/NaN values. Cluster labels must be non-missing for " "all observations to produce well-formed cluster-robust " "standard errors. Drop or impute the NA rows before fit." @@ -880,7 +910,7 @@ def fit( conley_lag_cutoff=self.conley_lag_cutoff, survey_design=survey_design, inference="analytical", - cluster=self.cluster, + cluster=user_cluster, ) if self.n_bootstrap > 0: raise ValueError( @@ -890,6 +920,18 @@ def fit( "analytical Conley SE, or vcov_type='hc1' with the bootstrap." ) + # 4.0 (M-080): the Bell-McCaffrey DOF lane is cluster-only (its contrast + # helper takes a cluster array), so the unclustered off-switch cannot pair + # with hc2_bm here; TODO.md tracks the one-way BM DOF port. The pairs + # bootstrap is NOT rejected: it resamples units on every lane regardless + # of `cluster` (the explicit-hc2 precedent), so False is a no-op there. + if self.cluster is False and self.vcov_type == "hc2_bm": + raise ValueError( + "SunAbraham(cluster=False, vcov_type='hc2_bm') is not supported: the " + "Bell-McCaffrey DOF lane is cluster-only. Use vcov_type='hc2' or 'hc1' " + "with cluster=False, or keep clustering (cluster=None auto-clusters at unit)." + ) + # Resolve survey design if provided from diff_diff.survey import ( _resolve_effective_cluster, @@ -1031,8 +1073,13 @@ def fit( # structure of the main fit. So the SA guard simplifies to # "explicit-vcov-only", without TWFE's `inference == "analytical"` # subguard. - if self.cluster is not None: - cluster_var: Optional[str] = self.cluster + if self.cluster is False: + # 4.0 (M-080): the uniform explicit off-switch - the plain HC + # sandwich (absorbed-FE full-K rescale inside LinearRegression), + # no auto-cluster, no survey PSU injection. + cluster_var: Optional[str] = None + elif user_cluster is not None: + cluster_var = user_cluster elif self.vcov_type == "conley": # Conley: never auto-cluster at unit. A unit-cluster product kernel # would zero every between-unit spatial pair, collapsing the spatial @@ -1061,7 +1108,7 @@ def fit( else: cluster_ids_raw = None effective_cluster_ids = _resolve_effective_cluster( - resolved_survey, cluster_ids_raw, cluster_var if self.cluster is not None else None + resolved_survey, cluster_ids_raw, cluster_var if user_cluster is not None else None ) if resolved_survey is not None and effective_cluster_ids is not None: from diff_diff.survey import _inject_cluster_as_psu, compute_survey_metadata @@ -1552,6 +1599,24 @@ def _refit_sa_cohort(w_r): _overall_df_final = float(_overall_df_used) if np.isfinite(_overall_df_used) else None # Store results + # 4.0 (M-080): cluster metadata describes the cluster ids actually + # passed to the analytical regression - None under a survey design + # (the survey block owns PSU reporting), under the pairs bootstrap + # (unit resampling, no analytical cluster) and on unclustered fits; + # Conley keeps the explicit product-kernel column (and its count). + if resolved_survey is not None or bootstrap_results is not None: + _sa_cluster_name: Optional[str] = None + _sa_n_clusters: Optional[int] = None + elif self.vcov_type == "conley": + _sa_cluster_name = user_cluster + _sa_n_clusters = _sa_fit_stats.n_clusters if user_cluster is not None else None + elif cluster_var is not None: + _sa_cluster_name = cluster_var + _sa_n_clusters = _sa_fit_stats.n_clusters + else: + _sa_cluster_name = None + _sa_n_clusters = None + self.results_ = SunAbrahamResults( event_study_effects=event_study_effects, att=overall_att, @@ -1576,11 +1641,12 @@ def _refit_sa_cohort(w_r): event_study_vcov_index=es_vcov_index, event_study_df=es_df_final, conley_lag_cutoff=(self.conley_lag_cutoff if self.vcov_type == "conley" else None), - cluster_name=(self.cluster if self.vcov_type == "conley" else None), + cluster_name=_sa_cluster_name, reference_period=self._reference_period, reference_observed=self._reference_observed, df_convention=self.df_convention, inference_df=_overall_df_final, + n_clusters=_sa_n_clusters, ) self.is_fitted_ = True diff --git a/diff_diff/triple_diff.py b/diff_diff/triple_diff.py index 511055bf1..71e25ff20 100644 --- a/diff_diff/triple_diff.py +++ b/diff_diff/triple_diff.py @@ -49,9 +49,11 @@ # is the class path 3.x pickles and existing readers use. from diff_diff.triple_diff_results import TripleDifferenceResults # noqa: F401 from diff_diff.utils import ( + ClusterArg, safe_inference, staggered_ddd_ctor_offenders, validate_anticipation, + validate_cluster, validate_n_bootstrap, validate_pscore_trim, ) @@ -97,7 +99,10 @@ class TripleDifference( model or propensity score model is correctly specified. - "reg": Regression adjustment (outcome regression). - "ipw": Inverse probability weighting. - cluster : str, optional + cluster : str, None or False, optional + ``False`` (4.0, M-080) is an accepted explicit no-op on both modes (the + 2x2x2 fit is HC-robust unless ``cluster=`` is given; the staggered mode + still rejects a column name). Column labels must be ``str``. Column name for cluster-robust standard errors. When provided, SEs are computed using the Liang-Zeger cluster-robust variance estimator on the influence function. @@ -273,7 +278,7 @@ class TripleDifference( def __init__( self, estimation_method: str = "dr", - cluster: Optional[str] = None, + cluster: ClusterArg = None, vcov_type: str = "hc1", alpha: float = 0.05, pscore_trim: float = 0.01, @@ -347,7 +352,7 @@ def __init__( self._validate_vcov_type(vcov_type) self.estimation_method = estimation_method - self.cluster = cluster + self.cluster = validate_cluster(cluster) self.vcov_type = vcov_type self.alpha = alpha self.pscore_trim = pscore_trim @@ -454,6 +459,12 @@ def fit( NotImplementedError If survey_design is used with wild_bootstrap inference. """ + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]): __init__ and set_params validate eagerly, so this only + # catches DIRECT attribute mutation. `False` is normalized to None + # locally so it never reaches a column lookup ([M-080]). + validate_cluster(self.cluster) + user_cluster = None if self.cluster is False else self.cluster # ---- Step 1: staggered mode --------------------------------------- # Raw sentinels forwarded: the 2x2x2 `time=` check must not run, because # `time=` is the calendar column on this path. @@ -569,9 +580,9 @@ def fit( T = data[time].values.astype(float) # Store cluster IDs for SE computation - self._cluster_ids = data[self.cluster].values if self.cluster is not None else None - if self._cluster_ids is not None and np.any(pd.isna(data[self.cluster])): - raise ValueError(f"Cluster column '{self.cluster}' contains missing values") + self._cluster_ids = data[user_cluster].values if user_cluster is not None else None + if self._cluster_ids is not None and np.any(pd.isna(data[user_cluster])): + raise ValueError(f"Cluster column '{user_cluster}' contains missing values") # Reject replicate-weight + cluster=: replicate IF variance is # computed by replicate reweighting (BRR / Fay / JK1 / JKn / SDR) @@ -582,12 +593,12 @@ def fit( # dishonestly. Fail-closed per feedback_no_silent_failures. # Mirrors CallawaySantAnna guard at staggered.py:1705-1719. if ( - self.cluster is not None + user_cluster is not None and survey_design is not None and getattr(survey_design, "replicate_weights", None) is not None ): raise NotImplementedError( - f"TripleDifference(cluster={self.cluster!r}) is not " + f"TripleDifference(cluster={user_cluster!r}) is not " "supported with replicate-weight survey designs. " "Replicate-weight variance is computed by replicate " "reweighting (BRR / Fay / JK1 / JKn / SDR) and ignores " @@ -601,7 +612,7 @@ def fit( # Resolve effective cluster and inject cluster-as-PSU for survey variance if resolved_survey is not None: effective_cluster_ids = _resolve_effective_cluster( - resolved_survey, self._cluster_ids, self.cluster + resolved_survey, self._cluster_ids, user_cluster ) if effective_cluster_ids is not None: resolved_survey = _inject_cluster_as_psu(resolved_survey, effective_cluster_ids) @@ -702,9 +713,9 @@ def fit( if resolved_survey is not None: cluster_name_for_results: Optional[str] = None n_clusters: Optional[int] = None - elif self.cluster is not None: - cluster_name_for_results = self.cluster - n_clusters = data[self.cluster].nunique() + elif user_cluster is not None: + cluster_name_for_results = user_cluster + n_clusters = data[user_cluster].nunique() else: cluster_name_for_results = None n_clusters = None @@ -807,7 +818,7 @@ def _fit_staggered( # no clustered analytical path. The removed StaggeredTripleDifference # class accepted-then-warned-then-ignored it; this surface raises rather # than shipping an accepted-but-unhonored parameter (3(a)'s precedent). - if self.cluster is not None: + if self.cluster is not None and self.cluster is not False: raise ValueError( "cluster= is not supported in staggered DDD mode: cluster-robust ANALYTICAL " "SEs are not implemented for the staggered engine. Use n_bootstrap > 0 for " @@ -848,7 +859,7 @@ def _validate_data( required_cols = [outcome, group, partition, time] if covariates: required_cols.extend(covariates) - if self.cluster is not None: + if isinstance(self.cluster, str): required_cols.append(self.cluster) missing_cols = [col for col in required_cols if col not in data.columns] diff --git a/diff_diff/twfe.py b/diff_diff/twfe.py index 5f6a9382b..8a070c980 100644 --- a/diff_diff/twfe.py +++ b/diff_diff/twfe.py @@ -25,6 +25,7 @@ fe_dummy_names, pre_demean_norms, snap_absorbed_regressors, + validate_cluster, validate_covariate_names, validate_design_term_names, ) @@ -42,7 +43,11 @@ class TwoWayFixedEffects(DifferenceInDifferences): Parameters ---------- - cluster : str, optional + cluster : str, None or False, optional + ``False`` (4.0, M-080) switches the unit auto-cluster off explicitly on + both modes: the plain HC sandwich (absorbed-FE full-K rescale, residual-df + t; one-way Bell-McCaffrey df under ``vcov_type="hc2_bm"``). Rejected with + ``inference="wild_bootstrap"``. Column labels must be ``str``. Column name for cluster-robust standard errors. If None, automatically clusters at the unit level (the `unit` parameter passed to `fit()`). This differs from @@ -258,6 +263,12 @@ def fit( # type: ignore[override] ``inference="wild_bootstrap"`` is combined with event-study mode (the wild cluster bootstrap covers the static ATT only). """ + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]): __init__ and set_params validate eagerly, so this only + # catches DIRECT attribute mutation. `False` is normalized to None + # locally so it never reaches a column lookup ([M-080]). + validate_cluster(self.cluster) + user_cluster = None if self.cluster is False else self.cluster # Per-fit bootstrap state: cleared up front so the result builder # labels inference from THIS fit only (see the matching reset in # DifferenceInDifferences.fit). @@ -356,7 +367,7 @@ def fit( # type: ignore[override] conley_lag_cutoff=self.conley_lag_cutoff, survey_design=survey_design, inference=self.inference, - cluster=self.cluster, + cluster=user_cluster, ) # Check for staggered treatment timing and warn if detected @@ -428,6 +439,15 @@ def fit( # type: ignore[override] f"for bootstrap inference; use inference='analytical' for " f"analytical SEs." ) + # 4.0 (M-080): cluster=False switches the clustering off, but the wild + # CLUSTER bootstrap draws at the cluster level - the combination is + # incoherent (same fail-closed rule as DiD's cluster=None guard). + if self.inference == "wild_bootstrap" and self.cluster is False: + raise ValueError( + "inference='wild_bootstrap' requires a cluster: the wild cluster " + "bootstrap resamples at the cluster level. Pass cluster=, leave " + "cluster=None for the unit auto-cluster, or use inference='analytical'." + ) # Unit-level clustering is the TWFE default when `cluster` is not # explicitly provided. But the one-way ``classical`` and ``hc2`` @@ -444,8 +464,12 @@ def fit( # type: ignore[override] # regardless of ``vcov_type`` (bootstrap consumes cluster structure # for resampling, independent of the analytical sandwich). # ``hc2_bm`` also keeps the auto-cluster (routes to CR2-BM at unit). - if self.cluster is not None: - cluster_var: Optional[str] = self.cluster + if self.cluster is False: + # 4.0 (M-080): the uniform explicit off-switch - the plain HC + # sandwich, no auto-cluster on any lane (wild rejected above). + cluster_var: Optional[str] = None + elif user_cluster is not None: + cluster_var = user_cluster elif ( self.vcov_type in ("classical", "hc2", "hc3") and self._vcov_type_explicit @@ -605,13 +629,13 @@ def fit( # type: ignore[override] # When survey PSU is present, it overrides cluster for variance estimation effective_cluster_ids = _resolve_effective_cluster( - resolved_survey, cluster_ids, self.cluster + resolved_survey, cluster_ids, user_cluster ) # For survey variance: only inject user-explicit cluster as PSU. # TWFE's default unit clustering should not override the documented # no-PSU survey path (implicit per-observation PSUs). - if resolved_survey is not None and self.cluster is None: + if resolved_survey is not None and user_cluster is None: survey_cluster_ids = None else: survey_cluster_ids = effective_cluster_ids @@ -674,7 +698,7 @@ def fit( # type: ignore[override] # Users who want the combined kernel must pass an above-unit # cluster (e.g. region) explicitly. _conley_cluster_override = ( - data[self.cluster].values if self.cluster is not None else None + data[user_cluster].values if user_cluster is not None else None ) else: _conley_coords_arr = None @@ -924,9 +948,15 @@ def _refit_twfe(w_r): n_bootstrap_used = self._bootstrap_results.n_bootstrap n_clusters_used = self._bootstrap_results.n_clusters p_val_type_used = self._bootstrap_results.p_val_type + elif resolved_survey is None: + # 4.0 (M-080): the analytical regression's own effective cluster + # count (None when it ran unclustered). Survey fits keep None - + # the survey metadata owns PSU reporting, and under a PSU design + # the regression received PSU labels, not the user's cluster. + n_clusters_used = reg.n_clusters_ # Cluster label for summary: TWFE auto-clusters at unit level when - # `self.cluster is None` AND the vcov family is cluster-compatible. + # `user_cluster is None` AND the vcov family is cluster-compatible. # One-way families (`classical`, `hc2`) disable the auto-cluster (see # the `cluster_var` block above); report None so summary() labels the # one-way family instead of "CR1 cluster-robust at unit". On the @@ -938,11 +968,16 @@ def _refit_twfe(w_r): # Either way, the cluster_name reflects the cluster IDs actually # passed to LinearRegression. if _fit_vcov_type == "conley": - _twfe_cluster_label: Optional[str] = self.cluster if self.cluster is not None else None - elif self.cluster is not None: - _twfe_cluster_label = self.cluster + _twfe_cluster_label: Optional[str] = user_cluster if user_cluster is not None else None + elif user_cluster is not None: + _twfe_cluster_label = user_cluster elif cluster_var is None: - # One-way family path: auto-cluster was intentionally dropped. + # One-way family path or cluster=False: no cluster ids were passed. + _twfe_cluster_label = None + elif resolved_survey is not None: + # 4.0 (M-080): under a survey design the auto-cluster is NOT + # injected (survey_cluster_ids=None above), so the regression + # received no cluster ids - the former "unit" label was false. _twfe_cluster_label = None else: _twfe_cluster_label = unit @@ -1101,8 +1136,12 @@ def _fit_event_study( # passes through on every lane; when survey_design carries its own # PSU the shared resolver still gives the PSU precedence (identical # on every class). - if self.cluster is not None: - cluster_override: Optional[str] = self.cluster + if self.cluster is False: + # 4.0 (M-080): explicit off - the core's unclustered lanes (plain + # HC sandwich, residual df, one-way hc2_bm BM df) do the rest. + cluster_override: Optional[str] = None + elif self.cluster is not None: + cluster_override = self.cluster elif self.vcov_type == "conley": # Static mirror: the implicit unit auto-cluster is silently # dropped on the Conley path - combining Conley with unit-level diff --git a/diff_diff/two_stage.py b/diff_diff/two_stage.py index e448a24b4..a38b021a4 100644 --- a/diff_diff/two_stage.py +++ b/diff_diff/two_stage.py @@ -50,7 +50,13 @@ TwoStageBootstrapResults, # noqa: F401 TwoStageDiDResults, ) # noqa: F401 (re-export) -from diff_diff.utils import safe_inference, validate_anticipation, validate_n_bootstrap +from diff_diff.utils import ( + ClusterArg, + safe_inference, + validate_anticipation, + validate_cluster, + validate_n_bootstrap, +) if TYPE_CHECKING: # Forward reference for the Wave E.1 survey-design path. Imported under @@ -1239,7 +1245,10 @@ class TwoStageDiD(TwoStageDiDBootstrapMixin, _TwoStageAggregationMixin, BaseEsti Must be a non-negative integer; ``bool`` is rejected. alpha : float, default=0.05 Significance level for confidence intervals. - cluster : str, optional + cluster : str, None or False, optional + ``False`` (4.0, M-080) is an accepted explicit no-op: the default + ``cluster_var = unit`` lives inside the unit-level influence-function + variance, so no unclustered variance exists. Column labels must be ``str``. Column name for cluster-robust standard errors. If None, clusters at the unit level by default. n_bootstrap : int, default=0 @@ -1318,7 +1327,7 @@ def __init__( self, anticipation: int = 0, alpha: float = 0.05, - cluster: Optional[str] = None, + cluster: ClusterArg = None, n_bootstrap: int = 0, bootstrap_weights: str = "rademacher", seed: Optional[int] = None, @@ -1341,7 +1350,7 @@ def __init__( self.anticipation = validate_anticipation(anticipation) self.alpha = alpha - self.cluster = cluster + self.cluster = validate_cluster(cluster) self.vcov_type = vcov_type validate_n_bootstrap(n_bootstrap) self.n_bootstrap = n_bootstrap @@ -1407,6 +1416,12 @@ def fit( # out-of-domain value silently changes the ESTIMAND); the assignment # also re-normalizes a mutated numpy scalar to a Python int. self.anticipation = validate_anticipation(self.anticipation) + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]): __init__ and set_params validate eagerly, so this only + # catches DIRECT attribute mutation. `False` is normalized to None + # locally so it never reaches a column lookup ([M-080]). + validate_cluster(self.cluster) + user_cluster = None if self.cluster is False else self.cluster # ---- Data validation ---- required_cols = [outcome, unit, time, first_treat] @@ -1438,7 +1453,7 @@ def fit( "Cannot use n_bootstrap > 0 with replicate-weight survey designs. " "Replicate weights provide their own variance estimation." ) - if _uses_replicate_ts and self.cluster is not None: + if _uses_replicate_ts and user_cluster is not None: raise NotImplementedError( "TwoStageDiD(cluster=...) with a replicate-weight survey design " "is not supported: replicate-weight variance " @@ -1575,10 +1590,10 @@ def fit( n_control_units = len(units_in_omega_0) # Cluster variable - cluster_var = self.cluster if self.cluster is not None else unit - if self.cluster is not None and self.cluster not in df.columns: + cluster_var = user_cluster if user_cluster is not None else unit + if user_cluster is not None and user_cluster not in df.columns: raise ValueError( - f"Cluster column '{self.cluster}' not found in data. " + f"Cluster column '{user_cluster}' not found in data. " f"Available columns: {list(df.columns)}" ) @@ -1599,7 +1614,7 @@ def fit( effective_cluster_ids = _resolve_effective_cluster( resolved_survey, cluster_ids_raw, - cluster_var if self.cluster is not None else None, + cluster_var if user_cluster is not None else None, ) resolved_survey = _inject_cluster_as_psu(resolved_survey, effective_cluster_ids) # When survey PSU is present, use it as the effective cluster for @@ -2036,14 +2051,14 @@ def _run_stage( # - `np.unique` (not `Series.nunique()`, which drops NaN) keeps the # same single NaN group the variance forms, so missing cluster IDs # cannot make the metadata undercount. - # `cluster_var` is `self.cluster`, or the `unit` column when + # `cluster_var` is `user_cluster`, or the `unit` column when # `cluster=None` (the Gardner sandwich always clusters at unit by # default), so the summary renders the unit-cluster CR1 label, not HC1. if resolved_survey is not None: _cluster_name_for_results: Optional[str] = None _n_clusters_for_results: Optional[int] = None else: - _cluster_name_for_results = self.cluster if self.cluster is not None else unit + _cluster_name_for_results = user_cluster if user_cluster is not None else unit _n_clusters_for_results = int(np.unique(df[cluster_var].values).size) # Event-study VCV / df persistence, gated by inference mode (spec diff --git a/diff_diff/utils.py b/diff_diff/utils.py index 3e9a49b37..366accd81 100644 --- a/diff_diff/utils.py +++ b/diff_diff/utils.py @@ -6,7 +6,7 @@ import os import warnings from dataclasses import dataclass, field -from typing import Any, Dict, Iterable, List, Optional, Tuple, Union +from typing import Any, Dict, Iterable, List, Literal, Optional, Tuple, Union import numpy as np import pandas as pd @@ -503,6 +503,42 @@ def validate_n_bootstrap(n_bootstrap: Any) -> None: raise ValueError(f"n_bootstrap must be a non-negative integer, got '{n_bootstrap}'") +#: The shared ``cluster`` constructor-argument contract (4.0, [M-155]): a column +#: name, ``None`` (the estimator's default clustering - unit auto-cluster where +#: one exists) or ``False`` (explicitly unclustered: the plain heteroskedasticity- +#: robust sandwich on the estimators whose clustering is a choice, an accepted +#: no-op on the by-construction unit-level estimators). +ClusterArg = Union[str, Literal[False], None] + + +def validate_cluster(cluster: object) -> ClusterArg: + """Validate the shared ``cluster`` constructor parameter and return it unchanged. + + Accepted values: ``None`` (the estimator's default clustering), ``False`` + (explicitly unclustered - the uniform 4.0 off-switch, [M-080]) or a column + name (``str``). Everything else - ``True``, numpy bools, integers or other + non-``str`` column labels, lists - raises ``ValueError`` with one shared + message ([M-155]; DMLDiD's pre-4.0 ``str``-only rule generalized). + + Adopted at ``__init__`` by every estimator exposing ``cluster`` except + ``StackedDiD`` (own ``'unit'``/``'unit_subexp'`` vocabulary validator) and + RE-RUN at the top of each ``fit()`` as the direct-mutation defense (the + :func:`validate_anticipation` pattern); ``BaseEstimator.set_params`` is + covered through its probe re-init. Callers normalize ``False`` to ``None`` + LOCALLY (``user_cluster = None if self.cluster is False else self.cluster``) + before any column lookup, so ``False`` never reaches ``data[...]``; the + stored attribute keeps the raw value so ``get_params`` round-trips. + """ + if cluster is None or cluster is False: + return cluster + if isinstance(cluster, str): + return cluster + raise ValueError( + "cluster must be a column name (str), None (the estimator's default clustering) " + f"or False (explicitly unclustered), got {cluster!r} (type {type(cluster).__name__})" + ) + + def validate_pscore_trim(value: Any) -> float: """Validate ``pscore_trim`` and return it coerced to ``float``. diff --git a/diff_diff/wooldridge.py b/diff_diff/wooldridge.py index b937c7b6e..7f299bbe4 100644 --- a/diff_diff/wooldridge.py +++ b/diff_diff/wooldridge.py @@ -22,6 +22,7 @@ from diff_diff._base import BaseEstimator from diff_diff.linalg import ( + _absorbed_fe_vcov_scale, compute_robust_vcov, effective_cluster_count, solve_logit, @@ -29,6 +30,7 @@ solve_poisson, ) from diff_diff.utils import ( + ClusterArg, absorbed_fe_cr1_k_increment, absorbed_fe_rank, pre_demean_norms, @@ -36,6 +38,7 @@ safe_inference, snap_absorbed_regressors, validate_anticipation, + validate_cluster, validate_df_convention, validate_n_bootstrap, within_transform, @@ -886,9 +889,16 @@ class WooldridgeDiD(BaseEstimator): replicate jwdid's ``xasis`` option. alpha : float Significance level for confidence intervals. - cluster : str or None + cluster : str, None or False Column name to use for cluster-robust SEs. Defaults to the ``unit`` - identifier passed to ``fit()``. + identifier passed to ``fit()``. ``False`` (4.0, M-080) switches the + clustering off explicitly: the plain heteroskedasticity-robust + sandwich on the OLS arms (within-transform hc1 carries the + absorbed-FE full-K rescale; residual-df t inference) and the plain + QMLE sandwich on the logit/poisson arms; rejected with + ``n_bootstrap > 0`` (the multiplier bootstrap draws at the cluster) + and with ``vcov_type="hc2_bm"`` (cluster-only Bell-McCaffrey DOF). + Column labels must be ``str``. n_bootstrap : int Number of bootstrap replications. 0 disables bootstrap. bootstrap_weights : {"rademacher", "webb", "mammen"} @@ -995,7 +1005,7 @@ def __init__( anticipation: int = 0, demean_covariates: bool = True, alpha: float = 0.05, - cluster: Optional[str] = None, + cluster: ClusterArg = None, n_bootstrap: int = 0, bootstrap_weights: str = "rademacher", seed: Optional[int] = None, @@ -1024,7 +1034,7 @@ def __init__( self.anticipation = validate_anticipation(anticipation) self.demean_covariates = demean_covariates self.alpha = alpha - self.cluster = cluster + self.cluster = validate_cluster(cluster) validate_n_bootstrap(n_bootstrap) self.n_bootstrap = n_bootstrap self.bootstrap_weights = bootstrap_weights @@ -1161,6 +1171,12 @@ def fit( # assignment also re-normalizes a mutated numpy scalar to int. self.anticipation = validate_anticipation(self.anticipation) self._validate_unsupported_period_action(self.unsupported_period_action) + # Fit-time re-check of the shared cluster contract (str | None | False, + # [M-155]): __init__ and set_params validate eagerly, so this only + # catches DIRECT attribute mutation. `False` is normalized to None + # locally so it never reaches a column lookup ([M-080]). + validate_cluster(self.cluster) + user_cluster = None if self.cluster is False else self.cluster df = data.copy() df = _warn_and_fill_nan_cohort(df, cohort, stacklevel=2) @@ -1268,6 +1284,23 @@ def fit( # error, so reject at the estimator boundary across both. The user # must drop bootstrap (``n_bootstrap=0``) or pick a cluster-compatible # ``vcov_type`` (``hc1`` or ``hc2_bm``). + # 4.0 (M-080): cluster=False switches the analytical clustering off, but + # the multiplier bootstrap draws one weight per analytical cluster - the + # combination is incoherent; and the Bell-McCaffrey DOF lane is + # cluster-only (TODO.md tracks the one-way BM DOF port). + if self.cluster is False and self.n_bootstrap > 0: + raise ValueError( + f"WooldridgeDiD(cluster=False, n_bootstrap={self.n_bootstrap}) is not " + "supported: the multiplier bootstrap draws one weight per cluster, so " + "it cannot run unclustered. Set n_bootstrap=0 for the plain HC sandwich, " + "or keep clustering (cluster=None auto-clusters at unit)." + ) + if self.cluster is False and self.vcov_type == "hc2_bm": + raise ValueError( + "WooldridgeDiD(cluster=False, vcov_type='hc2_bm') is not supported: the " + "Bell-McCaffrey DOF lane is cluster-only. Use vcov_type='hc2' or 'hc1' " + "with cluster=False, or keep clustering (cluster=None auto-clusters at unit)." + ) if self.n_bootstrap > 0 and self.vcov_type in ("hc2", "classical"): raise ValueError( f"WooldridgeDiD(vcov_type={self.vcov_type!r}, " @@ -1299,7 +1332,7 @@ def fit( conley_lag_cutoff=self.conley_lag_cutoff, survey_design=survey_design, inference="analytical", - cluster=self.cluster, + cluster=user_cluster, ) if self.n_bootstrap > 0: raise ValueError( @@ -2085,9 +2118,17 @@ def _fit_ols( # "classical"}`` explicitly (one-way families don't compose with # cluster_ids per the linalg validator). Explicit ``self.cluster=X`` # always wins. Mirrors SunAbraham PR #472 at ``sun_abraham.py:792-797``. - if self.cluster is not None: - cluster_col: Optional[str] = self.cluster - cluster_ids: Optional[np.ndarray] = sample[cluster_col].values + user_cluster = None if self.cluster is False else self.cluster + if self.cluster is False: + # 4.0 (M-080): the uniform explicit off-switch - plain HC sandwich + # (absorbed-FE full-K rescale below on the within lane), no + # auto-cluster, no survey PSU injection. Bootstrap / hc2_bm were + # rejected at fit(). + cluster_col: Optional[str] = None + cluster_ids: Optional[np.ndarray] = None + elif user_cluster is not None: + cluster_col = user_cluster + cluster_ids = sample[cluster_col].values elif self.vcov_type == "conley": # Conley: never auto-cluster at unit (a unit-cluster product kernel # would zero every between-unit spatial pair). Explicit cluster= @@ -2109,13 +2150,13 @@ def _fit_ols( # unit only kicks in when the user explicitly set ``cluster=X`` # (already handled above) — in that explicit-one-way + explicit-cluster # case the bootstrap matches the analytical cluster too. - bootstrap_cluster_col = self.cluster if self.cluster else unit + bootstrap_cluster_col = user_cluster if user_cluster else unit cluster_ids_bootstrap = sample[bootstrap_cluster_col].values # Resolve survey design, inject cluster as PSU only when user explicitly set cluster= - survey_cluster_ids = cluster_ids if self.cluster else None + survey_cluster_ids = cluster_ids if user_cluster else None resolved, survey_weights, survey_weight_type, survey_metadata, df_inf = ( - _resolve_survey_for_wooldridge(survey_design, sample, survey_cluster_ids, self.cluster) + _resolve_survey_for_wooldridge(survey_design, sample, survey_cluster_ids, user_cluster) ) # Branch design build on vcov_type. ``hc1`` keeps within-transform @@ -2323,6 +2364,32 @@ def _fit_ols( conley_lag_cutoff=self.conley_lag_cutoff, ) + # 4.0 (M-080): the unclustered hc1 within-transform lane (cluster=False) + # carries the absorbed-FE full-K rescale, so its SE equals the explicit + # full-dummy HC1 sandwich (fixest vcov="hetero") and matches the + # library's other within surfaces (LinearRegression, the TWFE + # event-study core). Conley also runs the within solve but is never + # rescaled (linalg's classical/hc1 gate); clustered and survey lanes + # carry their own K accounting. NaN-void on a non-positive full-K df. + _fe_rank_within: Optional[int] = None + if ( + vcov is not None + and cluster_ids is None + and resolved is None + and not use_full_dummy + and self.vcov_type == "hc1" + ): + _fe_rank_within = absorbed_fe_rank( + sample, [unit, time], has_intercept_col=False, weights=survey_weights + ) + _fe_scale_w = _absorbed_fe_vcov_scale( + float(X.shape[0]), int((~np.isnan(coefs)).sum()), _fe_rank_within + ) + if np.isnan(_fe_scale_w): + vcov = np.full_like(vcov, np.nan) + elif _fe_scale_w != 1.0: + vcov = vcov * _fe_scale_w + # Extract cohort-trend coefficients (paper W2025 Eq. 8.1 ``δ_g``). # Trend columns live at the tail of the full-dummy design after # the unit + time dummies. Empty dict when cohort_trends=False @@ -2458,11 +2525,15 @@ def _fit_ols( df_res = float( X.shape[0] - n_kept - - absorbed_fe_rank( - sample, - [unit, time], - has_intercept_col=False, - weights=survey_weights, + - ( + _fe_rank_within + if _fe_rank_within is not None + else absorbed_fe_rank( + sample, + [unit, time], + has_intercept_col=False, + weights=survey_weights, + ) ) ) _g_eff_w = ( @@ -2800,13 +2871,19 @@ def _fit_logit( f"method='logit' requires outcomes in [0, 1]. " f"Got range [{y.min():.4f}, {y.max():.4f}]." ) - cluster_col = self.cluster if self.cluster else unit - cluster_ids = sample[cluster_col].values + user_cluster = None if self.cluster is False else self.cluster + # 4.0 (M-080): cluster=False -> the plain QMLE HC1 sandwich + # (compute_robust_vcov with cluster_ids=None); inference stays + # normal-theory (or the survey df) - only the SE moves. + cluster_col: Optional[str] = None if self.cluster is False else (user_cluster or unit) + cluster_ids: Optional[np.ndarray] = ( + sample[cluster_col].values if cluster_col is not None else None + ) # Resolve survey design, inject cluster as PSU only when user explicitly set cluster= - survey_cluster_ids = cluster_ids if self.cluster else None + survey_cluster_ids = cluster_ids if user_cluster else None resolved, survey_weights, survey_weight_type, survey_metadata, df_inf = ( - _resolve_survey_for_wooldridge(survey_design, sample, survey_cluster_ids, self.cluster) + _resolve_survey_for_wooldridge(survey_design, sample, survey_cluster_ids, user_cluster) ) _has_survey = resolved is not None @@ -3006,7 +3083,9 @@ def _avg_ax0(a, cell_mask): # design-side structure instead. vcov_type=self.vcov_type, cluster_name=(None if _has_survey else cluster_col), - n_clusters=(None if _has_survey else int(np.unique(cluster_ids).size)), + n_clusters=( + None if (_has_survey or cluster_ids is None) else int(np.unique(cluster_ids).size) + ), _gt_weights=gt_weights, _n_g_per_cohort={ g: int(sample[sample[cohort] == g][unit].nunique()) @@ -3062,13 +3141,19 @@ def _fit_poisson( f"method='poisson' requires non-negative outcomes. " f"Got minimum value {y.min():.4f}." ) - cluster_col = self.cluster if self.cluster else unit - cluster_ids = sample[cluster_col].values + user_cluster = None if self.cluster is False else self.cluster + # 4.0 (M-080): cluster=False -> the plain QMLE HC1 sandwich + # (compute_robust_vcov with cluster_ids=None); inference stays + # normal-theory (or the survey df) - only the SE moves. + cluster_col: Optional[str] = None if self.cluster is False else (user_cluster or unit) + cluster_ids: Optional[np.ndarray] = ( + sample[cluster_col].values if cluster_col is not None else None + ) # Resolve survey design, inject cluster as PSU only when user explicitly set cluster= - survey_cluster_ids = cluster_ids if self.cluster else None + survey_cluster_ids = cluster_ids if user_cluster else None resolved, survey_weights, survey_weight_type, survey_metadata, df_inf = ( - _resolve_survey_for_wooldridge(survey_design, sample, survey_cluster_ids, self.cluster) + _resolve_survey_for_wooldridge(survey_design, sample, survey_cluster_ids, user_cluster) ) _has_survey = resolved is not None @@ -3268,7 +3353,9 @@ def _avg_ax0(a, cell_mask): # that wasn't actually applied. vcov_type=self.vcov_type, cluster_name=(None if _has_survey else cluster_col), - n_clusters=(None if _has_survey else int(np.unique(cluster_ids).size)), + n_clusters=( + None if (_has_survey or cluster_ids is None) else int(np.unique(cluster_ids).size) + ), _gt_weights=gt_weights, _n_g_per_cohort={ g: int(sample[sample[cohort] == g][unit].nunique()) diff --git a/docs/api/spillover.rst b/docs/api/spillover.rst index 33797ba18..60f6eeb6c 100644 --- a/docs/api/spillover.rst +++ b/docs/api/spillover.rst @@ -177,7 +177,8 @@ and planned follow-up enhancements: - **Gardner GMM first-stage correction at stage 2** — SHIPPED in Wave D. Stage-2 variance now applies the influence-function-based correction for stage-1 FE estimation uncertainty across all three ``vcov_type`` - paths (HC1, Conley, cluster) on both ``event_study=False`` AND + paths (the default CR1 at ``unit`` since 4.0 or an explicit ``cluster=``, + HC1 via ``cluster=False``, Conley) on both ``event_study=False`` AND ``event_study=True``. The IF formula is ``psi_i = gamma_hat' * X_{10,i} * eps_{10,i} - X_{2,i} * eps_{2,i}`` with ``gamma_hat = (X_10' X_10)^{-1} (X_1' X_2)``; the meat is diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index d691b59a9..a84d48767 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -131,6 +131,19 @@ inference through 3.x.) `TwoWayFixedEffects`' unit auto-cluster satisfies the cluster prerequisite; the survey and Conley `NotImplementedError` rejections keep precedence. The removed MultiPeriodDiD's warn-and-analytical-fallback went with the class at 4.0. + Since 4.0 ([M-080]) `cluster=False` is rejected on the same lane on BOTH classes: + the wild CLUSTER bootstrap draws at the cluster level, so an explicitly unclustered + fit has nothing to resample (TWFE's unit auto-cluster is switched off by `False`). +- **Note:** M-080 position (auto-cluster policy, category n/a in + `docs/methodology/variance-conventions.md` "Default clustering roster"; shipped item + 7b, 2026-10-29): the 2x2 `DifferenceInDifferences` stays HC-robust unless `cluster=` + is given, so the uniform 4.0 `cluster=False` off-switch is an accepted explicit no-op + here (plain HC = the default). The shared argument contract is `str | None | False` + ([M-155], `diff_diff.utils.validate_cluster`): `True` and non-`str` column labels raise + the shared `ValueError` at `__init__`, through `set_params` and again at `fit()`. + Analytical clustered fits report `n_clusters` from the regression's own effective + cluster count since 4.0 (non-survey fits; under a survey design the survey block owns + PSU reporting and `n_clusters` stays None). *Algorithm (test of H₀: τ = r, default r = 0):* 1. **Impose the null** by dropping the interaction column and re-fitting the reduced model; @@ -470,7 +483,10 @@ retired at 4.0 with it, row M-007 done). implicit-per-observation-PSU rule), and dropped for explicit one-way analytical families (`classical`/`hc2`). Explicit `cluster=` passes through and behaves exactly like the removed MultiPeriodDiD's explicit - cluster did on every lane (survey PSU injection included). + cluster did on every lane (survey PSU injection included). Since 4.0 + ([M-080]) `cluster=False` is the fourth carve-out: the explicit off-switch + runs the core's unclustered lanes (plain HC sandwich with the absorbed-FE + full-K rescale, residual-df t, one-way Bell-McCaffrey df under `hc2_bm`). - **Note:** `inference="wild_bootstrap"` in event-study mode raises `ValueError` from the mode's 3.9 birth - the WCR implementation covers the static ATT only, and the removed MultiPeriodDiD's silent analytical @@ -517,7 +533,26 @@ all regressors must be projected out of the same fixed effects space as the depe This matches the behavior of R's `fixest::feols()` with absorbed FE. *Standard errors:* -- Default: Cluster-robust at unit level (accounts for serial correlation) +- Default: Cluster-robust at unit level (accounts for serial correlation); + `cluster=False` (4.0, [M-080]) switches the clustering off explicitly - the plain + HC sandwich (see the M-080 Note below) +- **Note:** M-080 position (auto-cluster policy, category c1 in + `docs/methodology/variance-conventions.md` "Default clustering roster"; shipped item + 7b, 2026-10-29): the unit auto-cluster is the default on both modes; `cluster=False` + is the explicit off-switch - on the static mode the within-transform regression runs + with no cluster ids (`LinearRegression`'s absorbed-FE full-K rescale applies, so the + SE equals HC1 on the explicit unit + time dummy design and the t reference is the + full-K residual df; `df_convention` is inert without a cluster count), and + `vcov_type="hc2_bm"` + `cluster=False` runs the one-way (singleton-cluster) CR2 + Bell-McCaffrey lane - clubSandwich-verified on the static full-dummy surface (the hc2 + note below); on the event-study mode the same one-way hc2_bm lane carries no live + R pin yet (the open `TODO.md` row). `inference="wild_bootstrap"` + `cluster=False` + raises (the resampler needs a cluster). Metadata rule: `cluster_name` / `n_clusters` + describe the cluster ids actually passed to the analytical sandwich - `n_clusters` is + the regression's effective count on non-survey analytical fits (4.0; wild keeps the + bootstrap count), both None on unclustered fits, and under a survey design with no + explicit `cluster=` the auto label is None too (the auto-cluster is never injected as + a PSU, so the former `"unit"` label described ids the regression never received). - Degrees of freedom adjusted for absorbed fixed effects: component-aware rank via `diff_diff.utils.absorbed_fe_rank` — equal to the historical `n_units + n_times - 2` on a connected panel, and `sum(levels) - C - 1` in @@ -616,12 +651,13 @@ This matches the behavior of R's `fixest::feols()` with absorbed FE. unclustered, and conley alike — on lanes that were already t it is a new deliberate option, not a compatibility path). Fallback-level only under every value: survey/replicate df and per-coefficient Bell-McCaffrey DOF keep precedence, and point estimates/SEs/t-statistics - never move. **Since 4.0 the default is `"cluster"` on every surface** (LPDiD adopted it at + never move. **Since 4.0 the default is `"cluster"` on every knob surface** (LPDiD adopted it at 3.9; the other seven flipped at the 4.0 cut, item 7a, 2026-10-28 — lifecycle rows `docs/v4-deprecations.yaml` M-004..M-006 + M-128..M-131, pinned by `tests/test_v4_inference_defaults.py`; the flip moved every clustered p-value/CI, which is why it waited for a major version), and `"residual"` restores the 3.x residual-df inference - bit-for-bit. **Out of the knob's scope** (own inference stacks): the IF-based + bit-for-bit. **Out of the knob's scope** (own inference stacks): `SpilloverDiD` + (until [M-154], item 7c - its default CR1 lane runs residual-df t at 7b), the IF-based family — `CallawaySantAnna`, `TwoStageDiD`, `ChaisemartinDHaultfoeuille`, `EfficientDiD`, `ContinuousDiD`, `ChangesInChanges`, `SyntheticDiD`, `HeterogeneousAdoptionDiD` — and the OLS-based `TripleDifference` / `TROP` (own `safe_inference` df conventions, outside the @@ -697,7 +733,7 @@ This matches the behavior of R's `fixest::feols()` with absorbed FE. **Requirements checklist:** - [ ] Staggered adoption detection warning (only fires when `time` has >2 unique values; with binary `time`, staggering is undetectable) - [x] Multi-period time warning (fires when `time` has >2 unique values) -- [x] Auto-clusters standard errors at unit level +- [x] Auto-clusters standard errors at unit level (`cluster=False` = the uniform 4.0 off-switch) - [x] `decompose()` method returns BaconDecompositionResults - [x] Within-transformation correctly handles unbalanced panels - [x] Non-{0,1} binary time warning (fires when time has 2 unique values not in {0,1}) @@ -985,6 +1021,11 @@ Prior to the bare-`cluster=` wiring fix, `CallawaySantAnna(cluster="X")` was a s The `cluster_name` and `n_clusters` fields on `CallawaySantAnnaResults` report the effective clustering level: `survey_design.psu` (canonical column) when explicit PSU is provided, `self.cluster` when bare cluster synthesizes or injects. +- **Note:** M-080 position (auto-cluster policy, category b in + `docs/methodology/variance-conventions.md` "Default clustering roster"; shipped item + 7b, 2026-10-29): the per-unit influence functions are unit-level by construction, so the 4.0 auto-cluster flip is inert and no auto-cluster metadata is added (`cluster_name` stays None on the default; the RC-routed unbalanced-panel path keeps its synthesized unit PSU under `cluster=False` too - no-op means no-op). The shared `cluster` contract is `str | None | False` + ([M-155]): `False` is an accepted explicit no-op (bit-identical to `None`), `True` and + non-`str` labels raise the shared `ValueError`. - **Note (API decision — `cluster=` retained, NOT deprecated):** the `cluster=` → `SurveyDesign(psu=cluster)` synthesis above is an internal implementation detail, not a user-facing redundancy to be consolidated away. `cluster=` is the **canonical ergonomic single-level clustering kwarg** and is intentionally retained on `CallawaySantAnna` (and the sibling IF-based estimators `EfficientDiD` / `ImputationDiD` / `TwoStageDiD`): it matches the field's universal convention (R `fixest::feols(..., cluster = ~unit)`, Stata `vce(cluster id)`, statsmodels `cov_type="cluster"`), so users reach for `cluster=` first. `survey_design=SurveyDesign(psu=X, ...)` is the **advanced** entry point (adds strata / FPC / replicate weights / explicit weights); a bare `cluster=` is the shorthand for the common "just cluster at X" case and would be strictly less ergonomic if forced through `survey_design=`. This mirrors the HAD survey-API consolidation, which deprecated only the *redundant* `survey=` / `weights=` entry points in favor of `survey_design=` while deliberately keeping `cluster=`. Decision recorded 2026-07-04: do not deprecate `cluster=`; the former "decide whether to deprecate `CallawaySantAnna.cluster=X`" `TODO.md` row is closed as resolved (keep). *Estimator equation (as implemented):* @@ -1571,6 +1612,12 @@ The guard is fired by `_survey_se_from_group_if` (analytical and replicate) and *Notes #1, #2, #3, #4, #5, and #7 codify deviations from R `DIDmultiplegtDYN` (and from the paper's Equation 3 in the case of #1). Note #6 codifies a library extension with no R correspondence. The original scattered `**Note:**` and `**Note (deviation from R...):**` entries throughout the section above remain in place — this Deviations block is the canonical AI-review surface per CLAUDE.md "Documenting Deviations (AI Review Compatibility)" labels. Cross-references back to the existing Notes use semantic anchors (Phase / section names) rather than line numbers because the DCDH section is liable to shift as new contracts land; test-file references retain line numbers / class names because test files are more stable.* +- **Note:** M-080 position (auto-cluster policy, category b in + `docs/methodology/variance-conventions.md` "Default clustering roster"; shipped item + 7b, 2026-10-29): the cohort-recentred influence-function plug-in clusters at the group level by construction; a column name still raises `NotImplementedError` (custom clustering is reserved), while `cluster=False` passes the shared validator first and is an accepted no-op. The shared `cluster` contract is `str | None | False` + ([M-155]): `False` is an accepted explicit no-op (bit-identical to `None`), `True` and + non-`str` labels raise the shared `ValueError`. + 1. **Deviation from R / Deviation from the paper (Equation 3):** Equal-cell weighting — each `(g,t)` cell contributes equally regardless of within-cell observation count. AER 2020 Equation 3 prescribes `N_{d,d',t} = sum_g N_{g,t}` (observation sums); R `DIDmultiplegtDYN` weights by cell size. Phase 2 estimands (`DID_l`, `DID^{pl}_l`, `DID^n_l`, delta cost-benefit) inherit the same contract. Locked in `tests/test_chaisemartin_dhaultfoeuille.py::TestDropLargerLower::test_cell_count_weighting_unbalanced_input`. Cross-references the Phase 1 Theorem 3 equation block above (where `N_{a,b,t}` is documented as the count of `(g, t)` cells in each transition state) and `METHODOLOGY_REVIEW.md` § DCDH Deviations #1. 2. **Deviation from R:** Period-based stable-control sets (`stable_0(t)` = any cell with `D_{g,t-1} = D_{g,t} = 0` regardless of baseline `D_{g,1}`) — R uses cohort-based control sets that additionally require baseline `D_{g,1}` to match the side. Pure-direction panels agree exactly; ~1% point-estimate divergence on mixed-direction panels where joiners' post-switch cells could serve as leavers' controls. SE parity gap on pure-direction scenarios narrowed from ~18% to ~3% after the Round 2 full-IF fix. Cross-references the existing `**Note (deviation from R DIDmultiplegtDYN):**` in the period-vs-cohort discussion above and `METHODOLOGY_REVIEW.md` § DCDH Deviations #2. 3. **Deviation from R:** Balanced-baseline panel required + interior-gap drops + terminal-missingness retention + cell-period-allocator targeted `ValueError` — one composite deviation with four enforcement paths. Step 5b validation in `fit()` enforces the contract via `ValueError` (missing baseline) / `UserWarning` (interior gaps) / silent retention (terminal missingness). R accepts unbalanced panels. The cell-period allocator paths (Binder TSL with within-group-varying PSU, Rao-Wu replicate ATT, cell-level wild PSU bootstrap) have a targeted `ValueError` when cohort recentering would leak nonzero centered IF mass onto cells with no positive-weight observations. The four enforcement paths share a single underlying contract — "the panel must be balanced at baseline; terminal missingness is the only allowed unbalance; downstream variance machinery refuses to silently leak IF mass past the cell-period boundary". Cross-references the existing `**Note (deviation from R DIDmultiplegtDYN):**` in the ragged-panel discussion above (which itself details the three affected cell-period-allocator sub-paths) and `METHODOLOGY_REVIEW.md` § DCDH Deviations #3. @@ -1908,7 +1955,7 @@ where `q_{g,e} = pi_g / sum_{g' in G_{trt,e}} pi_{g'}`. - **Note:** Hausman pretest operates on the post-treatment event-study vector ES(e) per Theorem A.1. Both PT-All and PT-Post fits are aggregated to ES(e) using cohort-size weights before computing the test statistic H = delta' V^{-1} delta where delta = ES_post - ES_all and V = Cov(ES_post) - Cov(ES_all). Covariance is computed from aggregated ES(e)-level EIF values. The variance-difference matrix V is inverted via Moore-Penrose pseudoinverse to handle finite-sample non-positive-definiteness. Effective rank of V (number of positive eigenvalues) is used as degrees of freedom. - **Note:** Last-cohort-as-control (`control_group="last_cohort"`) reclassifies the latest treatment cohort as pseudo-never-treated and drops time periods at `t >= last_g - anticipation`, excluding anticipation-contaminated periods from the pseudo-control's pre-treatment window. This is distinct from CallawaySantAnna's `not_yet_treated` option which dynamically selects not-yet-treated units per (g,t) pair. - **Note:** `vcov_type` is permanently narrow to `{"hc1"}` per the Chen-Sant'Anna-Xie (2025) EIF-based variance achieving the semiparametric efficiency bound. Analytical-sandwich families `{classical, hc2, hc2_bm}` are rejected at `__init__` — the per-unit EIF aggregation has no equivalent single design matrix on which hat-matrix leverage or Bell-McCaffrey Satterthwaite DOF can be defined. `cluster=` invokes Liang-Zeger CR1 on cluster-aggregated EIF (`_compute_se_from_eif` with `cluster_indices`); `survey_design=` invokes TSL on the combined IF (`_compute_survey_eif_se`); both live in `diff_diff/efficient_did_aggregation.py` since the M-023 post-fit aggregate() extraction. `vcov_type='conley'` deferred to the EfficientDiD Conley follow-up row in DEFERRED.md. -- **Note:** Default `cluster=None` (no survey design) renders summary label "HC1 heteroskedasticity-robust" because the per-unit EIF SE `sqrt(mean(EIF²)/n)` is methodologically HC1-style (no Liang-Zeger G/(G-1) finite-sample correction). `EfficientDiDResults.cluster_name` and `n_clusters` stay None under unclustered fits. This diverges from `ImputationDiD` which auto-clusters at unit per Borusyak-Jaravel-Spiess (2024) Theorem 3 — there the default summary renders the CR1 unit-clustered label. +- **Note:** Default `cluster=None` (no survey design) renders summary label "HC1 heteroskedasticity-robust" because the per-unit EIF SE `sqrt(mean(EIF²)/n)` is methodologically HC1-style (no Liang-Zeger G/(G-1) finite-sample correction). M-080 position (category b; shipped item 7b, 2026-10-29): `cluster=False` is an accepted explicit no-op (bit-identical to the default) on `fit()` and on the `hausman_pretest` classmethod's own `cluster` argument; the shared `str | None | False` contract ([M-155]) applies. `EfficientDiDResults.cluster_name` and `n_clusters` stay None under unclustered fits. This diverges from `ImputationDiD` which auto-clusters at unit per Borusyak-Jaravel-Spiess (2024) Theorem 3 — there the default summary renders the CR1 unit-clustered label. - **Note:** `set_params(vcov_type=bad)` raises immediately on EVERY estimator: since the shared `BaseEstimator` mixin (`diff_diff/_base.py`, v4 2(c)-i), `set_params` validates transactionally by constructor probe re-init, so it enforces exactly `__init__`'s validation, eagerly, library-wide. The former split — EfficientDiD eager vs `ImputationDiD`/`TripleDifference`/`CallawaySantAnna` (and six more: SunAbraham, StackedDiD, the since-removed StaggeredTripleDifference, SpilloverDiD, TROP, PreTrendsPower) accepting constructor-rejected values until `fit()` — is retired; the fit-time re-validation layers remain as a second check against DIRECT attribute mutation (`est.vcov_type = ...`), which no setter can see. - **Note (post-fit aggregate() - rows M-023/M-120):** `fit(aggregate=, balance_e=)` was deprecated in 3.9 and REMOVED at 4.0 (PR 3b of the cut: `fit()` computes no event-study/group surface, the legacy `event_study_effects`/`group_effects`/`event_study_df` fields are None after EVERY fit, `to_dataframe(level='event_study'|'group')` raises on fresh fits until the M-093 schema rewire, and `survey_design`/`store_eif` are positional slots 7/8) in favor of post-fit `EfficientDiDResults.aggregate(type, balance_e=)` - a LAZY RECOMPUTING KIT (the CallawaySantAnna class, not a StackedDiD/dCDH view relay): `fit()` computes nothing extra, the results object retains an `AggregationKit`, and `aggregate('event_study'/'group', balance_e=)` re-runs the extracted `_EfficientAggregationMixin` aggregators on a throwaway host while `aggregate('simple')` relays the stored overall row bit-exact. (a) RETAINED BUFFERS (memory contract; phrased as maxima - optional design fields stay None when unsupplied): the per-(g,t) EIF dict, O(n_units x n_gt), the dominant payload - retained on EVERY fit regardless of `store_eif`, which since 3.9 governs only the public `influence_functions` field; `unit_cohorts` (cohort labels), `unit_level_weights`, factorized cluster codes - O(n_units) each; on ordinary (TSL) survey fits the unit-level `ResolvedSurveyDesign` adds `weights` plus, where supplied, `strata`/`psu`/`fpc` (factorized int codes / float values, never raw labels) - up to four O(n_units) arrays; on replicate designs it adds the O(n_units x n_replicates) replicate matrix plus, where supplied, `replicate_strata`/`replicate_rscales` (O(n_replicates)); per-row dict SNAPSHOTS of `group_time_effects` plus copies of the `groups`/`time_periods` lists and the scalar `pt_assumption`/`n_treated+n_control` provenance (aggregate() recomputes exclusively from these private snapshots, never from the mutable public result fields - a user edit of the public rows cannot mix altered point estimates with the retained EIF variance); scalars `n_units`, `cohort_fractions`, and the POST-OVERALL `df_survey` snapshot (captured after the overall inference and before the ES/group gates: the group pass can degenerate the working df to None on replicate designs with `n_valid <= 1`, and every fit-time aggregation seeds from the post-overall value, so recompute replays the exact seed). The data-minimization guarantee is scoped to unit identifiers - no unit-label container is retained. (b) `balance_e` uses the ANCHOR-HORIZON rule (keep cohorts with a finite effect at the anchor horizon, then retain all their horizons) - the same rule SHAPE CallawaySantAnna uses, with one keying-granularity difference (EDiD anchors on the `int(t - g)` bucket, CS on raw `t - g` - identical on integer-period panels; see the truncation Note below), divergent from ImputationDiD/TwoStageDiD's balanced-window rule; an anchor no cohort reaches warns and yields a legal zero-row container. (c) BOOTSTRAP fits: 'simple' (and 'total') RELAY the stored overall row verbatim (percentile se/p/CI beside the finite safe_inference t) with a NaN df column and never re-warn, while the RECOMPUTE levels (ES/group, balance_e included) REPLAY the fit-time multiplier bootstrap from the kit's `BootstrapReplaySpec` (the fit-captured RNG state + run params BY VALUE - seed=None fits replay; post-fit `set_params`/attribute mutation cannot alter it; pickles carry it): se/CI/t match a fit-time aggregation to BLAS reassociation (~1 ULP, `assert_allclose` - never bit-identity), the percentile p-value is a count statistic (compared at 2/n_bootstrap), and the replay publishes no analytical provenance (percentile overrides via the shared `bootstrap_utils` appliers; group rows' `df_used` cleared exactly as at fit time; EDiD has no sup-t cband). The spec stamps the weight backend at capture ('rust'/'numpy', or 'portable' for backend-independent branches - stratified survey generation, census-FPC zero weights, the single-PSU degenerate early return); a replay under a different backend fails closed naming both, and legacy pickles without the spec fail closed with a refit message. Replays RE-EMIT the fit-time bootstrap warnings for the replayed configuration (low n_bootstrap, single-PSU, empty balance_e anchor, non-finite-draw RuntimeWarnings); the relay levels stay silent. Per-call cost O(n_bootstrap x n_units x n_gt), no memoization. The M-027 rationale - no level publishes analytical-provenance fields beside percentile inference - is honored by the percentile overrides and NaN df channels. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is REJECTED BY DESIGN (both terminal TypeErrors state it): the primary ground is the absent joint event-study covariance (container `vcov=None`; the PER-ROW df column now threads the post-overall `df_survey` snapshot via the results-level `event_study_df` channel on analytical survey fits - M-092 completion; None on non-survey/bootstrapped/no-surface fits and for the replicate-undefined 0 sentinel - beside the scalar `df_survey` channel; see the CS post-fit Note's two-channel contract). Reference semantics are regime-dependent: under `pt_assumption="all"` there is NO reference row (universal first-period baseline; e=-1 is a genuine estimate); under `"post"` the per-cohort baseline cell is materialized as a mechanical zero anchor at `e = -1 - anticipation` whenever it is not the panel's first period, and the MEMBERSHIP-GATED `reference_period` property (the SunAbraham rule - never synthesized when the anchor cell was not estimated) marks it `is_reference` in the container and corrects `plot_event_study`'s inferred reference (previously the `-1` fallback) on PT-Post `anticipation>0` fits. (e) 'simple' relay conventions: `target="att"`, `n = n_treated_units + n_control_units` with `n_kind="units"` (DISJOINT by construction - `last_cohort` trimming reassigns before the counts, so a true total exists, unlike StackedDiD's overlapping sets), `df` = the post-overall snapshot (provenance-exact where `survey_metadata.df_survey` can diverge in the degenerate replicate state); 'group' relay: `n_kind="cells"`, `weight=None` (equal within-cohort weights, no cross-cohort mass), per-row `df_used` array captured at each row's `safe_inference` call (exact by construction; a stated divergence from CS's conservative-min scalar broadcast); 'event_study' rides the shared `_from_relative_dict` builder via a carrier whose `survey_metadata` copy carries the snapshot `df_survey` and whose `event_study_df` threads the same snapshot into the per-row df column (cleared on bootstrap replays and for the 0 sentinel). (f) `aggregate('total')` (3.10): the estimator-owned total incremental outcome - an exact relay `C x overall` CONDITIONAL on the realized aggregation mass, with `C = sum(n_treated)` over the kept post-anticipation FINITE-effect cells of the kit's deep-copied `group_time_effects` snapshot (an exact integer sum; NEVER the routinely-non-integral `n_units x sum(cohort_fractions)` float product). Unweighted fits only: ANY declared `survey_design=` fails closed via the kit's `unit_level_weights`/`resolved_survey_unit` markers (weight-type-agnostic - unweighted psu-only and analytic fweight designs, whose resolved weights stay RAW, gate identically; EDiD never synthesizes an internal design, so the markers are exact declaration provenance and post-fit mutation of the public `survey_metadata` cannot bypass them). No keepers -> NaN mass -> all-NaN row (no re-warn); with finite `C` the relay is verbatim (inherited NaNs pass through; att/n never blanked). Bootstrap fits relay with a NaN df column (M-027); df carrier = the post-overall `df_survey` snapshot; container: single `target='total'` row, `n_kind='obs'`, the per-level n-semantics note in the CS total Note applies. MMM: the container is admitted with NO scale (see the MMM section); the survey/RC totals remainder and its att*dC variance term are the DEFERRED row. - **Note (fractional-period truncation bucketing):** the event-study equation defines `ES(e)` at an exact relative time, but the implementation buckets horizons by `int(t - g)` - truncation TOWARD ZERO - on the analytical aggregator, the bootstrap ES prep (all three keying sites, the `balance_e` anchor filter included; aligned 2026-08), and `hausman_pretest`'s internal aggregation. On integer-period panels `int()` is the identity and nothing below applies. On fractional-period panels (accepted by `fit()` without an integrality check): (i) bucket 0 is DOUBLE-WIDTH, spanning `(-1, 1)`, so the pre-treatment horizon e=-0.5 pools into the post-treatment e=0 bucket; e=1.0/1.5 pool into bucket 1 and e=-1.5/-1.0 into bucket -1; (ii) under `pt_assumption="post"` the bucket at `-1 - anticipation` pools genuine estimated fractional pre-treatment horizons (e.g. raw e=-1.5) with the mechanical zero anchor, and the container's reference normalization then publishes the WHOLE pooled row as `att=0.0` with NaN inference (`is_reference=True`) - genuine fractional-horizon estimates are subsumed into the reference row (the fit-time `event_study_effects` dict keeps the pooled effect + percentile inference unrewritten; the rewrite is container-only); (iii) WEIGHTING is CELL-MASS within the bucket - the aggregator appends one `(effect, pi_g)` term per CELL and normalizes, so a cohort contributing k cells to a bucket carries k*pi_g mass, whereas the exact-relative-time equation has one term per cohort - while `n_groups` counts DISTINCT cohorts in the bucket (identity with the cell count on integer panels). Every aggregation that truncation-buckets a fractional horizon emits a UserWarning naming this Note. @@ -1946,7 +1993,24 @@ where weights ŵ_{g,e} = n_{g,e} / Σ_g n_{g,e} (sample share of cohort g at eve - `vcov_type ∈ {"classical","hc1","hc2","hc2_bm"}` supported as of Phase 1b PR 1/8 (mirrors the DiD/MPD/TWFE chain established in Phase 1a): - `"hc1"` (default): Eicker-Huber-White HC1 with cluster-at-unit default. - Auto-clusters at unit unless an explicit `cluster=` is passed. + Auto-clusters at unit unless an explicit `cluster=` is passed; + `cluster=False` (4.0, [M-080]) switches the clustering off explicitly (the + plain HC1 sandwich on the saturated regression - see the M-080 Note). +- **Note:** M-080 position (auto-cluster policy, category c1 in + `docs/methodology/variance-conventions.md` "Default clustering roster"; shipped item + 7b, 2026-10-29): `cluster=False` runs the saturated regression with no cluster ids + (absorbed-FE full-K rescale on the within spec, residual-df t; `df_convention` inert), + never injects a unit PSU under `survey_design=` (a design without its own PSU falls + to implicit per-observation PSUs: the TSL SE and `df_survey` move - this IS the + off-switch on the survey lane), and is REJECTED with `vcov_type="hc2_bm"` (the + Bell-McCaffrey DOF helper is cluster-only; the one-way port is a `TODO.md` row). The + pairs bootstrap ACCEPTS it as a no-op: it resamples units on every lane regardless + of `cluster` (the explicit-`hc2` precedent). Metadata (4.0): `SunAbrahamResults` + gains `n_clusters` (appended field) and `cluster_name` names the cluster ids the + analytical regression received (`"unit"` on the default, the explicit column, the + Conley product-kernel column); both None on unclustered, survey and pairs-bootstrap + fits. `summary()` renders the variance-family label on every analytical non-survey + fit (previously Conley only). - `"classical"`: homoskedastic OLS standard errors. Auto-cluster is dropped (one-way only). Routes through the full-dummy saturated design (see Implementation note below) for R-parity. @@ -2190,7 +2254,8 @@ where `W_it(h) = 1[K_it = h]` are lead indicators, estimated on `Omega_0` only. - Bootstrap does not update pre-period SEs (they are from the lead regression) - When `balance_e` is set, lead indicators are restricted to balanced cohorts; the full Omega_0 sample (including never-treated) is kept for within-transformation - Only affects event study aggregation; overall ATT and group aggregation unchanged -- **Note:** M-080 position (auto-cluster policy, re-scoped 2026-10-18 - category c2): +- **Note:** M-080 position (auto-cluster policy, re-scoped 2026-10-18 - category c2; + shipped item 7b, 2026-10-29 - `cluster=False` is accepted and bit-identical to the default): the default `cluster_var = unit` lives INSIDE the unit-level influence-function variance, so no unclustered variance exists; the uniform 4.0 `cluster=False` off-switch is an accepted explicit no-op (not an error, not a new SE). Roster in @@ -2276,7 +2341,8 @@ where `W_it(h) = 1[K_it = h]` are lead indicators, estimated on `Omega_0` only. **Primary source:** [Gardner, J. (2022). Two-stage differences in differences. arXiv:2207.05943.](https://arxiv.org/abs/2207.05943) -- **Note:** M-080 position (auto-cluster policy, re-scoped 2026-10-18 - category c2): +- **Note:** M-080 position (auto-cluster policy, re-scoped 2026-10-18 - category c2; + shipped item 7b, 2026-10-29 - `cluster=False` is accepted and bit-identical to the default): as for ImputationDiD, the default `cluster_var = unit` is internal to the unit-level influence-function machinery, so the uniform 4.0 `cluster=False` off-switch is an accepted explicit no-op. Roster in `docs/methodology/variance-conventions.md`. @@ -2484,7 +2550,9 @@ The paper text states a stricter bound (T_min + 1) but the R code by the co-auth - [x] Never-treated encoding (0 and inf) - [x] Survey design support (Phase 3): Q-weights compose multiplicatively with survey weights; TSL vcov on composed weights; survey design columns propagated through sub-experiments. Replicate weights supported via estimator-level refit with Q-weight composition (see Replicate Weight Variance section). - **Note:** M-080 position (auto-cluster policy, re-scoped 2026-10-18 - category c3 - in `docs/methodology/variance-conventions.md` "Default clustering roster"): the + in `docs/methodology/variance-conventions.md` "Default clustering roster"; 7b, + 2026-10-29: still rejected, unchanged - StackedDiD keeps its own vocabulary validator + outside the shared [M-155] contract): the stacked regression clusters by construction and `cluster` accepts only `'unit'` / `'unit_subexp'`; the uniform 4.0 `cluster=False` off-switch is therefore REJECTED here with the existing validator message rather than yielding an unclustered SE @@ -2582,12 +2650,27 @@ where `g(·)` is the link inverse (logistic or exp), `η_i` is the individual li - **Note (outcome-fit hint):** When `method="ols"` is used on a **binary** (`{0,1}`) or a **non-negative integer count** outcome, `WooldridgeDiD.fit()` emits a `UserWarning` noting that a matching nonlinear model (`method="logit"` / `method="poisson"`) is often the **more appropriate specification** for such outcomes. The framing is the paper's, not an efficiency heuristic: the nonlinear paths impose parallel trends on the **link/index scale** (logit index / Poisson log-mean) rather than in **levels**, and Wooldridge (2023) states the **linear-PT assumption is only valid for continuous/unbounded outcomes** (Eq. 2.5 vs the Index-PT Eq. 2.6–2.7). In the paper's **Section 5 simulations** the linear model is both **biased** (POLS −0.15 to −0.29 for binary; OLS >30%, >50% in places, for counts) **and less precise** (nonlinear SEs ~30–70% smaller) where the nonlinear mean holds, and the paper notes pre-trends tests often **fail to detect** the linear model's misspecification. This is a **different identifying assumption** than linear OLS — which one fits depends on which parallel-trends restriction holds — so the warning frames the nonlinear model as a **recommended comparison, not an automatic switch or a free upgrade**. Per **Table 1**, Gaussian/OLS remains a valid QMLE for *any* response, so the hint never asserts OLS is mechanically wrong. **Framing provenance (two-sided):** the earlier "canonical link requirement tied to Prop 3.1" reading was **incorrect** (corrected in **PR #453 R1**); the warning is equally careful *not* to over-correct into framing the nonlinear path as a pure efficiency add-on, since the paper's case is primarily about **bias / appropriateness** (codex R1, reconciled against `docs/methodology/papers/wooldridge-2023-review.md` §5). Detection is deliberately high-signal — binary requires support exactly `{0,1}`; the count branch suggests Poisson for non-negative integers with `>2` distinct values (Poisson is the natural *unbounded* nonnegative/count model per Table 1; bounded-support / **binomial** outcomes with a known upper bound are not separately detected — that routing would need user-supplied metadata). Fractional / bounded-continuous outcomes are **not flagged by this heuristic** (fractional responses are themselves a logistic/Bernoulli case in Table 1; the heuristic simply does not attempt to detect them — a coverage limit, not a claim that OLS is preferred for such outcomes). **Detection sample:** the hint reads the *full* outcome column passed to `fit()` — outcome *type* is a property of the variable, so the named column is the right signal — and it is read *before* any row filtering. `WooldridgeDiD._filter_sample` still expresses the control-group choice through the design matrix (`_build_interaction_matrix`), **not** by dropping rows (invariant pinned by `tests/test_wooldridge.py::TestOutcomeFitHint::test_filter_sample_preserves_outcome_support`), but as of 3.9 the full column and the fitted sample **no longer always share the same outcome support**: a cohort with no ETWFE reference period has its observations excluded downstream (see the unidentified-cohorts Note above and ledger row `M-123`). Classifying on the variable remains the correct choice precisely because of that — a count is a count regardless of which rows survive identification, so the hint must not flip on a sample reduction. Always-on (suppress via `warnings.filterwarnings`), implemented as the pure helper `_suggest_nonlinear_method` + the OLS-only fit-time gate `0g`; it never alters the fit or raises. *Standard errors:* -- OLS: Cluster-robust sandwich estimator at the unit level (default) +- OLS: Cluster-robust sandwich estimator at the unit level (default); + `cluster=False` (4.0, [M-080]) = the plain HC sandwich (see the M-080 Note below) - Logit/Poisson: QMLE sandwich `(X'WX)^{-1} meat (X'WX)^{-1}` via `compute_robust_vcov(..., weights=w, weight_type="aweight")` where `w = p_i(1-p_i)` for logit or `w = μ_i` for Poisson - Delta-method SEs for ATT(g,t) from nonlinear models: `Var(ATT) = ∇θ' Σ_β ∇θ` - Joint delta method for overall ATT: `agg_grad = Σ_k (w_k/w_total) * ∇θ_k` - **Deviation from R:** R's `etwfe` package uses `fixest` for nonlinear paths; this implementation uses direct QMLE via `compute_robust_vcov` to avoid a statsmodels/fixest dependency. -- **Note:** QMLE sandwich uses `weight_type="aweight"` which applies `(G/(G-1)) * ((n-1)/(n-k))` small-sample adjustment. Stata `jwdid` uses `G/(G-1)` only. The `(n-1)/(n-k)` term is conservative (inflates SEs slightly). For typical ETWFE panels where n >> k, the difference is negligible. +- **Note:** QMLE sandwich uses `weight_type="aweight"` which applies `(G/(G-1)) * ((n-1)/(n-k))` small-sample adjustment. Stata `jwdid` uses `G/(G-1)` only. The `(n-1)/(n-k)` term is conservative (inflates SEs slightly). For typical ETWFE panels where n >> k, the difference is negligible. Under `cluster=False` (4.0, [M-080]) the GLM arms run `compute_robust_vcov(..., cluster_ids=None)`: the plain QMLE HC1 sandwich with the `n/(n-k)` factor; GLM inference stays normal-theory (or the survey df) - only the SE moves. +- **Note:** M-080 position (auto-cluster policy, category c1 in + `docs/methodology/variance-conventions.md` "Default clustering roster"; shipped item + 7b, 2026-10-29): `cluster=False` switches the unit auto-cluster off on every arm. On + the OLS within-transform lane (`hc1`, `cohort_trends=False`) the unclustered vcov + carries the absorbed-FE full-K rescale (`linalg._absorbed_fe_vcov_scale`), so each + cohort-time cell SE equals HC1 on the explicit full-dummy design (fixest + `vcov = "hetero"`) and the residual df is `n - k_cells - rank(unit, time FE)` (NaN-void + inference when that df is non-positive; `df_convention` is inert without a cluster + count); the full-dummy routes (`classical`/`hc2`/`hc2_bm`, `cohort_trends=True`) need + no rescale. Rejected with `n_bootstrap > 0` (the multiplier bootstrap draws one weight + per analytical cluster) and with `vcov_type="hc2_bm"` (cluster-only Bell-McCaffrey + DOF; the one-way port is a `TODO.md` row). Under a survey design no PSU is injected + (unchanged: injection was always explicit-`cluster=`-only here). `cluster_name` / + `n_clusters` are None on the unclustered lanes. *Variance families (`vcov_type`, OLS path only):* - `hc1` (default) — CR1 Liang-Zeger cluster-robust on the within-transformed design. Point estimates, residuals, and the unscaled cluster-score meat are unchanged by the within-transform (FWL preserves the score); the CR1 finite-sample scalar uses `K_reference` (3.9). The natural R anchor is `fixest::feols(y ~ | unit + time, cluster=~unit)` or Stata `jwdid` (both within-transform). **Parity with Stata `jwdid` (K_reference convergence, 3.9; `tests/test_etwfe_cs_stata_parity.py`):** the ATT(g,t) POINT estimates match `jwdid` exactly (~3e-8 on the `mpdta` panel, i.e. Stata's log-output rounding), and the `hc1` SEs now match `jwdid`/reghdfe at machine precision (ratio 1.0, spreads ~1e-15..1e-14) on all three committed arms AND at every rung of the committed G≈20..500 subsample ladder (`etwfe_cs_stata_golden.json` `ladder` block). Historically the SEs were uniformly SMALLER by a cluster-count-dependent factor (1.0280 at G=20 down to 1.0010 at G=500) — defect D2 (`docs/methodology/variance-conventions.md`): the clustered CR1 factor's `k` counted only the visible treatment-cell columns, omitting the absorbed FE not nested in the cluster. The applied correction is the closed form `K_reference = explicit columns + (1 if no intercept column) + rank(non-nested FE | nested)` (here `cells + T`), threaded as `cluster_k_adjustment`; the ladder doubles as the K-accounting probe (reghdfe `df_a == absorbed_fe_cr1_k_increment − 1` per rung). This is distinct from the `lm + clubSandwich` deviation below, whose factor is `k`-based. `CallawaySantAnna` shows no such gap - its SEs match Stata `csdid` outright - which localizes this to the ETWFE path rather than a library-wide convention. **Deviation from R `lm + clubSandwich::vcovCR(type="CR1S")`:** the full-dummy `lm` SE differs by a factor of `sqrt((n - K_reference) / (n - k_total))` because clubSandwich's `(n-1)/(n-p)` finite-sample correction counts ALL columns (intercept + treatment + unit dummies + time dummies = `k_total`) while WooldridgeDiD's within-transform CR1 counts `K_reference` (cells + constant + the non-nested time-FE rank; the cluster-NESTED unit dummies are the remaining difference). On the 240-obs / 51-column R-parity fixture (`k_total = 51`, `K_reference = 12`) this is `sqrt(228/189)` ≈ 10% (was ~11% under the historical cells-only count); on typical larger panels (n >> k_total) the gap shrinks to <2%. No public WooldridgeDiD code path exposes the `lm + CR1S` (CR1 cluster-robust on the full-dummy design) finite-sample correction — `vcov_type="hc2_bm"` routes to the CR2 Bell-McCaffrey sandwich on the full-dummy design (different variance estimator entirely), not CR1S. Users who need exact `lm + clubSandwich::vcovCR(type="CR1S")` parity must call `solve_ols` directly on a full-dummy design or fit via R. Same deviation pattern as SunAbraham PR #472 (`fixest::sunab` vs `lm + clubSandwich`). @@ -2669,7 +2752,7 @@ where `g(·)` is the link inverse (logistic or exp), `η_i` is the individual li - [x] Saturated cohort×time interaction design matrix - [x] Unit + time FE absorption (within-transformation) - [x] OLS, logit (IRLS), and Poisson (IRLS) fitting methods -- [x] Cluster-robust SEs at unit level for all methods +- [x] Cluster-robust SEs at unit level for all methods (`cluster=False` = the uniform 4.0 off-switch) - [x] ASF-based ATT for nonlinear methods with delta-method SEs - [x] Joint delta-method SE for aggregate ATT in nonlinear models - [x] Four aggregation types: simple, group, calendar, event_study @@ -2779,7 +2862,7 @@ Eq. 12 reuses the absorbing clean control and only restricts the treated set (a The paper specifies no standard-error formula (Section 1 defers to "standard, well-understood techniques"); the reference Stata `lpdid` uses `vce(cluster unit)`. The entries below document diff-diff's inference and scope choices. -1. **Note:** Standard errors are **cluster-robust at the unit level by default** - `cluster=None` auto-clusters at the unit identifier and the results record `cluster_name`/`n_clusters` - with a `t(G-1)` reference distribution (G = realized clusters in each horizon's clean-control sample). Matches Stata `lpdid` `vce(cluster unit)` - measured on the Stata-anchored surfaces (absorbing via the R/`teffects` arms; non-absorbing per Deviation 4's scoping: Eq. 12 everywhere, Eq. 13 post + pooled-post on the convention-neutral subsample; elsewhere the two implementations' clean-sample conventions differ, see Deviation 4); the paper prescribes no SE. **Since 3.9 (M-127) the reference is the `df_convention` knob's `"cluster"` value — LPDiD's DEFAULT, so nothing moved; since the 4.0 flip (item 7a) it is the library-wide default too**: `"residual"` opts into t(per-horizon residual df `n_eff − k_kept`; on the RA path `n_total − k0_kept − 1`, the pooled M-estimator's parameter count — nuisance coefficients plus the ATT — a library convention with no external anchor since Stata `teffects ra` reports z), and `"normal"` into z. The G here stays the RAW unique cluster count (deliberately not `effective_cluster_count`: LPDiD's reweights are strictly positive and there is no user weight column, so the two cannot diverge). **Carve-out to the flat `t(G-1)` claim:** the degenerate lanes — the unclustered-refit fallback when the clustered solve fails, the RA `G<=1` lane, and the saturated-design early return (`n_obs <= k`) — keep the literal `df=None` (normal theory, NaN provenance) under ALL knob values, exactly as before 3.9. +1. **Note:** Standard errors are **cluster-robust at the unit level by default** - `cluster=None` auto-clusters at the unit identifier and the results record `cluster_name`/`n_clusters`; since 4.0 ([M-080], category c1) `cluster=False` is the explicit off-switch: every per-horizon regression runs unclustered (the plain HC1 sandwich on the explicit-dummy design with its residual-df t; the RA path's unclustered IF variance `sum_i psi_i^2 / n^2`, `vcov_type="if_robust"`, Note 2; implicit per-observation PSUs under a survey design, `n_psu = n_obs`), `cluster_name`/`n_clusters` are None and the `event_study`/`pooled` tables carry no cluster count (the 3.x `self.cluster or unit` read treated `False` as the auto-cluster) - with a `t(G-1)` reference distribution (G = realized clusters in each horizon's clean-control sample). Matches Stata `lpdid` `vce(cluster unit)` - measured on the Stata-anchored surfaces (absorbing via the R/`teffects` arms; non-absorbing per Deviation 4's scoping: Eq. 12 everywhere, Eq. 13 post + pooled-post on the convention-neutral subsample; elsewhere the two implementations' clean-sample conventions differ, see Deviation 4); the paper prescribes no SE. **Since 3.9 (M-127) the reference is the `df_convention` knob's `"cluster"` value — LPDiD's DEFAULT, so nothing moved; since the 4.0 flip (item 7a) it is the library-wide default too**: `"residual"` opts into t(per-horizon residual df `n_eff − k_kept`; on the RA path `n_total − k0_kept − 1`, the pooled M-estimator's parameter count — nuisance coefficients plus the ATT — a library convention with no external anchor since Stata `teffects ra` reports z), and `"normal"` into z. The G here stays the RAW unique cluster count (deliberately not `effective_cluster_count`: LPDiD's reweights are strictly positive and there is no user weight column, so the two cannot diverge). **Carve-out to the flat `t(G-1)` claim:** the degenerate lanes — the unclustered-refit fallback when the clustered solve fails, the RA `G<=1` lane, and the saturated-design early return (`n_obs <= k`) — keep the literal `df=None` (normal theory, NaN provenance) under ALL knob values, exactly as before 3.9. 2. **Note:** The regression-adjustment (RA) covariate path (`reweight=True` with covariates/absorb) reports an **influence-function cluster variance** `sum_c (sum_{i in c} psi_i)^2 / n^2`, in the same family as `ImputationDiD`'s Theorem-3 / BJS variance (see "IF-based variance estimators vs analytical-sandwich estimators" above). Its single Gram inversion is routed through `linalg._rank_guarded_inv` (finite SE on the identified subspace under near-collinearity; NaN at rank 0). Unlike the default/weighted `solve_ols` `hc1`-cluster path - which applies the `(G/(G-1))*((n-1)/(n-K_reference))` finite-sample factor (see the nested-dummy Note below) - the RA IF variance carries **no finite-sample factor**, while both paths share the `t(G-1)` reference. **PR-B2 validated this asymmetry as faithful to the authors' own tooling**, not a defect: the no-factor RA convention matches the canonical Stata `teffects ra ... atet vce(cluster)`, while the default path matches `feols`/`reghdfe`. **Originally inferred from the authors' `lpdid_regression_adjustment.do` degrees-of-freedom comments, this is now MEASURED against a runnable `teffects`:** `benchmarks/stata/generate_lpdid_ra_golden.do` runs `teffects ra (Dy x i.time) (tdiff), atet vce(cluster unit)` on an independently reconstructed clean sample (a Stata port of the `generate_lpdid_golden.R` `prep`/`clean_h` recipe), and the library RA IF SE matches it to **~1e-16 at all 7 event-study horizons** - the SE ratio is a flat 1.0 (no `G/(G-1)`, no `(n-1)/(n-k)`), directly confirming the no-finite-sample-factor convention (golden `benchmarks/data/lpdid_ra_stata_golden.json`; test `tests/test_lpdid_ra_stata_parity.py`, which also gates the Stata clean sample's realized size and cluster count `(e(N), e(N_clust))` against the library's `(n_obs, n_clusters)`). The RA *point* estimate is triple-anchored: Python == R full-interaction (`i.dtreat##(i.time c.x)`) == Stata `teffects`, to ~1e-10..1e-13 (`tests/test_methodology_lpdid.py::test_ra_covariate_point`). Because Stata is node-locked and cannot run in CI (goldens are committed, exactly like the R arm), the CI-runnable guards remain the regression pin (`test_ra_covariate_se_regression_pin` / `RA_SE_PIN`) and the ungated Monte-Carlo coverage study `benchmarks/python/coverage_lpdid_ra.py` (~0.95 empirical coverage of the true effect at cluster counts G in {30, 100, 300}). 3. **Note:** Direct covariate inclusion (`reweight=False` with covariates/absorb) emits a `UserWarning`: per online Appendix B.2.2 it preserves the non-negative LP-DiD weighting result only under linear and homogeneous covariate effects, so the regression-adjustment path (`reweight=True`) is preferred. 4. **Deviation from R:** Scope - non-absorbing treatment (Section 4.2) implements the **entry-effect** estimands (`non_absorbing="first_entry"` / `"effect_stabilization"`, PR-C1). **PR-C2 R-parity-validated both modes against an INDEPENDENT `fixest::feols` reconstruction of the paper's Eq. 12 / Eq. 13 clean-sample restrictions** (point and SE match to ~1e-13/~1e-15 for the variance-weighted variants; the `effect_stabilization` reweighted point matches and its SE is pinned as a regression guard - a small weighted-cluster convention difference vs feols; `tests/test_methodology_lpdid.py::TestLPDiDNonAbsorbingParityR`). The recipe's independence was demonstrated when an earlier draft's Eq. 12 control off-by-one diverged from the already-correct library and was corrected against the paper, plus a hand-computed Python micro-check. **`alexCardazzi/lpdid`'s `nonabsorbing_lag` is NOT a faithful Eq. 13** (it clamps `treat_diff[<0]<-0`, so its clean-control window blocks only treatment turn-*ons* - a violation of Eq. 13's stated `Delta-D = 0` both-directions control condition; it also holds two convention differences on surfaces Eq. 13 does not specify: a forward placebo window, and NA-exclusion of pre-panel-treated rows where the library clamps pre-`min_t` to untreated): it diverges ~0.01-0.05 from Eq. 13 even on a monotone no-off-switch panel, so it is **recorded in the golden `meta` as a divergent third-party reference, not a parity gate** (the alexCardazzi-pooled precedent; the golden's `meta.alex_note`/`meta.boundary_note` strings predate this two-tier framing and are generation-time text - this entry is the living statement). `first_entry` (Eq. 12) has no R-package analogue (anchored on the independent feols recipe, and - since the Stata non-absorbing arm - on the authors' Stata `lpdid` package end-to-end via `nonabsorbing(, firsttreat notyet)`). **The Stata canonical SE is now MEASURED against the authors' `lpdid` package (SSC, v1.0.3) end-to-end** - the first external anchor for the non-absorbing reweighted SE, the non-absorbing pooled windows (points and SEs), and the Eq. 12 reweighted point: Eq. 12 agrees on the full committed panel at every surface (all event-study horizons incl. placebos + both pooled windows; att ~1e-14 vw / ~1.3e-8 rw, SE ~6e-17 vw / ~2.3e-9 rw), and Eq. 13 agrees at post horizons + pooled post on a convention-neutral subsample (att ~1.1e-14 vw / ~1.9e-8 rw, SE ~1.8e-16 vw / ~1.6e-9 rw; golden `benchmarks/data/lpdid_nonabsorbing_stata_golden.json`, test `tests/test_lpdid_nonabsorbing_stata_parity.py`). **Three measured convention differences vs the package**, adjudicated against the paper's stated Eq. 13 restriction (verified against the open-access working-paper text, whose Eq. 18 is the published JAE Eq. 13): (a) *exact-L re-entry spells* - Eq. 13's treated condition is a LEVELS condition (`D = 0` on `[t-L, t-1]`), which the library implements; the package's switch-free lag window additionally excludes an exit AT `t-L`, effectively requiring `L+1` untreated periods - **the package is stricter than the paper here**; (b) *pre-panel boundary handling* - the package's missing-lag semantics admit always-treated units as controls at early `t` where the library's clamp excludes them (a paper-silent surface; both are documented conventions); (c) *placebo windows* - the package builds `h<0` clean samples by recursive lagged intersection of its `CCS_0` indicator, the library uses the backward window `[t-max(L,-h), t-1]` (also paper-silent; both are conventions). The full-panel Eq. 13 divergence from (a)+(b) and the placebo/pooled-pre divergence from (c) are recorded in the golden and locked by divergence-documentation gates (att floors + sample-size mismatch asserts), so parity claims stay scoped: Eq. 13 full-panel and placebo/pooled-pre inference remain OUTSIDE the Stata anchor (the full-panel reweighted SEs stay pinned via `RW_SE_PIN`; the pooled-pre SE has no pin or anchor). Appendix-C exit-event dynamics and the non-absorbing RA path (the package's RA engine is `listreg`, and the committed non-absorbing panel has no covariate) remain deferred follow-ups. @@ -2797,7 +2880,7 @@ The paper specifies no standard-error formula (Section 1 defers to "standard, we - [x] Regression-adjustment covariate path (recommended) + direct-inclusion path with homogeneity warning (PR-B1) - [x] PMD base period; pooled pre/post estimands (PR-B1) - [x] `no_composition` option (PR-B1) -- [x] Cluster-robust SE at unit level by default; NaN-consistent inference via `safe_inference` (PR-B1) +- [x] Cluster-robust SE at unit level by default; `cluster=False` unclustered lanes (4.0, [M-080]); NaN-consistent inference via `safe_inference` (PR-B1) - [x] `LPDiDResults` with `summary()` / `to_dict()` / cluster metadata (PR-B1) - [x] doc-deps.yaml mapping for `diff_diff/lpdid.py` + `lpdid_results.py`; llms.txt / llms-full.txt catalog entries (PR-B1, test-enforced) - [x] B1 pure-Python tests: analytical DGPs + cross-estimator equivalence (CS / BJS / DiD; Cengiz-stacked dropped, documented) + unbalanced / interior-gap / RA-overlap / pmd-missing edge cases (PR-B1) @@ -2821,6 +2904,11 @@ Exact reviewed artifacts (PDF SHA-256) and live-verified SSRN metadata are pinne Full maintainer paper reviews (equation-level detail, replication targets): `docs/methodology/papers/lee-wooldridge-2025-review.md`, `docs/methodology/papers/lee-wooldridge-2026-review.md`. +- **Note:** M-080 position (auto-cluster policy, category b in + `docs/methodology/variance-conventions.md` "Default clustering roster"; shipped item + 7b, 2026-10-29): the collapsed unit cross-section (`n_obs = n_units`) is unit-level by construction, so no auto-cluster metadata is added (`cluster_name` None on the default); `cluster=False` is an accepted no-op on `fit()` and on the `robustness_pre_periods` / `sensitivity_no_anticipation` helpers' own `cluster` argument. The shared `cluster` contract is `str | None | False` + ([M-155]): `False` is an accepted explicit no-op (bit-identical to `None`), `True` and + non-`str` labels raise the shared `ValueError`. - **Note:** 4.0 results-contract conformance (the cut's PR 1, 2026-10-18). `summary()` / `print_summary()` take `alpha=None` under the family-wide M-146 guard (a value other than the fit-time alpha raises; previously the methods took no alpha). `to_dataframe(level=)` @@ -3138,7 +3226,9 @@ the finite-dimensional `p_0` is handled by the variance correction below. `cluster=None` IS the per-unit IF variance by definition (see "IF-based variance estimators vs analytical-sandwich estimators"); the inherited value is family-consistent and truthful. No vocabulary widening. -- **Note:** M-080 position (auto-cluster policy, re-scoped 2026-10-18 - +- **Note:** M-080 position (auto-cluster policy, re-scoped 2026-10-18; shipped item 7b, + 2026-10-29 - `cluster=False` accepted and bit-identical to the default; the pre-4.0 + DMLDiD-local `str`-only check is now the shared [M-155] contract - category b in `docs/methodology/variance-conventions.md` "Default clustering roster") — DMLDiD's per-unit influence-function variance IS unit-level clustering, so the 4.0 auto-cluster DEFAULT flip is inert for it, as for @@ -4090,6 +4180,11 @@ contract changes. - [x] Survey design support: all methods (reg, IPW, DR) with weighted OLS/logit + TSL on combined influence functions. Weighted solve_logit() for propensity scores in IPW/DR paths. - **Note:** TripleDifference survey SE: for IPW/DR, pairwise IFs incorporate survey weights via weighted Riesz representers (`riesz *= weights`), so the combined IF is divided by per-observation survey weights (`inf / sw`) before passing to `compute_survey_vcov()` to prevent double-weighting. For regression (RA), pairwise IFs are already on the unweighted residual scale (WLS fits use weights internally but the IF is not Riesz-multiplied), so the combined IF passes directly to TSL without de-weighting. The OLS nuisance IF corrections in DR mode use weighted cross-products normalized by subgroup row count `n` (not `sum(weights)`). - **Note (vcov_type contract):** `vcov_type` is permanently narrow to `{"hc1"}` per the IF-based variance decomposition. Analytical-sandwich families `{classical, hc2, hc2_bm}` are rejected at `__init__` with a methodology-rooted message citing Ortiz-Villavicencio & Sant'Anna (2025) — the 3-pairwise-DiD decomposition has no single design matrix on which hat-matrix leverage or Bell-McCaffrey Satterthwaite DOF can be defined. `cluster=` continues to invoke Liang-Zeger CR1 on the combined influence function (`(G/(G-1)) · Σ_c (Σ_{i∈c} ψ_i)² / n²`, plain CR1 — no Stata-style `(n-1)/(n-p)` finite-sample factor because the IF has no design-matrix `p` in the OLS sense); `survey_design=` continues to invoke TSL on the combined IF. `vcov_type='conley'` is deferred to the TripleDifference Conley follow-up row in `DEFERRED.md`. See ["IF-based variance estimators vs analytical-sandwich estimators"](#if-based-variance-estimators-vs-analytical-sandwich-estimators) above for the structural taxonomy. +- **Note:** M-080 position (auto-cluster policy, category b in + `docs/methodology/variance-conventions.md` "Default clustering roster"; shipped item + 7b, 2026-10-29): on the 2x2x2 mode `cluster=False` is the plain HC default (category n/a: an accepted no-op); on the staggered mode (per-unit influence functions, bootstrap inference) a column name raises as before while `False` is an accepted no-op. The shared `cluster` contract is `str | None | False` + ([M-155]): `False` is an accepted explicit no-op (bit-identical to `None`), `True` and + non-`str` labels raise the shared `ValueError`. - **Note (`cluster=` + replicate-weight survey rejection):** `TripleDifference(cluster=X)` + `SurveyDesign(replicate_weights=[...], replicate_method=...)` is rejected at `fit()` with `NotImplementedError`. Replicate-weight variance is computed by replicate reweighting (BRR / Fay / JK1 / JKn / SDR) and ignores PSU/cluster entirely (the survey-side gate at `survey.py:104-109` enforces `replicate_weights` are mutually exclusive with `strata`/`psu`/`fpc`); honoring `cluster=` here would silently have no effect on the variance estimate while populating `cluster_name`/`n_clusters` on Results dishonestly. Mirrors the `CallawaySantAnna` guard at `staggered.py:1705-1719`. Either omit `cluster=` (the replicate weights encode the design structure implicitly) or use a non-replicate survey design with explicit `strata`/`psu`/`fpc`. --- @@ -5136,6 +5231,11 @@ Shipped in `diff_diff/had_pretests.py` as `stute_joint_pretest()` (residuals-in - [x] Phase 2a: Panel validator (`diff_diff.had._validate_had_panel`) verifies `D_{g,1} = 0` for all units, rejects negative post-period doses (`D_{g,2} < 0`) front-door on the original (unshifted) scale, rejects `>2` time periods on the two-period (overall) path (multi-period panels run the event-study mode, Phase 2b), and rejects unbalanced panels and NaN in outcome/dose/unit columns. Both Design 1 paths (`continuous_near_d_lower` and `mass_point`) additionally require `d_lower == float(d.min())` within float tolerance; mismatched overrides raise with a pointer to the unsupported (LATE-like / off-support) estimand. - [x] Phase 2a: NaN-propagation tests covering constant-y, degenerate-mass-point, and single-cluster-CR1 inputs. The guaranteed NaN coupling is on the DOWNSTREAM triple (`t_stat`, `p_value`, `conf_int`) via the `safe_inference()` gate, which returns NaN on all three whenever `se` is non-finite, zero, or negative. `att` and `se` themselves are raw estimator outputs: on constant-y / no-dose-variation / divide-by-zero the fit paths return `(att=nan, se=nan)` so all five fields move to NaN together; on the degenerate single-cluster CR1 configuration on the mass-point path, `_fit_mass_point_2sls` returns `(att=beta_hat, se=nan)` - `att` is finite (Wald-IV is well defined) while `se` is NaN, so the downstream triple is NaN while `att` remains the raw 2SLS coefficient. The `assert_nan_inference` fixture in `tests/conftest.py` checks the downstream triple against this contract without requiring `att` to be NaN. - **Note (mass-point SE):** Standard errors on the mass-point path use the structural-residual 2SLS sandwich `[Z'X]^{-1} · Ω · [Z'X]^{-T}` with `Ω` built from the structural residuals `u = ΔY - α̂ - β̂·D` (not the reduced-form residuals from an OLS-on-indicator shortcut). Supported: `classical`, `hc1`, and CR1 (cluster-robust) when `cluster=` is supplied. `hc2` and `hc2_bm` raise `NotImplementedError` pending a 2SLS-specific leverage derivation (the OLS leverage `x_i' (X'X)^{-1} x_i` is wrong for 2SLS; the correct finite-sample correction depends on `(Z'X)^{-1}` rather than `(X'X)^{-1}`) plus a dedicated R parity anchor. Queued for the follow-up PR. + - **Note:** M-080 position (auto-cluster policy, category b in + `docs/methodology/variance-conventions.md` "Default clustering roster"; shipped item + 7b, 2026-10-29): HC1/CCT on unit-level first differences is unit-level by construction, so no auto-cluster metadata is added (`cluster_name` None on the default) and `cluster=False` is an accepted no-op on the static and event-study paths. The shared `cluster` contract is `str | None | False` + ([M-155]): `False` is an accepted explicit no-op (bit-identical to `None`), `True` and + non-`str` labels raise the shared `ValueError`. - **Note (continuous cluster-robust SE, Phase 2a):** On the continuous designs (`continuous_at_zero` / `continuous_near_d_lower`), `cluster=` threads the per-unit cluster IDs into `bias_corrected_local_linear`, whose CCT-2014 robust variance then becomes the cluster-robust nonparametric SE; the β̂-scale SE is `se_robust / |den|`. Because the β-scale rescale is a deterministic linear transform, the estimator-level clustered SE equals the direct `bias_corrected_local_linear(cluster=...).se_robust / |den|` to machine precision — this is the in-library validation anchor, and the clustered CCT SE is itself golden-tested against `nprobust` on DGP 4 (see the Phase 1c note above: `atol=1e-14` in first-appearance cluster order). Cluster IDs must be unit-constant (validated up front; a nonexistent column, NaN, or within-unit-varying cluster now raises rather than being silently ignored, mirroring the mass-point path). Cluster-robust inference is **unidentified with fewer than two clusters in the active kernel window** (`eC = cluster[ind]`, the in-bandwidth subset the CCT variance is actually computed on — a stricter condition than the global cluster count, since clusters can be separated from the boundary by the bandwidth): the guard lives in `_nprobust_port.lprobust` and NaNs `se_rb`/`se_cl`, so `bias_corrected_local_linear.se_robust` is NaN and HAD's `safe_inference` NaNs the t-stat / p-value / CI while the point estimate stays finite — the same single-cluster NaN contract as the mass-point CR1 path (`_fit_mass_point_2sls`), applied at the variance-computation site so it also covers the direct `bias_corrected_local_linear` API. A window with ≥2 distinct clusters is bit-identical (DGP-4 golden parity preserved). A bare `cluster=` gives unweighted cluster-robust inference; the `cluster=` + `survey_design=` composition raises `NotImplementedError`: the Binder (1983) TSL survey variance is composed from the per-unit influence function via `compute_survey_if_variance` and would silently override the cluster-robust SE — for weighted clustering route through `survey_design=SurveyDesign(weights='', psu='')` instead. Result metadata reports `vcov_type="cr1"` + `cluster_name=` with `inference_method="analytical_nonparametric"` (distinguishing the clustered CCT variance from the mass-point 2SLS CR1 sandwich). Estimator-level cluster threading also extends to the **Phase 2b event-study** path (cluster-robust per-horizon pointwise CIs on both designs plus a cluster-robust simultaneous sup-t band; the per-horizon variance-family reconciliation is documented in "Note (HAD clustered event-study sup-t band)" above) — the former "cluster ignored" `UserWarning` is removed. - **Note (Design 1 identification):** `continuous_near_d_lower` and `mass_point` fits emit a `UserWarning` surfacing that `WAS_{d̲}` identification requires Assumption 6 (or Assumption 5 for sign identification only) beyond parallel trends, and that neither is testable via pre-trends. `continuous_at_zero` (Design 1', Assumption 3 only) does not emit this warning. - **Note (CI endpoints):** Because the continuous-path `att` is `(mean(ΔY) - τ̂_bc) / den`, the beta-scale CI endpoints reverse relative to the Phase 1c boundary-limit CI: `CI_lower(β̂) = (mean(ΔY) - CI_upper(τ̂_bc)) / den` and `CI_upper(β̂) = (mean(ΔY) - CI_lower(τ̂_bc)) / den`. The `HeterogeneousAdoptionDiD.fit()` implementation computes `att ± z · se` directly via `safe_inference`, which handles the reversal naturally from the transformed point estimate. @@ -6963,12 +7063,21 @@ should be a deliberate user choice. **Note:** This estimator is a **documented synthesis** of ingredients — no single published software package implements the exact recipe. `did2s` (R/Stata, Butts & Gardner 2022) implements the Gardner two-stage residualization but does NOT support ring covariates. The Butts (2021/2023) paper proposes the ring estimator in Equations 5/6/8 (non-staggered) and Section 5 / Table 2 (staggered) but does not ship reference software. The diff-diff implementation combines: (a) Butts (2021) Section 5 / Table 2 identification, (b) Gardner (2022) two-stage residualize-then-fit, (c) Wave A's Conley spatial-HAC vcov. -- **Note:** M-080 position (auto-cluster policy, re-scoped 2026-10-18 - category a): - SpilloverDiD is the ONE required-unit estimator still running observation-level HC - when `cluster=` is omitted, so its default SE flips to unit clustering at 4.0 with - TWFE's carve-outs mirrored (explicit `cluster=` wins; Conley drops the auto-cluster; - never injected as a survey PSU). 3.x numbers are reproduced with `cluster=False`. - Roster in `docs/methodology/variance-conventions.md` "Default clustering roster". +- **Note:** M-080 position (auto-cluster policy, re-scoped 2026-10-18 - category a; + SHIPPED item 7b, 2026-10-29): SpilloverDiD was the ONE required-unit estimator still + running observation-level HC when `cluster=` was omitted; since 4.0 it auto-clusters + at `unit` (CR1 with the `G/(G-1) * (n-1)/(n-p_2)` multiplier on the default lane) with + TWFE's carve-outs mirrored: explicit `cluster=` wins on every lane (incl. the Conley + product kernel and survey PSU injection), the auto-cluster is dropped on the Conley + path and is never injected as a survey PSU (implicit per-observation PSUs). 3.x + numbers are reproduced with `cluster=False` (the observation-level HC1 lane; the + shared `str | None | False` contract, [M-155]). Results report `cluster_name="unit"` + / `n_clusters=G` on the default lane. Roster in + `docs/methodology/variance-conventions.md` "Default clustering roster". +- **Note:** inference df unchanged at 7b - `t(n_obs_eff - k_effective)` on every lane + INCLUDING the new default CR1 lane (SpilloverDiD has no `df_convention` knob yet, so + it is the one auto-clustered default not on t(G-1)); [M-154] / item 7c adds the knob + with the 4.0 default `"cluster"` (t(G-1) on the clustered lanes, inert on Conley). - **Note (rank-guarded Wave D bread):** The Wave D variance bread `A_22 = (X_2'WX_2)^{-1}` is inverted by the shared `_rank_guarded_inv` (`diff_diff/linalg.py`) on the already-column-dropped kept submatrix. `np.linalg.solve` raised only on an exactly-singular bread (prior fallback: dense `lstsq`); a **near**-singular kept Gram would otherwise flow a garbage inverse (~1e13) into the SE. The rank-guard truncates redundant directions → finite SE on the identified subspace (NaN at rank 0), re-inflated to (k, k) with NaN at the dropped positions, and warns. See the CallawaySantAnna "rank-guarded IF standard errors" Note. Sibling of axis-A finding #17. **Identification spec (committed):** @@ -7054,8 +7163,8 @@ vcov = (X_2' X_2)^{-1} @ meat @ (X_2' X_2)^{-1} where the kernel `K` is path-dependent: -- **HC1**: `K = I_n` → `meat = Psi' Psi` with `n / (n - p_2)` finite-sample multiplier. -- **Cluster CR1**: `K_ij = 1{cluster_i = cluster_j}` → per-cluster sum + outer product, with `G / (G-1) * (n-1) / (n - p_2)` finite-sample multiplier. +- **HC1**: `K = I_n` → `meat = Psi' Psi` with `n / (n - p_2)` finite-sample multiplier (the `cluster=False` lane since 4.0; the default before 4.0). +- **Cluster CR1**: `K_ij = 1{cluster_i = cluster_j}` → per-cluster sum + outer product, with `G / (G-1) * (n-1) / (n - p_2)` finite-sample multiplier (the 4.0 default at `unit`, or an explicit `cluster=` column). - **Conley**: `K_ij = kernel(d_ij / cutoff) * 1{cluster_i = cluster_j}` (cross-sectional) or panel-block decomposed (`conley_time` / `conley_unit` / `conley_lag_cutoff` set). No finite-sample multiplier — matches `conleyreg` convention. The correction applies unconditionally (no opt-out kwarg). Point estimates (`tau_total`, `delta_j`, event-study `tau_k` / `delta_jk`) are byte-identical to the pre-Wave-D path; SE values shift upward by 1-few percent. @@ -7068,7 +7177,7 @@ The correction applies unconditionally (no opt-out kwarg). Point estimates (`tau ### Variance (Wave E.1 — survey-design integration via Binder TSL) -`survey_design=` is now supported on `vcov_type ∈ {"hc1"}` and CR1 (`cluster=`) paths. The vcov is the design-consistent sandwich `V = A_22^{-1} @ meat @ A_22^{-1}` where: +`survey_design=` is now supported on `vcov_type ∈ {"hc1"}` and CR1 (`cluster=`) paths (the 4.0 unit auto-cluster is NOT injected as a PSU: a design without its own PSU keeps implicit per-observation PSUs; only an explicit `cluster=` becomes the PSU). The vcov is the design-consistent sandwich `V = A_22^{-1} @ meat @ A_22^{-1}` where: - `A_22 = X_2' W X_2` (bread with weighted-OLS structure when `survey_weights` provided) - `meat = sum_h (1-f_h) * n_h/(n_h-1) * sum_j (S_psu_hj - S_psu_h_bar) (S_psu_hj - S_psu_h_bar)'` diff --git a/docs/methodology/variance-conventions.md b/docs/methodology/variance-conventions.md index a80b73c99..f7692f0d0 100644 --- a/docs/methodology/variance-conventions.md +++ b/docs/methodology/variance-conventions.md @@ -192,27 +192,31 @@ and the out-of-scope roster (IF-based estimators, TripleDifference/TROP, Wooldridge GLM arms) is enumerated in the REGISTRY TwoWayFixedEffects deviation note. -## Default clustering roster (decided 2026-10-18, [M-080]) +## Default clustering roster (decided 2026-10-18, [M-080]; shipped item 7b, 2026-10-29) Measured by execution at v3.13.0 for the 4.0-cut framing (50 units x 8 periods, `cluster=` omitted vs `cluster="unit"`). The ledger row M-080 was written on the premise that static `TwoWayFixedEffects` runs HC-robust by default; it has auto-clustered at unit since 2026-01, so the row was re-scoped to this roster -(spec `docs/v4-design.md` section 7). Categories: **a** default SE moves at +(spec `docs/v4-design.md` section 7). Categories: **a** default SE moved at 4.0; **b** unit-level by construction with a `cluster` param; **c1** already clusters at unit by a regression-level default; **c2** defaults to unit inside unit-level influence-function machinery; **c3** clusters by construction with a restricted `cluster` vocabulary; **d** no `cluster` param; **n/a** -cross-sectional or no clustering support. +cross-sectional or no clustering support. The shared argument contract since 4.0 +is `str | None | False` ([M-155], `diff_diff.utils.validate_cluster`): `True` and +non-`str` labels raise one `ValueError` everywhere (StackedDiD keeps its own +vocabulary validator), and `False` is normalized locally so it never reaches a +column lookup. -| Class | `cluster` param? | Default when omitted (v3.13.0) | `cluster=False` at 4.0 | Category | +| Class | `cluster` param? | Default when omitted (v3.13.0 -> 4.0) | `cluster=False` since 4.0 | Category | |---|---|---|---|---| -| SpilloverDiD | yes | observation-level HC1 | plain HC SE (= 3.x default) | **a** - default flips to unit | -| TwoWayFixedEffects (static) | yes | CR1 at unit (`cluster_name="unit"`, `n_clusters=None` on the analytical path - gap closed at 4.0) | plain HC sandwich (new behaviour) | c1 | -| TwoWayFixedEffects (event-study) | yes | CR1 at unit since 3.9 (M-010) | plain HC sandwich (new behaviour) | c1 | -| SunAbraham | yes | CR1 at unit (reports `cluster_name=None`, no `n_clusters` field - gap closed at 4.0) | plain HC sandwich (new behaviour) | c1 | -| WooldridgeDiD | yes | CR1 at unit (`"unit"`, G) | plain HC sandwich (new behaviour) | c1 | -| LPDiD | yes | CR1 at unit (`self.cluster or unit`; `False` currently treated as auto-cluster - bug fixed at 4.0) | plain HC sandwich (new behaviour) | c1 | +| SpilloverDiD | yes | observation-level HC1 -> **CR1 at unit since 4.0** (`"unit"`, G; TWFE's carve-outs mirrored; inference df still the residual df until [M-154]) | plain HC1 SE (= the 3.x default) | **a** - default flipped to unit | +| TwoWayFixedEffects (static) | yes | CR1 at unit (`cluster_name="unit"`; reports `n_clusters` since 4.0 on non-survey analytical fits; the survey-lane auto label is None since 4.0) | plain HC sandwich (absorbed-FE full-K rescale, residual-df t; one-way CR2-BM under `hc2_bm`); rejected under `inference="wild_bootstrap"` | c1 | +| TwoWayFixedEffects (event-study) | yes | CR1 at unit since 3.9 (M-010) | plain HC sandwich (the core's unclustered lanes) | c1 | +| SunAbraham | yes | CR1 at unit (reports `cluster_name="unit"` and the appended `n_clusters` since 4.0) | plain HC sandwich (residual-df t); no unit PSU under `survey_design=`; pairs bootstrap unaffected; rejected with `hc2_bm` | c1 | +| WooldridgeDiD | yes | CR1 at unit (`"unit"`, G) | plain HC sandwich (OLS within lane carries the absorbed-FE full-K rescale; GLM arms the plain QMLE sandwich); rejected with `n_bootstrap > 0` and with `hc2_bm` | c1 | +| LPDiD | yes | CR1 at unit (`False` was treated as the auto-cluster before 4.0) | plain HC1 per horizon; `if_robust` unclustered IF variance on the RA path; implicit per-observation PSUs under a survey design | c1 | | ImputationDiD | yes | `cluster_var = unit` inside the unit-level IF variance (`"unit"`, G) | accepted explicit no-op | c2 | | TwoStageDiD | yes | `cluster_var = unit` inside the unit-level IF variance (`"unit"`, G) | accepted explicit no-op | c2 | | StackedDiD | yes (`'unit'`/`'unit_subexp'` only) | CR1 at unit (`"unit"`, G) | REJECTED (validator) | c3 | @@ -235,6 +239,15 @@ metadata - `cluster_name` stays `None` where the field exists and no field is added where it does not. Their SE formulas lack the G/(G-1) factor the explicit `cluster="unit"` path applies (verified: CS and EDiD `se(cluster="unit")^2 / se(default)^2 = 50/49` on the probe), so labelling them `"unit"` would misstate -the formula. The 4.0 flip is therefore the SpilloverDiD default, the uniform -`cluster=False` off-switch on categories a/b/c1/c2, and the metadata gaps named -in the c1 rows. +the formula. The 4.0 flip SHIPPED as item 7b (2026-10-29): the SpilloverDiD +default, the uniform `cluster=False` off-switch on categories a/b/c1/c2 (an +accepted bit-identical no-op on b/c2 and on the 2x2 surfaces, incl. the CS +RC auto-PSU path, `EfficientDiD.hausman_pretest` and the LWDiD sensitivity +helpers), and the metadata gaps named in the c1 rows; pinned by +`tests/test_v4_cluster_policy.py`. Two c1 lanes reject the off-switch rather +than open an unverified variance: `cluster=False` + `vcov_type="hc2_bm"` on +SunAbraham / WooldridgeDiD (their Bell-McCaffrey DOF helper is cluster-only; the +one-way port is a `TODO.md` row - TWFE's one-way CR2-BM lane already exists), and +the bootstraps that draw AT the analytical cluster (DiD/TWFE wild, WooldridgeDiD +multiplier). SpilloverDiD's new default lane keeps residual-df inference until +[M-154] (item 7c) adds its `df_convention` knob. diff --git a/docs/migration-4.0.md b/docs/migration-4.0.md index b66789b0f..c35f4f160 100644 --- a/docs/migration-4.0.md +++ b/docs/migration-4.0.md @@ -76,8 +76,8 @@ Two things are easy to get wrong here, and both change numbers: mode auto-clusters at `unit` whenever `unit=` is passed and the fit is not an explicit `classical`/`hc2`/`hc3`, Conley or survey fit (`hc1`, the default, is NOT an opt-out). To reproduce MultiPeriodDiD's unclustered HC1 numbers, drop `unit=`; an explicit `cluster=` - reproduces a MultiPeriodDiD fit that passed the same `cluster=`. (`cluster=False` as the - uniform off-switch arrives with [M-080].) + reproduces a MultiPeriodDiD fit that passed the same `cluster=`. (`cluster=False` is the + uniform off-switch since 4.0, [M-080].) ```python # 4.0 - bit-exact with the 3.x unclustered MultiPeriodDiD default (no unit=) @@ -350,11 +350,40 @@ Two changes alter results without changing any call: `cluster=False` becomes the explicit off-switch on every estimator whose clustering is a choice (in 3.x it was accepted nowhere): on `SpilloverDiD`, `TwoWayFixedEffects`, `SunAbraham`, `WooldridgeDiD` and `LPDiD` it yields the plain HC sandwich — on the - last four that is **new** unclustered behaviour, not a 3.x reproduction; on the - by-construction estimators and on `ImputationDiD`/`TwoStageDiD` it is an accepted - no-op (their variance is unit-level by construction, so nothing can be switched off); - `StackedDiD` keeps rejecting it (its `cluster` accepts only `"unit"`/`"unit_subexp"`, - the stacked regression clusters by construction). + last four that is **new** unclustered behaviour, not a 3.x reproduction (the + within-transform lanes carry the absorbed-FE full-K rescale, so the SE equals HC1 on + the explicit full-dummy design; the t reference is the residual df, `df_convention` + being inert without a cluster count; `SunAbraham`'s survey lane stops injecting the + unit PSU, so its TSL SE moves too; `WooldridgeDiD`'s logit/poisson arms give the plain + QMLE sandwich under an unchanged normal-theory reference distribution - the SE and + hence the p-value move); on the by-construction estimators + and on `ImputationDiD`/`TwoStageDiD` it is an accepted no-op (their variance is + unit-level by construction, so nothing can be switched off); `StackedDiD` keeps + rejecting it (its `cluster` accepts only `"unit"`/`"unit_subexp"`, the stacked + regression clusters by construction). Three combinations are rejected rather than + given an unverified variance: `cluster=False` with `inference="wild_bootstrap"` + (DiD/TWFE) or with `WooldridgeDiD(n_bootstrap > 0)` - those bootstraps draw at the + analytical cluster - and `cluster=False` with `vcov_type="hc2_bm"` on `SunAbraham` / + `WooldridgeDiD` (their Bell-McCaffrey DOF lane is cluster-only; `TwoWayFixedEffects` + does run the one-way CR2-BM lane). `SunAbraham`'s pairs bootstrap accepts it as a no-op + (it resamples units regardless of `cluster`). +- **`cluster` accepts `str | None | False` only** ([M-155]). The shared validator + replaces 13 per-class lookups that accepted any hashable column label: an integer- or + tuple-labelled cluster column must be renamed to a string, and `cluster=True` raises + (previously a `KeyError` or silent unit clustering, depending on the class). The check + runs at construction, through `set_params` and again at `fit()`. +- **Cluster metadata is populated on analytical fits.** `DifferenceInDifferences` and + `TwoWayFixedEffects` report `n_clusters` from the regression's own effective cluster + count on non-survey analytical fits (it was `None` unless the wild bootstrap ran), and + `SunAbrahamResults` gains `n_clusters` and reports `cluster_name` on every clustered + analytical fit (it was `None` off the Conley path). Under a survey design with no + explicit `cluster=`, `TwoWayFixedEffects` now reports `cluster_name=None` (the + auto-cluster is never injected as a PSU, so the former `"unit"` label described ids + the regression never received); summaries render ", G=..." and `SunAbraham` summaries + gain a "Std. errors:" line on analytical non-survey fits. +- **`SpilloverDiD` p-values and CIs on the new default lane stay on the residual df** in + this release step; the next PR adds its `df_convention` knob with the 4.0 `"cluster"` + default ([M-154]), which moves them to t(G-1) like every other auto-clustered default. ## Removed functions and aliases @@ -371,7 +400,9 @@ at 4.0, removed at 5.0), the `covariates=` constructor-to-`fit()` move, a retire Bacon roster re-homing, and the family-wide `anticipation` validation below. The appendix lists the ledger-derived removals/flips; [M-144], [M-145], and [M-146] are behavior tightenings, [M-147] is an additive policy, and [M-153] is a results-frame reshape. These have no -removal/deprecation fields and appear here only. +removal/deprecation fields and appear here only. [M-155] (the shared `cluster` +argument contract, see "Inference defaults that move numbers") is the same kind of +behavior tightening. - `WooldridgeDiD(unsupported_period_action="error")` ([M-147]) refuses periods lacking comparison support before removing them. The default `"drop"` preserves filtering and @@ -571,7 +602,7 @@ does not mean no action is required, so read the `Fix` cell. | M-002 | obligation-sdid-params | `diff_diff:SyntheticDiD[zeta]` | — | Drop it. It has been IGNORED since 3.0.0, so 3.x already auto-computes regularization - do NOT copy its value into `zeta_lambda=`, which activates an override that was inert and changes weights, ATT and inference. Set `zeta_lambda=` only as a deliberate new choice. | | M-003 | obligation-sdid-params | `diff_diff:SyntheticDiDResults.placebo_effects` | `diff_diff:SyntheticDiDResults.variance_effects` | Read `variance_effects` instead of `placebo_effects`. | | M-007 | obligation-warning-retirements | `diff_diff.estimators` | — | The MultiPeriodDiD `e=-1` transition FutureWarning is no longer emitted (the class is gone; the event-study mode defaults to the last pre-period silently); drop any warning filter that suppressed it. | -| M-080 | policy-auto-cluster | `diff_diff:SpilloverDiD[cluster]` | — | `SpilloverDiD` now clusters at `unit` when `cluster=` is omitted; pass `cluster=False` for the 3.x observation-level HC standard errors (`None`/omission means auto-cluster, not off). Every other panel estimator already clustered at unit or is unit-level by construction, so its numbers do not move. `cluster=False` is also newly accepted on `TwoWayFixedEffects`/`SunAbraham`/`WooldridgeDiD`/`LPDiD` (plain HC sandwich - new behaviour) and as a no-op on the by-construction estimators and `ImputationDiD`/`TwoStageDiD`; `StackedDiD` still rejects it. | +| M-080 | policy-auto-cluster | `diff_diff:SpilloverDiD[cluster]` | — | `SpilloverDiD` now clusters at `unit` when `cluster=` is omitted (TWFE's carve-outs mirrored: explicit `cluster=` wins, dropped on the Conley path, never injected as a survey PSU; its p-values/CIs keep the residual df until [M-154]); pass `cluster=False` for the 3.x observation-level HC standard errors (`None`/omission means auto-cluster, not off). Every other panel estimator already clustered at unit or is unit-level by construction, so its numbers do not move. `cluster=False` is also newly accepted on `TwoWayFixedEffects`/`SunAbraham`/`WooldridgeDiD`/`LPDiD` (plain HC sandwich - new behaviour; residual-df t), as a no-op on the by-construction estimators, on `ImputationDiD`/`TwoStageDiD` and on the 2x2 `DifferenceInDifferences` / 2x2x2 `TripleDifference` surfaces; `StackedDiD` still rejects it. Rejected with `inference="wild_bootstrap"` (DiD/TWFE), `WooldridgeDiD(n_bootstrap > 0)` and `vcov_type="hc2_bm"` on `SunAbraham`/`WooldridgeDiD`. `cluster` accepts `str | None | False` only ([M-155]). | | M-032 | renames-cohort | `diff_diff:WooldridgeDiD.fit[cohort]` | `diff_diff:WooldridgeDiD.fit[first_treat]` | Rename the keyword: `cohort=` becomes `first_treat=`. | | M-035 | renames-col-suffix | `diff_diff:HeterogeneousAdoptionDiD.fit[outcome_col]` | `diff_diff:HeterogeneousAdoptionDiD.fit[outcome]` | Rename the keyword: `outcome_col=` becomes `outcome=`. | | M-036 | renames-col-suffix | `diff_diff:HeterogeneousAdoptionDiD.fit[dose_col]` | `diff_diff:HeterogeneousAdoptionDiD.fit[dose]` | Rename the keyword: `dose_col=` becomes `dose=`. | diff --git a/docs/tutorials/23_spillover_tva.ipynb b/docs/tutorials/23_spillover_tva.ipynb index c6f359244..59755a2a9 100644 --- a/docs/tutorials/23_spillover_tva.ipynb +++ b/docs/tutorials/23_spillover_tva.ipynb @@ -381,6 +381,9 @@ "separates the direct effect (-7.34, very close to true -7.4) from the\n", "spillover on near-controls (-4.53, very close to true -4.5). The\n", "spillover-aware estimator removes the bias the naive TWFE suffered.\n", + "The standard errors in the table are the 4.0 default: cluster-robust\n", + "(CR1) at the unit level, like `TwoWayFixedEffects` - `SpilloverDiD`\n", + "auto-clusters at `unit` when `cluster=` is omitted.\n", "\n", "The interpretation is straightforward: every period in the post window,\n", "treated units lose 7.4 units of $y$ on average from the policy, AND\n", @@ -397,20 +400,25 @@ "source": [ "## Robust inference with Conley spatial-HAC\n", "\n", - "The HC1 standard error in the `SpilloverDiD` fit section treats each observation's score as\n", - "independent of every other. On spatially-correlated panels \u2014 and this\n", - "is the canonical case for place-based interventions \u2014 that\n", - "independence assumption is violated. Conley (1999) provides a\n", - "spatial-HAC variance estimator that downweights pairs by their\n", - "geographic distance through a Bartlett kernel out to a `conley_cutoff_km`\n", - "bandwidth. Newey-West (1987) provides the analogous serial term via\n", - "`conley_lag_cutoff`.\n", + "The default standard error in the `SpilloverDiD` fit section is\n", + "cluster-robust at the unit level (CR1 at `unit`, the 4.0 default): it\n", + "treats each unit's scores as one block and assumes independence\n", + "*between* units. Passing `cluster=False` switches clustering off and\n", + "gives the observation-level HC1 sandwich, which treats every\n", + "observation's score as independent of every other. On\n", + "spatially-correlated panels - and this is the canonical case for\n", + "place-based interventions - both independence assumptions are violated\n", + "across nearby units. Conley (1999) provides a spatial-HAC variance\n", + "estimator that downweights pairs by their geographic distance through a\n", + "Bartlett kernel out to a `conley_cutoff_km` bandwidth. Newey-West (1987)\n", + "provides the analogous serial term via `conley_lag_cutoff`.\n", "\n", "Per Butts (2021) \u00a73.1, the Conley bandwidth is typically chosen to\n", - "match the spillover horizon `d_bar`. Below we compare the fit's HC1 SE\n", - "against Conley with `conley_cutoff_km = d_bar = 100`, both with\n", - "`conley_lag_cutoff = 0` (spatial only) and `conley_lag_cutoff = 1` (add\n", - "within-unit serial correlation across consecutive periods).\n" + "match the spillover horizon `d_bar`. Below we compare the default\n", + "unit-clustered SE and the HC1 SE (`cluster=False`) against Conley with\n", + "`conley_cutoff_km = d_bar = 100`, both with `conley_lag_cutoff = 0`\n", + "(spatial only) and `conley_lag_cutoff = 1` (add within-unit serial\n", + "correlation across consecutive periods).\n" ] }, { @@ -420,8 +428,13 @@ "metadata": {}, "outputs": [], "source": [ - "def fit_with(vcov_type, lag=None):\n", - " kwargs = dict(rings=[0.0, D_BAR_KM], conley_coords=(\"lat\", \"lon\"), vcov_type=vcov_type)\n", + "def fit_with(vcov_type, lag=None, cluster=None):\n", + " kwargs = dict(\n", + " rings=[0.0, D_BAR_KM],\n", + " conley_coords=(\"lat\", \"lon\"),\n", + " vcov_type=vcov_type,\n", + " cluster=cluster, # None = the 4.0 default (CR1 at unit); False = HC1\n", + " )\n", " if vcov_type == \"conley\":\n", " kwargs[\"conley_cutoff_km\"] = D_BAR_KM\n", " kwargs[\"conley_lag_cutoff\"] = lag\n", @@ -434,16 +447,22 @@ " return est.fit(df, outcome=\"y\", unit=\"unit\", time=\"time\", treatment=\"D\")\n", "\n", "\n", - "res_hc1 = fit_with(\"hc1\")\n", + "res_cr1 = fit_with(\"hc1\") # the default: CR1 at unit\n", + "res_hc1 = fit_with(\"hc1\", cluster=False) # explicitly unclustered: observation-level HC1\n", "res_c0 = fit_with(\"conley\", lag=0)\n", "res_c1 = fit_with(\"conley\", lag=1)\n", "\n", "se_table = pd.DataFrame(\n", " {\n", - " \"tau_total\": [res_hc1.att, res_c0.att, res_c1.att],\n", - " \"SE\": [res_hc1.se, res_c0.se, res_c1.se],\n", + " \"tau_total\": [res_cr1.att, res_hc1.att, res_c0.att, res_c1.att],\n", + " \"SE\": [res_cr1.se, res_hc1.se, res_c0.se, res_c1.se],\n", " },\n", - " index=[\"HC1\", \"Conley (lag=0, spatial only)\", \"Conley (lag=1, spatial + serial)\"],\n", + " index=[\n", + " \"CR1 at unit (4.0 default)\",\n", + " \"HC1 (cluster=False)\",\n", + " \"Conley (lag=0, spatial only)\",\n", + " \"Conley (lag=1, spatial + serial)\",\n", + " ],\n", ")\n", "print(se_table.to_string(float_format=lambda x: f\"{x:>8.4f}\"))\n" ] @@ -453,9 +472,11 @@ "id": "e6d5ee1b", "metadata": {}, "source": [ - "Point estimates are identical across all three rows \u2014 the variance\n", + "Point estimates are identical across all four rows \u2014 the variance\n", "choice doesn't move the coefficients. But the standard errors differ:\n", - "on this DGP, the Conley spatial-HAC SE comes in *lower* than HC1.\n", + "the unit-clustered default (CR1 at `unit`) is slightly *wider* than\n", + "HC1 here (each unit's four periods are positively correlated), and on\n", + "this DGP the Conley spatial-HAC SE comes in *lower* than both.\n", "\n", "The mechanism is the pairwise covariance structure of the\n", "influence-function score matrix. With `conley_cutoff_km = 100`, the\n", @@ -487,14 +508,16 @@ "sandwich ends up above or below HC1 depends on both the missing\n", "multiplier AND the net sign and magnitude of the weighted\n", "off-diagonal cross-products under the specific DGP and design.) On\n", - "this seed they net out to a smaller ATT variance, so Conley < HC1,\n", + "this seed they net out to a smaller ATT variance, so Conley < HC1 <\n", + "CR1-at-unit,\n", "but the direction is a feature of the data and the sign mechanism\n", "is not easily attributable to any one pair class. Adding the serial\n", "term (`lag = 1`) further reshapes the SE through a separate\n", "within-unit cross-period score covariance sum.\n", "\n", "The takeaway: Conley vcov is a *correction*. It can move the SE in\n", - "either direction relative to HC1 depending on the residual covariance\n", + "either direction relative to the unit-clustered default (or to HC1 via\n", + "`cluster=False`) depending on the residual covariance\n", "structure within the kernel's support, but it's the methodologically\n", "correct choice when scores are spatially or serially correlated. On\n", "real applications the correction often inflates the SE (and the\n", @@ -529,8 +552,8 @@ "**When Conley matters.** Whenever your residuals have spatial or\n", "serial correlation. On clustered-treatment designs (treatment radius\n", "$\\ll$ panel extent) the spatial-HAC adjustment can move the SE in\n", - "either direction relative to HC1; the direction is a feature of the\n", - "DGP. Per Butts (2021) \u00a73.1, set `conley_cutoff_km = d_bar` as the\n", + "either direction relative to the default unit-clustered SE (and to HC1\n", + "via `cluster=False`); the direction is a feature of the DGP. Per Butts (2021) \u00a73.1, set `conley_cutoff_km = d_bar` as the\n", "default and run a sensitivity grid on the cutoff if the SE matters\n", "for your inference.\n", "\n", diff --git a/docs/v4-deprecations.yaml b/docs/v4-deprecations.yaml index 69c77c8ba..dc364e3b2 100644 --- a/docs/v4-deprecations.yaml +++ b/docs/v4-deprecations.yaml @@ -1054,10 +1054,11 @@ rows: new: null deprecated_in: "4.0" removed_in: null - status: planned + status: done phase: 5 + test_ref: tests/test_v4_cluster_policy.py code_refs: [diff_diff/spillover.py, diff_diff/twfe.py, diff_diff/estimators.py, diff_diff/sun_abraham.py, diff_diff/lpdid.py, diff_diff/stacked_did.py, diff_diff/wooldridge.py, diff_diff/imputation.py, diff_diff/two_stage.py, diff_diff/staggered.py, diff_diff/efficient_did.py, diff_diff/dml_did.py, diff_diff/lwdid.py, diff_diff/had.py, diff_diff/triple_diff.py, diff_diff/_staggered_triple_diff_engine.py, diff_diff/chaisemartin_dhaultfoeuille.py, docs/methodology/variance-conventions.md] - notes: "RE-SCOPED 2026-10-18 (4.0-cut framing, user decision; measured roster in docs/methodology/variance-conventions.md and spec section 7). The original premise - static TwoWayFixedEffects runs HC-robust by default - was already false: static TWFE auto-clusters at unit (twfe.py, since 2026-01). Measured categories: (a) SpilloverDiD is the ONE required-unit estimator whose default SE moves (observation-level HC -> unit clustering, mirroring TWFE's carve-outs: explicit cluster= wins, Conley drops it, never injected as survey PSU); (c1) TWFE/SunAbraham/WooldridgeDiD/LPDiD already cluster at unit by regression-level default - cluster=False yields the plain HC sandwich; (c2) ImputationDiD/TwoStageDiD default cluster_var=unit INSIDE unit-level influence-function machinery - no unclustered variance exists, cluster=False is an accepted explicit no-op; (c3) StackedDiD clusters by construction and its validator accepts only unit/unit_subexp - cluster=False is REJECTED (REGISTRY Note); (b) CS/EfficientDiD/DMLDiD/dCDH/staggered-DDD (per-unit IF) and LWDiD/HAD (regression on a unit-level cross-section) are unit-level by construction - cluster=False is an accepted explicit no-op; (d) ContinuousDiD/DurationDiD/SDiD/SC/TROP/CiC have no cluster param; (n/a) cross-sectional DiD/2x2x2 DDD/RDD stay HC-robust or unclustered. Categories b and d carry NO auto-cluster metadata by decision (cluster_name stays None where the field exists; no field is added where it does not). The 4.0 flip therefore = the SpilloverDiD default + the uniform cluster=False off-switch on a/b/c1/c2 (today accepted nowhere: LPDiD's `self.cluster or unit` treats False as auto-cluster, most others look up a column named False, DMLDiD/StackedDiD reject on type, dCDH raises on any non-None) + metadata gaps (TWFE/DiD n_clusters on the analytical path; SunAbraham cluster_name + n_clusters). ORIGINAL NOTE: ONE auto-cluster policy at 4.0: panel estimators (required unit column) default to clustering at unit with cluster_name/n_clusters metadata; cluster=False disables; 2x2 cross-sectional estimators stay HC-robust. Changes some default SEs - migration guide with [M-004..M-006]. SCOPE NOTE (Phase 3(a), user decision 2026-08-07): the TWFE EVENT-STUDY mode shipped already auto-clustered at 3.9 (new API adopts the end-state immediately, with the static carve-outs mirrored - see [M-010]), so this 4.0 flip does not re-touch it." + notes: "DONE at the 4.0 cut (item 7b, 2026-10-29): SpilloverDiD auto-clusters at unit when cluster= is omitted (TWFE carve-outs mirrored: explicit cluster= wins, dropped on the Conley path, never injected as a survey PSU; inference df unchanged - residual df - pending [M-154]); uniform cluster=False via utils.validate_cluster - the shared contract is str | None | False ([M-155]), so non-str column labels are rejected everywhere (DMLDiD existing rule generalized) - plain HC on a/c1 incl. the new LPDiD / WooldridgeDiD-GLM unclustered lanes, the Wooldridge within-transform absorbed-FE full-K rescale and TWFE one-way hc2_bm; accepted no-op on b/c2 and the 2x2 surfaces (incl. the CS RC auto-PSU path, EfficientDiD.hausman_pretest and the LWDiD sensitivity helpers); rejected on StackedDiD (own vocabulary); the DiD/TWFE wild and WooldridgeDiD multiplier bootstraps and the cluster-only hc2_bm lanes of SunAbraham / WooldridgeDiD reject it (SunAbraham pairs bootstrap: accepted no-op, the hc2 precedent); metadata: DiD/TWFE analytical n_clusters from the regression (non-survey fits), SunAbrahamResults.n_clusters (appended) + cluster_name on clustered analytical fits, TWFE survey auto-label corrected to None. Pinned by tests/test_v4_cluster_policy.py; notebook 23 re-executed. RE-SCOPED 2026-10-18 (4.0-cut framing, user decision; measured roster in docs/methodology/variance-conventions.md and spec section 7). The original premise - static TwoWayFixedEffects runs HC-robust by default - was already false: static TWFE auto-clusters at unit (twfe.py, since 2026-01). Measured categories: (a) SpilloverDiD is the ONE required-unit estimator whose default SE moves (observation-level HC -> unit clustering, mirroring TWFE's carve-outs: explicit cluster= wins, Conley drops it, never injected as survey PSU); (c1) TWFE/SunAbraham/WooldridgeDiD/LPDiD already cluster at unit by regression-level default - cluster=False yields the plain HC sandwich; (c2) ImputationDiD/TwoStageDiD default cluster_var=unit INSIDE unit-level influence-function machinery - no unclustered variance exists, cluster=False is an accepted explicit no-op; (c3) StackedDiD clusters by construction and its validator accepts only unit/unit_subexp - cluster=False is REJECTED (REGISTRY Note); (b) CS/EfficientDiD/DMLDiD/dCDH/staggered-DDD (per-unit IF) and LWDiD/HAD (regression on a unit-level cross-section) are unit-level by construction - cluster=False is an accepted explicit no-op; (d) ContinuousDiD/DurationDiD/SDiD/SC/TROP/CiC have no cluster param; (n/a) cross-sectional DiD/2x2x2 DDD/RDD stay HC-robust or unclustered. Categories b and d carry NO auto-cluster metadata by decision (cluster_name stays None where the field exists; no field is added where it does not). The 4.0 flip therefore = the SpilloverDiD default + the uniform cluster=False off-switch on a/b/c1/c2 (today accepted nowhere: LPDiD's `self.cluster or unit` treats False as auto-cluster, most others look up a column named False, DMLDiD/StackedDiD reject on type, dCDH raises on any non-None) + metadata gaps (TWFE/DiD n_clusters on the analytical path; SunAbraham cluster_name + n_clusters). ORIGINAL NOTE: ONE auto-cluster policy at 4.0: panel estimators (required unit column) default to clustering at unit with cluster_name/n_clusters metadata; cluster=False disables; 2x2 cross-sectional estimators stay HC-robust. Changes some default SEs - migration guide with [M-004..M-006]. SCOPE NOTE (Phase 3(a), user decision 2026-08-07): the TWFE EVENT-STUDY mode shipped already auto-clustered at 3.9 (new API adopts the end-state immediately, with the static carve-outs mirrored - see [M-010]), so this 4.0 flip does not re-touch it." - id: M-081 kind: behavior group: policy-n-bootstrap @@ -1910,3 +1911,28 @@ rows: test_ref: tests/test_lwdid.py code_refs: [diff_diff/lwdid_results.py, docs/methodology/REGISTRY.md, docs/api/lwdid.rst] notes: "LWDiDResults.to_dataframe() reshaped onto the uniform results contract at the 4.0 cut (spec section 5: to_dataframe's selector is level= everywhere): level in {'simple', 'event_study', 'group'} (the class's aggregate() vocabulary), each returning aggregate(level).to_dataframe() - the shared AGGREGATION_SCHEMA / EVENT_STUDY_SCHEMA - plus the three LWDiD config columns rolling / estimation_method / vcov_type; default 'simple'; to_csv() relays the same level. The 3.10-3.13 frame had no selector and a bespoke shape: a term (common timing) / cohort (staggered) column -> now label; a trailing 'Overall' row on staggered fits -> now level='simple'; n_treated -> the schema's n; n_obs and n_control -> no longer in the frame (to_dict() keeps both); ci_lower/ci_upper -> conf_int_lower/conf_int_upper. A 3.10-born frame reshaped at the major, so no deprecation window - the [M-144]..[M-147] post-cut behavior shape (introduced_in 4.0, done). Documented in the migration guide's prose (not the ledger-derived appendix)." + - id: M-154 + kind: behavior + group: tail-df-consolidation + old: "diff_diff:SpilloverDiD[df_convention]" + new: null + introduced_in: "4.0" + deprecated_in: null + removed_in: null + status: planned + phase: 5 + code_refs: [diff_diff/spillover.py, diff_diff/utils.py, docs/methodology/variance-conventions.md] + notes: "PLANNED (item 7c, the PR after 7b; user decision 2026-10-29): SpilloverDiD gains the three-value df_convention knob with the 4.0 default 'cluster' - t(G-1) on its clustered analytical lanes (the 4.0 unit auto-cluster and explicit cluster=), residual df under 'residual' and on unclustered (cluster=False) fits, z under 'normal'; inert on Conley (G forced to None even with an explicit product-kernel cluster, the SunAbraham precedent); survey df keeps precedence. Closes the gap 7b documents: at 7b the default CR1 lane runs t(n_obs_eff - k_effective) while every other auto-clustered default runs t(G-1). Moves p/CI on clustered SpilloverDiD fits; notebook 23 re-executed again in 7c; the REGISTRY TWFE knob Note moves SpilloverDiD from its out-of-scope list to the knob roster." + - id: M-155 + kind: behavior + group: policy-auto-cluster + old: "diff_diff.utils:validate_cluster" + new: null + introduced_in: "4.0" + deprecated_in: null + removed_in: null + status: done + phase: 5 + test_ref: tests/test_v4_cluster_policy.py + code_refs: [diff_diff/utils.py] + notes: "Shared cluster-argument contract (item 7b, 2026-10-29; the [M-144]/[M-145] validation-tightening precedent): cluster is str | None | False on every estimator exposing it (StackedDiD keeps its 'unit'/'unit_subexp' vocabulary); non-str column labels, True and other types raise the shared ValueError at __init__, through set_params and again at fit (direct-mutation defense). Previously 13 classes accepted any hashable label and only DMLDiD type-checked. Also covers EfficientDiD.hausman_pretest and the LWDiD sensitivity helpers' own cluster arguments." diff --git a/docs/v4-design.md b/docs/v4-design.md index 9c9965f6d..8030ac5f2 100644 --- a/docs/v4-design.md +++ b/docs/v4-design.md @@ -853,7 +853,29 @@ domain vocabulary, not drift. and DiD report `n_clusters=None` on the analytical path; SunAbraham auto-clusters but reports `cluster_name=None` and has no `n_clusters` field). The [M-080] row's locator and `code_refs` were re-pointed - accordingly. + accordingly. (SHIPPED, item 7b, 2026-10-29: SpilloverDiD auto-clusters at + unit with the carve-outs mirrored; the uniform `cluster=False` via the shared + `utils.validate_cluster` - the contract is `str | None | False` [M-155], so + non-`str` column labels are rejected everywhere (DMLDiD's rule generalized); + plain HC on a/c1 incl. the new LPDiD and Wooldridge-GLM unclustered lanes, + the Wooldridge within-transform absorbed-FE full-K rescale and TWFE's one-way + hc2_bm lane; accepted no-op on b/c2 and the 2x2 surfaces; rejected on + StackedDiD, under the DiD/TWFE wild and Wooldridge multiplier bootstraps, and + with `hc2_bm` on SunAbraham/WooldridgeDiD pending the one-way BM DOF port + (TODO.md); DiD/TWFE analytical `n_clusters`, `SunAbrahamResults.n_clusters` + + `cluster_name`, the TWFE survey auto-label corrected to None; pinned by + `tests/test_v4_cluster_policy.py`. SpilloverDiD's default CR1 lane keeps + residual-df inference until [M-154].) +- **SpilloverDiD df_convention** [M-154] (item 7c, the PR after 7b; user + decision 2026-10-29): SpilloverDiD gains the three-value `df_convention` knob + with the 4.0 default `"cluster"` - t(G-1) on its clustered analytical lanes + (the unit auto-cluster and explicit `cluster=`), residual df under + `"residual"` and on unclustered (`cluster=False`) fits, z under `"normal"`; + inert on Conley (G forced to None even with an explicit product-kernel + cluster, the SunAbraham precedent); survey df keeps precedence. Closes the + one gap 7b leaves: at 7b the default CR1 lane runs t(n_obs_eff - + k_effective) while every other auto-clustered default runs t(G-1). Moves + p/CI on clustered SpilloverDiD fits; notebook 23 is re-executed again there. - **df_convention default flip** [M-004]..[M-006] + [M-128]..[M-131]: `"residual"` -> `"cluster"` (G-1) at 4.0 - the locked #663 direction. PR C ([M-127], 3.9) resolved the "evaluate extending the knob" item EARLY: @@ -1016,7 +1038,7 @@ above; anything only one PR cares about stays in that PR's plan.** | 2: contract foundations | 3.9 | (a) results base + unified event-study representation [M-092] + to_dict completion + the Diagnostic marker base on the diagnostic result roster [M-091] (section 3.5); (b) `aggregate()` + fit(aggregate=) shims [M-020..M-027] [M-139] (M-020's shim already shipped; M-139 is the HAD workflow twin, a pre-cut amendment); (c) param renames [M-030..M-047] [M-084] [M-086..M-089] + their results-field mirrors [M-094] [M-095] (section 8 rule 9) + the public-function completeness sweep [M-097..M-113] (section 8 rule 10) + the dCDH results mirror [M-114] + the fourth `robust` site [M-115] + the 2(c)-ii missed-rename amendments [M-136..M-138] (LPDiD `level` value; the two post-dummy diagnostics params) + BaseEstimator mixin + ContinuousDiD covariates move; (d) alias introduction [M-062] (the Spillover introduction is cancelled [M-063]) + the alias-diet `__getattr__` warning shim [M-135] + wrapper deprecations [M-070..M-077] + the two inference-surface policies: `n_bootstrap` semantic unification [M-081] and the wild-cluster-bootstrap roster guard [M-096]; shipped insertions (all done): the aggregate contract [M-122], the ETWFE reference-period family [M-123] [M-124] [M-125], and the variance-consolidation program [M-126] [M-127] | | 3: merges | 3.9 | (a) TWFE event-study mode [M-010] + EventStudy warn [M-060] + the fit `time`->`post` rename [M-082] (gates: section 4.1's equivalence/divergence/pooled-parity test triple) (shipped: tests/test_v4_merge_mpd.py; consumer ports incl. HonestDiD/PreTrendsPower calendar routes); (b) TripleDifference facade [M-013] + the SDDD alias [M-064] (shipped: tests/test_v4_merge_ddd.py; the engine relocation into the private shared mixin, the keyword-only staggered fit params, and the pscore_trim tightening [M-142]. The fit-time aggregate=/balance_e= carve-out rows are scheduled REMOVALS and so are cited in the phase-5 cell, not here - their only lifecycle version is removed_in 4.0); (c) CiC method= [M-015] + its results-field mirror [M-143] (shipped: tests/test_v4_merge_cic.py; method= is keyword-only and lowercase-only, the QDiD CLASS is deprecated while the METHOD is not, and ChangesInChangesResults.estimator -> .method carries a dual-key to_dict() window through 3.9) | | 4: release + soak | 3.9 cut | Migration guide written (skeleton: section 10); maintainer cuts 3.9; maint/3.8 rule active | -| 5: enforcement | 4.0 | Removals [M-010..M-015, M-020..M-027, M-139, M-030, M-032..M-047 old names, M-060, M-061, M-064, M-070..M-077, M-084, M-086..M-089, M-001..M-003, M-117, M-118, M-119, M-120, M-140, M-141, M-143] + the alias diet [M-132]..[M-134] + the amendment's old names [M-094] [M-095] [M-097..M-115] [M-136..M-138] [M-150] (incl. their consumer migrations and the `clean_control` serialized reporting key); M-031's old `time` name persists as the merged class's calendar column, so it is deliberately absent from the removal roster (its 4.0 enforcement is the M-085 behavior entry below); property window: [M-016] property-flips at 4.0 (removal at 5.0); storage flips [M-050..M-058]; default policies [M-004..M-006, M-128..M-131, M-080]; merged-class behavior enforcements [M-083] [M-085]; warning retirement [M-007]; fastpath go/no-go [M-008]; diagnostic-family docs/roster reorganization [M-090]; sentinel retirement [M-093]; the family-wide anticipation validation [M-144], the family-wide pscore_trim validation [M-145], and the family-wide summary-alpha guard [M-146] (staggered family at the initial cut; extended to the non-staggered summaries post-cut), the LWDiD to_dataframe reshape [M-153] (behavior tightenings landing at 4.0 - shipped post-3.9-cut, terminal `done`, no removal/deprecation fields); the additive Wooldridge comparison-support policy [M-147] (`unsupported_period_action="drop"` preserves filtering, `"error"` refuses before period removal; same post-cut lifecycle); the inert SyntheticControl inner-budget pair [M-148] [M-149] (warn-and-discard shims at 4.0 once the donor weights moved to an exact NNLS, removal at 5.0); the LWDiD result-alias pair [M-151] [M-152] (FutureWarning shims at 4.0, removal at 5.0); docs/llms.txt/README refresh | +| 5: enforcement | 4.0 | Removals [M-010..M-015, M-020..M-027, M-139, M-030, M-032..M-047 old names, M-060, M-061, M-064, M-070..M-077, M-084, M-086..M-089, M-001..M-003, M-117, M-118, M-119, M-120, M-140, M-141, M-143] + the alias diet [M-132]..[M-134] + the amendment's old names [M-094] [M-095] [M-097..M-115] [M-136..M-138] [M-150] (incl. their consumer migrations and the `clean_control` serialized reporting key); M-031's old `time` name persists as the merged class's calendar column, so it is deliberately absent from the removal roster (its 4.0 enforcement is the M-085 behavior entry below); property window: [M-016] property-flips at 4.0 (removal at 5.0); storage flips [M-050..M-058]; default policies [M-004..M-006, M-128..M-131, M-080] + the shared cluster contract [M-155] (7b) and the SpilloverDiD df_convention knob [M-154] (7c); merged-class behavior enforcements [M-083] [M-085]; warning retirement [M-007]; fastpath go/no-go [M-008]; diagnostic-family docs/roster reorganization [M-090]; sentinel retirement [M-093]; the family-wide anticipation validation [M-144], the family-wide pscore_trim validation [M-145], and the family-wide summary-alpha guard [M-146] (staggered family at the initial cut; extended to the non-staggered summaries post-cut), the LWDiD to_dataframe reshape [M-153] (behavior tightenings landing at 4.0 - shipped post-3.9-cut, terminal `done`, no removal/deprecation fields); the additive Wooldridge comparison-support policy [M-147] (`unsupported_period_action="drop"` preserves filtering, `"error"` refuses before period removal; same post-cut lifecycle); the inert SyntheticControl inner-budget pair [M-148] [M-149] (warn-and-discard shims at 4.0 once the donor weights moved to an exact NNLS, removal at 5.0); the LWDiD result-alias pair [M-151] [M-152] (FutureWarning shims at 4.0, removal at 5.0); docs/llms.txt/README refresh | | 6: front door | 4.1 | `event_study(data, outcome, unit, time, first_treat, estimator=...)` comparison entry point over the staggered family (sketch only; specified in its own plan) | Citation semantic for the table: a cell may cite a row whose current `phase` @@ -1208,7 +1230,10 @@ tests-first PR that flips no rows). 7. Inference defaults: the df_convention flips + [M-080] as re-scoped + the uniform `cluster=False` off-switch. (7a SHIPPED 2026-10-28: the df_convention flips [M-004]..[M-006] + [M-128]..[M-131], pinned by - `tests/test_v4_inference_defaults.py`; 7b = [M-080] pending.) + `tests/test_v4_inference_defaults.py`; 7b SHIPPED 2026-10-29: [M-080] + + the shared cluster contract [M-155], pinned by + `tests/test_v4_cluster_policy.py`, notebook 23 re-executed; 7c = [M-154] + SpilloverDiD df_convention, the next PR.) 8. [M-093] sentinel retirement + event-study schema enforcement. 9. [M-090] docs reorganization: Bacon to the diagnostics family, catalogue counts, JOSS prose, REGISTRY archive framing for the removed classes. @@ -1219,12 +1244,15 @@ tests-first PR that flips no rows). 04_parallel_trends, 05_honest_did, 07_pretrends_power, 09_real_world_examples, 11_imputation_did, 13_stacked_did, 16_wooldridge_etwfe (NOT 16_survey_did: the survey df wins) - plus any - 7b/8/9 movers; 24 is handled in 7a and 23 in 7b (each re-executed there - if its outputs moved). + 7c/8/9 movers; 24 is handled in 7a and 23 in 7b and again in 7c (each + re-executed there). 7b's label changes also drift every committed tutorial + that renders an analytical DiD/TWFE summary (", G=" on the variance line) + or a SunAbraham summary (the new "Std. errors:" line, e.g. 02) - all + within the nine-notebook set above. Ordering constraints: 4a before 4b before 4c-i before 4c-ii; 4 before 5 and 6; 5 before 6 -(the unified DDD container is born canonical); 7 and 8 independent; 9 after -4/6; 10 last. +(the unified DDD container is born canonical); 7a before 7b before 7c; 7 and 8 +independent; 9 after 4/6; 10 last. Waves 2 and 3 are independent of each other and of 4. **3.9-cut checklist (un-rowed obligations).** `test_due_rows_are_terminal` diff --git a/tests/test_spillover.py b/tests/test_spillover.py index 862c6f7f8..35d069305 100644 --- a/tests/test_spillover.py +++ b/tests/test_spillover.py @@ -3579,6 +3579,10 @@ def test_event_study_false_matches_wave_b_golden(self): d_bar=200.0, conley_coords=("lat", "lon"), event_study=False, + # 4.0 (M-080): the default now auto-clusters at unit; these goldens + # were captured on the observation-level HC1 lane, which + # cluster=False pins explicitly. + cluster=False, ) import warnings as _w @@ -3669,6 +3673,10 @@ def test_wave_d_se_inflates_relative_to_wave_b_uncorrected(self): d_bar=200.0, conley_coords=("lat", "lon"), event_study=False, + # 4.0 (M-080): the default now auto-clusters at unit; these goldens + # were captured on the observation-level HC1 lane, which + # cluster=False pins explicitly. + cluster=False, ) import warnings as _w @@ -4476,18 +4484,27 @@ def test_cluster_kwarg_routes_to_cr1_not_hc1(self): with _w.catch_warnings(): _w.simplefilter("ignore", UserWarning) - est_hc1 = SpilloverDiD(**common) # vcov_type="hc1" default, no cluster + # 4.0 (M-080): cluster=False is the explicit observation-level HC1 lane + # (the 3.x default); the default and cluster="unit" are both CR1 at unit. + est_hc1 = SpilloverDiD(cluster=False, **common) res_hc1 = est_hc1.fit(df, outcome="y", unit="unit", time="time", treatment="D") est_cr1 = SpilloverDiD(cluster="unit", **common) res_cr1 = est_cr1.fit(df, outcome="y", unit="unit", time="time", treatment="D") + res_default = SpilloverDiD(**common).fit( + df, outcome="y", unit="unit", time="time", treatment="D" + ) # Point estimates match (cluster kwarg only affects variance). - assert res_hc1.att == res_cr1.att + assert res_hc1.att == res_cr1.att == res_default.att # SE values must DIFFER — if equal, the cluster kwarg was a no-op. assert res_hc1.se != res_cr1.se, ( f"HC1 SE {res_hc1.se!r} == CR1 SE {res_cr1.se!r}; " f"cluster= appears to be silently ignored" ) + # The 4.0 default IS the unit auto-cluster: bit-equal to cluster="unit". + np.testing.assert_allclose(res_default.se, res_cr1.se, rtol=1e-14, atol=1e-14) + assert res_default.cluster_name == "unit" and res_default.n_clusters == res_cr1.n_clusters + assert res_hc1.cluster_name is None and res_hc1.n_clusters is None def test_single_cluster_sample_raises(self): """CR1 path on a single-cluster sample raises ValueError per diff --git a/tests/test_t23_spillover_tva_drift.py b/tests/test_t23_spillover_tva_drift.py index a4c131177..57d88d7a6 100644 --- a/tests/test_t23_spillover_tva_drift.py +++ b/tests/test_t23_spillover_tva_drift.py @@ -171,6 +171,16 @@ def spillover_fit(panel): return est.fit(panel, outcome="y", unit="unit", time="time", treatment="D") +@pytest.fixture(scope="module") +def spillover_hc1_fit(panel): + """The observation-level HC1 lane (the 3.x default): ``cluster=False`` is the + 4.0 off-switch; the module's ``spillover_fit`` is the 4.0 default (CR1 at unit).""" + est = SpilloverDiD(rings=[0.0, D_BAR_KM], conley_coords=("lat", "lon"), cluster=False) + with warnings.catch_warnings(): + _silence_spillover_matmul_warnings() + return est.fit(panel, outcome="y", unit="unit", time="time", treatment="D") + + @pytest.fixture(scope="module") def spillover_conley_lag0_fit(panel): est = SpilloverDiD( @@ -383,21 +393,69 @@ def test_rings_grid_d_bar_100_to_200_identical(panel): np.testing.assert_allclose(results, results[0] * np.ones(3), atol=1e-10) -def test_conley_se_differs_from_hc1(spillover_fit, spillover_conley_lag0_fit): - """§6 sanity: Conley vcov produces a different SE than HC1 by more than - floating-point noise. Pairs with `test_conley_se_less_than_hc1` which - pins the direction of the difference for this specific DGP.""" +def test_conley_se_differs_from_default( + spillover_fit, spillover_hc1_fit, spillover_conley_lag0_fit +): + """§6 sanity: Conley vcov produces a different SE than both the default + (CR1 at unit, 4.0) and the HC1 lane by more than floating-point noise.""" assert abs(spillover_conley_lag0_fit.se - spillover_fit.se) > 1e-6 + assert abs(spillover_conley_lag0_fit.se - spillover_hc1_fit.se) > 1e-6 -def test_conley_se_less_than_hc1(spillover_fit, spillover_conley_lag0_fit): +def test_conley_se_less_than_default_cr1( + spillover_fit, spillover_hc1_fit, spillover_conley_lag0_fit +): """§6 prose claim: 'on this DGP, the Conley spatial-HAC SE comes in - *lower* than HC1'. Pin the direction so the narrative doesn't go - stale if a future library change flips the sign of the per-pair - score covariance and reverses the inequality.""" + *lower* than both HC1 and the unit-clustered default'. Pin the direction + so the narrative doesn't go stale if a future library change flips the + sign of the per-pair score covariance and reverses the inequality.""" + assert spillover_conley_lag0_fit.se < spillover_hc1_fit.se assert spillover_conley_lag0_fit.se < spillover_fit.se +def test_cr1_unit_default_exceeds_hc1(spillover_fit, spillover_hc1_fit): + """§6 table ordering: on this DGP the 4.0 default (CR1 at unit) is wider + than the observation-level HC1 lane; point estimates identical.""" + assert spillover_fit.att == spillover_hc1_fit.att + assert spillover_fit.se > spillover_hc1_fit.se + + +def test_default_reports_unit_cluster(spillover_fit, spillover_hc1_fit, panel): + """4.0 (M-080): the default fit is CR1 at unit and says so; the HC1 lane + carries no cluster metadata.""" + assert spillover_fit.cluster_name == "unit" + assert spillover_fit.n_clusters == panel["unit"].nunique() == N_TREATED + N_NEAR + N_FAR + assert spillover_hc1_fit.cluster_name is None and spillover_hc1_fit.n_clusters is None + + +def test_notebook_section6_source_pins(): + """The §6 comparison cell must build the four-row table (CR1 at unit = the 4.0 + default, HC1 via ``cluster=False``, Conley lag 0, Conley lag 1) - source-level + pins so a notebook-only edit cannot silently drop the off-switch row. Skips + without source-tree access (see the DGP-constants guard below).""" + import json + from pathlib import Path + + nb_path = Path(__file__).resolve().parents[1] / "docs" / "tutorials" / "23_spillover_tva.ipynb" + if not nb_path.exists(): + pytest.skip("Notebook source not found; meaningful only in local dev.") + with nb_path.open() as f: + nb = json.load(f) + cells = ["".join(c["source"]) for c in nb["cells"] if c["cell_type"] == "code"] + matches = [c for c in cells if "def fit_with(" in c] + assert len(matches) == 1, "exactly one §6 comparison cell expected" + src = matches[0] + assert "cluster=None" in src and "cluster=cluster" in src + assert 'fit_with("hc1", cluster=False)' in src + for label in ( + "CR1 at unit (4.0 default)", + "HC1 (cluster=False)", + "Conley (lag=0, spatial only)", + "Conley (lag=1, spatial + serial)", + ): + assert label in src, label + + def test_conley_se_point_estimates_invariant( spillover_fit, spillover_conley_lag0_fit, spillover_conley_lag1_fit ): diff --git a/tests/test_v4_cluster_policy.py b/tests/test_v4_cluster_policy.py new file mode 100644 index 000000000..5f91a6be1 --- /dev/null +++ b/tests/test_v4_cluster_policy.py @@ -0,0 +1,900 @@ +"""Item 7b of the 4.0 cut - the uniform cluster policy (ledger rows M-080 and M-155). + +Pins, in one place: + +1. The shared ``cluster`` contract (``diff_diff.utils.validate_cluster``): ``str`` + (a column name), ``None`` (the estimator's default clustering) or ``False`` + (explicitly unclustered) on every ``BaseEstimator`` subclass exposing the + parameter; ``True`` / non-``str`` labels raise ONE shared ``ValueError`` at + ``__init__``, through ``set_params`` and again at ``fit()`` (direct-mutation + defense). ``StackedDiD`` keeps its own ``'unit'``/``'unit_subexp'`` vocabulary. +2. ``SpilloverDiD`` auto-clusters at ``unit`` when ``cluster=`` is omitted (the ONE + default SE that moves at 4.0) with ``TwoWayFixedEffects``' carve-outs mirrored: + explicit ``cluster=`` wins, the auto-cluster is dropped on the Conley path and is + never injected as a survey PSU; ``cluster=False`` reproduces the 3.x + observation-level HC1 lane (its goldens are pinned in ``tests/test_spillover.py``). +3. ``cluster=False`` on the estimators whose clustering is a CHOICE (category c1: + TWFE static + event-study, SunAbraham, WooldridgeDiD OLS + GLM, LPDiD): the plain + heteroskedasticity-robust sandwich with residual-df t inference (the + ``df_convention`` knob is inert without a cluster count), no cluster metadata; the + lanes whose bootstrap draws AT the analytical cluster (DiD / TWFE wild, + WooldridgeDiD multiplier) and the cluster-only Bell-McCaffrey DOF lanes of + SunAbraham / WooldridgeDiD reject it. +4. ``cluster=False`` on the unit-level-by-construction estimators (category b / c2) + and on the 2x2 surfaces is a bit-identical no-op. +5. Cluster metadata (4.0): DiD / TWFE analytical fits report ``n_clusters`` from the + regression's own effective cluster count on non-survey fits; ``SunAbrahamResults`` + gains ``n_clusters`` and reports ``cluster_name`` on clustered analytical fits; + TWFE's survey-lane auto label is None (the regression received no cluster ids). +6. The ledger rows read ``done`` with this file as ``test_ref``. +""" + +from __future__ import annotations + +import inspect +import pickle +import warnings + +import numpy as np +import pandas as pd +import pytest +from scipy import stats + +import diff_diff +from diff_diff import ( + CallawaySantAnna, + ChaisemartinDHaultfoeuille, + DifferenceInDifferences, + EfficientDiD, + LPDiD, + SpilloverDiD, + StackedDiD, + SunAbraham, + SurveyDesign, + TripleDifference, + TwoWayFixedEffects, + WooldridgeDiD, + generate_ddd_data, + generate_reversible_did_data, + generate_staggered_ddd_data, +) +from diff_diff._base import BaseEstimator +from diff_diff.linalg import LinearRegression +from diff_diff.lwdid_sensitivity import robustness_pre_periods, sensitivity_no_anticipation +from diff_diff.utils import ClusterArg, validate_cluster +from tests.test_dml_did import FIT_KW as DML_FIT_KW +from tests.test_dml_did import make_staggered_dml_data +from tests.test_lwdid import _make_common_timing_panel +from tests.test_t23_spillover_tva_drift import D_BAR_KM, _build_t23_panel +from tests.test_v4_matrix import ROWS as LEDGER_ROWS + +THIS_TEST_REF = "tests/test_v4_cluster_policy.py" + +CONTRACT_MSG = r"cluster must be a column name \(str\), None" + + +def _t_tail(t, df): + return 2 * stats.t.sf(abs(t), df) + + +def _quintet(r): + return (r.att, r.se, r.t_stat, r.p_value, tuple(r.conf_int)) + + +def _meta(r): + return (getattr(r, "cluster_name", "absent"), getattr(r, "n_clusters", "absent")) + + +# --------------------------------------------------------------------------- # +# Panels +# --------------------------------------------------------------------------- # + + +@pytest.fixture(scope="module") +def panel(): + """60 units x 8 periods, cohorts {never, 4, 6}; unit-constant survey weight.""" + rng = np.random.default_rng(7) + rows = [] + for u in range(60): + ft = [0, 4, 6][u % 3] + lat, lon = 40.0 + 0.05 * (u // 6), -100.0 + 0.05 * (u % 6) + w = rng.uniform(0.5, 1.5) + for t in range(1, 9): + eff = 0.6 if (ft and t >= ft) else 0.0 + rows.append( + dict( + unit=u, + time=t, + first_treat=ft, + grp=int(ft > 0), + post=int(t >= 4), + outcome=0.3 * u / 60 + 0.1 * t + eff + rng.standard_normal() * 0.6, + lat=lat, + lon=lon, + x=rng.standard_normal(), + w=w, + row=len(rows), + ) + ) + df = pd.DataFrame(rows) + df["d"] = df["grp"] * df["post"] + df["treated"] = ((df["first_treat"] > 0) & (df["time"] >= df["first_treat"])).astype(int) + df["yb"] = (df["outcome"] > df["outcome"].median()).astype(float) + return df + + +@pytest.fixture(scope="module") +def spill_panel(): + return _build_t23_panel() + + +SA_KW = dict(outcome="outcome", unit="unit", time="time", first_treat="first_treat") +TW_KW = dict(outcome="outcome", treatment="d", post="post", unit="unit") +ES_KW = dict( + outcome="outcome", + treatment="grp", + unit="unit", + time="time", + event_study=True, + post_periods=[4, 5, 6, 7, 8], +) +LP_KW = dict(outcome="outcome", unit="unit", time="time", treatment="treated") +LP_CTOR = dict(pre_window=2, post_window=2) +SP_KW = dict(outcome="y", unit="unit", time="time", treatment="D") + + +def _spill(**kw): + return SpilloverDiD(rings=[0.0, D_BAR_KM], conley_coords=("lat", "lon"), **kw) + + +def _fit_quiet(est, df, **kw): + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + return est.fit(df, **kw) + + +# --------------------------------------------------------------------------- # +# (1) The shared contract +# --------------------------------------------------------------------------- # + + +def _cluster_roster(): + """Every exported BaseEstimator subclass (deduplicated by class object; + aliases such as CS / SA / TWFE point at the same classes) whose __init__ + exposes ``cluster``. Functions such as bias_corrected_local_linear (an + ndarray ``cluster``) are not estimators and are excluded.""" + seen, out = set(), [] + for name in diff_diff.__all__: + obj = getattr(diff_diff, name) + if not inspect.isclass(obj) or id(obj) in seen or not issubclass(obj, BaseEstimator): + continue + seen.add(id(obj)) + if "cluster" in inspect.signature(obj.__init__).parameters: + out.append(obj) + return sorted(out, key=lambda c: c.__name__) + + +ROSTER = _cluster_roster() +CTOR_KW = {"SpilloverDiD": {"rings": [0.0, D_BAR_KM]}, "LPDiD": LP_CTOR} + + +def test_roster_is_the_sixteen_documented_classes(): + assert {c.__name__ for c in ROSTER} == { + "CallawaySantAnna", + "ChaisemartinDHaultfoeuille", + "DMLDiD", + "DifferenceInDifferences", + "EfficientDiD", + "HeterogeneousAdoptionDiD", + "ImputationDiD", + "LPDiD", + "LWDiD", + "SpilloverDiD", + "StackedDiD", + "SunAbraham", + "TripleDifference", + "TwoStageDiD", + "TwoWayFixedEffects", + "WooldridgeDiD", + } + + +class TestValidateCluster: + @pytest.mark.parametrize("ok", [None, False, "unit", "state"]) + def test_accepts_and_returns_unchanged(self, ok): + assert validate_cluster(ok) is ok + + @pytest.mark.parametrize("bad", [True, 1, 0, 1.5, [], ("a",), b"x", np.bool_(False)]) + def test_rejects_everything_else(self, bad): + with pytest.raises(ValueError, match=CONTRACT_MSG): + validate_cluster(bad) + + def test_alias_is_the_documented_union(self): + import typing + + assert set(typing.get_args(ClusterArg)) == {str, typing.Literal[False], type(None)} + + +@pytest.mark.parametrize("cls", ROSTER, ids=lambda c: c.__name__) +def test_false_accepted_and_round_trips_except_stacked(cls): + kw = CTOR_KW.get(cls.__name__, {}) + if cls is StackedDiD: + with pytest.raises(ValueError, match="cluster must be 'unit' or 'unit_subexp'"): + StackedDiD(cluster=False) + return + est = cls(cluster=False, **kw) + assert est.cluster is False + assert est.get_params()["cluster"] is False + est2 = cls(**kw) + est2.set_params(cluster=False) + assert est2.cluster is False + est2.set_params(cluster=None) + assert est2.cluster is None + + +@pytest.mark.parametrize("cls", ROSTER, ids=lambda c: c.__name__) +def test_true_and_non_str_labels_rejected_everywhere(cls): + kw = CTOR_KW.get(cls.__name__, {}) + if cls is StackedDiD: + for bad in (True, 7): + with pytest.raises(ValueError, match="cluster must be 'unit' or 'unit_subexp'"): + StackedDiD(cluster=bad) + return + for bad in (True, 7): + with pytest.raises(ValueError, match=CONTRACT_MSG): + cls(cluster=bad, **kw) + est = cls(**kw) + with pytest.raises(ValueError, match=CONTRACT_MSG): + est.set_params(cluster=bad) + assert est.cluster is None # transactional rollback + + +def test_dcdh_column_name_still_raises_not_implemented(): + with pytest.raises(NotImplementedError, match="user-specified clustering"): + ChaisemartinDHaultfoeuille(cluster="unit") + + +def _fit_recipes(panel, spill_panel): + """Per-class (ctor kwargs, data, fit kwargs) for a minimal fit.""" + had_g = 200 + rng = np.random.default_rng(3) + d = rng.uniform(0.0, 1.0, had_g) + d[0] = 0.0 + dy = 0.3 * d + 0.1 * rng.standard_normal(had_g) + had = pd.DataFrame( + { + "unit": np.repeat(np.arange(had_g), 2), + "period": np.tile([1, 2], had_g), + "dose": np.column_stack([np.zeros(had_g), d]).ravel(), + "outcome": np.column_stack([np.zeros(had_g), dy]).ravel(), + } + ) + return { + "DifferenceInDifferences": ( + {}, + panel, + dict(outcome="outcome", treatment="grp", post="post"), + ), + "TwoWayFixedEffects": ({}, panel, TW_KW), + "SunAbraham": ({}, panel, SA_KW), + "WooldridgeDiD": ({}, panel, SA_KW), + "ImputationDiD": ({}, panel, SA_KW), + "TwoStageDiD": ({}, panel, SA_KW), + "CallawaySantAnna": ({}, panel, SA_KW), + "EfficientDiD": ({}, panel, SA_KW), + "LPDiD": (LP_CTOR, panel, LP_KW), + "SpilloverDiD": ( + dict(rings=[0.0, D_BAR_KM], conley_coords=("lat", "lon")), + spill_panel, + SP_KW, + ), + "DMLDiD": ( + dict(seed=0, n_folds=2), + make_staggered_dml_data(), + dict(covariates=["x1", "x2"], **DML_FIT_KW), + ), + "ChaisemartinDHaultfoeuille": ( + {}, + generate_reversible_did_data(n_groups=40, n_periods=5, seed=1), + dict(outcome="outcome", unit="group", time="period", treatment="treatment"), + ), + "TripleDifference": ( + {}, + generate_ddd_data(n_per_cell=100, treatment_effect=2.0, seed=42), + dict(outcome="outcome", group="group", partition="partition", post="time"), + ), + "LWDiD": ( + {}, + _make_common_timing_panel(), + dict(outcome="y", unit="unit", time="time", treatment="treat"), + ), + "HeterogeneousAdoptionDiD": ( + dict(design="continuous_at_zero"), + had, + dict(outcome="outcome", dose="dose", time="period", unit="unit"), + ), + } + + +@pytest.fixture(scope="module") +def recipes(panel, spill_panel): + return _fit_recipes(panel, spill_panel) + + +@pytest.mark.parametrize( + "name", sorted(c.__name__ for c in ROSTER if c is not StackedDiD), ids=lambda n: n +) +def test_direct_mutation_to_true_raises_at_fit_on_every_class(name, recipes): + """The fit-time re-check is wired on EVERY fit path, not only the c1 lanes: + without it ``True`` would fall through to a column lookup (KeyError).""" + ctor, data, fit_kw = recipes[name] + est = getattr(diff_diff, name)(**ctor) + est.cluster = True # bypasses __init__ and set_params on purpose + with pytest.raises(ValueError, match=CONTRACT_MSG): + _fit_quiet(est, data, **fit_kw) + + +# --------------------------------------------------------------------------- # +# (2) SpilloverDiD auto-cluster (category a) +# --------------------------------------------------------------------------- # + + +class TestSpilloverAutoCluster: + @pytest.fixture(scope="class") + def default(self, spill_panel): + return _fit_quiet(_spill(), spill_panel, **SP_KW) + + @pytest.fixture(scope="class") + def explicit_unit(self, spill_panel): + return _fit_quiet(_spill(cluster="unit"), spill_panel, **SP_KW) + + @pytest.fixture(scope="class") + def off(self, spill_panel): + return _fit_quiet(_spill(cluster=False), spill_panel, **SP_KW) + + def test_default_is_cr1_at_unit(self, default, explicit_unit, spill_panel): + G = spill_panel["unit"].nunique() + assert default.cluster_name == "unit" + assert default.n_clusters == G == explicit_unit.n_clusters + assert default.att == explicit_unit.att + np.testing.assert_allclose(default.se, explicit_unit.se, rtol=1e-14, atol=1e-14) + np.testing.assert_allclose(default.p_value, explicit_unit.p_value, rtol=1e-12) + assert "CR1 cluster-robust at unit, G=" in default.summary() + + def test_false_is_the_observation_level_hc1_lane(self, default, off): + assert off.cluster_name is None and off.n_clusters is None + assert off.att == default.att + assert off.se != default.se + assert "HC1 heteroskedasticity-robust" in off.summary() + + def test_false_differs_from_none(self, default, off): + assert default.se != off.se # None is NOT the off-switch + + def test_conley_drops_the_auto_cluster(self, spill_panel): + kw = dict(vcov_type="conley", conley_cutoff_km=D_BAR_KM, conley_lag_cutoff=0) + auto = _fit_quiet(_spill(**kw), spill_panel, **SP_KW) + off = _fit_quiet(_spill(cluster=False, **kw), spill_panel, **SP_KW) + assert auto.cluster_name is None and auto.n_clusters is None + assert auto.se == off.se and auto.p_value == off.p_value + assert off.cluster_name is None + + def test_conley_explicit_cluster_keeps_the_product_kernel(self, spill_panel): + kw = dict(vcov_type="conley", conley_cutoff_km=D_BAR_KM, conley_lag_cutoff=0) + auto = _fit_quiet(_spill(**kw), spill_panel, **SP_KW) + prod = _fit_quiet(_spill(cluster="unit", **kw), spill_panel, **SP_KW) + assert prod.cluster_name == "unit" and prod.n_clusters == spill_panel["unit"].nunique() + assert prod.se != auto.se + + def test_survey_never_injects_the_auto_cluster(self, spill_panel): + df = spill_panel.copy() + codes = {u: i for i, u in enumerate(df["unit"].unique())} + df["w"] = df["unit"].map(lambda u: 0.5 + (codes[u] % 7) / 7) + auto = _fit_quiet(_spill(), df, survey_design=SurveyDesign(weights="w"), **SP_KW) + off = _fit_quiet( + _spill(cluster=False), df, survey_design=SurveyDesign(weights="w"), **SP_KW + ) + explicit = _fit_quiet( + _spill(cluster="unit"), df, survey_design=SurveyDesign(weights="w"), **SP_KW + ) + # weights-only design: implicit per-observation PSUs under auto AND False + assert auto.n_psu == len(df) == off.n_psu + assert auto.survey_metadata.df_survey == len(df) - 1 + assert auto.cluster_name is None and auto.n_clusters is None + assert auto.se == off.se + # explicit cluster= still becomes the PSU + assert explicit.n_psu == df["unit"].nunique() == explicit.n_clusters + assert explicit.cluster_name == "unit" + assert explicit.se != auto.se + + def test_point_estimates_never_move(self, default, off, explicit_unit): + for a, b in ((default, off), (default, explicit_unit)): + assert a.att == b.att + pd.testing.assert_frame_equal( + a.spillover_effects[["coef"]], b.spillover_effects[["coef"]] + ) + + +# --------------------------------------------------------------------------- # +# (3) The c1 off-switch +# --------------------------------------------------------------------------- # + + +class TestTWFEOffSwitch: + def test_static_plain_hc1_full_k_parity(self, panel): + """cluster=False on the within-transform TWFE = HC1 on the explicit + unit + post dummy design (the absorbed-FE full-K rescale), residual-df t.""" + default = TwoWayFixedEffects().fit(panel, **TW_KW) + off = TwoWayFixedEffects(cluster=False).fit(panel, **TW_KW) + assert off.cluster_name is None and off.n_clusters is None + assert off.att == default.att and off.se != default.se + X = pd.concat( + [ + panel[["d"]].astype(float), + pd.get_dummies(panel["unit"], prefix="u", dtype=float), + pd.get_dummies(panel["post"], prefix="p", drop_first=True, dtype=float), + ], + axis=1, + ) + reg = LinearRegression(include_intercept=False).fit(X.values, panel["outcome"].values) + inf = reg.get_inference(0) + np.testing.assert_allclose(off.se, inf.se, rtol=1e-10) + np.testing.assert_allclose(off.p_value, inf.p_value, rtol=1e-10) + assert off.inference_df == pytest.approx(inf.df) + n, k_full = len(panel), X.shape[1] + assert off.inference_df == n - k_full + + @pytest.mark.parametrize("knob", ["cluster", "residual"]) + def test_static_knob_inert_without_a_cluster(self, panel, knob): + off = TwoWayFixedEffects(cluster=False, df_convention=knob).fit(panel, **TW_KW) + base = TwoWayFixedEffects(cluster=False).fit(panel, **TW_KW) + assert _quintet(off) == _quintet(base) + z = TwoWayFixedEffects(cluster=False, df_convention="normal").fit(panel, **TW_KW) + assert z.inference_df is None and z.p_value != base.p_value + + def test_static_hc2_bm_one_way_lane(self, panel): + default = TwoWayFixedEffects(vcov_type="hc2_bm").fit(panel, **TW_KW) + off = TwoWayFixedEffects(cluster=False, vcov_type="hc2_bm").fit(panel, **TW_KW) + assert off.cluster_name is None and off.n_clusters is None + assert default.cluster_name == "unit" + assert np.isfinite(off.se) and np.isfinite(off.p_value) + assert off.inference_df is not None and off.inference_df != default.inference_df + X = pd.concat( + [ + panel[["d"]].astype(float), + pd.get_dummies(panel["unit"], prefix="u", dtype=float), + pd.get_dummies(panel["post"], prefix="p", drop_first=True, dtype=float), + ], + axis=1, + ) + reg = LinearRegression(include_intercept=False, vcov_type="hc2_bm").fit( + X.values, panel["outcome"].values + ) + np.testing.assert_allclose(off.se, reg.get_inference(0).se, rtol=1e-10) + + def test_event_study_off_switch(self, panel): + default = TwoWayFixedEffects().fit(panel, **ES_KW) + off = TwoWayFixedEffects(cluster=False).fit(panel, **ES_KW) + keep = ~off.is_reference + np.testing.assert_allclose(off.att[keep], default.att[keep], rtol=1e-12) + assert np.all(off.se[keep] != default.se[keep]) + # residual df on every row (knob inert), G-1 on the default + assert np.all(off.df[keep] == off.df[keep][0]) and off.df[keep][0] > panel["unit"].nunique() + assert np.all(default.df[keep] == panel["unit"].nunique() - 1) + res = TwoWayFixedEffects(cluster=False, df_convention="residual").fit(panel, **ES_KW) + np.testing.assert_array_equal(res.df[keep], off.df[keep]) + np.testing.assert_allclose(res.p_value[keep], off.p_value[keep], rtol=1e-12) + + def test_event_study_hc2_bm_one_way_lane_is_finite(self, panel): + default = TwoWayFixedEffects(vcov_type="hc2_bm").fit(panel, **ES_KW) + off = TwoWayFixedEffects(cluster=False, vcov_type="hc2_bm").fit(panel, **ES_KW) + keep = ~off.is_reference + assert np.all(np.isfinite(off.se[keep])) and np.all(np.isfinite(off.df[keep])) + assert np.all(off.df[keep] != default.df[keep]) + + def test_wild_bootstrap_rejects_false(self, panel): + with pytest.raises(ValueError, match="requires a cluster"): + TwoWayFixedEffects(cluster=False, inference="wild_bootstrap", n_bootstrap=9).fit( + panel, **TW_KW + ) + with pytest.raises(ValueError, match="requires cluster="): + DifferenceInDifferences(cluster=False, inference="wild_bootstrap", n_bootstrap=9).fit( + panel, outcome="outcome", treatment="grp", post="post" + ) + + def test_front_doors_keep_precedence_over_the_wild_reject(self, panel): + est = TwoWayFixedEffects(cluster=False, inference="wild_bootstrap", n_bootstrap=9) + with pytest.raises(NotImplementedError): + est.fit(panel, survey_design=SurveyDesign(weights="w"), **TW_KW) + + def test_conley_plus_false_equals_conley_default(self, panel): + kw = dict( + vcov_type="conley", + conley_coords=("lat", "lon"), + conley_cutoff_km=20.0, + conley_lag_cutoff=0, + ) + a = TwoWayFixedEffects(**kw).fit(panel, **TW_KW) + b = TwoWayFixedEffects(cluster=False, **kw).fit(panel, **TW_KW) + assert _quintet(a) == _quintet(b) + assert a.cluster_name is None and b.cluster_name is None + assert a.n_clusters is None and b.n_clusters is None + + def test_survey_default_has_no_psu_injection(self, panel): + r = TwoWayFixedEffects().fit(panel, survey_design=SurveyDesign(weights="w"), **TW_KW) + assert r.survey_metadata.n_psu == len(panel) + assert r.cluster_name is None and r.n_clusters is None # 4.0: honest label + off = TwoWayFixedEffects(cluster=False).fit( + panel, survey_design=SurveyDesign(weights="w"), **TW_KW + ) + assert _quintet(off) == _quintet(r) + + +class TestSunAbrahamOffSwitch: + def test_plain_hc1_residual_df(self, panel): + default = SunAbraham().fit(panel, **SA_KW) + off = SunAbraham(cluster=False).fit(panel, **SA_KW) + assert off.att == default.att and off.se != default.se + assert off.cluster_name is None and off.n_clusters is None + assert off.inference_df > panel["unit"].nunique() # residual df, not G-1 + np.testing.assert_allclose(off.p_value, _t_tail(off.t_stat, off.inference_df), rtol=1e-12) + res = SunAbraham(cluster=False, df_convention="residual").fit(panel, **SA_KW) + assert _quintet(res) == _quintet(off) + assert "HC1 heteroskedasticity-robust" in off.summary() + + def test_hc2_bm_rejected(self, panel): + with pytest.raises(ValueError, match="Bell-McCaffrey DOF lane is cluster-only"): + SunAbraham(cluster=False, vcov_type="hc2_bm").fit(panel, **SA_KW) + + def test_pairs_bootstrap_accepts_false_as_a_no_op(self, panel): + a = SunAbraham(n_bootstrap=19, seed=1).fit(panel, **SA_KW) + b = SunAbraham(cluster=False, n_bootstrap=19, seed=1).fit(panel, **SA_KW) + assert a.se == b.se and a.p_value == b.p_value + assert b.cluster_name is None and b.n_clusters is None + assert a.cluster_name is None and a.n_clusters is None # bootstrap: no analytical cluster + assert "CR1 cluster-robust" not in a.summary() + + def test_survey_false_means_no_unit_psu_injection(self, panel): + default = SunAbraham().fit(panel, survey_design=SurveyDesign(weights="w"), **SA_KW) + off = SunAbraham(cluster=False).fit(panel, survey_design=SurveyDesign(weights="w"), **SA_KW) + assert default.survey_metadata.n_psu == panel["unit"].nunique() + assert off.survey_metadata.n_psu == len(panel) + assert off.survey_metadata.df_survey == len(panel) - 1 + assert off.se != default.se + assert default.cluster_name is None and default.n_clusters is None + assert off.cluster_name is None and off.n_clusters is None + assert "Std. errors:" not in default.summary() + + def test_conley_plus_false_equals_conley_default(self, panel): + kw = dict( + vcov_type="conley", + conley_coords=("lat", "lon"), + conley_cutoff_km=20.0, + conley_lag_cutoff=0, + ) + a = SunAbraham(**kw).fit(panel, **SA_KW) + b = SunAbraham(cluster=False, **kw).fit(panel, **SA_KW) + assert _quintet(a) == _quintet(b) and a.cluster_name is None and a.n_clusters is None + + +class TestWooldridgeOffSwitch: + def test_ols_within_plain_hc1_cell_parity(self, panel): + """The unclustered within lane carries the absorbed-FE full-K rescale, so + each cohort-time cell SE equals HC1 on the explicit full-dummy design.""" + default = WooldridgeDiD().fit(panel, **SA_KW) + off = WooldridgeDiD(cluster=False).fit(panel, **SA_KW) + assert off.att == pytest.approx(default.att, rel=1e-12) and off.se != default.se + assert off.cluster_name is None and off.n_clusters is None + cells = sorted(off.group_time_effects) + cell_cols = { + (g, t): ((panel["first_treat"] == g) & (panel["time"] == t)).astype(float).values + for (g, t) in cells + } + X = np.column_stack( + [cell_cols[c] for c in cells] + + [pd.get_dummies(panel["unit"], dtype=float).values] + + [pd.get_dummies(panel["time"], drop_first=True, dtype=float).values] + ) + reg = LinearRegression(include_intercept=False).fit(X, panel["outcome"].values) + for i, c in enumerate(cells): + np.testing.assert_allclose( + off.group_time_effects[c]["se"], reg.get_inference(i).se, rtol=1e-9 + ) + + def test_ols_knob_inert_without_a_cluster(self, panel): + off = WooldridgeDiD(cluster=False).fit(panel, **SA_KW) + res = WooldridgeDiD(cluster=False, df_convention="residual").fit(panel, **SA_KW) + assert _quintet(off) == _quintet(res) + z = WooldridgeDiD(cluster=False, df_convention="normal").fit(panel, **SA_KW) + assert z.p_value != off.p_value + np.testing.assert_allclose(z.p_value, 2 * stats.norm.sf(abs(z.t_stat)), rtol=1e-12) + + def test_ols_cohort_trends_full_dummy_lane(self, panel): + default = WooldridgeDiD(cohort_trends=True).fit(panel, **SA_KW) + off = WooldridgeDiD(cluster=False, cohort_trends=True).fit(panel, **SA_KW) + assert off.se != default.se and off.cluster_name is None and off.n_clusters is None + + @pytest.mark.parametrize("method", ["logit", "poisson"]) + def test_glm_plain_qmle_sandwich_normal_theory(self, panel, method): + kw = dict(outcome="yb", unit="unit", time="time", first_treat="first_treat") + default = WooldridgeDiD(method=method).fit(panel, **kw) + off = WooldridgeDiD(method=method, cluster=False).fit(panel, **kw) + assert off.att == pytest.approx(default.att, rel=1e-10) and off.se != default.se + assert off.cluster_name is None and off.n_clusters is None + assert default.cluster_name == "unit" and default.n_clusters == panel["unit"].nunique() + # GLM inference stays normal-theory (knob-independent): only the SE moved + np.testing.assert_allclose(off.p_value, 2 * stats.norm.sf(abs(off.t_stat)), rtol=1e-12) + + def test_bootstrap_rejects_false(self, panel): + with pytest.raises(ValueError, match="multiplier bootstrap draws one weight per cluster"): + WooldridgeDiD(cluster=False, n_bootstrap=9).fit(panel, **SA_KW) + + def test_hc2_bm_rejects_false(self, panel): + with pytest.raises(ValueError, match="Bell-McCaffrey DOF lane is cluster-only"): + WooldridgeDiD(cluster=False, vcov_type="hc2_bm").fit(panel, **SA_KW) + + def test_conley_plus_false_equals_conley_default(self, panel): + kw = dict( + vcov_type="conley", + conley_coords=("lat", "lon"), + conley_cutoff_km=20.0, + conley_lag_cutoff=0, + ) + a = WooldridgeDiD(**kw).fit(panel, **SA_KW) + b = WooldridgeDiD(cluster=False, **kw).fit(panel, **SA_KW) + assert _quintet(a) == _quintet(b) and a.cluster_name is None + + def test_survey_default_has_no_psu_injection(self, panel): + r = WooldridgeDiD().fit(panel, survey_design=SurveyDesign(weights="w"), **SA_KW) + assert r.survey_metadata.n_psu == len(panel) + assert r.cluster_name is None and r.n_clusters is None + + def test_saturated_within_full_k_df_voids_inference(self): + """Full-K residual df <= 0 on the unclustered within lane -> all-NaN + inference (the helper's NaN contract), never a finite under-counted SE.""" + # 2 units x 2 periods, one treated cell: n = 4, k = 1 cell dummy, absorbed unit + + # time rank = 3 -> full-K residual df = 0. + rows = [] + for u, ft in enumerate([0, 2]): + for t in (1, 2): + rows.append( + dict( + unit=u, + time=t, + first_treat=ft, + outcome=float(0.7 * u + t + (ft and t >= ft)), + ) + ) + df = pd.DataFrame(rows) + r = _fit_quiet(WooldridgeDiD(cluster=False), df, **SA_KW) + assert np.isfinite(r.att) + assert np.isnan(r.se) and np.isnan(r.t_stat) and np.isnan(r.p_value) + assert np.isnan(r.conf_int[0]) and np.isnan(r.conf_int[1]) + + +class TestLPDiDOffSwitch: + def test_ols_lane(self, panel): + default = LPDiD(**LP_CTOR).fit(panel, **LP_KW) + off = LPDiD(cluster=False, **LP_CTOR).fit(panel, **LP_KW) + assert off.att == default.att and off.se != default.se + assert off.cluster_name is None and off.n_clusters is None + assert off.vcov_type == "hc1" + assert off.event_study["n_clusters"].isna().all() + assert default.event_study["n_clusters"].notna().any() + # residual df per horizon (knob inert), G-1 on the default + assert all(v > panel["unit"].nunique() for v in off.event_study_df.values()) + res = LPDiD(cluster=False, df_convention="residual", **LP_CTOR).fit(panel, **LP_KW) + assert _quintet(res) == _quintet(off) + assert "HC1 heteroskedasticity-robust" in off.summary() + assert off.to_dict()["inference_method"] == "heteroskedasticity_robust" + + def test_ra_lane_unclustered_if_variance(self, panel): + default = LPDiD(reweight=True, **LP_CTOR).fit(panel, covariates=["x"], **LP_KW) + off = LPDiD(cluster=False, reweight=True, **LP_CTOR).fit(panel, covariates=["x"], **LP_KW) + assert off.vcov_type == "if_robust" and default.vcov_type == "if_cluster" + assert off.cluster_name is None and off.n_clusters is None + assert off.se != default.se + # Singleton clusters make sum_g s_g^2 == sum_i phi_i^2 and neither lane + # carries a finite-sample factor: the SEs agree EXACTLY, the p/CI differ + # (t(G-1 = n-1) vs the residual df). + single = LPDiD(cluster="row", reweight=True, **LP_CTOR).fit( + panel, covariates=["x"], **LP_KW + ) + np.testing.assert_allclose(off.se, single.se, rtol=1e-14) + assert off.p_value != single.p_value + assert "Influence-function heteroskedasticity-robust (unclustered)" in off.summary() + + def test_survey_false_implicit_observation_psus(self, panel): + default = LPDiD(**LP_CTOR).fit(panel, survey_design=SurveyDesign(weights="w"), **LP_KW) + off = LPDiD(cluster=False, **LP_CTOR).fit( + panel, survey_design=SurveyDesign(weights="w"), **LP_KW + ) + assert default.n_psu == panel["unit"].nunique() == default.n_clusters + assert off.n_psu == len(panel) + assert off.cluster_name is None and off.n_clusters is None + assert off.se != default.se + + +# --------------------------------------------------------------------------- # +# (4) The b / c2 / n-a no-op +# --------------------------------------------------------------------------- # + + +NOOP = [ + "CallawaySantAnna", + "EfficientDiD", + "DMLDiD", + "ChaisemartinDHaultfoeuille", + "LWDiD", + "HeterogeneousAdoptionDiD", + "ImputationDiD", + "TwoStageDiD", + "DifferenceInDifferences", + "TripleDifference", +] + + +@pytest.mark.parametrize("name", NOOP) +def test_false_is_a_bit_identical_no_op(name, recipes): + ctor, data, fit_kw = recipes[name] + cls = getattr(diff_diff, name) + a = _fit_quiet(cls(**ctor), data, **fit_kw) + b = _fit_quiet(cls(cluster=False, **ctor), data, **fit_kw) + assert _quintet(a) == _quintet(b) + assert _meta(a) == _meta(b) + + +def test_false_no_op_on_the_staggered_ddd_mode(): + stag = generate_staggered_ddd_data(n_units=96, n_periods=6, cohort_periods=[3, 4], seed=42) + kw = dict( + outcome="outcome", + unit="unit", + time="period", + first_treat="first_treat", + partition="eligibility", + ) + a = _fit_quiet(TripleDifference(n_bootstrap=19, seed=1), stag, **kw) + b = _fit_quiet(TripleDifference(cluster=False, n_bootstrap=19, seed=1), stag, **kw) + assert _quintet(a) == _quintet(b) and _meta(a) == _meta(b) + with pytest.raises(ValueError, match="not supported in staggered DDD mode"): + _fit_quiet(TripleDifference(cluster="unit", n_bootstrap=19, seed=1), stag, **kw) + + +def test_false_no_op_on_the_cs_rc_auto_psu_path(panel): + unbalanced = panel.drop(panel.index[::17]) + kw = dict(allow_unbalanced_panel=True) + a = _fit_quiet(CallawaySantAnna(**kw), unbalanced, **SA_KW) + b = _fit_quiet(CallawaySantAnna(cluster=False, **kw), unbalanced, **SA_KW) + assert a.cluster_name == "unit" == b.cluster_name # the RC unit PSU survives False + assert _quintet(a) == _quintet(b) and a.n_clusters == b.n_clusters + + +def test_hausman_pretest_accepts_false(panel): + a = EfficientDiD.hausman_pretest(panel, **SA_KW) + b = EfficientDiD.hausman_pretest(panel, cluster=False, **SA_KW) + assert a.statistic == b.statistic and a.p_value == b.p_value + + +def test_lwdid_sensitivity_helpers_accept_false(): + df = _make_common_timing_panel() + kw = dict(outcome="y", unit="unit", time="time", treatment="treat") + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + a = robustness_pre_periods(df, **kw) + b = robustness_pre_periods(df, cluster=False, **kw) + c = sensitivity_no_anticipation(df, **kw) + d = sensitivity_no_anticipation(df, cluster=False, **kw) + pd.testing.assert_frame_equal(a.to_dataframe(), b.to_dataframe()) + pd.testing.assert_frame_equal(c.to_dataframe(), d.to_dataframe()) + with pytest.raises(ValueError, match=CONTRACT_MSG): + robustness_pre_periods(df, cluster=True, **kw) + + +def test_stacked_did_still_rejects_false(): + with pytest.raises(ValueError, match="cluster must be 'unit' or 'unit_subexp'"): + StackedDiD(cluster=False) + + +# --------------------------------------------------------------------------- # +# (5) Cluster metadata +# --------------------------------------------------------------------------- # + + +class TestClusterMetadata: + def test_did_analytical_n_clusters(self, panel): + G = panel["unit"].nunique() + r = DifferenceInDifferences(cluster="unit").fit( + panel, outcome="outcome", treatment="grp", post="post" + ) + assert (r.cluster_name, r.n_clusters) == ("unit", G) + assert r.inference_df == G - 1 + plain = DifferenceInDifferences().fit( + panel, outcome="outcome", treatment="grp", post="post" + ) + assert plain.n_clusters is None and plain.cluster_name is None + + def test_did_explicit_cluster_under_psu_survey_keeps_none_count(self, panel): + df = panel.copy() + df["psu"] = df["unit"] // 2 + r = DifferenceInDifferences(cluster="unit").fit( + df, + outcome="outcome", + treatment="grp", + post="post", + survey_design=SurveyDesign(weights="w", psu="psu"), + ) + assert r.cluster_name == "unit" and r.n_clusters is None + + def test_twfe_metadata_lanes(self, panel): + G = panel["unit"].nunique() + auto = TwoWayFixedEffects().fit(panel, **TW_KW) + assert (auto.cluster_name, auto.n_clusters) == ("unit", G) + assert auto.summary().count(f"G={G}") == 1 + wild = TwoWayFixedEffects(inference="wild_bootstrap", n_bootstrap=9, seed=1).fit( + panel, **TW_KW + ) + assert wild.n_clusters == G and wild.inference_method == "wild_bootstrap" + kw = dict( + vcov_type="conley", + conley_coords=("lat", "lon"), + conley_cutoff_km=20.0, + conley_lag_cutoff=0, + ) + c_auto = TwoWayFixedEffects(**kw).fit(panel, **TW_KW) + assert (c_auto.cluster_name, c_auto.n_clusters) == (None, None) + panel2 = panel.assign(region=panel["unit"] % 5) + c_exp = TwoWayFixedEffects(cluster="region", **kw).fit(panel2, **TW_KW) + assert (c_exp.cluster_name, c_exp.n_clusters) == ("region", 5) + one_way = TwoWayFixedEffects(vcov_type="hc2").fit(panel, **TW_KW) + assert (one_way.cluster_name, one_way.n_clusters) == (None, None) + + def test_sun_abraham_metadata_lanes(self, panel): + G = panel["unit"].nunique() + auto = SunAbraham().fit(panel, **SA_KW) + assert (auto.cluster_name, auto.n_clusters) == ("unit", G) + assert f"CR1 cluster-robust at unit, G={G}" in auto.summary() + d = auto.to_dict() + assert d["cluster_name"] == "unit" and d["n_clusters"] == G + panel2 = panel.assign(state=panel["unit"] % 5) + exp = SunAbraham(cluster="state").fit(panel2, **SA_KW) + assert (exp.cluster_name, exp.n_clusters) == ("state", 5) + off = SunAbraham(cluster=False).fit(panel, **SA_KW) + assert (off.cluster_name, off.n_clusters) == (None, None) + assert "n_clusters" not in off.to_dict() + one_way = SunAbraham(vcov_type="hc2").fit(panel, **SA_KW) + assert (one_way.cluster_name, one_way.n_clusters) == (None, None) + kw = dict( + vcov_type="conley", + conley_coords=("lat", "lon"), + conley_cutoff_km=20.0, + conley_lag_cutoff=0, + ) + c_exp = SunAbraham(cluster="state", **kw).fit(panel2, **SA_KW) + assert (c_exp.cluster_name, c_exp.n_clusters) == ("state", 5) + back = pickle.loads(pickle.dumps(auto)) + assert back.n_clusters == G and back.cluster_name == "unit" + + def test_sun_abraham_results_field_is_appended_last(self): + import dataclasses + + from diff_diff.sun_abraham import SunAbrahamResults + + names = [f.name for f in dataclasses.fields(SunAbrahamResults)] + assert names[-1] == "n_clusters" and names[-2] == "inference_df" + + +# --------------------------------------------------------------------------- # +# (6) Ledger +# --------------------------------------------------------------------------- # + + +def test_ledger_rows_are_done_and_point_here(): + rows = {r["id"]: r for r in LEDGER_ROWS} + for rid in ("M-080", "M-155"): + row = rows[rid] + assert row["kind"] == "behavior", rid + assert row["status"] == "done", rid + assert row["test_ref"] == THIS_TEST_REF, rid + assert rows["M-080"]["old"] == "diff_diff:SpilloverDiD[cluster]" + assert rows["M-155"]["old"] == "diff_diff.utils:validate_cluster" + assert rows["M-154"]["status"] == "planned" # 7c: the SpilloverDiD df_convention knob + assert rows["M-154"]["old"] == "diff_diff:SpilloverDiD[df_convention]" diff --git a/tests/test_v4_inference_policy.py b/tests/test_v4_inference_policy.py index 7876c822c..e640e3f87 100644 --- a/tests/test_v4_inference_policy.py +++ b/tests/test_v4_inference_policy.py @@ -63,9 +63,10 @@ INFERENCE_MSG = "inference must be one of ('analytical', 'wild_bootstrap'), got {value!r}" CLUSTER_REQUIRED_MSG = ( - "inference='wild_bootstrap' requires cluster=. The wild cluster " - "bootstrap resamples at the cluster level; pass cluster= or use " - "inference='analytical'." + "inference='wild_bootstrap' requires cluster=. The wild cluster " + "bootstrap resamples at the cluster level; cluster=None and " + "cluster=False both leave a DifferenceInDifferences fit unclustered, " + "so pass a cluster column or use inference='analytical'." ) @@ -354,11 +355,12 @@ def test_did_refit_transition_clears_bootstrap_metadata(self, clustered_panel, c did.set_params(inference="analytical") analytical = _fit_did(did, clustered_panel) - # All four metadata fields flow from the one _bootstrap_results - # conditional - pin the full set. + # The bootstrap-only metadata clears; n_clusters is re-filled from the + # analytical regression's own cluster count (4.0, M-080), not left over + # from the wild fit. assert analytical.inference_method == "analytical" assert analytical.n_bootstrap is None - assert analytical.n_clusters is None + assert analytical.n_clusters == clustered_panel["cluster"].nunique() assert analytical.p_val_type is None _assert_finite_quintet(analytical) @@ -377,7 +379,8 @@ def test_twfe_refit_transition_clears_bootstrap_metadata(self, clustered_panel, analytical = _fit_twfe(twfe, clustered_panel) assert analytical.inference_method == "analytical" assert analytical.n_bootstrap is None - assert analytical.n_clusters is None + # 4.0 (M-080): the analytical refit reports the unit auto-cluster's count. + assert analytical.n_clusters == clustered_panel["unit"].nunique() assert analytical.p_val_type is None twfe.set_params(inference="wild_bootstrap") diff --git a/tests/test_v4_matrix.py b/tests/test_v4_matrix.py index c4d53580b..155fc5fd4 100644 --- a/tests/test_v4_matrix.py +++ b/tests/test_v4_matrix.py @@ -142,11 +142,12 @@ # shipped by #838 without extending this floor) = 131; + the Wooldridge # to_dataframe level twin (M-150, 4.0-cut framing) = 132; + the LWDiD # results-contract rows (M-151/M-152 alias deprecations, M-153 to_dataframe -# reshape; 4.0-cut PR 1) = 135. +# reshape; 4.0-cut PR 1) = 135; + the 7b cluster-policy pair (M-154 SpilloverDiD +# df_convention, planned for 7c; M-155 the shared cluster contract) = 137. # Ids are never reused and terminal rows are never deleted, so the ledger # only grows - raise the floor when rows are added; a lower parse count # means scanner/format drift or an illegal row deletion. -ROW_COUNT_FLOOR = 135 +ROW_COUNT_FLOOR = 137 # Committed snapshot of the shipped id set ("ids are never deleted or reused" # contract - a delete-one-add-one edit keeps the count above the floor but trips @@ -205,7 +206,7 @@ (147, 147), (148, 149), (150, 150), - (151, 153), + (151, 155), ] EXPECTED_INITIAL_IDS = frozenset( f"M-{n:03d}" for lo, hi in _INITIAL_ID_RANGES for n in range(lo, hi + 1) @@ -710,10 +711,10 @@ def test_initial_ids_never_deleted(): M-136..M-138 + M-139 + M-140..M-142 + M-143 + M-144 + M-145/M-146 + M-147 + M-148/M-149 (shipped by #838 without extending the snapshot; added at the 4.0-cut framing) + M-150 + M-151..M-153 (the LWDiD results-contract rows, - 4.0-cut PR 1)).""" + 4.0-cut PR 1) + M-154/M-155 (the 7b cluster-policy pair)).""" missing = sorted(EXPECTED_INITIAL_IDS - set(_ROW_IDS)) assert not missing, f"ledger rows deleted (ids are permanent): {missing}" - assert len(EXPECTED_INITIAL_IDS) == 135 + assert len(EXPECTED_INITIAL_IDS) == 137 def test_version_tuple_pads_to_three_components(): @@ -1367,18 +1368,19 @@ def _changes_at_4_0(row): Keyed on the two LIFECYCLE version fields only - a symbol removed at 4.0, or one whose deprecation warning starts firing at 4.0 (the ``field-flip`` family, removed - at 5.0). 113 of the 135 rows qualify. + at 5.0). 113 of the 137 rows qualify. - The 22 that do not, and why (this enumeration is the contract - a reader of the + The 24 that do not, and why (this enumeration is the contract - a reader of the guide must be able to trust that nothing 4.0-relevant was dropped): - 12 ``behavior`` rows with ``introduced_in: 3.9`` and no dep/rem: already shipped in 3.9, so there is no 4.0 action. They get their own guide section, not an appendix row. - - 5 ``behavior`` rows with ``introduced_in: 4.0`` and no deprecation/removal + - 7 ``behavior`` rows with ``introduced_in: 4.0`` and no deprecation/removal fields (``M-144`` anticipation validation, ``M-145`` pscore_trim validation, ``M-146`` summary-alpha guard, ``M-147`` ETWFE unsupported-period policy, - ``M-153`` LWDiD to_dataframe reshape - the post-cut behavior changes): they land + ``M-153`` LWDiD to_dataframe reshape, ``M-154`` SpilloverDiD df_convention (7c), + ``M-155`` the shared cluster contract - the post-cut behavior changes): they land AT 4.0 but remove/deprecate nothing, so they appear in the guide's "Remaining 4.0 changes" prose, not the ledger-derived appendix. - ``M-062``, ``M-063``: aliases, introduce-only / all lifecycle fields null. diff --git a/tests/test_wild_bootstrap.py b/tests/test_wild_bootstrap.py index 1071c081b..d602cac36 100644 --- a/tests/test_wild_bootstrap.py +++ b/tests/test_wild_bootstrap.py @@ -466,9 +466,10 @@ def test_did_wild_bootstrap_requires_cluster(self, clustered_did_data, ci_params with pytest.raises( ValueError, match=re.escape( - "inference='wild_bootstrap' requires cluster=. The wild cluster " - "bootstrap resamples at the cluster level; pass cluster= or use " - "inference='analytical'." + "inference='wild_bootstrap' requires cluster=. The wild cluster " + "bootstrap resamples at the cluster level; cluster=None and " + "cluster=False both leave a DifferenceInDifferences fit unclustered, " + "so pass a cluster column or use inference='analytical'." ), ): did.fit(clustered_did_data, outcome="outcome", treatment="treated", post="post") From ec9c18e5d42b0425e6304a91d9aa3f5845c8cfaf Mon Sep 17 00:00:00 2001 From: igerber Date: Thu, 29 Oct 2026 15:15:58 -0400 Subject: [PATCH 2/2] docs(v4): qualify SpilloverDiD's residual-df description by the survey-df precedence The SpilloverDiD cluster docstring and the REGISTRY 7b Note said inference uses the residual df "on every lane"; the survey-design df has always taken precedence under survey_design= (spillover.py df_for_inference). Both now say "every non-survey lane" and name the precedence. Documentation only. --- diff_diff/spillover.py | 5 +++-- docs/methodology/REGISTRY.md | 10 ++++++---- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/diff_diff/spillover.py b/diff_diff/spillover.py index d526733dd..087d6b36f 100644 --- a/diff_diff/spillover.py +++ b/diff_diff/spillover.py @@ -1701,8 +1701,9 @@ class SpilloverDiD(BaseEstimator): on every lane: CR1, the combined Conley cluster product kernel when paired with ``vcov_type="conley"``, or the effective PSU under a ``survey_design`` without its own PSU. Column labels must be ``str``. - Inference uses the residual df on every lane (``df_convention`` - arrives with [M-154]). + Inference uses the residual df on every non-survey lane, the + default CR1 lane included; under a ``survey_design`` the design df + takes precedence (``df_convention`` arrives with [M-154]). alpha : float, default=0.05 Significance level for confidence intervals. anticipation : int, default=0 diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index a84d48767..5d74aea0e 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -7074,10 +7074,12 @@ should be a deliberate user choice. shared `str | None | False` contract, [M-155]). Results report `cluster_name="unit"` / `n_clusters=G` on the default lane. Roster in `docs/methodology/variance-conventions.md` "Default clustering roster". -- **Note:** inference df unchanged at 7b - `t(n_obs_eff - k_effective)` on every lane - INCLUDING the new default CR1 lane (SpilloverDiD has no `df_convention` knob yet, so - it is the one auto-clustered default not on t(G-1)); [M-154] / item 7c adds the knob - with the 4.0 default `"cluster"` (t(G-1) on the clustered lanes, inert on Conley). +- **Note:** inference df unchanged at 7b - `t(n_obs_eff - k_effective)` on every + non-survey lane INCLUDING the new default CR1 lane (the survey-design df keeps its + precedence under `survey_design=`, as before); SpilloverDiD has no `df_convention` + knob yet, so it is the one auto-clustered default not on t(G-1); [M-154] / item 7c + adds the knob with the 4.0 default `"cluster"` (t(G-1) on the clustered lanes, inert + on Conley, survey df still first). - **Note (rank-guarded Wave D bread):** The Wave D variance bread `A_22 = (X_2'WX_2)^{-1}` is inverted by the shared `_rank_guarded_inv` (`diff_diff/linalg.py`) on the already-column-dropped kept submatrix. `np.linalg.solve` raised only on an exactly-singular bread (prior fallback: dense `lstsq`); a **near**-singular kept Gram would otherwise flow a garbage inverse (~1e13) into the SE. The rank-guard truncates redundant directions → finite SE on the identified subspace (NaN at rank 0), re-inflated to (k, k) with NaN at the dropped positions, and warns. See the CallawaySantAnna "rank-guarded IF standard errors" Note. Sibling of axis-A finding #17. **Identification spec (committed):**