diff --git a/TODO.md b/TODO.md index 277c05fa..2a925f44 100644 --- a/TODO.md +++ b/TODO.md @@ -21,7 +21,6 @@ Related tracking surfaces: | Issue | Location | Origin | Effort | Priority | |-------|----------|--------|--------|----------| -| Non-staggered `summary(alpha=...)` mislabel audit (the M-146 staggered-family guard's remaining siblings): `results.py:199` and `continuous_did_results.py:250` are VERIFIED mislabels (requested-alpha `{conf_level}%` header over stored fit-time `conf_int` — dual-review round 4); `results.py:792/:1290`, `synthetic_control_results.py:499`, `triple_diff.py:163`, `trop_results.py:207` carry the same `alpha = alpha or self.alpha` idiom and still need verification (a genuine recomputation from the SE at the requested alpha is legitimate on analytical fits). Apply the shared `results_base._require_fit_alpha` guard (or a real recomputation) per site. Include `plot_dose_response`'s raw-DataFrame CI reconstruction (`diff_diff/visualization/_continuous.py:105-108`): zero-SE rows draw a zero-width band — a weaker instance (user-supplied `se` column, no stored inference to contradict) of the M-146-adjacent zero-SE gate | `diff_diff/results.py`, `diff_diff/continuous_did_results.py`, `diff_diff/synthetic_control_results.py`, `diff_diff/triple_diff.py`, `diff_diff/trop_results.py`, `diff_diff/visualization/_continuous.py` | #794 follow-ups review | Mid | Medium | | Consolidate the remaining estimator-entangled DR/logit score variants (`staggered.py::_doubly_robust` + RC twins, `triple_diff.py`, `lwdid.py`, `wooldridge.py`) onto the shared `_dr_scores.py` module, each migration with its own committed oracle capture (the ContinuousDiD lift's two-tier pattern in `tests/test_dr_scores.py`); and add a ridge vcov path to `solve_ridge` if an estimator ever needs analytical ridge inference | `diff_diff/_dr_scores.py` | dml-b0 | Mid | Low | | hc2/hc2_bm floor `1 - h_ii` at 1e-10 in the shared leverage meat, fabricating finite (if inflated) variances for leverage-one observations - hc3 now fails closed there (LWDiD fix wave) but the pre-existing hc2 family behavior is released surface; decide fail-closed vs keep-floor for hc2/hc2_bm | `diff_diff/linalg.py` | #588 | Quick | Low | | Numeric between-period cohorts (e.g. `first_treat=4.5` with integer times) are rejected by LWDiD while CallawaySantAnna estimates them and LWDiD's own datetime/Period cohorts map to the next observed period — close the dtype asymmetry by adopting the next-observed-period mapping for numeric cohorts too (contract documented in REGISTRY cohort-encodings Note + `docs/api/lwdid.rst` Input Contract). Lands only after PR #588 merges | `diff_diff/lwdid.py` | #588 | Quick | Low | @@ -58,6 +57,7 @@ Related tracking surfaces: | `WooldridgeDiD` does not apply the W2025 Sec 5.4 `D_{G_max} x X` covariate normalization, and three sibling covariate rank deficiencies are pre-existing. Measured with the period range pinned and only the never-treated units toggled: (1) time-invariant `exovar` is absorbed by the unit FE, 4 of 26 columns, IDENTICALLY with and without never-treated units; (2) `xgvar`'s cell x covariate block, 19 of 41, identical on both panels; (3) `xtvar` under `demean_covariates=False` does exhibit the `sum_g D_g x = x` dependency that the default demeaning removes; (4) the newly-reachable case -- time-VARYING data passed through `exovar`, which its own docstring reserves for time-invariant covariates -- where the paper's `dT_i` rule would give a deterministic `D_{G_max} x X` drop instead of QR's arbitrary pick (coefficients unaffected, `1.35e-14`; `rank_deficient_action="error"` raises). REGISTRY's narrowed Sec 5.4 note cross-references this row. **Trap for whoever takes it:** `xtvar` under the DEFAULT `demean_covariates=True` is FULL RANK -- the raw block carries demeaned values while `D_g x X` carries raw ones -- and forcing the drop there moves `overall_att` 1.11903 -> 1.46269. Pinned as-is by `TestComparisonSupportFiltering::test_cells_derived_groups_did_not_leak_into_the_design`. | `diff_diff/wooldridge.py` | #729-followup | Heavy | Medium | | `WooldridgeDiD.n_control_units` counts never-treated UNITS on `control_group="never_treated"` regardless of method, but on the nonlinear paths (`logit`/`poisson`) treated units' pre-treatment rows ARE the identifying comparison -- only the OLS path absorbs them into their own cells. So the reported count under-states the comparison pool exactly where the REGISTRY control-pool asymmetry note applies. Widen to `not_yet_treated or (never_treated and method != "ols")`, or document the count as never-treated-units-by-definition. Behavior is PRE-EXISTING; documented for now in the REGISTRY control-pool Note rather than changed, because widening moves a public results field and wants its own ledger row and test matrix. | `diff_diff/wooldridge.py` | #729-followup | Mid | Low | | `WooldridgeDiD` has no opt-out for comparison-support period filtering: a user who would rather see the refusal than a reduced sample cannot ask for it. Adding one means a constructor parameter (`get_params`/`set_params` propagation, transactional validation), a ledger row, and a test matrix across both predicate branches and all three `rank_deficient_action` modes -- deliberately out of scope for the change that introduced the filter. The always-on warning is the interim answer. | `diff_diff/wooldridge.py` | #729-followup | Mid | Low | +| Fractional confidence-level display unification: every `summary()` header computes `int((1 - alpha) * 100)` (~19 truncating sites incl. the shipped M-146 staggered family), so a fit `alpha=0.025` prints "97% Confidence Interval" over a 97.5% interval; `plot_dose_response`'s band label now renders exact fractional coverage via its `_coverage_label` helper - unify the summary surfaces on the same formatter (cross-family display change: moves every summary golden/doctest that pins a header, so it wants one sweep with its own test recapture, not a per-estimator drip) | `diff_diff/results_base.py`, `diff_diff/results.py` | alpha-guard review | Quick | Low | ### Performance diff --git a/changelog.d/20260830-alpha-mislabel-guard.md b/changelog.d/20260830-alpha-mislabel-guard.md new file mode 100644 index 00000000..88e4f751 --- /dev/null +++ b/changelog.d/20260830-alpha-mislabel-guard.md @@ -0,0 +1,20 @@ +### Behavioral Changes +- **`summary(alpha=...)` never recomputes or relabels stored inference, + family-wide** ([M-146] completion): a non-fit `alpha` now raises `ValueError` + at seven more sites - `DiDResults` (and `SpilloverDiDResults` by + inheritance), `MultiPeriodDiDResults`, `SyntheticDiDResults`, + `TripleDifferenceResults`, `TROPResults`, `ContinuousDiDResults` (all + previously printed a requested-alpha header over fit-time stored intervals), + and `SyntheticControlResults` (previously a silent no-op `alpha`); `alpha=0.0`, + previously swallowed by a falsy-`or` default, raises too. Re-fit at the + desired alpha instead. +- **`plot_dose_response` honest bands and labels**: DataFrame-`se` input masks + non-positive/non-finite `se` rows from the confidence band with a warning + (previously a zero-SE row drew a finite zero-width band) and validates + `alpha` strictly inside (0, 1); the band legend is alpha-derived on the `se` + branch, `results.alpha`-derived on `results=` input, and the level-free + "CI" for bare-curve/explicit-CI input (previously hard-coded "95% CI" + regardless of the requested alpha); an explicitly passed `alpha` on + non-`se` input warns instead of being silently ignored; the plotly band + polygon filters non-finite-CI rows (a NaN vertex previously mangled the + `toself` band) and both renderers suppress an all-masked band. diff --git a/diff_diff/continuous_did_results.py b/diff_diff/continuous_did_results.py index a88c4176..96cad667 100644 --- a/diff_diff/continuous_did_results.py +++ b/diff_diff/continuous_did_results.py @@ -15,7 +15,12 @@ from diff_diff.aggregation import AggregationMixin, AggregationResult from diff_diff.continuous_did_aggregation import _ContinuousDiDAggregationMixin from diff_diff.results import _format_survey_block, _get_significance_stars -from diff_diff.results_base import BaseResults, build_event_study_surface +from diff_diff.results_base import ( + _SUMMARY_ALPHA_MESSAGE, + BaseResults, + _require_fit_alpha, + build_event_study_surface, +) from diff_diff.utils import safe_inference __all__ = ["ContinuousDiDResults", "DoseResponseCurve"] @@ -258,8 +263,17 @@ def coef_var(self) -> float: return self.overall_att_se / abs(self.overall_att) def summary(self, alpha: Optional[float] = None) -> str: - """Generate formatted summary.""" - alpha = alpha or self.alpha + """Generate formatted summary. + + Parameters + ---------- + alpha : float, optional + Accepted for signature uniformity. The stored intervals were + computed at fit time; a value different from the stored + ``alpha`` raises ValueError rather than silently recomputing + or relabeling. Re-fit at the desired alpha instead. + """ + alpha = _require_fit_alpha(alpha, self.alpha, message=_SUMMARY_ALPHA_MESSAGE) conf_level = int((1 - alpha) * 100) w = 85 diff --git a/diff_diff/results.py b/diff_diff/results.py index 202d2c3b..87dd318e 100644 --- a/diff_diff/results.py +++ b/diff_diff/results.py @@ -11,7 +11,7 @@ import numpy as np import pandas as pd -from diff_diff.results_base import BaseResults +from diff_diff.results_base import _SUMMARY_ALPHA_MESSAGE, BaseResults, _require_fit_alpha def _format_survey_block(sm, width: int) -> list: @@ -188,15 +188,17 @@ def summary(self, alpha: Optional[float] = None) -> str: Parameters ---------- alpha : float, optional - Significance level for confidence intervals. Defaults to the - alpha used during estimation. + Accepted for signature uniformity. The stored intervals were + computed at fit time; a value different from the stored + ``alpha`` raises ValueError rather than silently recomputing + or relabeling. Re-fit at the desired alpha instead. Returns ------- str Formatted summary table. """ - alpha = alpha or self.alpha + alpha = _require_fit_alpha(alpha, self.alpha, message=_SUMMARY_ALPHA_MESSAGE) conf_level = int((1 - alpha) * 100) lines = [ @@ -473,7 +475,12 @@ class SpilloverDiDResults(DiDResults): def summary(self, alpha: Optional[float] = None) -> str: """Extended summary with ATT row, per-event-time direct block, and - per-(ring, event-time) spillover block.""" + per-(ring, event-time) spillover block. + + ``alpha`` must equal the fit alpha (None = fit alpha): the inherited + guard raises on any other value rather than relabeling stored + intervals. + """ base = super().summary(alpha=alpha) insert_blocks: List[str] = [] @@ -781,15 +788,17 @@ def summary(self, alpha: Optional[float] = None) -> str: Parameters ---------- alpha : float, optional - Significance level for confidence intervals. Defaults to the - alpha used during estimation. + Accepted for signature uniformity. The stored intervals were + computed at fit time; a value different from the stored + ``alpha`` raises ValueError rather than silently recomputing + or relabeling. Re-fit at the desired alpha instead. Returns ------- str Formatted summary table. """ - alpha = alpha or self.alpha + alpha = _require_fit_alpha(alpha, self.alpha, message=_SUMMARY_ALPHA_MESSAGE) conf_level = int((1 - alpha) * 100) lines = [ @@ -1279,15 +1288,17 @@ def summary(self, alpha: Optional[float] = None) -> str: Parameters ---------- alpha : float, optional - Significance level for confidence intervals. Defaults to the - alpha used during estimation. + Accepted for signature uniformity. The stored intervals were + computed at fit time; a value different from the stored + ``alpha`` raises ValueError rather than silently recomputing + or relabeling. Re-fit at the desired alpha instead. Returns ------- str Formatted summary table. """ - alpha = alpha or self.alpha + alpha = _require_fit_alpha(alpha, self.alpha, message=_SUMMARY_ALPHA_MESSAGE) conf_level = int((1 - alpha) * 100) lines = [ diff --git a/diff_diff/results_base.py b/diff_diff/results_base.py index 7b83f51d..5719273a 100644 --- a/diff_diff/results_base.py +++ b/diff_diff/results_base.py @@ -92,16 +92,35 @@ class BaseResults: __slots__ = () -def _require_fit_alpha(alpha: Optional[float], fit_alpha: float) -> float: +# Family-wide non-staggered override for _require_fit_alpha: the staggered +# default's rationale tail (bootstrap percentile intervals) is false for the +# analytic-default siblings, so they share this accurate generic wording. +# May use exactly the {alpha}/{fit_alpha} placeholders (helper-formatted). +_SUMMARY_ALPHA_MESSAGE = ( + "This result stores intervals computed at fit time (alpha={fit_alpha}); " + "summary() never recomputes or relabels stored inference " + "(requested alpha={alpha}); re-fit with the desired alpha." +) + + +def _require_fit_alpha( + alpha: Optional[float], fit_alpha: float, message: Optional[str] = None +) -> float: """Reject a non-fit ``alpha``; summaries never recompute stored inference. - Shared by the staggered-family ``summary()`` methods (CallawaySantAnna - and siblings): stored intervals were computed at fit time, and bootstrap - percentile intervals cannot be reconstructed from the reported SE, so a + The family-wide ``summary()`` contract (staggered and non-staggered + results classes alike): stored intervals were computed at fit time, so a requested alpha other than the fit alpha raises instead of silently - relabeling the confidence-interval header. + relabeling the confidence-interval header. ``message`` replaces the + ENTIRE default error text for classes where the default rationale would + be inaccurate; overrides may use exactly the named placeholders + ``{alpha}`` and ``{fit_alpha}`` (this helper formats them — escape any + other literal brace as ``{{``/``}}``) and must keep the phrase + "never recomputes" (the family test pin). """ if alpha is not None and alpha != fit_alpha: + if message is not None: + raise ValueError(message.format(alpha=alpha, fit_alpha=fit_alpha)) raise ValueError( f"This result stores intervals computed at alpha={fit_alpha}; " f"summary() never recomputes or relabels stored inference " diff --git a/diff_diff/synthetic_control.py b/diff_diff/synthetic_control.py index 8f856671..526a3842 100644 --- a/diff_diff/synthetic_control.py +++ b/diff_diff/synthetic_control.py @@ -126,7 +126,10 @@ class SyntheticControl(BaseEstimator): standard deviation across donors+treated (ddof=1), matching R ``Synth``. ``"none"`` is a deviation from R (see REGISTRY). alpha : float, default 0.05 - Significance level recorded for downstream (placebo) inference. + Significance level recorded on the results for signature + uniformity. Placebo and test-inversion entry points take their own + independent ``gamma``/``alpha`` arguments; the results + ``summary(alpha=)`` accepts only this stored value. seed : int, optional Seed for the multistart random (Dirichlet) starting points. v_cv_t0 : int, optional diff --git a/diff_diff/synthetic_control_results.py b/diff_diff/synthetic_control_results.py index d5e7ce34..bca37931 100644 --- a/diff_diff/synthetic_control_results.py +++ b/diff_diff/synthetic_control_results.py @@ -22,7 +22,7 @@ import pandas as pd from diff_diff.results import _format_survey_block, _get_significance_stars -from diff_diff.results_base import BaseResults +from diff_diff.results_base import BaseResults, _require_fit_alpha __all__ = ["SyntheticControlResults"] @@ -203,7 +203,11 @@ class SyntheticControlResults(BaseResults): standardize : str ``"std"`` (per-row SD scaling) or ``"none"``. alpha : float - Significance level recorded for downstream (placebo) inference. + Significance level recorded at fit time for signature uniformity. + Placebo and test-inversion entry points take their own independent + ``gamma``/``alpha`` arguments; this field feeds only + ``is_significant`` (always False here, since the Wald fields are + NaN by design). rmspe_ratio : float The treated unit's post/pre RMSPE ratio = ``sqrt(MSPE_post / MSPE_pre)`` — the in-space placebo test statistic (ADH 2010 §2.4), computed at fit time. @@ -489,14 +493,31 @@ def summary(self, alpha: Optional[float] = None) -> str: Parameters ---------- alpha : float, optional - Significance level; defaults to the alpha used during estimation. + Accepted for signature uniformity; must equal the stored fit + ``alpha`` (None = fit alpha) or ValueError is raised. This + summary reports no alpha-based interval (the Wald inference + fields are NaN by design); the displayed confidence set is + keyed on its own stored ``gamma``. Returns ------- str Formatted summary table. """ - alpha = alpha or self.alpha + # Tailored message: this class stores NO alpha-based interval (Wald + # fields NaN by design; the confidence set is gamma-keyed), so the + # shared messages' "stores intervals computed at alpha=..." would be + # factually wrong here. + alpha = _require_fit_alpha( + alpha, + self.alpha, + message=( + "This summary reports no alpha-based interval (Wald inference " + "fields are NaN by design; the displayed confidence set uses " + "its own stored gamma) and never recomputes stored inference " + "(requested alpha={alpha}, fit alpha={fit_alpha})." + ), + ) n_top = min(5, len(self.donor_weights)) top_donors = sorted(self.donor_weights.items(), key=lambda kv: kv[1], reverse=True)[:n_top] diff --git a/diff_diff/triple_diff.py b/diff_diff/triple_diff.py index 34ee6f8b..80d05279 100644 --- a/diff_diff/triple_diff.py +++ b/diff_diff/triple_diff.py @@ -44,7 +44,7 @@ from diff_diff._staggered_triple_diff_engine import _StaggeredTripleDiffEngineMixin from diff_diff.linalg import _rank_guarded_inv, solve_logit, solve_ols from diff_diff.results import _format_survey_block, _get_significance_stars -from diff_diff.results_base import BaseResults +from diff_diff.results_base import _SUMMARY_ALPHA_MESSAGE, BaseResults, _require_fit_alpha from diff_diff.staggered_aggregation import CallawaySantAnnaAggregationMixin from diff_diff.staggered_bootstrap import CallawaySantAnnaBootstrapMixin from diff_diff.staggered_triple_diff_results import StaggeredTripleDiffResults @@ -153,15 +153,17 @@ def summary(self, alpha: Optional[float] = None) -> str: Parameters ---------- alpha : float, optional - Significance level for confidence intervals. Defaults to the - alpha used during estimation. + Accepted for signature uniformity. The stored intervals were + computed at fit time; a value different from the stored + ``alpha`` raises ValueError rather than silently recomputing + or relabeling. Re-fit at the desired alpha instead. Returns ------- str Formatted summary table. """ - alpha = alpha or self.alpha + alpha = _require_fit_alpha(alpha, self.alpha, message=_SUMMARY_ALPHA_MESSAGE) conf_level = int((1 - alpha) * 100) lines = [ diff --git a/diff_diff/trop_results.py b/diff_diff/trop_results.py index 8cc9a478..7822bb81 100644 --- a/diff_diff/trop_results.py +++ b/diff_diff/trop_results.py @@ -17,7 +17,7 @@ from typing_extensions import TypedDict from diff_diff.results import _format_survey_block, _get_significance_stars -from diff_diff.results_base import BaseResults +from diff_diff.results_base import BaseResults, _require_fit_alpha __all__ = [ "_LAMBDA_INF", @@ -196,15 +196,29 @@ def summary(self, alpha: Optional[float] = None) -> str: Parameters ---------- alpha : float, optional - Significance level for confidence intervals. Defaults to the - alpha used during estimation. + Accepted for signature uniformity. The stored interval was + computed at fit time; a value different from the stored + ``alpha`` raises ValueError rather than silently recomputing + or relabeling. Re-fit at the desired alpha instead. Returns ------- str Formatted summary table. """ - alpha = alpha or self.alpha + # Tailored message: TROP's interval is a t interval at the fit alpha + # (arithmetically reconstructible), so the guard holds purely by the + # family-wide never-recompute contract - never claim otherwise. + alpha = _require_fit_alpha( + alpha, + self.alpha, + message=( + "This result stores a t interval computed at the fit alpha " + "(alpha={fit_alpha}); summary() never recomputes or relabels " + "stored inference by the family-wide contract " + "(requested alpha={alpha}); re-fit with the desired alpha." + ), + ) conf_level = int((1 - alpha) * 100) lines = [ diff --git a/diff_diff/visualization/_continuous.py b/diff_diff/visualization/_continuous.py index 70344385..00021072 100644 --- a/diff_diff/visualization/_continuous.py +++ b/diff_diff/visualization/_continuous.py @@ -1,20 +1,33 @@ """Continuous DiD visualization functions (dose-response curves).""" +import numbers +import warnings from typing import TYPE_CHECKING, Any, Optional, Tuple +import numpy as np import pandas as pd if TYPE_CHECKING: from diff_diff.continuous_did_results import ContinuousDiDResults, DoseResponseCurve +def _coverage_label(alpha: float) -> str: + """Exact coverage label: '97.5% CI' for alpha=0.025, '95% CI' for 0.05. + + Rounds to 6 decimals first so a float32-noised alpha (0.0500000007...) + still reads 95 rather than 94.9999999255, then trims trailing zeros. + """ + level = round(100.0 * (1.0 - float(alpha)), 6) + return f"{level:g}% CI" + + def plot_dose_response( results: Optional["ContinuousDiDResults"] = None, *, curve: Optional["DoseResponseCurve"] = None, data: Optional[pd.DataFrame] = None, target: str = "att", - alpha: float = 0.05, + alpha: Optional[float] = None, figsize: Tuple[float, float] = (10, 6), title: Optional[str] = None, xlabel: str = "Dose", @@ -41,11 +54,20 @@ def plot_dose_response( A DoseResponseCurve object directly. data : pd.DataFrame, optional DataFrame with columns ``dose``, ``effect``, ``se`` (and optionally - ``conf_int_lower``, ``conf_int_upper``). + ``conf_int_lower``, ``conf_int_upper``). Rows whose ``se`` is + non-positive or non-finite carry no defined interval: their band is + masked (with a warning) rather than drawn zero-width. target : str, default="att" Which dose-response curve: ``"att"`` or ``"acrt"``. - alpha : float, default=0.05 - Significance level for confidence intervals (used with DataFrame input). + alpha : float, optional + Significance level for the reconstructed band on DataFrame-``se`` + input ONLY (default 0.05 there; must be strictly inside (0, 1)). + Stored (``results=``/``curve=``) and explicit-CI intervals keep the + level they were built at — an explicitly passed ``alpha`` warns and + is ignored on those inputs. The band legend states the level where + it is knowable (the requested alpha on the ``se`` branch, + ``results.alpha`` on ``results=`` input) and the level-free "CI" + otherwise. figsize : tuple, default=(10, 6) Figure size (width, height) in inches. title : str, optional @@ -86,6 +108,41 @@ def plot_dose_response( else: raise ValueError(f"target must be 'att' or 'acrt', got '{target}'") + # ``alpha`` constructs an interval ONLY on the DataFrame-``se`` branch; + # everywhere else the band's level is fixed by how it was built, so an + # explicitly passed alpha is a no-op — warn instead of silently ignoring. + se_branch = ( + data is not None + and "se" in data.columns + and not ("conf_int_lower" in data.columns and "conf_int_upper" in data.columns) + ) + if alpha is not None and not se_branch: + warnings.warn( + "alpha= only applies to DataFrame input with an 'se' column; the " + "displayed band keeps its stored/unknown confidence level.", + UserWarning, + stacklevel=2, + ) + + # Band legend: state the level only where it is knowable. + resolved_alpha = 0.05 if alpha is None else alpha + if se_branch: + if not 0.0 < resolved_alpha < 1.0: + raise ValueError(f"alpha must be strictly between 0 and 1, got {resolved_alpha}") + ci_label = _coverage_label(resolved_alpha) + elif results is not None: + fit_alpha = getattr(results, "alpha", None) + # numbers.Real (not (int, float)): np.float32 etc. are real numeric + # fit alphas too; the (0, 1) range check excludes bools. + if isinstance(fit_alpha, numbers.Real) and 0.0 < float(fit_alpha) < 1.0: + ci_label = _coverage_label(fit_alpha) + else: + ci_label = "CI" + else: + # Bare curve= (DoseResponseCurve carries no alpha) or explicit-CI / + # CI-less DataFrame input: no knowable level. + ci_label = "CI" + if curve is not None: # Infer target from curve when passed directly (not via results) if results is None and hasattr(curve, "target") and curve.target: @@ -103,9 +160,22 @@ def plot_dose_response( ci_lower = data["conf_int_lower"].values ci_upper = data["conf_int_upper"].values elif "se" in data.columns: - z = scipy_stats.norm.ppf(1 - alpha / 2) - ci_lower = effects - z * data["se"].values - ci_upper = effects + z * data["se"].values + se = np.asarray(data["se"].values, dtype=float) + invalid = ~(np.isfinite(se) & (se > 0)) + z = scipy_stats.norm.ppf(1 - resolved_alpha / 2) + ci_lower = np.asarray(effects - z * se, dtype=float) + ci_upper = np.asarray(effects + z * se, dtype=float) + if invalid.any(): + # A zero/negative/non-finite SE carries no defined interval: + # masking beats drawing a zero-width band asserting certainty. + warnings.warn( + f"{int(invalid.sum())} row(s) with non-positive or " + "non-finite 'se' have no confidence band.", + UserWarning, + stacklevel=2, + ) + ci_lower[invalid] = np.nan + ci_upper[invalid] = np.nan else: ci_lower = None ci_upper = None @@ -125,6 +195,7 @@ def plot_dose_response( effects=effects, ci_lower=ci_lower, ci_upper=ci_upper, + ci_label=ci_label, title=title, xlabel=xlabel, ylabel=ylabel, @@ -139,6 +210,7 @@ def plot_dose_response( effects=effects, ci_lower=ci_lower, ci_upper=ci_upper, + ci_label=ci_label, figsize=figsize, title=title, xlabel=xlabel, @@ -157,6 +229,7 @@ def _render_dose_response_mpl( effects, ci_lower, ci_upper, + ci_label, figsize, title, xlabel, @@ -181,17 +254,22 @@ def _render_dose_response_mpl( if show_zero_line: ax.axhline(y=0, color="gray", linestyle="--", linewidth=1, alpha=0.5) - # Confidence band + # Confidence band. Skip entirely when no CI row is finite - an + # all-masked band would still register a stray legend entry. if ci_lower is not None and ci_upper is not None: - band_color = ci_color or color - ax.fill_between( - dose_grid, - ci_lower, - ci_upper, - alpha=0.15, - color=band_color, - label="95% CI", + finite_band = np.isfinite(np.asarray(ci_lower, dtype=float)) & np.isfinite( + np.asarray(ci_upper, dtype=float) ) + if finite_band.any(): + band_color = ci_color or color + ax.fill_between( + dose_grid, + ci_lower, + ci_upper, + alpha=0.15, + color=band_color, + label=ci_label, + ) # Effect line ax.plot(dose_grid, effects, color=color, linewidth=2, label="Effect") @@ -216,6 +294,7 @@ def _render_dose_response_plotly( effects, ci_lower, ci_upper, + ci_label, title, xlabel, ylabel, @@ -239,21 +318,31 @@ def _render_dose_response_plotly( if show_zero_line: fig.add_hline(y=0, line_dash="dash", line_color="gray", line_width=1, opacity=0.5) - # Confidence band + # Confidence band. A NaN vertex splits a fill="toself" polygon into + # independently-closed sub-polygons, so filter non-finite CI rows out + # before building the trace (the _event_study.py precedent - CI VALUES + # only, never the dose, which may be non-numeric) and skip the trace + # entirely when nothing survives (else an empty stray legend entry). if ci_lower is not None and ci_upper is not None: - band_color = ci_color or color - dose_list = list(dose_grid) - fig.add_trace( - go.Scatter( - x=dose_list + dose_list[::-1], - y=list(ci_upper) + list(ci_lower)[::-1], - fill="toself", - fillcolor=_color_to_rgba(band_color, 0.15), - line=dict(color="rgba(0,0,0,0)"), - name="95% CI", - hoverinfo="skip", + lo = np.asarray(ci_lower, dtype=float) + hi = np.asarray(ci_upper, dtype=float) + keep = np.isfinite(lo) & np.isfinite(hi) + if keep.any(): + band_color = ci_color or color + dose_list = [d for d, k in zip(dose_grid, keep) if k] + hi_list = list(hi[keep]) + lo_list = list(lo[keep]) + fig.add_trace( + go.Scatter( + x=dose_list + dose_list[::-1], + y=hi_list + lo_list[::-1], + fill="toself", + fillcolor=_color_to_rgba(band_color, 0.15), + line=dict(color="rgba(0,0,0,0)"), + name=ci_label, + hoverinfo="skip", + ) ) - ) # Effect line fig.add_trace( diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 947ead41..7e700e3c 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -1230,7 +1230,9 @@ The multiplier bootstrap uses random weights w_i with E[w]=0 and Var(w)=1: - [x] Repeated cross-sections (`panel=False`) for non-panel surveys (Phase 7b) - **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. -- **Note (`summary(alpha=)` never recomputes stored inference, ledger row M-146):** `CallawaySantAnnaResults.summary`/`print_summary` — and the sibling staggered-family results classes (StaggeredTripleDiffResults, ChaisemartinDHaultfoeuilleResults, ImputationDiDResults, EfficientDiDResults, TwoStageDiDResults, StackedDiDResults, SunAbrahamResults) — reject an `alpha` different from the fit-time value via the shared `results_base._require_fit_alpha` guard (the DMLDiDResults/EventStudyResults precedent). Previously the requested alpha relabeled the confidence-interval header while the FIT-TIME stored intervals were printed — silent coverage mislabeling (bootstrap percentile intervals cannot be reconstructed from the SE at all); `alpha=0.0` was additionally swallowed by a falsy-`or` default and now raises. Re-fit at the desired alpha instead. The non-staggered summaries with the same idiom are tracked as a TODO.md audit row. +- **Note (`summary(alpha=)` never recomputes stored inference, ledger row M-146):** `CallawaySantAnnaResults.summary`/`print_summary` — and the sibling staggered-family results classes (StaggeredTripleDiffResults, ChaisemartinDHaultfoeuilleResults, ImputationDiDResults, EfficientDiDResults, TwoStageDiDResults, StackedDiDResults, SunAbrahamResults) — reject an `alpha` different from the fit-time value via the shared `results_base._require_fit_alpha` guard (the DMLDiDResults/EventStudyResults precedent). Previously the requested alpha relabeled the confidence-interval header while the FIT-TIME stored intervals were printed — silent coverage mislabeling (bootstrap percentile intervals cannot be reconstructed from the SE at all); `alpha=0.0` was additionally swallowed by a falsy-`or` default and now raises. Re-fit at the desired alpha instead. The guard is now FAMILY-WIDE: the non-staggered summaries adopted it too (DiDResults incl. SpilloverDiDResults by inheritance, MultiPeriodDiDResults, SyntheticDiDResults, TripleDifferenceResults, TROPResults, ContinuousDiDResults, SyntheticControlResults), closing the audit row this sentence used to track. +- **Note (SyntheticControl summary-alpha posture):** `SyntheticControlResults.summary(alpha=)` applies the same guard although the class prints no alpha-based Wald interval (`se`/`t_stat`/`p_value`/`conf_int` NaN by design; the displayed Firpo-Possebom confidence set is keyed on its own stored `gamma`): the previously dead `alpha` parameter now raises on a non-fit value, with a class-accurate message, instead of silently doing nothing. +- **Note (TROP summary-alpha posture):** `TROPResults.summary(alpha=)` applies the guard purely by the family-wide never-recompute contract - its stored interval is a t interval at the fit alpha around a bootstrap SE and WOULD be arithmetically reconstructible; the tailored error message states the uniform-contract rationale rather than a non-reconstructibility claim. --- ## ChaisemartinDHaultfoeuille @@ -6069,7 +6071,8 @@ should be a deliberate user choice. reference carries `se = NaN` and keeps its hollow-marker-no-bar rendering, unchanged. Plotly caveat: the plotly renderer filters NaN-CI rows out of the CI band and interpolates the polygon across the gap (pre-existing NaN-SE convention); the matplotlib - backend is where the gated interval visibly disappears. `plot_honest_event_study`'s raw + backend is where the gated interval visibly disappears. +- **Note (`plot_dose_response` zero-SE gate and band labeling):** the raw-DataFrame `se` branch adopts the same gate: rows with non-positive or non-finite `se` have their reconstructed band masked to NaN with a `UserWarning` (previously a zero-SE row drew a finite zero-width band); the plotly renderer filters non-finite-CI rows out of the `toself` band polygon and bridges the gap (the `plot_group_effects` convention above; CI values only - the dose axis may be non-numeric), and both renderers suppress an all-masked band entirely (no stray legend entry). Band LABELING states the confidence level only where it is knowable: the requested `alpha` on the DataFrame-`se` branch (validated strictly inside (0, 1)), `results.alpha` on `results=` input, and the level-free "CI" for bare-curve and explicit-CI input. Alpha semantics deliberately DIVERGE from `plot_group_effects`: that surface recomputes `effect +/- z*SE` at a requested alpha, while `plot_dose_response` warns-and-ignores an explicit `alpha` on non-`se` input, because stored/explicit-CI dose-response inputs cannot be recomputed. `plot_honest_event_study`'s raw (non-container) routes mirror the container's retained-row semantics: zero/non-finite-SE rows are excluded up front (explicitly requesting one raises), the reference row - now auto-inferred on raw routes from a `reference_period` attribute or HonestDiD's own diff --git a/docs/migration-4.0.md b/docs/migration-4.0.md index 58a0f1f7..dc34ddbf 100644 --- a/docs/migration-4.0.md +++ b/docs/migration-4.0.md @@ -243,11 +243,16 @@ no removal/deprecation fields and appear here only. coerced to built-in `float`; `LWDiD`'s message wording changed. The deprecated `StaggeredTripleDifference` keeps its permissive construction shape. -- `summary(alpha=...)` / `print_summary(alpha=...)` on the staggered-family results classes - ([M-146], landing at 4.0): a value different from the fit-time `alpha` now raises `ValueError` - instead of silently relabeling the confidence-interval header over fit-time stored intervals - (bootstrap percentile intervals cannot be reconstructed from the SE); `alpha=0.0`, previously +- `summary(alpha=...)` / `print_summary(alpha=...)` on the results classes family-wide + ([M-146], landing at 4.0; the staggered family first, then the non-staggered summaries - + DiDResults incl. SpilloverDiDResults, MultiPeriodDiDResults, SyntheticDiDResults, + TripleDifferenceResults, TROPResults, ContinuousDiDResults): a value different from the + fit-time `alpha` now raises `ValueError` instead of silently relabeling the + confidence-interval header over fit-time stored intervals; `alpha=0.0`, previously swallowed by a falsy-`or` default, raises too. Re-fit at the desired alpha instead. + `SyntheticControlResults.summary` joins the contract as a carve-out: its `alpha` was a + dead no-op (it prints no alpha-based interval; the displayed confidence set is keyed on + its own stored `gamma`), and a non-fit value now raises like the rest of the family. One pending decision: the `DIFF_DIFF_SOLVE_OLS_FASTPATH` environment default has a go/no-go due at 4.0 that has not been made. If it lands on, it is a numerics change and will be documented diff --git a/docs/v4-deprecations.yaml b/docs/v4-deprecations.yaml index 1a33518b..45b7fb7d 100644 --- a/docs/v4-deprecations.yaml +++ b/docs/v4-deprecations.yaml @@ -1770,5 +1770,5 @@ rows: status: done phase: 5 test_ref: tests/test_staggered.py - code_refs: [diff_diff/results_base.py, diff_diff/staggered_results.py, diff_diff/staggered_triple_diff_results.py, diff_diff/chaisemartin_dhaultfoeuille_results.py, diff_diff/imputation_results.py, diff_diff/efficient_did_results.py, diff_diff/two_stage_results.py, diff_diff/stacked_did_results.py, diff_diff/sun_abraham.py, docs/methodology/REGISTRY.md] - notes: "Staggered-family summary(alpha=)/print_summary(alpha=) tightening via the shared results_base._require_fit_alpha guard (the DMLDiDResults/EventStudyResults precedent): eight results classes (CallawaySantAnnaResults, StaggeredTripleDiffResults, ChaisemartinDHaultfoeuilleResults, ImputationDiDResults, EfficientDiDResults, TwoStageDiDResults, StackedDiDResults, SunAbrahamResults) previously did alpha = alpha or self.alpha and relabeled the confidence-interval header at the REQUESTED alpha while always printing the FIT-TIME stored intervals - silent coverage mislabeling on any non-fit alpha (bootstrap percentile intervals cannot be reconstructed from the SE at all). Behavior delta: accepted-and-mislabeled -> loud ValueError; alpha=0.0, previously swallowed by the falsy `or` idiom, now raises too. StaggeredTripleDiffResults is INCLUDED although its parent estimator is removed at 4.0 ([M-013]/[M-014]) - the mislabel is a live 3.x rendering bug, distinct from the estimator's frozen construction shape. The non-staggered summaries with the same idiom are tracked as a TODO.md audit row, not silently left behind. introduced_in 4.0 per the [M-144] rationale; status done is terminal." + code_refs: [diff_diff/results_base.py, diff_diff/staggered_results.py, diff_diff/staggered_triple_diff_results.py, diff_diff/chaisemartin_dhaultfoeuille_results.py, diff_diff/imputation_results.py, diff_diff/efficient_did_results.py, diff_diff/two_stage_results.py, diff_diff/stacked_did_results.py, diff_diff/sun_abraham.py, diff_diff/results.py, diff_diff/triple_diff.py, diff_diff/trop_results.py, diff_diff/continuous_did_results.py, diff_diff/synthetic_control_results.py, docs/methodology/REGISTRY.md] + notes: "Staggered-family summary(alpha=)/print_summary(alpha=) tightening via the shared results_base._require_fit_alpha guard (the DMLDiDResults/EventStudyResults precedent): eight results classes (CallawaySantAnnaResults, StaggeredTripleDiffResults, ChaisemartinDHaultfoeuilleResults, ImputationDiDResults, EfficientDiDResults, TwoStageDiDResults, StackedDiDResults, SunAbrahamResults) previously did alpha = alpha or self.alpha and relabeled the confidence-interval header at the REQUESTED alpha while always printing the FIT-TIME stored intervals - silent coverage mislabeling on any non-fit alpha (bootstrap percentile intervals cannot be reconstructed from the SE at all). Behavior delta: accepted-and-mislabeled -> loud ValueError; alpha=0.0, previously swallowed by the falsy `or` idiom, now raises too. StaggeredTripleDiffResults is INCLUDED although its parent estimator is removed at 4.0 ([M-013]/[M-014]) - the mislabel is a live 3.x rendering bug, distinct from the estimator's frozen construction shape. Resolved - the guard is applied family-wide: the non-staggered summaries (DiDResults incl. SpilloverDiDResults by inheritance, MultiPeriodDiDResults, SyntheticDiDResults, TripleDifferenceResults, TROPResults, ContinuousDiDResults, SyntheticControlResults - the last previously a dead no-op alpha, now an honest raise) adopted it via the same helper (an optional message= override carries class-accurate wording where the staggered default's bootstrap-percentile rationale would be false; the staggered default message is byte-unchanged). Sibling behavioral evidence lives in tests/test_estimators.py, tests/test_triple_diff.py, tests/test_trop.py, tests/test_continuous_did.py, tests/test_methodology_synthetic_control.py, and tests/test_spillover.py (test_ref stays the staggered anchor: the schema's test_ref is a single path and this row remains the one logical contract). introduced_in 4.0 per the [M-144] rationale; status done is terminal." diff --git a/docs/v4-design.md b/docs/v4-design.md index ca5b2b04..6dbe5c78 100644 --- a/docs/v4-design.md +++ b/docs/v4-design.md @@ -851,7 +851,7 @@ above; anything only one PR cares about stays in that PR's plan.** | 2: contract foundations | 3.9 | (a) results base + unified event-study representation [M-092] + to_dict completion + the Diagnostic marker base on the diagnostic result roster [M-091] (section 3.5); (b) `aggregate()` + fit(aggregate=) shims [M-020..M-027] [M-139] (M-020's shim already shipped; M-139 is the HAD workflow twin, a pre-cut amendment); (c) param renames [M-030..M-047] [M-084] [M-086..M-089] + their results-field mirrors [M-094] [M-095] (section 8 rule 9) + the public-function completeness sweep [M-097..M-113] (section 8 rule 10) + the dCDH results mirror [M-114] + the fourth `robust` site [M-115] + the 2(c)-ii missed-rename amendments [M-136..M-138] (LPDiD `level` value; the two post-dummy diagnostics params) + BaseEstimator mixin + ContinuousDiD covariates move; (d) alias introduction [M-062] (the Spillover introduction is cancelled [M-063]) + the alias-diet `__getattr__` warning shim [M-135] + wrapper deprecations [M-070..M-077] + the two inference-surface policies: `n_bootstrap` semantic unification [M-081] and the wild-cluster-bootstrap roster guard [M-096]; shipped insertions (all done): the aggregate contract [M-122], the ETWFE reference-period family [M-123] [M-124] [M-125], and the variance-consolidation program [M-126] [M-127] | | 3: merges | 3.9 | (a) TWFE event-study mode [M-010] + EventStudy warn [M-060] + the fit `time`->`post` rename [M-082] (gates: section 4.1's equivalence/divergence/pooled-parity test triple) (shipped: tests/test_v4_merge_mpd.py; consumer ports incl. HonestDiD/PreTrendsPower calendar routes); (b) TripleDifference facade [M-013] + the SDDD alias [M-064] (shipped: tests/test_v4_merge_ddd.py; the engine relocation into the private shared mixin, the keyword-only staggered fit params, and the pscore_trim tightening [M-142]. The fit-time aggregate=/balance_e= carve-out rows are scheduled REMOVALS and so are cited in the phase-5 cell, not here - their only lifecycle version is removed_in 4.0); (c) CiC method= [M-015] + its results-field mirror [M-143] (shipped: tests/test_v4_merge_cic.py; method= is keyword-only and lowercase-only, the QDiD CLASS is deprecated while the METHOD is not, and ChangesInChangesResults.estimator -> .method carries a dual-key to_dict() window through 3.9) | | 4: release + soak | 3.9 cut | Migration guide written (skeleton: section 10); maintainer cuts 3.9; maint/3.8 rule active | -| 5: enforcement | 4.0 | Removals [M-010..M-015, M-020..M-027, M-139, M-030, M-032..M-047 old names, M-060, M-061, M-064, M-070..M-077, M-084, M-086..M-089, M-001..M-003, M-117, M-118, M-119, M-120, M-140, M-141, M-143] + the alias diet [M-132]..[M-134] + the amendment's old names [M-094] [M-095] [M-097..M-115] [M-136..M-138] (incl. their consumer migrations and the `clean_control` serialized reporting key); M-031's old `time` name persists as the merged class's calendar column, so it is deliberately absent from the removal roster (its 4.0 enforcement is the M-085 behavior entry below); property window: [M-016] property-flips at 4.0 (removal at 5.0); storage flips [M-050..M-058]; default policies [M-004..M-006, M-128..M-131, M-080]; merged-class behavior enforcements [M-083] [M-085]; warning retirement [M-007]; fastpath go/no-go [M-008]; diagnostic-family docs/roster reorganization [M-090]; sentinel retirement [M-093]; the family-wide anticipation validation [M-144], the family-wide pscore_trim validation [M-145], and the staggered-family summary-alpha guard [M-146] (behavior tightenings landing at 4.0 - shipped post-3.9-cut, terminal `done`, no removal/deprecation fields); docs/llms.txt/README refresh | +| 5: enforcement | 4.0 | Removals [M-010..M-015, M-020..M-027, M-139, M-030, M-032..M-047 old names, M-060, M-061, M-064, M-070..M-077, M-084, M-086..M-089, M-001..M-003, M-117, M-118, M-119, M-120, M-140, M-141, M-143] + the alias diet [M-132]..[M-134] + the amendment's old names [M-094] [M-095] [M-097..M-115] [M-136..M-138] (incl. their consumer migrations and the `clean_control` serialized reporting key); M-031's old `time` name persists as the merged class's calendar column, so it is deliberately absent from the removal roster (its 4.0 enforcement is the M-085 behavior entry below); property window: [M-016] property-flips at 4.0 (removal at 5.0); storage flips [M-050..M-058]; default policies [M-004..M-006, M-128..M-131, M-080]; merged-class behavior enforcements [M-083] [M-085]; warning retirement [M-007]; fastpath go/no-go [M-008]; diagnostic-family docs/roster reorganization [M-090]; sentinel retirement [M-093]; the family-wide anticipation validation [M-144], the family-wide pscore_trim validation [M-145], and the family-wide summary-alpha guard [M-146] (staggered family at the initial cut; extended to the non-staggered summaries post-cut) (behavior tightenings landing at 4.0 - shipped post-3.9-cut, terminal `done`, no removal/deprecation fields); docs/llms.txt/README refresh | | 6: front door | 4.1 | `event_study(data, outcome, unit, time, first_treat, estimator=...)` comparison entry point over the staggered family (sketch only; specified in its own plan) | Citation semantic for the table: a cell may cite a row whose current `phase` diff --git a/tests/test_continuous_did.py b/tests/test_continuous_did.py index 18341bba..367d1dae 100644 --- a/tests/test_continuous_did.py +++ b/tests/test_continuous_did.py @@ -2245,3 +2245,25 @@ def test_idempotent_refit(self): assert est.get_params() == cfg # config unchanged by fit r2 = est.fit(df, **_DKW) np.testing.assert_allclose(r1.dose_response_att.effects, r2.dose_response_att.effects) + + +class TestSummaryAlphaContract: + """summary(alpha=...) never recomputes stored inference (M-146 family-wide).""" + + @pytest.fixture + def alpha_fitted(self): + data = generate_continuous_did_data(n_units=100, n_periods=3, seed=42, noise_sd=0.1) + return ContinuousDiD(n_bootstrap=49, seed=42).fit( + data, "outcome", "unit", "period", "first_treat", "dose" + ) + + @pytest.mark.parametrize("bad_alpha", [0.10, 0.0]) + def test_summary_rejects_non_fit_alpha(self, alpha_fitted, bad_alpha): + with pytest.raises(ValueError, match="never recomputes") as exc: + alpha_fitted.summary(alpha=bad_alpha) + msg = str(exc.value) + assert "re-fit with the desired alpha" in msg + assert "bootstrap percentile" not in msg + + def test_summary_accepts_fit_alpha(self, alpha_fitted): + assert alpha_fitted.summary(alpha=alpha_fitted.alpha) == alpha_fitted.summary() diff --git a/tests/test_estimators.py b/tests/test_estimators.py index 048f0049..62d30afa 100644 --- a/tests/test_estimators.py +++ b/tests/test_estimators.py @@ -4006,3 +4006,149 @@ def test_sun_abraham_joint_span_covariate_dropped_att_stable(self): covariates=["xspan"], ) assert res.att == pytest.approx(base.att, abs=1e-8) + + +# --------------------------------------------------------------------------- +# Family-wide summary(alpha=) contract (M-146 completion) +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="module") +def did_alpha_fitted(): + np.random.seed(11) + n = 40 + rows = [] + for unit in range(n): + for post in (0, 1): + rows.append( + { + "unit": unit, + "post": post, + "treated": int(unit < n // 2), + "outcome": np.random.normal() + 2.0 * (unit < n // 2) * post, + } + ) + data = pd.DataFrame(rows) + return DifferenceInDifferences().fit(data, outcome="outcome", treatment="treated", time="post") + + +@pytest.fixture(scope="module") +def mpd_alpha_fitted(): + np.random.seed(12) + n, t = 40, 6 + rows = [] + for unit in range(n): + for period in range(t): + rows.append( + { + "unit": unit, + "period": period, + "treated": int(unit < n // 2), + "outcome": np.random.normal() + + 0.3 * period + + 1.5 * (unit < n // 2) * (period >= 3), + } + ) + data = pd.DataFrame(rows) + return MultiPeriodDiD().fit( + data, outcome="outcome", treatment="treated", time="period", post_periods=[3, 4, 5] + ) + + +@pytest.fixture(scope="module") +def sdid_alpha_fitted(ci_params): + np.random.seed(13) + n, t = 30, 8 + rows = [] + for unit in range(n): + ue = np.random.normal(0, 3) + for period in range(t): + rows.append( + { + "unit": unit, + "period": period, + "treated": int(unit < 5), + "outcome": 10.0 + + ue + + 0.5 * period + + 5.0 * (unit < 5) * (period >= 4) + + np.random.normal(0, 0.5), + } + ) + data = pd.DataFrame(rows) + n_boot = ci_params.bootstrap(20) + return SyntheticDiD(n_bootstrap=n_boot, seed=42).fit( + data, + outcome="outcome", + treatment="treated", + unit="unit", + time="period", + post_periods=[4, 5, 6, 7], + ) + + +class TestSummaryAlphaContractNonStaggered: + """summary(alpha=...) never recomputes stored inference (M-146 family-wide). + + A non-fit alpha raises instead of silently relabeling the + confidence-interval header over fit-time stored intervals; alpha=0.0 + (previously swallowed by the falsy `alpha or self.alpha` idiom) raises + too. The five generic-message sites must emit the accurate shared + message, never the staggered default's bootstrap-percentile rationale. + """ + + @pytest.mark.parametrize("bad_alpha", [0.10, 0.0]) + @pytest.mark.parametrize( + "fixture", ["did_alpha_fitted", "mpd_alpha_fitted", "sdid_alpha_fitted"] + ) + def test_summary_rejects_non_fit_alpha(self, request, fixture, bad_alpha): + fitted = request.getfixturevalue(fixture) + with pytest.raises(ValueError, match="never recomputes") as exc: + fitted.summary(alpha=bad_alpha) + msg = str(exc.value) + assert "re-fit with the desired alpha" in msg + assert "bootstrap percentile" not in msg + + @pytest.mark.parametrize( + "fixture", ["did_alpha_fitted", "mpd_alpha_fitted", "sdid_alpha_fitted"] + ) + def test_summary_accepts_fit_alpha(self, request, fixture): + fitted = request.getfixturevalue(fixture) + assert fitted.summary(alpha=fitted.alpha) == fitted.summary() + + def test_print_summary_relays_the_guard(self, did_alpha_fitted): + with pytest.raises(ValueError, match="never recomputes"): + did_alpha_fitted.print_summary(alpha=0.10) + + +class TestRequireFitAlphaMessageOverride: + """The helper's message= override replaces the whole message; the default + stays byte-identical (staggered pins depend on it).""" + + def test_default_message_unchanged(self): + from diff_diff.results_base import _require_fit_alpha + + with pytest.raises(ValueError) as exc: + _require_fit_alpha(0.10, 0.05) + assert str(exc.value) == ( + "This result stores intervals computed at alpha=0.05; " + "summary() never recomputes or relabels stored inference " + "(requested alpha=0.1). Re-fit with the desired alpha " + "(bootstrap percentile intervals cannot be reconstructed from " + "the reported SE)." + ) + + def test_message_override_replaces_and_formats(self): + from diff_diff.results_base import _require_fit_alpha + + with pytest.raises(ValueError) as exc: + _require_fit_alpha( + 0.10, 0.05, message="custom never recomputes a={alpha} f={fit_alpha}" + ) + assert str(exc.value) == "custom never recomputes a=0.1 f=0.05" + + def test_none_and_fit_alpha_pass(self): + from diff_diff.results_base import _require_fit_alpha + + assert _require_fit_alpha(None, 0.05, message="x never recomputes") == 0.05 + assert _require_fit_alpha(0.05, 0.05) == 0.05 diff --git a/tests/test_methodology_synthetic_control.py b/tests/test_methodology_synthetic_control.py index 093eae3f..c30732b7 100644 --- a/tests/test_methodology_synthetic_control.py +++ b/tests/test_methodology_synthetic_control.py @@ -4865,3 +4865,23 @@ def test_permutation_floor_still_holds_with_covariates(self): warnings.simplefilter("ignore", UserWarning) s = res.conformal_test(0.0, covariates=["z"], scheme="moving_block") assert float(s["p_value"]) >= 1.0 / float(s["n_perms"]) - 1e-12 + + +def test_summary_alpha_guard_rejects_non_fit_alpha(): + # M-146 family-wide summary(alpha=) contract, SyntheticControl carve-out: + # the previously DEAD alpha parameter (assigned, never used) now raises on + # a non-fit value with a class-accurate message - this class prints no + # alpha-based interval, so the message must not claim stored intervals. + res = _exact_combo_fit(effect=3.0) + for bad_alpha in (0.10, 0.0): + with pytest.raises(ValueError, match="never recomputes") as exc: + res.summary(alpha=bad_alpha) + msg = str(exc.value) + assert "no alpha-based interval" in msg + assert "stores intervals" not in msg + # Fit alpha and None still render, confidence set included. + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + res.confidence_set(family="constant", gamma=0.25) + assert res.summary(alpha=res.alpha) == res.summary() + assert "Confidence set" in res.summary() or "confidence set" in res.summary().lower() diff --git a/tests/test_spillover.py b/tests/test_spillover.py index f1d724f5..862c6f7f 100644 --- a/tests/test_spillover.py +++ b/tests/test_spillover.py @@ -9066,3 +9066,15 @@ def test_fit_sparse_matches_dense_end_to_end(self, monkeypatch): atol=1e-12, equal_nan=True, ) + + +class TestSummaryAlphaContract: + """SpilloverDiDResults.summary relays DiDResults' inherited M-146 guard.""" + + def test_summary_rejects_non_fit_alpha(self): + df = _make_butts_2period_dgp(seed=42) + est = SpilloverDiD(rings=[0.0, 100.0], conley_coords=("lat", "lon")) + result = est.fit(df, outcome="y", unit="unit", time="time", treatment="D") + with pytest.raises(ValueError, match="never recomputes"): + result.summary(alpha=0.10) + assert result.summary(alpha=result.alpha) == result.summary() diff --git a/tests/test_triple_diff.py b/tests/test_triple_diff.py index 331c79ab..7bfcd958 100644 --- a/tests/test_triple_diff.py +++ b/tests/test_triple_diff.py @@ -1602,3 +1602,28 @@ def test_triple_difference_convenience_func_threads_valid_vcov_type(self): assert res.vcov_type == "hc1" assert np.isfinite(res.att) assert np.isfinite(res.se) + + +class TestSummaryAlphaContract: + """summary(alpha=...) never recomputes stored inference (M-146 family-wide).""" + + @pytest.fixture + def alpha_fitted(self, simple_ddd_data): + return TripleDifference().fit( + simple_ddd_data, + outcome="outcome", + group="group", + partition="partition", + post="time", + ) + + @pytest.mark.parametrize("bad_alpha", [0.10, 0.0]) + def test_summary_rejects_non_fit_alpha(self, alpha_fitted, bad_alpha): + with pytest.raises(ValueError, match="never recomputes") as exc: + alpha_fitted.summary(alpha=bad_alpha) + msg = str(exc.value) + assert "re-fit with the desired alpha" in msg + assert "bootstrap percentile" not in msg + + def test_summary_accepts_fit_alpha(self, alpha_fitted): + assert alpha_fitted.summary(alpha=alpha_fitted.alpha) == alpha_fitted.summary() diff --git a/tests/test_trop.py b/tests/test_trop.py index 3c2d8371..032e8c76 100644 --- a/tests/test_trop.py +++ b/tests/test_trop.py @@ -4281,3 +4281,57 @@ def matching(needle: str): assert ( "of" in msg and "per-observation fits" in msg ), f"LOOCV warning is not in aggregate format (fan-out not reduced): {msg}" + + +class TestSummaryAlphaContract: + """summary(alpha=...) never recomputes stored inference (M-146 family-wide). + + TROP's tailored message states the uniform-contract rationale - its t + interval WOULD be reconstructible, so it must not claim otherwise. + """ + + @pytest.fixture(scope="class") + def alpha_fitted(self): + # Same tiny config as TestTROPResults.fitted_results (class-scoped + # there, so re-declared rather than reused). + rng = np.random.default_rng(123) + n_units, n_treated, n_pre, n_post, true_att = 20, 5, 5, 3, 3.0 + data = [] + for i in range(n_units): + is_treated = i < n_treated + for t in range(n_pre + n_post): + post = t >= n_pre + y = 10.0 + i * 0.1 + t * 0.5 + if is_treated and post: + y += true_att + y += rng.normal(0, 0.5) + data.append( + { + "unit": i, + "period": t, + "outcome": y, + "treated": 1 if (is_treated and post) else 0, + } + ) + panel = pd.DataFrame(data) + trop_est = TROP( + lambda_time_grid=[0.0, 1.0], + lambda_unit_grid=[0.0, 1.0], + lambda_nn_grid=[0.0, 0.1], + n_bootstrap=10, + seed=42, + ) + return trop_est.fit( + panel, outcome="outcome", treatment="treated", unit="unit", time="period" + ) + + @pytest.mark.parametrize("bad_alpha", [0.10, 0.0]) + def test_summary_rejects_non_fit_alpha(self, alpha_fitted, bad_alpha): + with pytest.raises(ValueError, match="never recomputes") as exc: + alpha_fitted.summary(alpha=bad_alpha) + msg = str(exc.value) + assert "family-wide contract" in msg + assert "cannot be reconstructed" not in msg + + def test_summary_accepts_fit_alpha(self, alpha_fitted): + assert alpha_fitted.summary(alpha=alpha_fitted.alpha) == alpha_fitted.summary() diff --git a/tests/test_v4_matrix.py b/tests/test_v4_matrix.py index b8d1a52d..c56f046d 100644 --- a/tests/test_v4_matrix.py +++ b/tests/test_v4_matrix.py @@ -131,7 +131,7 @@ # pscore_trim tightening) = 124; + the CiC results-field rename (M-143) = 125; # + the family-wide anticipation validation row (M-144) = 126; + the DML # review-follow-ups pair (M-145 family-wide pscore_trim validation, M-146 -# staggered-family summary-alpha guard) = 128. +# family-wide summary-alpha guard) = 128. # Ids are never reused and terminal rows are never deleted, so the ledger # only grows - raise the floor when rows are added; a lower parse count # means scanner/format drift or an illegal row deletion. diff --git a/tests/test_visualization_new.py b/tests/test_visualization_new.py index 2f955222..5f79b93e 100644 --- a/tests/test_visualization_new.py +++ b/tests/test_visualization_new.py @@ -139,6 +139,7 @@ def dose_response_curve(): def continuous_results(dose_response_curve): """Mock ContinuousDiDResults.""" results = MagicMock() + results.alpha = 0.05 # real numeric fit alpha (drives the band label) results.dose_response_att = dose_response_curve acrt = MagicMock() acrt.dose_grid = np.array([1.0, 2.0, 3.0]) @@ -388,6 +389,133 @@ def test_multiple_inputs_raises(self, continuous_results, dose_response_curve): with pytest.raises(ValueError, match="exactly one"): plot_dose_response(continuous_results, curve=dose_response_curve, show=False) + @staticmethod + def _band_vertices(ax): + import numpy as np + + polys = [c for c in ax.collections if hasattr(c, "get_paths")] + if not polys: + return np.empty((0, 2)) + return np.vstack([p.vertices for c in polys for p in c.get_paths()]) + + def test_invalid_se_rows_masked_with_warning(self): + """Zero/negative/non-finite se rows lose their band (not zero-width) + while the effect line keeps every row.""" + from diff_diff import plot_dose_response + + df = pd.DataFrame( + { + "dose": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0], + "effect": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6], + "se": [0.05, 0.0, np.nan, -0.1, np.inf, 0.06], + } + ) + with pytest.warns(UserWarning, match="4 row\\(s\\) with non-positive or non-finite 'se'"): + ax = plot_dose_response(data=df, show=False) + verts = self._band_vertices(ax) + finite = verts[np.isfinite(verts).all(axis=1)] + for masked_x in (2.0, 3.0, 4.0, 5.0): + at_x = finite[np.isclose(finite[:, 0], masked_x)] + # No band interval may survive at a masked dose (degenerate + # single-y fill endpoints are tolerated; a real interval is not). + assert at_x.size == 0 or np.allclose(at_x[:, 1], at_x[0, 1]) + # Effects untouched: the effect line still carries all six rows. + assert len(ax.lines[-1].get_xdata()) == 6 + + def test_valid_se_no_warning_and_alpha_bounds(self): + import warnings + + from diff_diff import plot_dose_response + + df = pd.DataFrame({"dose": [1.0, 2.0], "effect": [0.1, 0.2], "se": [0.05, 0.06]}) + with warnings.catch_warnings(): + warnings.simplefilter("error") + plot_dose_response(data=df, show=False) + for bad in (1.0, 0.0): + with pytest.raises(ValueError, match="strictly between 0 and 1"): + plot_dose_response(data=df, alpha=bad, show=False) + + def test_band_labels_state_level_only_where_knowable( + self, continuous_results, dose_response_curve + ): + from diff_diff import plot_dose_response + + def label_of(ax): + return [t.get_text() for t in ax.get_legend().get_texts()] + + df_se = pd.DataFrame({"dose": [1.0, 2.0], "effect": [0.1, 0.2], "se": [0.05, 0.06]}) + assert "90% CI" in label_of(plot_dose_response(data=df_se, alpha=0.10, show=False)) + assert "95% CI" in label_of(plot_dose_response(data=df_se, show=False)) + # results= carries a knowable level (results.alpha). + assert "95% CI" in label_of(plot_dose_response(continuous_results, show=False)) + # Bare curve and explicit-CI input: level-free. + labels_curve = label_of(plot_dose_response(curve=dose_response_curve, show=False)) + assert "CI" in labels_curve and "95% CI" not in labels_curve + df_ci = pd.DataFrame( + { + "dose": [1.0, 2.0], + "effect": [0.1, 0.2], + "conf_int_lower": [0.0, 0.1], + "conf_int_upper": [0.2, 0.3], + } + ) + labels_ci = label_of(plot_dose_response(data=df_ci, show=False)) + assert "CI" in labels_ci and "95% CI" not in labels_ci + + def test_results_numpy_scalar_alpha_labels_level(self, continuous_results): + # np.float32 is a real numeric fit alpha (not a float subclass) - + # the level is knowable and must be labeled. + from diff_diff import plot_dose_response + + continuous_results.alpha = np.float32(0.05) + ax = plot_dose_response(continuous_results, show=False) + assert "95% CI" in [t.get_text() for t in ax.get_legend().get_texts()] + + def test_fractional_coverage_labeled_exactly(self, continuous_results): + # alpha=0.025 is a 97.5% interval - the label must not truncate or + # round it to a whole percentage. + from diff_diff import plot_dose_response + + df = pd.DataFrame({"dose": [1.0, 2.0], "effect": [0.1, 0.2], "se": [0.05, 0.06]}) + ax = plot_dose_response(data=df, alpha=0.025, show=False) + assert "97.5% CI" in [t.get_text() for t in ax.get_legend().get_texts()] + continuous_results.alpha = 0.025 + ax_r = plot_dose_response(continuous_results, show=False) + assert "97.5% CI" in [t.get_text() for t in ax_r.get_legend().get_texts()] + + def test_explicit_alpha_warns_on_every_no_op_path( + self, continuous_results, dose_response_curve + ): + from diff_diff import plot_dose_response + + df_ci = pd.DataFrame( + { + "dose": [1.0, 2.0], + "effect": [0.1, 0.2], + "conf_int_lower": [0.0, 0.1], + "conf_int_upper": [0.2, 0.3], + } + ) + df_bare = pd.DataFrame({"dose": [1.0, 2.0], "effect": [0.1, 0.2]}) + for kwargs in ( + {"results": continuous_results}, + {"curve": dose_response_curve}, + {"data": df_ci}, + {"data": df_bare}, + ): + with pytest.warns(UserWarning, match="only applies to DataFrame input"): + plot_dose_response(alpha=0.10, show=False, **kwargs) + + def test_all_masked_se_no_band_no_stray_legend(self): + from diff_diff import plot_dose_response + + df = pd.DataFrame( + {"dose": [1.0, 2.0, 3.0], "effect": [0.1, 0.2, 0.3], "se": [0.0, np.nan, -1.0]} + ) + with pytest.warns(UserWarning, match="3 row"): + ax = plot_dose_response(data=df, show=False) + assert [t.get_text() for t in ax.get_legend().get_texts()] == ["Effect"] + def test_no_input_raises(self): from diff_diff import plot_dose_response diff --git a/tests/test_visualization_plotly.py b/tests/test_visualization_plotly.py index 46dab1d7..d3c4d774 100644 --- a/tests/test_visualization_plotly.py +++ b/tests/test_visualization_plotly.py @@ -550,3 +550,115 @@ class _Fake: # ordinal x: -1 -> 0.0 (reference, retained), 0 -> 1.0 (gated), 1 -> 2.0 assert 1.0 not in band_x, "zero-SE non-reference row still in the CI band" assert 0.0 in band_x and 2.0 in band_x + + +# ── Dose-response band tests (plotly backend) ──────────────────────────────── + + +class TestPlotDoseResponsePlotly: + """The toself band polygon must never carry a non-finite vertex, must + vanish entirely when all rows are masked, and must state the confidence + level only where it is knowable.""" + + @staticmethod + def _band_traces(fig): + return [t for t in fig.data if getattr(t, "fill", None) == "toself"] + + def test_masked_se_rows_filtered_from_band(self): + import pandas as pd + + from diff_diff import plot_dose_response + + df = pd.DataFrame( + { + "dose": [1.0, 2.0, 3.0, 4.0], + "effect": [0.1, 0.2, 0.3, 0.4], + "se": [0.05, 0.0, np.nan, 0.06], + } + ) + with pytest.warns(UserWarning, match="2 row"): + fig = plot_dose_response(data=df, backend="plotly", show=False) + (band,) = self._band_traces(fig) + xs = [float(x) for x in band.x] + ys = np.asarray(band.y, dtype=float) + assert np.isfinite(ys).all() + assert 2.0 not in xs and 3.0 not in xs + assert 1.0 in xs and 4.0 in xs + + def test_explicit_ci_nan_and_inf_rows_filtered(self): + import pandas as pd + + from diff_diff import plot_dose_response + + df = pd.DataFrame( + { + "dose": [1.0, 2.0, 3.0, 4.0], + "effect": [0.1, 0.2, 0.3, 0.4], + "conf_int_lower": [0.0, np.nan, 0.2, -np.inf], + "conf_int_upper": [0.2, 0.3, 0.4, 0.5], + } + ) + fig = plot_dose_response(data=df, backend="plotly", show=False) + (band,) = self._band_traces(fig) + xs = [float(x) for x in band.x] + assert np.isfinite(np.asarray(band.y, dtype=float)).all() # isfinite, not isnan + assert 2.0 not in xs and 4.0 not in xs + + def test_string_dose_still_renders(self): + import pandas as pd + + from diff_diff import plot_dose_response + + df = pd.DataFrame( + { + "dose": ["low", "mid", "high"], + "effect": [0.1, 0.2, 0.3], + "conf_int_lower": [0.0, 0.1, 0.2], + "conf_int_upper": [0.2, 0.3, 0.4], + } + ) + fig = plot_dose_response(data=df, backend="plotly", show=False) + (band,) = self._band_traces(fig) + assert band.name == "CI" + assert set(band.x) == {"low", "mid", "high"} + + def test_all_masked_se_no_band_trace(self): + import pandas as pd + + from diff_diff import plot_dose_response + + df = pd.DataFrame({"dose": [1.0, 2.0], "effect": [0.1, 0.2], "se": [0.0, np.nan]}) + with pytest.warns(UserWarning, match="2 row"): + fig = plot_dose_response(data=df, backend="plotly", show=False) + assert self._band_traces(fig) == [] + assert [t.name for t in fig.data] == ["Effect"] + + def test_band_labels(self): + import pandas as pd + + from diff_diff import plot_dose_response + + df_se = pd.DataFrame({"dose": [1.0, 2.0], "effect": [0.1, 0.2], "se": [0.05, 0.06]}) + fig = plot_dose_response(data=df_se, alpha=0.10, backend="plotly", show=False) + assert self._band_traces(fig)[0].name == "90% CI" + results = MagicMock() + results.alpha = 0.05 + curve = MagicMock() + curve.dose_grid = np.array([1.0, 2.0]) + curve.effects = np.array([0.1, 0.2]) + curve.conf_int_lower = np.array([0.0, 0.1]) + curve.conf_int_upper = np.array([0.2, 0.3]) + curve.target = "att" + results.dose_response_att = curve + fig_r = plot_dose_response(results, backend="plotly", show=False) + assert self._band_traces(fig_r)[0].name == "95% CI" + results.alpha = np.float32(0.05) # numpy scalar fit alpha labels too + fig_r32 = plot_dose_response(results, backend="plotly", show=False) + assert self._band_traces(fig_r32)[0].name == "95% CI" + fig_c = plot_dose_response(curve=curve, backend="plotly", show=False) + assert self._band_traces(fig_c)[0].name == "CI" + fig_frac = plot_dose_response(data=df_se, alpha=0.025, backend="plotly", show=False) + assert self._band_traces(fig_frac)[0].name == "97.5% CI" + results.alpha = 0.025 + fig_r_frac = plot_dose_response(results, backend="plotly", show=False) + assert self._band_traces(fig_r_frac)[0].name == "97.5% CI"