Skip to content

fix(results): family-wide summary(alpha=) guard + honest dose-response bands (M-146 completion) - #811

Merged
igerber merged 1 commit into
mainfrom
fix/summary-alpha-mislabel-guard
Aug 30, 2026
Merged

fix(results): family-wide summary(alpha=) guard + honest dose-response bands (M-146 completion)#811
igerber merged 1 commit into
mainfrom
fix/summary-alpha-mislabel-guard

Conversation

@igerber

@igerber igerber commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • Family-wide summary(alpha=) guard (M-146 completion): the non-staggered results containers printed a requested-alpha {level}% header over the FIT-TIME stored conf_int — silent coverage mislabeling. The shared results_base._require_fit_alpha guard now covers DiDResults (SpilloverDiDResults by inheritance), MultiPeriodDiDResults, SyntheticDiDResults, TripleDifferenceResults, TROPResults, ContinuousDiDResults, and SyntheticControlResults (whose alpha was a dead assignment — now an honest raise). alpha=0.0, previously swallowed by the falsy alpha or self.alpha idiom, raises everywhere. Default-call summary() output verified byte-identical pre/post on all seven containers.
  • Accurate error messages: the helper gains a message= override (helper-formatted {alpha}/{fit_alpha}; the staggered default message is byte-unchanged, pinned by an exact-string test). The five analytic-default sites share an accurate generic message; TROP's states the uniform-contract rationale (its t interval WOULD be reconstructible — no false non-reconstructibility claim); SyntheticControl's states it reports no alpha-based interval (the confidence set is gamma-keyed).
  • plot_dose_response honest bands/labels: non-positive/non-finite se rows are masked from the band with a warning (previously a zero-SE row drew a finite zero-width band); alpha is validated strictly inside (0, 1) (alpha=1.0 previously drew a zero-width band via z=0); the band legend (previously hard-coded "95% CI") is derived via a _coverage_label helper — exact fractional coverage (97.5% CI for alpha=0.025), tolerant of numpy-scalar fit alphas — from the requested alpha on the se branch and results.alpha on results= input, with the level-free "CI" where no level is knowable; an explicitly passed alpha on non-se input warns instead of being silently ignored; the plotly toself band filters non-finite-CI rows (a NaN vertex split the polygon) and both renderers suppress an all-masked band.
  • Docs/ledger: M-146 note amended + code_refs extended (note-prose only, no lifecycle change; test_ref stays the staggered anchor per the single-path schema); docs/v4-design.md, docs/migration-4.0.md, and the REGISTRY M-146 note extended family-wide with SyntheticControl and TROP posture Notes; the plot_group_effects zero-SE Note extended to the dose-response twin (including the deliberate alpha-semantics divergence). The audit TODO row is closed; the summaries' pre-existing truncating {level}% idiom (~19 sites incl. the shipped staggered family) is deliberately unchanged and recorded as its own TODO row.

Methodology references (required if estimator / math changes)

  • Method name(s): N/A — no estimation, weighting, or inference computation changed; this is a reporting-contract change (summary/plot labeling and fail-closed display), documented under the M-146 family contract in docs/methodology/REGISTRY.md
  • Paper / source link(s): N/A
  • Any intentional deviations from the source (and why): None — REGISTRY Notes added for the SyntheticControl/TROP guard postures and the dose-response zero-SE gate/labeling rule (incl. the deliberate alpha-semantics divergence from plot_group_effects)

Validation

  • Tests added/updated: tests/test_estimators.py (family contract + _require_fit_alpha unit tests), tests/test_triple_diff.py, tests/test_trop.py, tests/test_continuous_did.py, tests/test_methodology_synthetic_control.py, tests/test_spillover.py, tests/test_visualization_new.py, tests/test_visualization_plotly.py (new dose-response class), tests/test_v4_matrix.py (commentary wording)
  • All targeted suites green (incl. v4 matrix, tracking guards, staggered M-146 pins untouched); black/ruff over diff_diff tests and mypy at zero errors; pre/post oracle capture of all seven summaries byte-identical on default calls
  • Backtest / simulation / notebook evidence (if applicable): N/A

Security / privacy

  • Confirm no secrets/PII in this PR: Yes

…e bands (M-146 completion)

summary(alpha=...) on the non-staggered results containers printed a
requested-alpha "{level}% CI" header over the FIT-TIME stored conf_int -
silent coverage mislabeling. Apply the shared
results_base._require_fit_alpha guard (M-146, previously staggered-only)
to DiDResults (SpilloverDiDResults by inheritance), MultiPeriodDiDResults,
SyntheticDiDResults, TripleDifferenceResults, TROPResults, and
ContinuousDiDResults; SyntheticControlResults joins too (its alpha was a
dead assignment - now an honest raise). alpha=0.0, previously swallowed by
the falsy `alpha or self.alpha` idiom, raises everywhere.

The helper gains an optional message= override (helper-formatted
{alpha}/{fit_alpha} placeholders; the staggered default message is
byte-unchanged, pinned by an exact-string test) because the default's
bootstrap-percentile rationale is false for the new sites: the five
generic sites share an accurate never-recompute message, TROP states the
uniform-contract rationale (its t interval WOULD be reconstructible), and
SyntheticControl states that it reports no alpha-based interval at all
(the confidence set is gamma-keyed).

plot_dose_response fixes in the same audit class:
- DataFrame-`se` rows with non-positive/non-finite se are masked from the
  band with a UserWarning (previously a zero-SE row drew a finite
  zero-width band); the resolved alpha is validated strictly inside (0,1)
  (alpha=1.0 previously drew a zero-width band via z=0).
- Band legend was hard-coded "95% CI" in both renderers regardless of the
  requested alpha: it is now derived via _coverage_label (exact
  fractional coverage - alpha=0.025 reads "97.5% CI" - tolerant of
  numpy-scalar fit alphas incl. float32) from the requested alpha on the
  se branch and results.alpha on results= input, and the level-free "CI"
  for bare-curve/explicit-CI input (no knowable level).
- alpha: Optional[float] = None; an explicitly passed alpha on any
  non-`se` path warns instead of being silently ignored.
- The plotly toself band polygon filters non-finite-CI rows (a NaN vertex
  split it into independently-closed sub-polygons) - CI values only, the
  dose axis may be non-numeric; both renderers suppress an all-masked band
  (no stray legend entry).

Docs: M-146 ledger note amended (code_refs extended; test_ref stays the
staggered anchor per the single-path schema), v4-design/migration-4.0/
REGISTRY wording extended family-wide with the SyntheticControl and TROP
posture Notes, and the plot_group_effects zero-SE Note extended to the
dose-response twin (including the deliberate alpha-semantics divergence:
plot_group_effects recomputes, plot_dose_response warns-and-ignores).
Behavior fragment in changelog.d; the audit TODO row is closed, and the
summaries' pre-existing truncating {level}% idiom (all ~19 sites incl.
the shipped staggered family) is deliberately unchanged - unifying it on
the exact formatter is recorded as its own TODO row.

Tests: TestSummaryAlphaContract per container (message-content pins so a
site that forgot message= fails), _require_fit_alpha override unit tests,
and matplotlib+plotly dose-response suites (mask contract incl. negative/
inf se, all-masked suppression, string-dose rendering, label matrix incl.
fractional and numpy-scalar alphas, explicit-alpha warnings on all four
no-op paths). Pre/post capture of all seven summaries verified
byte-identical on default calls.
@github-actions

Copy link
Copy Markdown

Overall Assessment

Looks good — no unmitigated P0 or P1 findings.

Executive Summary

  • No estimator, weighting, identification, variance, or SE computation changed.
  • Fit-alpha guards match the documented M-146 family-wide contract.
  • Invalid dose-response SEs now fail closed without misleading bands.
  • Behavioral changes have tests, documentation, and a valid changelog fragment.
  • Fractional summary labels remain tracked as P3 technical debt.

Methodology

  • Severity: P3 — informational, documented.
  • Impact: Summary and plotting behavior follows explicit Registry Notes for M-146, Synthetic Control, TROP, and dose-response bands (REGISTRY.md:L1233, REGISTRY.md:L6075). No paper-defined estimation or inference calculation is altered.
  • Concrete fix: None required.

Code Quality

No findings. The shared guard and vectorized finite-SE masking are consistent and avoid inline inference anti-patterns.

Performance

No findings. Added finite-value checks are linear in plotted rows and negligible relative to rendering.

Maintainability

No findings. Centralizing the summary contract in _require_fit_alpha reduces cross-container drift (results_base.py:L106).

Tech Debt

  • Severity: P3 — informational, tracked in TODO.md.
  • Impact: Fractional confidence levels in summaries remain truncated, such as 97.5% displaying as 97% (TODO.md:L60).
  • Concrete fix: Later reuse the exact coverage formatter across summary surfaces; no action required for this PR.

Security

No findings. No secrets, unsafe inputs, or sensitive-data handling changes were introduced.

Documentation / Tests

No findings. Both plotting backends, invalid/all-masked SEs, alpha interactions, inherited summaries, and class-specific messages receive coverage. The changelog fragment passes its checker.

Full tests could not be rerun because this environment lacks pytest and numerical dependencies; all 17 changed Python files passed static AST parsing.

@igerber igerber added the ready-for-ci Triggers CI test workflows label Aug 30, 2026
@igerber
igerber merged commit e11d414 into main Aug 30, 2026
39 of 40 checks passed
@igerber
igerber deleted the fix/summary-alpha-mislabel-guard branch August 30, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ci Triggers CI test workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant