test: recalibrate Windows warm P50 budget (Fixes #513)

[Karthik Nadig (karthiknadig)]

Summary

Recalibrate only the Windows schema-v2 warm full-refresh P50 budget after unchanged product runs exposed an unusually low exact baseline. The new limit accepts the measured 105-182ms range while still blocking a sustained median above 255ms against that baseline.

Changes

• change Windows warm full-refresh P50 from 50ms/30% to 150ms/50%
• preserve Linux/macOS, P95, cold, startup, coverage, and schema gates
• prove the observed 182ms run passes
• prove a 300ms sustained warm median fails without implicating P95 or cold P50
• document six-run calibration provenance

Validation

• python -m unittest discover -s scripts/tests -p 'test_*.py' -v (35 passed)
• observed 182ms Windows artifact passes against exact baseline ad7ca14

Stacked on #512; retarget to main after #512 merges.

Fixes #513

[Copilot]

Pull request overview

This PR adjusts the Windows warm Full refresh P50 regression budget used by the quality snapshot comparator to reflect observed schema-v2 variance on hosted runners, while keeping the rest of the performance and coverage gates unchanged.

Changes:

• Update the Windows warm full-refresh P50 budget from 50ms/30% to 150ms/50% in the comparator.
• Update unit tests to cover the observed schema-v2 Windows warm P50 variance and ensure a material regression still fails.
• Update docs/QUALITY_SNAPSHOTS.md to document the recalibration provenance.

Show a summary per file

File	Description
scripts/tests/test_quality_snapshot.py	Updates/extends tests to validate the new Windows warm P50 budget behavior (pass for observed variance; fail for material regression).
scripts/quality_snapshot.py	Adjusts Windows warm full-refresh P50 regression budget to 150ms/50%.
docs/QUALITY_SNAPSHOTS.md	Updates performance budget table and adds provenance text for the Windows warm P50 recalibration.

Review details

Tip

Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

docs/QUALITY_SNAPSHOTS.md:31

• This calibration note says "repeated unchanged-code pull-request measurements" but the PR description calls out a specific six-run provenance. To avoid ambiguity (and to match how the surrounding paragraphs document exact sample counts), spell out the number of runs used for this recalibration.

The Windows warm full-refresh P50 budget was recalibrated in issue #513 from repeated unchanged-code pull-request measurements plus the exact schema-v2 baseline at `ad7ca14`. It retains nearly twice the observed absolute range while blocking a sustained median above 255ms against that baseline.

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

scripts/tests/test_quality_snapshot.py:178

• With the new Windows Full refresh P50 budget (150ms / 50%), the existing test_relative_budget_must_also_be_exceeded no longer exercises the intended case (absolute budget exceeded but relative budget not exceeded). Adding an explicit schema-v2 test where delta >150ms but percent change <50% will keep the “must exceed both budgets” behavior covered.

    def test_schema_v2_windows_warm_p50_material_regression_fails(self):
        baseline = performance_snapshot(schema_version=2, refresh_p50=105)
        current = performance_snapshot(schema_version=2, refresh_p50=300)

        _, failures = compare_performance(current, baseline, 'Windows')

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite