test: recalibrate Windows warm P50 budget (Fixes #513)

[Karthik Nadig (karthiknadig)]

Summary

Recalibrate only the Windows schema-v2 warm full-refresh P50 budget after unchanged product runs exposed an unusually low exact baseline. The new limit accepts the measured 105-182ms range while still blocking a sustained median above 255ms against that baseline.

Changes

• change Windows warm full-refresh P50 from 50ms/30% to 150ms/50%
• preserve Linux/macOS, P95, cold, startup, coverage, and schema gates
• prove the observed 182ms run passes
• prove a 300ms sustained warm median fails without implicating P95 or cold P50
• preserve explicit coverage that both absolute and relative limits must be exceeded
• document the six-measurement calibration set

Validation

• python -m unittest discover -s scripts/tests -p 'test_*.py' -v (35 passed)
• observed 182ms Windows artifact passes against exact baseline ad7ca14
• all-platform performance run 31524812300 passed after retrying a transient baseline-download certificate failure
• clean Copilot review on superseded stacked PR test: recalibrate Windows warm P50 budget (Fixes #513) #514

Supersedes #514, which GitHub automatically closed when its stacked base branch was deleted.

Fixes #513

[github-actions]

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	80.815%	80.815%	+0.000pp
Functions	84.806%	84.806%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	1ms	1ms	+0ms	+0.0%	>5ms and >100%	➖
Server startup P95	1ms	1ms	+0ms	+0.0%	>50ms and >200%	➖
Full refresh P50	59ms	60ms	-1ms	-1.7%	>25ms and >30%	✅
Full refresh P95	62ms	62ms	+0ms	+0.0%	>50ms and >50%	➖
Time to first environment P50	13ms	14ms	-1ms	-7.1%	>20ms and >100%	✅
Time to first environment P95	16ms	16ms	+0ms	+0.0%	>25ms and >100%	➖
Cold refresh P50	150ms	156ms	-6ms	-3.8%	>100ms and >50%	✅

Workload	PR	Baseline
Environments	5	5
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	53ms	65ms	-12ms	-18.5%	>100ms and >50%	✅
Server startup P95	559ms	706ms	-147ms	-20.8%	>750ms and >100%	✅
Full refresh P50	93ms	112ms	-19ms	-17.0%	>100ms and >50%	✅
Full refresh P95	107ms	137ms	-30ms	-21.9%	>300ms and >100%	✅
Time to first environment P50	70ms	92ms	-22ms	-23.9%	>150ms and >50%	✅
Time to first environment P95	82ms	109ms	-27ms	-24.8%	>250ms and >100%	✅
Cold refresh P50	257ms	337ms	-80ms	-23.7%	>250ms and >50%	✅

Workload	PR	Baseline
Environments	10	10
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	78.207%	78.207%	+0.000pp
Functions	81.533%	81.533%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	9ms	8ms	+1ms	+12.5%	>10ms and >50%	🔺
Server startup P95	12ms	11ms	+1ms	+9.1%	>50ms and >100%	🔺
Full refresh P50	163ms	137ms	+26ms	+19.0%	>150ms and >50%	🔺
Full refresh P95	180ms	144ms	+36ms	+25.0%	>250ms and >100%	🔺
Time to first environment P50	25ms	17ms	+8ms	+47.1%	>25ms and >50%	🔺
Time to first environment P95	38ms	30ms	+8ms	+26.7%	>100ms and >100%	🔺
Cold refresh P50	157ms	138ms	+19ms	+13.8%	>150ms and >50%	🔺

Workload	PR	Baseline
Environments	10	10
Managers	2	2

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[Copilot]

Pull request overview

This PR recalibrates the Windows warm Full refresh P50 regression gate used by the quality snapshot comparator, aligning it with observed schema-v2 runner variance while keeping other platform/percentile/cold/coverage/schema behaviors unchanged.

Changes:

• Updated the Windows warm Full refresh P50 dual regression budget from 50ms/30% to 150ms/50%.
• Expanded unit tests to prove a representative 182ms schema-v2 Windows run passes while a sustained 300ms median fails (without triggering unrelated gates).
• Documented the updated Windows warm P50 calibration provenance and the new budget in docs/QUALITY_SNAPSHOTS.md.

Show a summary per file

File	Description
scripts/quality_snapshot.py	Raises the Windows Full refresh P50 regression budget to reflect observed schema-v2 variance.
scripts/tests/test_quality_snapshot.py	Adds/adjusts tests to validate the new Windows warm P50 behavior (pass at 182ms, fail at 300ms, and “both budgets must be exceeded” coverage).
docs/QUALITY_SNAPSHOTS.md	Updates the published budget table and records the six-measurement Windows calibration set for issue #513.

Review details

Tip

Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite