Make cold-refresh performance snapshots statistically robust

[karthiknadig]

Problem

The performance snapshot currently runs 10 fresh PET processes against one shared cache directory. In practice, only iteration 1 exercises the cold cache; iterations 2-10 are warm. The interpolated P95 is therefore dominated by a single cold sample, which is not statistically robust.

PR #508 changes only the Python comparator's macOS budget tuple, tests, and documentation, yet unchanged Linux/Windows product paths produced these failures:

• Run 31504572420, Linux: iteration 1 = 3,035ms, iterations 2-10 = 48-54ms, untimed diagnostics = 48-135ms; reported P95 = 1,694ms and failed.
• Run 31505511672, Windows: iteration 1 = 16,567ms, iterations 2-10 = 133-150ms, untimed diagnostics = 140-176ms; reported P95 = 9,179ms and failed.

Inventories, medians, time-to-first, and all diagnostic locator/phase distributions remained healthy. This makes it difficult to distinguish a genuine PET cold-start regression from a one-off hosted-runner/cache/antivirus event.

Scope

• Measure cold refresh with multiple independent cold-cache iterations rather than one sample.
• Keep warm-cache P50/P95 separate from cold-refresh metrics.
• Gate both cold and warm performance with budgets derived from repeated exact-base runs.
• Preserve inventory checks, interpreter-timeout counts, and phase/locator diagnostics.
• Associate diagnostics with the corresponding measured workload closely enough to attribute a cold failure.
• Do not solve this by simply widening or disabling the existing budgets.

Acceptance criteria

• Repeated unchanged-head hosted runs do not fail because of one uncorrelated cold sample.
• An injected multi-second delay in every cold iteration fails a tested gate.
• Warm median/tail regressions remain independently blocking.
• Missing samples, inventory mismatches, and malformed metrics still fail closed.
• Linux, Windows, and macOS workflows plus comparator unit tests pass.

[karthiknadig]

Cold/warm calibration runs 31509651276 and 31510433777 both passed on b96da55.

Platform	Cold P50 runs	Cold P95 runs	Warm P50 runs	Warm P95 runs
Linux	147 / 141ms	514 / 1,113ms	58 / 55ms	60 / 58ms
Windows	155 / 131ms	1,665 / 1,285ms	160 / 133ms	171 / 143ms
macOS	404 / 496ms	2,019 / 2,059ms	160 / 198ms	198 / 270ms

Cold P50 varied by only 6ms, 24ms, and 92ms, while cold P95 remained dominated by one host-cold sample (maxima 0.8-3.3s). This supports gating systematic cache-cold regressions with cold P50 and retaining cold P95 as diagnostic.

Selected schema-v2 cold P50 dual budgets:

• Linux: 100ms / 50%
• Windows: 150ms / 50%
• macOS: 250ms / 50%

For the one-time schema-v2-current/schema-v1-base transition, explicit absolute ceilings are 500ms / 750ms / 1,000ms. Both real schema-v2 runs pass against their exact legacy main baselines. Once main publishes schema-v2 baselines, normal dual comparisons apply automatically.

[karthiknadig]

Final quality conclusion for PR #510 at 9bc387f:

• Exact-head Copilot review: 4/4 files, 0 comments, 0 unresolved threads.
• Full CI, lint, CodeQL, Linux/Windows coverage, and all performance jobs pass.
• Linux: warm refresh P50/P95 64/69ms vs legacy baseline 47/363ms; cold P50 158ms vs 500ms migration ceiling.
• Windows: warm 182/206ms vs 158/1,624ms; cold P50 179ms vs 750ms ceiling.
• macOS: warm 151/171ms vs 117/1,092ms; cold P50 409ms vs 1,000ms ceiling.
• Each platform has 10 cold, 10 warm, and 10 independently cold diagnostic samples; inventories match and interpreter timeout maps are empty.
• Cold P95 remains diagnostic (268ms Linux, 1,720ms Windows, 2,226ms macOS) while cold P50 is stable and blocking, preventing one host-cold sample from defining the result.
• Line/function coverage is unchanged on Linux and Windows.
• Local ci-perf execution ran all 14 tests concurrently with 8 threads; every test passed using isolated named cache sandboxes.
• Comparator suite: 31/31 passed; two real schema-v2 all-platform runs pass against schema-v1 exact baselines, and synthetic systematic cold regressions fail.