test: make cold-refresh performance snapshots robust (Fixes #509)

[Karthik Nadig (karthiknadig)]

Summary

Make cold-refresh quality snapshots statistically robust by measuring ten independent cache-cold refreshes and ten paired cache-warm refreshes. Cold P50 now catches systematic regressions without allowing one host-cold event to dominate the gate; warm P50/P95 remain independently enforced.

Changes

• use a unique cache directory for each cold/warm pair and fresh PET processes for both members
• keep existing full_refresh and time-to-first fields as warm compatibility metrics
• add schema-v2 cold refresh/time-to-first distributions and independently cold phase/locator diagnostics
• enforce cold P50 with calibrated platform dual budgets once both snapshots are v2
• use explicit fail-closed absolute ceilings while the exact base has legacy schema-v1 metrics
• require all cold diagnostics, stable inventories, and privacy-safe timeout counts
• document workload semantics, migration behavior, and measured budgets

Validation

• cargo test -p pet --test e2e_performance (6 passed)
• local release cold/warm benchmark completed with 10 samples per workload
• python -m unittest discover -s scripts/tests -p 'test_*.py' -v (31 passed)
• cargo test --all
• mandatory format/clippy pre-commit checks
• unchanged-head all-platform runs 31509651276 and 31510433777 passed
• both real schema-v2 runs pass against exact schema-v1 main baselines

Fixes #509

[github-actions]

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	1ms	0ms	+1ms	+inf%	>5ms and >100%	🔺
Server startup P95	1ms	1ms	+0ms	+0.0%	>50ms and >200%	➖
Full refresh P50	64ms	47ms	+17ms	+36.2%	>25ms and >30%	🔺
Full refresh P95	69ms	363ms	-294ms	-81.0%	>1000ms and >100%	✅
Time to first environment P50	14ms	10ms	+4ms	+40.0%	>20ms and >100%	🔺
Time to first environment P95	15ms	24ms	-9ms	-37.5%	>250ms and >100%	✅
Cold refresh P50	158ms	legacy schema	n/a	n/a	>500ms absolute	✅

Workload	PR	Baseline
Environments	5	5
Managers	1	1

Cold refresh uses a platform absolute ceiling while the exact base has legacy metrics.

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	80.815%	80.815%	+0.000pp
Functions	84.806%	84.806%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	83ms	59ms	+24ms	+40.7%	>100ms and >50%	🔺
Server startup P95	978ms	538ms	+440ms	+81.8%	>750ms and >100%	🔺
Full refresh P50	151ms	117ms	+34ms	+29.1%	>100ms and >50%	🔺
Full refresh P95	171ms	1092ms	-921ms	-84.3%	>1000ms and >50%	✅
Time to first environment P50	111ms	73ms	+38ms	+52.1%	>150ms and >50%	🔺
Time to first environment P95	136ms	571ms	-435ms	-76.2%	>750ms and >100%	✅
Cold refresh P50	409ms	legacy schema	n/a	n/a	>1000ms absolute	✅

Workload	PR	Baseline
Environments	10	10
Managers	1	1

Cold refresh uses a platform absolute ceiling while the exact base has legacy metrics.

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	78.207%	78.207%	+0.000pp
Functions	81.533%	81.533%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	10ms	9ms	+1ms	+11.1%	>10ms and >50%	🔺
Server startup P95	15ms	13ms	+2ms	+15.4%	>50ms and >100%	🔺
Full refresh P50	182ms	158ms	+24ms	+15.2%	>50ms and >30%	🔺
Full refresh P95	206ms	1624ms	-1418ms	-87.3%	>5000ms and >100%	✅
Time to first environment P50	25ms	23ms	+2ms	+8.7%	>25ms and >50%	🔺
Time to first environment P95	47ms	34ms	+13ms	+38.2%	>500ms and >100%	🔺
Cold refresh P50	179ms	legacy schema	n/a	n/a	>750ms absolute	✅

Workload	PR	Baseline
Environments	10	10
Managers	2	2

Cold refresh uses a platform absolute ceiling while the exact base has legacy metrics.

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[Copilot]

Pull request overview

This PR updates PET’s performance snapshot generation and comparison logic to make cold-refresh gating statistically robust by measuring multiple independent cold-cache refreshes and keeping warm-cache metrics independently enforced. It introduces a schema v2 for performance snapshots that adds cold refresh/time-to-first distributions while maintaining legacy schema v1 compatibility via explicit platform ceilings during migration.

Changes:

• Update the Rust E2E performance snapshot to run 10 cold/warm cache pairs with isolated cache directories per pair and add schema-v2 cold distributions.
• Extend the Python comparator to understand schema v2, enforce cold P50 (budgeted when both sides are v2; absolute ceilings when the baseline is still v1), and render legacy-ceiling behavior in reports.
• Update tests and documentation to reflect the new sampling semantics, schema behavior, and gating rules.

Show a summary per file

File	Description
scripts/tests/test_quality_snapshot.py	Adds schema-v2 test coverage for cold metrics, legacy-ceiling behavior, and report rendering.
scripts/quality_snapshot.py	Implements schema versioning, cold metric validation, legacy ceiling comparisons, and report formatting for schema-v2 snapshots.
docs/QUALITY_SNAPSHOTS.md	Documents paired cold/warm sampling, schema-v2 migration rules, budgets, and diagnostic workload semantics.
crates/pet/tests/e2e_performance.rs	Produces schema-v2 performance snapshots with isolated cold/warm cache pairs and cold diagnostic iterations.

Review details

Tip

Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

crates/pet/tests/e2e_performance.rs:1485

• Same as the cold path: the warm time-to-first sample is treated as optional, but the test later requires exactly one sample per iteration. Record it unconditionally (and include the iteration index in the panic) so failures are actionable.

        if let Some(ttfe) = warm_client.time_to_first_env() {
            warm_time_to_first_env_stats.add(ttfe.as_millis());
        }

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite