fix: enforce trustworthy quality snapshots

[Karthik Nadig (karthiknadig)]

Summary

Turn PET performance and coverage snapshots into trustworthy, enforceable quality gates instead of advisory comments.

• compare against artifacts for the exact PR base commit
• fail malformed/missing snapshots, inventory mismatches, and platform-budget regressions
• evaluate P50, P95, time-to-first-environment, line coverage, and function coverage
• centralize tested cross-platform comparison/report logic
• preserve one buffered JSON-RPC stdout reader and continuously drain bounded stderr
• document budgets, local validation, and the macOS tail-latency follow-up

Validation

• python -B -m unittest discover -s scripts/tests -p 'test_*.py' -v
• parsed all changed workflows with existing PyYAML
• validated all platform comparators against real baseline artifacts
• cargo test --all
• ./scripts/rust-precommit.ps1

Fixes #503

[github-actions]

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	80.176%	80.181%	-0.006pp
Functions	84.548%	84.548%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	74ms	66ms	+8ms	+12.1%	>100ms and >50%	🔺
Server startup P95	5462ms	4540ms	+922ms	+20.3%	>10000ms and >100%	🔺
Full refresh P50	124ms	142ms	-18ms	-12.7%	>100ms and >50%	✅
Full refresh P95	30310ms	30291ms	+19ms	+0.1%	>5000ms and >25%	🔺
Time to first environment P50	95ms	101ms	-6ms	-5.9%	>150ms and >50%	✅
Time to first environment P95	5489ms	4570ms	+919ms	+20.1%	>10000ms and >100%	🔺

Workload	PR	Baseline
Environments	10	10
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	1ms	1ms	+0ms	+0.0%	>5ms and >100%	➖
Server startup P95	1ms	1ms	+0ms	+0.0%	>50ms and >200%	➖
Full refresh P50	54ms	54ms	+0ms	+0.0%	>25ms and >30%	➖
Full refresh P95	168ms	284ms	-116ms	-40.8%	>1000ms and >100%	✅
Time to first environment P50	14ms	12ms	+2ms	+16.7%	>20ms and >100%	🔺
Time to first environment P95	22ms	21ms	+1ms	+4.8%	>250ms and >100%	🔺

Workload	PR	Baseline
Environments	5	5
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[github-actions]

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	77.535%	77.535%	+0.000pp
Functions	81.164%	81.164%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	9ms	9ms	+0ms	+0.0%	>10ms and >50%	➖
Server startup P95	12ms	12ms	+0ms	+0.0%	>50ms and >100%	➖
Full refresh P50	153ms	152ms	+1ms	+0.7%	>50ms and >30%	🔺
Full refresh P95	691ms	1572ms	-881ms	-56.0%	>5000ms and >100%	✅
Time to first environment P50	23ms	24ms	-1ms	-4.2%	>25ms and >50%	✅
Time to first environment P95	25ms	30ms	-5ms	-16.7%	>500ms and >100%	✅

Workload	PR	Baseline
Environments	10	10
Managers	2	2

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly.

[Copilot]

Pull request overview

This PR turns PET’s existing performance and coverage “quality snapshots” into enforceable CI gates by validating snapshot integrity, comparing PR results against artifacts from the exact PR base commit, and failing the workflow on regressions or invalid comparisons. It also hardens the E2E JSON-RPC client to avoid stdout/stderr handling artifacts that could skew latency measurements.

Changes:

• Add a centralized Python snapshot comparator (scripts/quality_snapshot.py) with unit tests and markdown reporting for performance (P50/P95/TTFE) and coverage (line/function).
• Update perf/coverage workflows to download baseline artifacts for the PR base SHA, validate comparator logic, and fail on malformed/missing snapshots, inventory mismatches, or budget regressions.
• Update the Rust E2E client to keep a single buffered stdout reader and continuously drain bounded stderr to prevent backpressure or buffered read-ahead loss.

Show a summary per file

File	Description
scripts/tests/test_quality_snapshot.py	Adds unit coverage for snapshot parsing, comparison decisions, and CLI exit/report behavior.
scripts/quality_snapshot.py	Implements snapshot loading/validation, cross-platform budgeted comparisons, and report generation used by CI gates.
docs/QUALITY_SNAPSHOTS.md	Documents the new gating rules, budgets, and local validation commands.
crates/pet/tests/e2e_performance.rs	Fixes JSON-RPC client IO handling to preserve stdout buffering and drain stderr, improving measurement trustworthiness.
.github/workflows/perf-tests.yml	Enforces performance snapshot extraction/validation and compares against exact PR base artifacts (blocking on regressions/invalid snapshots).
.github/workflows/perf-baseline.yml	Ensures baseline artifacts are valid (and expands baseline triggers) so PR comparisons are trustworthy.
.github/workflows/coverage.yml	Enforces coverage comparisons against exact PR base artifacts and blocks on regressions/invalid snapshots.
.github/workflows/coverage-baseline.yml	Validates baseline coverage artifacts (and expands baseline triggers) to support enforceable gating.

Review details

Tip

Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 8/8 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

scripts/quality_snapshot.py:206

• parse_lcov() accepts negative summary counts (e.g. LF:-1, LH:-1). Those are invalid LCOV values and can lead to nonsensical percentages or incorrect pass/fail decisions; the snapshot validator should reject them explicitly.

    if lines_found == 0 or functions_found == 0:
        raise SnapshotError(f'Coverage file has no line/function summary data: {path}')
    if lines_hit > lines_found or functions_hit > functions_found:
        raise SnapshotError(f'Coverage file has invalid hit totals: {path}')

• Files reviewed: 8/8 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 0 new
• Review effort level: Lite