Skip to content

Recalibrate warm P95 budgets after schema-v2 baselines #511

Description

Problem

PR #510 changes the existing full_refresh and time_to_first_env distributions from one cold + nine warm samples to ten independently paired warm samples. The current P95 budgets were calibrated against the old mixed distribution and will become too permissive once schema-v2 baselines land.

Three schema-v2 hosted runs report:

Platform Warm refresh P95 Warm time-to-first P95 Current absolute budgets
Linux 60 / 58 / 69ms 15 / 16 / 15ms 1,000ms / 250ms
Windows 171 / 143 / 206ms 38 / 28 / 47ms 5,000ms / 500ms
macOS 198 / 270 / 171ms 166 / 228 / 136ms 1,000ms / 750ms

After #510, those old limits could allow multi-second warm-cache drift even though cold performance is gated separately.

Scope

After #510 merges and exact schema-v2 main baselines exist:

  • inspect repeated warm refresh/time-to-first baseline variance on all platforms;
  • tighten warm P95 dual budgets with measured hosted-runner headroom;
  • keep P50, server-startup, and cold-refresh budgets unchanged unless independent evidence requires adjustment;
  • update comparator tests and docs/QUALITY_SNAPSHOTS.md;
  • retain the schema-v1 migration behavior only where still required.

Acceptance criteria

  • Repeated unchanged-head schema-v2 runs stay green.
  • Synthetic multi-second warm refresh and warm time-to-first regressions fail on every platform.
  • Cold P50 gating remains independent and unchanged.
  • Missing metrics, inventory mismatches, and malformed snapshots still fail closed.
  • Full CI, coverage, performance, and CodeQL workflows pass.

Depends on #510.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestimportantIssue identified as high-priority

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions