Add performance budget regression guardrails

.github/workflows/quality.yml

@@ -53,6 +53,32 @@ jobs:
       - name: Run root policy tests
         run: php vendor/bin/phpunit --configuration phpunit.xml.dist tests/Architecture tests/Documentation
 
+  benchmark-policy:
+    name: Benchmark policy (PHP 8.4)
+    runs-on: ubuntu-24.04
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+
+      - name: Set up PHP
+        uses: shivammathur/setup-php@f3e473d116dcccaddc5834248c87452386958240 # 2.37.2
+        with:
+          php-version: '8.4'
+          tools: composer:v2
+          coverage: none
+
+      - name: Validate benchmark Composer manifest and lockfile
+        run: composer validate --working-dir=benchmarks --strict --check-lock
+
+      - name: Install benchmark dependencies
+        run: composer install --working-dir=benchmarks --no-interaction --no-progress --prefer-dist
+
+      - name: Run CI-safe benchmark policy checks
+        run: composer --working-dir=benchmarks ci:policy
+
   workspace-quality:
     name: Workspace quality (PHP ${{ matrix.php }})
     runs-on: ubuntu-24.04

benchmarks/README.md

@@ -1,6 +1,6 @@
 # EvolvePHP 2 Benchmarks
 
-EvolvePHP performance work starts from measurements instead of guesses. This benchmark harness provides correctness checks, environment fingerprinting, local baseline output, and measured evidence for later optimization work.
+EvolvePHP performance work starts from measurements instead of guesses. This benchmark harness provides correctness checks, environment fingerprinting, local baseline output, measured evidence for later optimization work, and benchmark-only budget evaluation for controlled EvolvePHP evidence.
 
 This harness is benchmark infrastructure only. It does not optimize production framework code and it does not make a fastest-framework claim. There is no current fastest-framework claim and no current top-three performance claim.
 
@@ -18,7 +18,9 @@ Benchmark-only dependencies are installed inside `benchmarks/vendor/` and locked
 
 The primary comparison lane is PHP 8.4 with OPcache enabled, JIT disabled, production-like framework configuration, and no debugging or profiling extension altering timings. PHP 8.5 results may be useful, but they must not be combined with PHP 8.4 results as if they came from the same environment.
 
-one-off stopwatch results are not performance evidence. Warmup is required. Multiple iterations are required. The environment fingerprint must match before two baselines are treated as comparable. Shared GitHub-hosted runners are useful for smoke validation, but they are not authoritative sources for absolute wall-clock regression budgets. Future blocking budgets must come from a controlled or proven low-noise environment.
+one-off stopwatch results are not performance evidence. Warmup is required. Multiple iterations are required. The environment fingerprint must match before two baselines are treated as comparable. Shared GitHub-hosted runners are useful for smoke validation and benchmark policy checks, but they are not authoritative sources for absolute wall-clock regression budgets.
+
+The initial performance budget uses p50 as the blocking metric. p95, p99, mean, relative standard deviation, throughput, and memory remain diagnostic evidence. No blocking memory budget is active because the accepted calibration did not establish a cross-run memory noise floor.
 
 ## Commands
 
@@ -52,6 +54,27 @@ Run the fast benchmark smoke:
 php benchmarks\bin\benchmark-smoke.php
 ```
 
+Validate the tracked performance budget and compact reference summary without running a benchmark:
+
+```powershell
+php benchmarks\bin\performance-budget.php --budget benchmarks\budgets\performance-budget.json --validate-reference
+```
+
+Evaluate a controlled EvolvePHP comparator candidate directory:
+
+```powershell
+php benchmarks\bin\performance-budget.php --budget benchmarks\budgets\performance-budget.json --candidate benchmarks\results\local\comparator-candidate
+```
+
+Budget evaluation states:
+
+- `pass`: p50 is within the accepted observed calibration envelope.
+- `warn`: p50 is outside the observed calibration envelope but not a blocking regression under the scenario policy.
+- `fail`: a blocking warm HTTP p50 threshold was exceeded.
+- `incomparable`: identity, protocol, availability, source-cleanliness, sample-count, or normalized-result requirements do not allow timing comparison.
+
+Exit code `0` means pass or non-blocking warning. Exit code `1` means a blocking regression. Exit code `2` means incomparable evidence or invalid policy/reference data.
+
 Run internal Core scenarios:
 
 ```powershell
@@ -112,7 +135,9 @@ A result qualifies as a reference baseline only when the documented PHP 8.4 prot
 
 Baseline comparison must reject casual comparisons when the environment fingerprint differs. The normalized result schema includes scenario identifiers, sample counts, timing statistics, percentiles when enough samples exist, relative standard deviation, throughput where derivable, memory fields, environment fingerprint, source SHA, and schema version.
 
-Cross-framework comparison, optimization work, and regression budgets are intentionally outside this initial harness. They should be introduced only after baseline measurements are reproducible and the relevant comparison methodology is defined.
+Cross-framework comparison, optimization work, and regression budgets require reproducible controlled evidence and matching comparison identity. The benchmark-only budget policy lives under `benchmarks/budgets/performance-budget.json`, and the compact reference artifacts live under `benchmarks/results/reference/`.
+
+Warm HTTP p50 scenarios have blocking thresholds derived from the accepted EvolvePHP-only controlled calibration. `application_boot` is currently monitor-only because its accepted calibration showed high within-run relative standard deviation and volatile tail timing. It can warn when it exceeds the observed envelope or the provisional observation boundary, but it does not produce a blocking timing failure in this policy version.
 
 ## Cross-Framework Comparator Infrastructure
 
@@ -130,7 +155,7 @@ Install the benchmark harness explicitly:
 composer install --working-dir=benchmarks --no-interaction
 ```
 
-Install comparator fixtures only when doing framework-maintainer comparator work:
+Install comparator fixtures only when doing controlled comparator work:
 
 ```powershell
 composer install --working-dir=benchmarks/comparators/evolvephp --no-interaction
@@ -253,4 +278,13 @@ Candidate evidence is useful while developing or reviewing the harness. Store it
 
 Canonical reference evidence requires PHP exactly 8.4.25, OPcache enabled for CLI, JIT disabled, the same php.ini/configuration and extension set for all comparator processes, and ext-phalcon 5.20.3 loaded when the five-framework lane is claimed. Shared GitHub-hosted runner wall-clock timing is not authoritative comparator evidence.
 
+The tracked reference directory contains compact, intentional artifacts:
+
+- `benchmarks/results/reference/performance-summary.json`
+- `benchmarks/results/reference/performance-report.md`
+
+Raw 100-sample process records, command streams, and disposable candidate directories remain local or externally archived. They should not be committed to the repository.
+
+The budget evaluator rejects blind timing comparisons. It requires matching execution-environment fingerprint, comparator identity, scenario identity, matrix hash, EvolvePHP comparator lock hash, fixture identity hash, sample protocol, repeated-warm operations protocol, availability state, clean candidate evidence, valid source SHA, and well-formed normalized results.
+
 The public reporting policy is a non-ranking policy. Per-scenario evidence and limitations may be published, but broad claims such as fastest framework, top-three placement, composite rankings, or one framework generally beating another belong to later accepted performance-budget work.

benchmarks/bin/performance-budget.php

@@ -0,0 +1,215 @@
+<?php
+
+declare(strict_types=1);
+
+use Evolve\Benchmarks\Support\PerformanceBudgetEvaluator;
+
+require dirname(__DIR__) . DIRECTORY_SEPARATOR . 'vendor' . DIRECTORY_SEPARATOR . 'autoload.php';
+
+$options = getopt('', [
+    'budget:',
+    'candidate:',
+    'validate-reference',
+]);
+
+if (!isset($options['budget']) || !is_string($options['budget'])) {
+    fwrite(STDERR, "Missing required --budget option.\n");
+    exit(2);
+}
+
+$budgetPath = $options['budget'];
+$evaluator = new PerformanceBudgetEvaluator();
+
+try {
+    $budget = readJsonObject($budgetPath);
+
+    if (array_key_exists('validate-reference', $options)) {
+        $evaluator->assertValidBudget($budget);
+        validateReferenceSummary($budget);
+
+        echo json_encode([
+            'schema_version' => PerformanceBudgetEvaluator::EVALUATION_SCHEMA_VERSION,
+            'status' => 'pass',
+            'validation' => 'reference policy is valid',
+        ], JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR) . PHP_EOL;
+        exit(0);
+    }
+
+    if (!isset($options['candidate']) || !is_string($options['candidate'])) {
+        fwrite(STDERR, "Missing required --candidate option unless --validate-reference is used.\n");
+        exit(2);
+    }
+
+    $evaluation = $evaluator->evaluate($budget, readCandidateEvidence($options['candidate']));
+
+    echo json_encode($evaluation, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR) . PHP_EOL;
+
+    exit(match ($evaluation['status']) {
+        'pass', 'warn' => 0,
+        'fail' => 1,
+        default => 2,
+    });
+} catch (Throwable $exception) {
+    fwrite(STDERR, $exception->getMessage() . PHP_EOL);
+    exit(2);
+}
+
+/**
+ * @return array<string, mixed>
+ */
+function readJsonObject(string $path): array
+{
+    if (!is_file($path)) {
+        throw new RuntimeException("JSON file not found: {$path}");
+    }
+
+    $decoded = json_decode((string) file_get_contents($path), true, flags: JSON_THROW_ON_ERROR);
+
+    if (!is_array($decoded)) {
+        throw new RuntimeException("JSON file must decode to an object: {$path}");
+    }
+
+    return $decoded;
+}
+
+/**
+ * @return array{manifest: array<string, mixed>, results: list<array<string, mixed>>}
+ */
+function readCandidateEvidence(string $candidateDir): array
+{
+    if (!is_dir($candidateDir)) {
+        throw new RuntimeException("Candidate directory not found: {$candidateDir}");
+    }
+
+    $candidateDir = realpath($candidateDir);
+    if ($candidateDir === false) {
+        throw new RuntimeException('Candidate directory could not be resolved.');
+    }
+
+    $manifest = readJsonObject($candidateDir . DIRECTORY_SEPARATOR . 'manifest.json');
+    $results = [];
+
+    foreach ($manifest['results'] ?? [] as $result) {
+        if (!is_array($result) || !isset($result['normalized_result']['path']) || !is_string($result['normalized_result']['path'])) {
+            continue;
+        }
+
+        $hash = $result['normalized_result']['sha256'] ?? null;
+        if (!is_string($hash) || preg_match('/\A[a-f0-9]{64}\z/', $hash) !== 1) {
+            throw new RuntimeException('Manifest normalized_result.sha256 must be a 64-character lowercase hexadecimal hash.');
+        }
+
+        $normalizedPath = resolveCandidateFile($candidateDir, $result['normalized_result']['path']);
+        $actualHash = hash_file('sha256', $normalizedPath);
+
+        if ($actualHash !== $hash) {
+            throw new RuntimeException('Normalized result hash does not match manifest evidence.');
+        }
+
+        $results[] = readJsonObject($normalizedPath);
+    }
+
+    return [
+        'manifest' => $manifest,
+        'results' => $results,
+    ];
+}
+
+function resolveCandidateFile(string $candidateDir, string $relativePath): string
+{
+    if (preg_match('/\A(?:[A-Za-z]:[\\\\\/]|[\\\\\/])/', $relativePath) === 1) {
+        throw new RuntimeException('Normalized result path must be relative to the candidate directory.');
+    }
+
+    $path = realpath($candidateDir . DIRECTORY_SEPARATOR . str_replace(['/', '\\'], DIRECTORY_SEPARATOR, $relativePath));
+
+    if ($path === false || !is_file($path)) {
+        throw new RuntimeException('Normalized result file does not exist.');
+    }
+
+    $root = str_replace('\\', '/', rtrim($candidateDir, DIRECTORY_SEPARATOR));
+    $resolved = str_replace('\\', '/', $path);
+
+    if (!str_starts_with($resolved, $root . '/')) {
+        throw new RuntimeException('Normalized result path escapes the candidate directory.');
+    }
+
+    return $path;
+}
+
+/**
+ * @param array<string, mixed> $budget
+ */
+function validateReferenceSummary(array $budget): void
+{
+    $referencePath = dirname(__DIR__) . DIRECTORY_SEPARATOR . 'results' . DIRECTORY_SEPARATOR . 'reference' . DIRECTORY_SEPARATOR . 'performance-summary.json';
+    $summary = readJsonObject($referencePath);
+
+    if (($summary['schema_version'] ?? null) !== 'evolvephp.performance-reference-summary.v1') {
+        throw new RuntimeException('Reference summary schema_version is not supported.');
+    }
+
+    foreach ([
+        'regression_baseline_source_sha' => $budget['baseline_source_sha'],
+        'canonical_environment_fingerprint' => $budget['comparison_identity']['execution_environment_fingerprint'],
+    ] as $field => $expected) {
+        if (($summary[$field] ?? null) !== $expected) {
+            throw new RuntimeException("Reference summary {$field} does not match the performance budget.");
+        }
+    }
+
+    $protocol = is_array($summary['calibration_protocol'] ?? null) ? $summary['calibration_protocol'] : [];
+    foreach ([
+        'run_count' => $budget['calibration']['run_count'],
+        'sample_count' => $budget['calibration']['sample_count'],
+        'php_version' => $budget['canonical_runtime_policy']['php_version'],
+        'opcache_cli_enabled' => $budget['canonical_runtime_policy']['opcache_cli_enabled'],
+        'jit_enabled' => $budget['canonical_runtime_policy']['jit_enabled'],
+        'primary_metric' => $budget['primary_metric'],
+        'repeated_warm_request_count' => $budget['sample_protocol']['request_count'],
+    ] as $field => $expected) {
+        if (($protocol[$field] ?? null) !== $expected) {
+            throw new RuntimeException("Reference summary calibration_protocol.{$field} does not match the performance budget.");
+        }
+    }
+
+    foreach ($budget['scenarios'] as $scenarioId => $policy) {
+        if (!isset($summary['scenarios'][$scenarioId]) || !is_array($summary['scenarios'][$scenarioId])) {
+            throw new RuntimeException("Reference summary is missing scenario {$scenarioId}.");
+        }
+
+        foreach ([
+            'p50_microseconds_by_run' => $policy['observed_p50_microseconds'],
+            'reference_median_p50_microseconds' => $policy['reference_p50_microseconds'],
+            'observed_maximum_p50_microseconds' => $policy['observed_maximum_p50_microseconds'],
+            'observed_range_percent' => $policy['cross_run_range_percent'],
+            'budget_classification' => $policy['mode'],
+        ] as $field => $expected) {
+            if (($summary['scenarios'][$scenarioId][$field] ?? null) !== $expected) {
+                throw new RuntimeException("Reference summary {$scenarioId}.{$field} does not match the performance budget.");
+            }
+        }
+
+        if (($policy['mode'] ?? null) === 'blocking') {
+            if (($summary['scenarios'][$scenarioId]['blocking_threshold_p50_microseconds'] ?? null) !== $policy['blocking_threshold_p50_microseconds']) {
+                throw new RuntimeException("Reference summary {$scenarioId}.blocking_threshold_p50_microseconds does not match the performance budget.");
+            }
+        } elseif (($summary['scenarios'][$scenarioId]['observation_threshold_p50_microseconds'] ?? null) !== $policy['observation_threshold_p50_microseconds']) {
+            throw new RuntimeException("Reference summary {$scenarioId}.observation_threshold_p50_microseconds does not match the performance budget.");
+        }
+
+        if (
+            $scenarioId === 'application_boot'
+            && (($summary['scenarios'][$scenarioId]['diagnostic_rsd_percent_by_run'] ?? null) !== ($policy['diagnostic_rsd_percent'] ?? null))
+        ) {
+            throw new RuntimeException('Reference summary application_boot.diagnostic_rsd_percent_by_run does not match the performance budget.');
+        }
+
+        if (
+            $scenarioId === 'http_repeated_warm'
+            && (($summary['scenarios'][$scenarioId]['operations_per_sample'] ?? null) !== $budget['sample_protocol']['repeated_warm_operations_per_sample'])
+        ) {
+            throw new RuntimeException('Reference summary http_repeated_warm.operations_per_sample does not match the performance budget.');
+        }
+    }
+}