diff --git a/benchmarks/README.md b/benchmarks/README.md index 6e36d77..6755043 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -76,6 +76,22 @@ Normalize an XML result: php benchmarks\bin\normalize-results.php --input benchmarks\results\local\phpbench.xml --output benchmarks\results\local\normalized-results.json ``` +Run the controlled comparator preflight: + +```powershell +php benchmarks\bin\comparator-preflight.php +``` + +Run controlled comparator execution: + +```powershell +php benchmarks\bin\comparator-run.php --output=benchmarks\results\local\comparator-candidate --samples=100 --warmups=5 --request-count=25 +``` + +Each controlled comparator run requires a fresh output directory. The output path must be absent or empty before execution starts; the runner fails before writing new evidence when the path already contains files or directories. + +The runner performs the controlled-lane preflight before any measured worker is started. If PHP version, OPcache CLI state, JIT state, Phalcon extension version, or the required extension lane does not match, the command writes machine-readable preflight evidence and exits non-zero without writing raw or normalized timing artifacts. + ## Scenarios Internal scenarios cover application boot/component preparation, service resolution for application, execution and transient lifetimes, execution orchestration with no sink versus a no-op sink, reset participant overhead, and persistent-style sequential execution evidence. @@ -136,13 +152,31 @@ php benchmarks\bin\comparator-smoke.php The smoke command loads `benchmarks/comparators/matrix.json`, validates fixture paths and lockfiles, exercises the common scenarios, reports Phalcon unavailable when the extension is missing, and exits non-zero if an available comparator is broken. It emits no rankings and makes no performance claims. +Smoke output is correctness output only. It is deliberately separate from controlled comparator execution and must not be treated as timed evidence. + +### Controlled Comparator Execution + +Controlled comparator execution uses `benchmarks/bin/comparator-run.php`. The parent runner loads the matrix, verifies preflight, selects one comparator or the full matrix, selects accepted scenarios, and launches a subprocess per measured sample. Each sample subprocess loads only the minimal worker harness and the selected comparator fixture root, executes bounded warmups, records one measured sample, and returns raw JSON to the parent runner. + +The runner writes raw result files, normalized result files, and `manifest.json` under the selected output directory. The manifest records the exact command for every subprocess, execution order, source SHA and dirty state, matrix SHA-256, fixture identity, comparator lock SHA-256, implementation model, execution environment identity/fingerprint, worker runtime identity, availability state, raw result path/hash, per-sample raw hashes, and normalized result path/hash. + +Broken available comparators produce a non-zero runner exit. Unavailable comparators, including Phalcon without the required extension, are recorded as unavailable with an explicit reason and no fabricated timing samples. + +The process-isolation model is subprocess per measured sample. Laravel, Symfony, Slim, Phalcon, and EvolvePHP comparator dependencies are not loaded into one long-lived measurement process. Accepted samples must report the same worker runtime identity as the accepted preflight lane before they are normalized. Comparator evidence uses the full execution environment identity for shared machine/tooling conditions and uses worker runtime identity only to prove child-process conformance. + +`application_boot` is intentionally cold inside each measured worker: it receives zero in-process subject warmups, so the measured boot is the first framework/application construction in that fresh measured PHP process. Immediately before every measured `application_boot` sample, the runner starts one separate discarded `application_boot` worker for the same comparator. That discarded process can prime host-level filesystem and code paths, but it cannot warm the measured process in-process and its timing is excluded from statistics. + +Application boot samples use deterministic rotating round-robin scheduling across the selected comparators. For selected comparators `[A, B, C]`, sample round 1 runs `A, B, C`; sample round 2 runs `B, C, A`; sample round 3 runs `C, A, B`; and the sequence repeats. Every comparator slot runs discarded worker, then measured worker. The manifest records the actual worker-order provenance with discarded/measured roles, comparator ID, scenario ID, sample index, process ID, worker runtime identity hash, exit code and availability state. + +For `application_boot`, p50 is the primary central comparison statistic. Mean, p95, p99 and relative standard deviation remain visible as tail and noise evidence. Every measured sample is retained; no arbitrary outlier trimming or post-hoc sample deletion is permitted. Warm HTTP and repeated-warm scenarios keep the configured in-process subject warmups because their timing boundary starts after framework preparation, and they do not run discarded boot workers. + ### Comparator Versions The comparator matrix records these selected versions and lockfile hashes: | Comparator | Package | Version | Constraint | Lockfile SHA-256 | | --- | --- | --- | --- | --- | -| EvolvePHP | `evolvephp/http` | `2.0.0-dev+9a0e741` | `^2.0@dev` | `f792575ec5491c8d3aa171ba5f7de3b38558bfbd82b977beea45e603fd79e491` | +| EvolvePHP | `evolvephp/http` | `2.0.x-dev` | `^2.0@dev` | `f792575ec5491c8d3aa171ba5f7de3b38558bfbd82b977beea45e603fd79e491` | | Laravel | `laravel/framework` | `13.29.0` | `13.29.0` | `33b4d04706fa39dffc1d71a7d2d03f09651555afead629e31f9229adcdc86354` | | Symfony | `symfony/http-kernel` | `8.1.5` | `8.1.5` | `d93fdac19b2cdd5379e5700a8146bb705c9b516c9ec9a0709dcd785be9b1e1d6` | | Slim | `slim/slim` | `4.15.2` | `4.15.2` | `87370678970fe51c62c6a4cd4e4ca7b3600b22c84a2a5b920e2b8527a2e089a7` | @@ -154,15 +188,19 @@ The Symfony fixture represents the Symfony 8.1 framework line using `symfony/eve The matrix uses only these stable cross-framework scenario IDs: -- `application_boot`: application/bootstrap setup for the selected fixture model. -- `http_static`: `GET /benchmark`, routed through the normal framework path, HTTP 200, deterministic body. -- `http_parameterized`: `GET /benchmark/123`, with route parameter capture proven by the response and smoke metadata. -- `http_middleware`: `GET /benchmark-middleware`, with five ordered middleware/listener layers proven by smoke metadata. -- `http_not_found`: `GET /benchmark-missing`, a genuinely unmatched path using the normal not-found path. -- `http_repeated_warm`: repeated `GET /benchmark` requests against one pre-booted reusable app/kernel/container. +- `application_boot`: application/bootstrap setup for the selected fixture model. Framework construction is inside the measured subject. +- `http_static`: `GET /benchmark`, routed through the normal framework path, HTTP 200, deterministic body. Application/router/kernel preparation is outside the measured subject. +- `http_parameterized`: `GET /benchmark/123`, with route parameter capture proven by the response and smoke metadata. Application/router/kernel preparation is outside the measured subject. +- `http_middleware`: `GET /benchmark-middleware`, with five ordered middleware/listener layers proven by smoke metadata. Application/router/kernel preparation is outside the measured subject. +- `http_not_found`: `GET /benchmark-missing`, a genuinely unmatched path using the normal not-found path. Application/router/kernel preparation is outside the measured subject. +- `http_repeated_warm`: repeated `GET /benchmark` requests against one pre-booted reusable app/kernel/container. One prepared framework instance is reused for the configured request count. The timed workload for these scenarios must not perform database access, network calls, template rendering, filesystem I/O, session storage, external cache access, queues, or application business logic. +The runner stores warmup counts and sample counts in the manifest. Percentiles in normalized output follow the schema rules: p95 requires at least 20 samples and p99 requires at least 100 samples, otherwise the percentile status is `insufficient_samples`. + +For `http_repeated_warm`, one raw sample records the batch duration for the configured request count. Normalized latency and throughput are reported per request through `operations_per_sample`; the raw batch durations remain in raw evidence. + ### Dependency Isolation Each comparator owns an isolated Composer root: @@ -182,14 +220,18 @@ benchmarks/comparators/phalcon/composer.lock Laravel, Symfony, Slim and Phalcon comparator dependencies are not installed into `benchmarks/composer.json`, any `packages/*/composer.json`, the root production dependency graph, or the application skeleton. +Measurement workers do not load `benchmarks/vendor/autoload.php`. They load only minimal benchmark harness files needed by the worker and the selected comparator's own autoload root. + ### Environment Identity vs Fixture Identity -Execution environment identity represents shared benchmark conditions: PHP version, SAPI, operating system, CPU, memory, OPcache, JIT, loaded extensions and benchmark execution tooling. It intentionally excludes comparator lockfile hashes. +Execution environment identity represents shared benchmark conditions: PHP version, SAPI, operating system, CPU, memory, Composer version, PHPBench version, OPcache, JIT, loaded extensions and benchmark execution tooling. It intentionally excludes comparator lockfile hashes. Fixture identity represents comparator-specific state: comparator/framework, exact version, fixture version, configuration, Composer lock hash and implementation model. Two comparators with different dependency lockfiles can share the same execution environment identity when they run under the same controlled PHP 8.4 benchmark environment. +The EvolvePHP comparator framework identity uses the stable development line `2.0.x-dev`. The exact measured source SHA and dirty state are recorded separately in evidence metadata and the manifest. + ### Phalcon Availability Phalcon is extension-backed. The Phalcon fixture records two deterministic states: @@ -199,6 +241,16 @@ Phalcon is extension-backed. The Phalcon fixture records two deterministic state Local development machines without the extension are allowed to report Phalcon unavailable. That is not performance evidence and not a fixture failure. +The middleware scenario uses Phalcon Micro before handlers to represent five framework event layers before the final route handler. + ### Local Comparator Results Any local comparator output is local and non-canonical. It may prove fixture correctness, availability handling, schema shape and dependency isolation, but it must not be used as an authoritative performance comparison. Controlled cross-framework measurement, normalization, publication and remediation decisions must use the documented PHP 8.4 benchmark protocol. + +### Candidate and Reference Evidence + +Candidate evidence is useful while developing or reviewing the harness. Store it under ignored local paths such as `benchmarks/results/local/comparator-candidate/`. Candidate evidence must be labelled local or candidate unless it is generated from the committed implementation reference in the controlled lane. + +Canonical reference evidence requires PHP exactly 8.4.25, OPcache enabled for CLI, JIT disabled, the same php.ini/configuration and extension set for all comparator processes, and ext-phalcon 5.20.3 loaded when the five-framework lane is claimed. Shared GitHub-hosted runner wall-clock timing is not authoritative comparator evidence. + +The public reporting policy is a non-ranking policy. Per-scenario evidence and limitations may be published, but broad claims such as fastest framework, top-three placement, composite rankings, or one framework generally beating another belong to later accepted performance-budget work. diff --git a/benchmarks/bin/comparator-preflight.php b/benchmarks/bin/comparator-preflight.php new file mode 100644 index 0000000..9527b30 --- /dev/null +++ b/benchmarks/bin/comparator-preflight.php @@ -0,0 +1,15 @@ +run($matrixPath, $outputDir, [ + 'comparators' => $options['comparator'] ?? 'all', + 'scenarios' => $options['scenario'] ?? 'all', + 'samples' => (int) ($options['samples'] ?? 100), + 'warmups' => (int) ($options['warmups'] ?? 5), + 'request_count' => (int) ($options['request-count'] ?? 25), + ]); + + echo json_encode($manifest, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR) . PHP_EOL; + + if (($manifest['status'] ?? 'failed') !== 'completed') { + exit(1); + } + + foreach ($manifest['results'] as $result) { + if (($result['availability'] ?? null) === 'failed' || ($result['exit_code'] ?? 1) !== 0) { + exit(1); + } + } + + exit(0); +} catch (Throwable $exception) { + fwrite(STDERR, $exception->getMessage() . PHP_EOL); + exit(1); +} diff --git a/benchmarks/bin/comparator-worker.php b/benchmarks/bin/comparator-worker.php new file mode 100644 index 0000000..c392a06 --- /dev/null +++ b/benchmarks/bin/comparator-worker.php @@ -0,0 +1,52 @@ +getMessage() . PHP_EOL); + exit(1); +} diff --git a/benchmarks/comparators/evolvephp/bootstrap.php b/benchmarks/comparators/evolvephp/bootstrap.php index 6a03cf1..b3ec99b 100644 --- a/benchmarks/comparators/evolvephp/bootstrap.php +++ b/benchmarks/comparators/evolvephp/bootstrap.php @@ -2,6 +2,11 @@ declare(strict_types=1); +$autoload = __DIR__ . DIRECTORY_SEPARATOR . 'vendor' . DIRECTORY_SEPARATOR . 'autoload.php'; +if (is_file($autoload)) { + require_once $autoload; +} + require_once __DIR__ . DIRECTORY_SEPARATOR . 'src' . DIRECTORY_SEPARATOR . 'EvolvePhpComparatorFixture.php'; return new Benchmark\EvolvePHP\EvolvePhpComparatorFixture(); diff --git a/benchmarks/comparators/evolvephp/src/EvolvePhpComparatorFixture.php b/benchmarks/comparators/evolvephp/src/EvolvePhpComparatorFixture.php index 40be031..951550e 100644 --- a/benchmarks/comparators/evolvephp/src/EvolvePhpComparatorFixture.php +++ b/benchmarks/comparators/evolvephp/src/EvolvePhpComparatorFixture.php @@ -4,9 +4,10 @@ namespace Benchmark\EvolvePHP; -use Evolve\Benchmarks\Comparator\ComparatorFixture; -use Evolve\Core\Execution\ExecutionOrchestrator; +use Evolve\Benchmarks\Comparator\PreparedComparatorFixture; +use Evolve\Benchmarks\Comparator\PreparedScenario; use Evolve\Core\Container\ServiceRegistry; +use Evolve\Core\Execution\ExecutionOrchestrator; use Evolve\Http\Exception\RouteNotFound; use Evolve\Http\HttpKernel; use Evolve\Http\Routing\Route; @@ -20,7 +21,7 @@ use Psr\Http\Server\MiddlewareInterface; use Psr\Http\Server\RequestHandlerInterface; -final class EvolvePhpComparatorFixture implements ComparatorFixture +final class EvolvePhpComparatorFixture implements PreparedComparatorFixture { public function id(): string { @@ -34,52 +35,138 @@ public function availability(): array public function applicationBoot(): array { - $this->createKernel(); - - return ['scenario_id' => 'application_boot', 'status' => 'ok']; + return $this->prepareScenario('application_boot')->runOnce(); } public function httpStatic(): array { - return $this->handle($this->createKernel(), '/benchmark'); + return $this->prepareScenario('http_static')->runOnce(); } public function httpParameterized(string $id): array { - $result = $this->handle($this->createKernel(), '/benchmark/' . $id); - $result['parameters'] = ['id' => $id]; - - return $result; + return $this->prepareScenario('http_parameterized', ['id' => $id])->runOnce(); } public function httpMiddleware(): array { - $state = (object) ['order' => []]; - $result = $this->handle($this->createKernel($state), '/benchmark-middleware'); - $result['middleware_order'] = $state->order; - - return $result; + return $this->prepareScenario('http_middleware')->runOnce(); } public function httpNotFound(): array { - return $this->handle($this->createKernel(), '/benchmark-missing'); + return $this->prepareScenario('http_not_found')->runOnce(); } public function httpRepeatedWarm(int $requestCount): array + { + return $this->prepareScenario('http_repeated_warm', ['request_count' => $requestCount])->runOnce(); + } + + public function prepareScenario(string $scenarioId, array $options = []): PreparedScenario + { + return match ($scenarioId) { + 'application_boot' => new PreparedScenario( + 'application_boot', + 'application_boot_constructs_framework', + 0, + function (): array { + $this->createKernel(); + + return [ + 'scenario_id' => 'application_boot', + 'status' => 'ok', + 'framework_constructed_in_subject' => true, + 'framework_prepared_outside_subject' => false, + ]; + }, + ), + 'http_static' => $this->warmHttpSubject('http_static', '/benchmark'), + 'http_parameterized' => $this->warmHttpSubject( + 'http_parameterized', + '/benchmark/' . (string) ($options['id'] ?? '123'), + ['parameters' => ['id' => (string) ($options['id'] ?? '123')]], + ), + 'http_middleware' => $this->middlewareSubject(), + 'http_not_found' => $this->warmHttpSubject('http_not_found', '/benchmark-missing'), + 'http_repeated_warm' => $this->repeatedWarmSubject((int) ($options['request_count'] ?? 25)), + default => throw new \InvalidArgumentException("Unknown comparator scenario '{$scenarioId}'."), + }; + } + + /** + * @param array $extra + */ + private function warmHttpSubject(string $scenarioId, string $path, array $extra = []): PreparedScenario { $kernel = $this->createKernel(); - $last = null; - for ($i = 0; $i < $requestCount; ++$i) { - $last = $this->handle($kernel, '/benchmark'); - } + return new PreparedScenario( + $scenarioId, + 'prepared_warm_http_request', + 1, + function () use ($kernel, $path, $extra): array { + return $this->withPreparedHttpMetadata($this->handle($kernel, $path) + $extra); + }, + ); + } + + private function middlewareSubject(): PreparedScenario + { + $state = (object) ['order' => []]; + $kernel = $this->createKernel($state); + + return new PreparedScenario( + 'http_middleware', + 'prepared_warm_http_request', + 1, + function () use ($kernel, $state): array { + $state->order = []; + $result = $this->handle($kernel, '/benchmark-middleware'); + $result['middleware_order'] = $state->order; + + return $this->withPreparedHttpMetadata($result); + }, + ); + } - $last ??= []; - $last['request_count'] = $requestCount; - $last['bootstrap_count'] = 1; + private function repeatedWarmSubject(int $requestCount): PreparedScenario + { + $kernel = $this->createKernel(); - return $last; + return new PreparedScenario( + 'http_repeated_warm', + 'prepared_repeated_warm_requests', + 1, + function () use ($kernel, $requestCount): array { + $last = null; + + for ($i = 0; $i < max(1, $requestCount); ++$i) { + $last = $this->handle($kernel, '/benchmark'); + } + + $last ??= []; + $last['request_count'] = max(1, $requestCount); + $last['bootstrap_count'] = 1; + $last['framework_constructed_in_subject'] = false; + $last['normal_framework_path_executed'] = true; + $last['prepared_framework_instance_reused'] = true; + + return $last; + }, + ); + } + + /** + * @param array $result + * @return array + */ + private function withPreparedHttpMetadata(array $result): array + { + $result['framework_constructed_in_subject'] = false; + $result['normal_framework_path_executed'] = true; + + return $result; } private function createKernel(?object $middlewareState = null): HttpKernel @@ -138,7 +225,7 @@ private function handle(HttpKernel $kernel, string $path): array private function handler(Psr17Factory $factory, string $scenario): RequestHandlerInterface { - return new class($factory, $scenario) implements RequestHandlerInterface { + return new class ($factory, $scenario) implements RequestHandlerInterface { public function __construct( private Psr17Factory $factory, private string $scenario, @@ -171,7 +258,7 @@ private function middleware(object $state): array $middleware = []; for ($i = 1; $i <= 5; ++$i) { - $middleware[] = new class($state, $i) implements MiddlewareInterface { + $middleware[] = new class ($state, $i) implements MiddlewareInterface { public function __construct(private object $state, private int $index) {} public function process(ServerRequestInterface $request, RequestHandlerInterface $handler): ResponseInterface diff --git a/benchmarks/comparators/laravel/src/LaravelComparatorFixture.php b/benchmarks/comparators/laravel/src/LaravelComparatorFixture.php index 3d0cdda..d38b853 100644 --- a/benchmarks/comparators/laravel/src/LaravelComparatorFixture.php +++ b/benchmarks/comparators/laravel/src/LaravelComparatorFixture.php @@ -4,7 +4,8 @@ namespace Benchmark\Laravel; -use Evolve\Benchmarks\Comparator\ComparatorFixture; +use Evolve\Benchmarks\Comparator\PreparedComparatorFixture; +use Evolve\Benchmarks\Comparator\PreparedScenario; use Illuminate\Container\Container; use Illuminate\Events\Dispatcher; use Illuminate\Http\Request; @@ -14,7 +15,7 @@ use Illuminate\Routing\Router; use Symfony\Component\HttpKernel\Exception\NotFoundHttpException; -final class LaravelComparatorFixture implements ComparatorFixture +final class LaravelComparatorFixture implements PreparedComparatorFixture { public function id(): string { @@ -28,52 +29,135 @@ public function availability(): array public function applicationBoot(): array { - $this->createRouter(); - - return ['scenario_id' => 'application_boot', 'status' => 'ok']; + return $this->prepareScenario('application_boot')->runOnce(); } public function httpStatic(): array { - return $this->dispatch($this->createRouter(), '/benchmark'); + return $this->prepareScenario('http_static')->runOnce(); } public function httpParameterized(string $id): array { - $result = $this->dispatch($this->createRouter(), '/benchmark/' . $id); - $result['parameters'] = ['id' => $id]; - - return $result; + return $this->prepareScenario('http_parameterized', ['id' => $id])->runOnce(); } public function httpMiddleware(): array { - MiddlewareState::$order = []; - $result = $this->dispatch($this->createRouter(), '/benchmark-middleware'); - $result['middleware_order'] = MiddlewareState::$order; - - return $result; + return $this->prepareScenario('http_middleware')->runOnce(); } public function httpNotFound(): array { - return $this->dispatch($this->createRouter(), '/benchmark-missing'); + return $this->prepareScenario('http_not_found')->runOnce(); } public function httpRepeatedWarm(int $requestCount): array + { + return $this->prepareScenario('http_repeated_warm', ['request_count' => $requestCount])->runOnce(); + } + + public function prepareScenario(string $scenarioId, array $options = []): PreparedScenario + { + return match ($scenarioId) { + 'application_boot' => new PreparedScenario( + 'application_boot', + 'application_boot_constructs_framework', + 0, + function (): array { + $this->createRouter(); + + return [ + 'scenario_id' => 'application_boot', + 'status' => 'ok', + 'framework_constructed_in_subject' => true, + 'framework_prepared_outside_subject' => false, + ]; + }, + ), + 'http_static' => $this->warmHttpSubject('http_static', '/benchmark'), + 'http_parameterized' => $this->warmHttpSubject( + 'http_parameterized', + '/benchmark/' . (string) ($options['id'] ?? '123'), + ['parameters' => ['id' => (string) ($options['id'] ?? '123')]], + ), + 'http_middleware' => $this->middlewareSubject(), + 'http_not_found' => $this->warmHttpSubject('http_not_found', '/benchmark-missing'), + 'http_repeated_warm' => $this->repeatedWarmSubject((int) ($options['request_count'] ?? 25)), + default => throw new \InvalidArgumentException("Unknown comparator scenario '{$scenarioId}'."), + }; + } + + /** + * @param array $extra + */ + private function warmHttpSubject(string $scenarioId, string $path, array $extra = []): PreparedScenario { $router = $this->createRouter(); - $last = null; - for ($i = 0; $i < $requestCount; ++$i) { - $last = $this->dispatch($router, '/benchmark'); - } + return new PreparedScenario( + $scenarioId, + 'prepared_warm_http_request', + 1, + fn(): array => $this->withPreparedHttpMetadata($this->dispatch($router, $path) + $extra), + ); + } - $last ??= []; - $last['request_count'] = $requestCount; - $last['bootstrap_count'] = 1; + private function middlewareSubject(): PreparedScenario + { + $router = $this->createRouter(); + + return new PreparedScenario( + 'http_middleware', + 'prepared_warm_http_request', + 1, + function () use ($router): array { + MiddlewareState::$order = []; + $result = $this->dispatch($router, '/benchmark-middleware'); + $result['middleware_order'] = MiddlewareState::$order; + + return $this->withPreparedHttpMetadata($result); + }, + ); + } + + private function repeatedWarmSubject(int $requestCount): PreparedScenario + { + $router = $this->createRouter(); + + return new PreparedScenario( + 'http_repeated_warm', + 'prepared_repeated_warm_requests', + 1, + function () use ($router, $requestCount): array { + $last = null; + + for ($i = 0; $i < max(1, $requestCount); ++$i) { + $last = $this->dispatch($router, '/benchmark'); + } + + $last ??= []; + $last['request_count'] = max(1, $requestCount); + $last['bootstrap_count'] = 1; + $last['framework_constructed_in_subject'] = false; + $last['normal_framework_path_executed'] = true; + $last['prepared_framework_instance_reused'] = true; + + return $last; + }, + ); + } - return $last; + /** + * @param array $result + * @return array + */ + private function withPreparedHttpMetadata(array $result): array + { + $result['framework_constructed_in_subject'] = false; + $result['normal_framework_path_executed'] = true; + + return $result; } private function createRouter(): Router diff --git a/benchmarks/comparators/matrix.json b/benchmarks/comparators/matrix.json index 8dc4c9e..e6cfd60 100644 --- a/benchmarks/comparators/matrix.json +++ b/benchmarks/comparators/matrix.json @@ -49,7 +49,7 @@ "id": "evolvephp", "name": "EvolvePHP", "framework_package": "evolvephp/http", - "framework_version": "2.0.0-dev+9a0e741", + "framework_version": "2.0.x-dev", "composer_constraint": "^2.0@dev", "fixture_version": 1, "fixture_path": "evolvephp", diff --git a/benchmarks/comparators/phalcon/src/PhalconComparatorFixture.php b/benchmarks/comparators/phalcon/src/PhalconComparatorFixture.php index f2ec75e..84d25d4 100644 --- a/benchmarks/comparators/phalcon/src/PhalconComparatorFixture.php +++ b/benchmarks/comparators/phalcon/src/PhalconComparatorFixture.php @@ -4,10 +4,11 @@ namespace Benchmark\Phalcon; -use Evolve\Benchmarks\Comparator\ComparatorFixture; use Evolve\Benchmarks\Comparator\PhalconAvailability; +use Evolve\Benchmarks\Comparator\PreparedComparatorFixture; +use Evolve\Benchmarks\Comparator\PreparedScenario; -final class PhalconComparatorFixture implements ComparatorFixture +final class PhalconComparatorFixture implements PreparedComparatorFixture { public function id(): string { @@ -21,59 +22,144 @@ public function availability(): array public function applicationBoot(): array { - $this->createApplication(); - - return ['scenario_id' => 'application_boot', 'status' => 'ok']; + return $this->prepareScenario('application_boot')->runOnce(); } public function httpStatic(): array { - return $this->handle($this->createApplication(), '/benchmark'); + return $this->prepareScenario('http_static')->runOnce(); } public function httpParameterized(string $id): array { - $result = $this->handle($this->createApplication(), '/benchmark/' . $id); - $result['parameters'] = ['id' => $id]; - - return $result; + return $this->prepareScenario('http_parameterized', ['id' => $id])->runOnce(); } public function httpMiddleware(): array { - $state = (object) ['order' => []]; - $result = $this->handle($this->createApplication($state), '/benchmark-middleware'); - $result['middleware_order'] = $state->order; - - return $result; + return $this->prepareScenario('http_middleware')->runOnce(); } public function httpNotFound(): array { - return $this->handle($this->createApplication(), '/benchmark-missing'); + return $this->prepareScenario('http_not_found')->runOnce(); } public function httpRepeatedWarm(int $requestCount): array + { + return $this->prepareScenario('http_repeated_warm', ['request_count' => $requestCount])->runOnce(); + } + + public function prepareScenario(string $scenarioId, array $options = []): PreparedScenario + { + return match ($scenarioId) { + 'application_boot' => new PreparedScenario( + 'application_boot', + 'application_boot_constructs_framework', + 0, + function (): array { + $this->createApplication(); + + return [ + 'scenario_id' => 'application_boot', + 'status' => 'ok', + 'framework_constructed_in_subject' => true, + 'framework_prepared_outside_subject' => false, + ]; + }, + ), + 'http_static' => $this->warmHttpSubject('http_static', '/benchmark'), + 'http_parameterized' => $this->warmHttpSubject( + 'http_parameterized', + '/benchmark/' . (string) ($options['id'] ?? '123'), + ['parameters' => ['id' => (string) ($options['id'] ?? '123')]], + ), + 'http_middleware' => $this->middlewareSubject(), + 'http_not_found' => $this->warmHttpSubject('http_not_found', '/benchmark-missing'), + 'http_repeated_warm' => $this->repeatedWarmSubject((int) ($options['request_count'] ?? 25)), + default => throw new \InvalidArgumentException("Unknown comparator scenario '{$scenarioId}'."), + }; + } + + /** + * @param array $extra + */ + private function warmHttpSubject(string $scenarioId, string $path, array $extra = []): PreparedScenario { $application = $this->createApplication(); - $last = null; - for ($i = 0; $i < $requestCount; ++$i) { - $last = $this->handle($application, '/benchmark'); - } + return new PreparedScenario( + $scenarioId, + 'prepared_warm_http_request', + 1, + fn(): array => $this->withPreparedHttpMetadata($this->handle($application, $path) + $extra), + ); + } + + private function middlewareSubject(): PreparedScenario + { + $state = (object) ['order' => []]; + $application = $this->createApplication($state); + + return new PreparedScenario( + 'http_middleware', + 'prepared_warm_http_request', + 1, + function () use ($application, $state): array { + $state->order = []; + $result = $this->handle($application, '/benchmark-middleware'); + $result['middleware_order'] = $state->order; + $result['middleware_model'] = 'before'; + + return $this->withPreparedHttpMetadata($result); + }, + ); + } - $last ??= []; - $last['request_count'] = $requestCount; - $last['bootstrap_count'] = 1; + private function repeatedWarmSubject(int $requestCount): PreparedScenario + { + $application = $this->createApplication(); + + return new PreparedScenario( + 'http_repeated_warm', + 'prepared_repeated_warm_requests', + 1, + function () use ($application, $requestCount): array { + $last = null; + + for ($i = 0; $i < max(1, $requestCount); ++$i) { + $last = $this->handle($application, '/benchmark'); + } - return $last; + $last ??= []; + $last['request_count'] = max(1, $requestCount); + $last['bootstrap_count'] = 1; + $last['framework_constructed_in_subject'] = false; + $last['normal_framework_path_executed'] = true; + $last['prepared_framework_instance_reused'] = true; + + return $last; + }, + ); + } + + /** + * @param array $result + * @return array + */ + private function withPreparedHttpMetadata(array $result): array + { + $result['framework_constructed_in_subject'] = false; + $result['normal_framework_path_executed'] = true; + + return $result; } private function createApplication(?object $middlewareState = null): object { $applicationClass = 'Phalcon\\Mvc\\Micro'; $responseClass = 'Phalcon\\Http\\Response'; - $notFoundHandler = static function () use ($responseClass): object { + $notFoundHandler = function () use ($responseClass): object { $response = new $responseClass(); $response->setStatusCode(404); $response->setContent('phalcon:not-found'); @@ -83,27 +169,32 @@ private function createApplication(?object $middlewareState = null): object $app = new $applicationClass(); $app->notFound($notFoundHandler); - $app->get('/benchmark', static function () use ($responseClass): object { + + if ($middlewareState !== null) { + foreach ([1, 2, 3, 4, 5] as $index) { + $app->before(function () use ($middlewareState, $index): bool { + $middlewareState->order[] = $index; + + return true; + }); + } + } + + $app->get('/benchmark', function () use ($responseClass): object { $response = new $responseClass(); $response->setStatusCode(200); $response->setContent('phalcon:static'); return $response; }); - $app->get('/benchmark/{id}', static function (string $id) use ($responseClass): object { + $app->get('/benchmark/{id}', function (string $id) use ($responseClass): object { $response = new $responseClass(); $response->setStatusCode(200); $response->setContent('phalcon:parameterized:' . $id); return $response; }); - $app->get('/benchmark-middleware', static function () use ($responseClass, $middlewareState): object { - if ($middlewareState !== null) { - for ($i = 1; $i <= 5; ++$i) { - $middlewareState->order[] = $i; - } - } - + $app->get('/benchmark-middleware', function () use ($responseClass): object { $response = new $responseClass(); $response->setStatusCode(200); $response->setContent('phalcon:middleware'); diff --git a/benchmarks/comparators/slim/src/SlimComparatorFixture.php b/benchmarks/comparators/slim/src/SlimComparatorFixture.php index e03ebd8..f0590a5 100644 --- a/benchmarks/comparators/slim/src/SlimComparatorFixture.php +++ b/benchmarks/comparators/slim/src/SlimComparatorFixture.php @@ -4,7 +4,8 @@ namespace Benchmark\Slim; -use Evolve\Benchmarks\Comparator\ComparatorFixture; +use Evolve\Benchmarks\Comparator\PreparedComparatorFixture; +use Evolve\Benchmarks\Comparator\PreparedScenario; use Nyholm\Psr7\Factory\Psr17Factory; use Psr\Http\Message\ResponseInterface; use Psr\Http\Message\ServerRequestInterface; @@ -12,7 +13,7 @@ use Slim\App; use Slim\Factory\AppFactory; -final class SlimComparatorFixture implements ComparatorFixture +final class SlimComparatorFixture implements PreparedComparatorFixture { public function id(): string { @@ -26,52 +27,136 @@ public function availability(): array public function applicationBoot(): array { - $this->createApp(); - - return ['scenario_id' => 'application_boot', 'status' => 'ok']; + return $this->prepareScenario('application_boot')->runOnce(); } public function httpStatic(): array { - return $this->handle($this->createApp(), '/benchmark'); + return $this->prepareScenario('http_static')->runOnce(); } public function httpParameterized(string $id): array { - $result = $this->handle($this->createApp(), '/benchmark/' . $id); - $result['parameters'] = ['id' => $id]; - - return $result; + return $this->prepareScenario('http_parameterized', ['id' => $id])->runOnce(); } public function httpMiddleware(): array { - $state = (object) ['order' => []]; - $result = $this->handle($this->createApp($state), '/benchmark-middleware'); - $result['middleware_order'] = $state->order; - - return $result; + return $this->prepareScenario('http_middleware')->runOnce(); } public function httpNotFound(): array { - return $this->handle($this->createApp(), '/benchmark-missing'); + return $this->prepareScenario('http_not_found')->runOnce(); } public function httpRepeatedWarm(int $requestCount): array + { + return $this->prepareScenario('http_repeated_warm', ['request_count' => $requestCount])->runOnce(); + } + + public function prepareScenario(string $scenarioId, array $options = []): PreparedScenario + { + return match ($scenarioId) { + 'application_boot' => new PreparedScenario( + 'application_boot', + 'application_boot_constructs_framework', + 0, + function (): array { + $this->createApp(); + + return [ + 'scenario_id' => 'application_boot', + 'status' => 'ok', + 'framework_constructed_in_subject' => true, + 'framework_prepared_outside_subject' => false, + ]; + }, + ), + 'http_static' => $this->warmHttpSubject('http_static', '/benchmark'), + 'http_parameterized' => $this->warmHttpSubject( + 'http_parameterized', + '/benchmark/' . (string) ($options['id'] ?? '123'), + ['parameters' => ['id' => (string) ($options['id'] ?? '123')]], + ), + 'http_middleware' => $this->middlewareSubject(), + 'http_not_found' => $this->warmHttpSubject('http_not_found', '/benchmark-missing'), + 'http_repeated_warm' => $this->repeatedWarmSubject((int) ($options['request_count'] ?? 25)), + default => throw new \InvalidArgumentException("Unknown comparator scenario '{$scenarioId}'."), + }; + } + + /** + * @param array $extra + */ + private function warmHttpSubject(string $scenarioId, string $path, array $extra = []): PreparedScenario { $app = $this->createApp(); - $last = null; - for ($i = 0; $i < $requestCount; ++$i) { - $last = $this->handle($app, '/benchmark'); - } + return new PreparedScenario( + $scenarioId, + 'prepared_warm_http_request', + 1, + fn(): array => $this->withPreparedHttpMetadata($this->handle($app, $path) + $extra), + ); + } + + private function middlewareSubject(): PreparedScenario + { + $state = (object) ['order' => []]; + $app = $this->createApp($state); + + return new PreparedScenario( + 'http_middleware', + 'prepared_warm_http_request', + 1, + function () use ($app, $state): array { + $state->order = []; + $result = $this->handle($app, '/benchmark-middleware'); + $result['middleware_order'] = $state->order; + + return $this->withPreparedHttpMetadata($result); + }, + ); + } + + private function repeatedWarmSubject(int $requestCount): PreparedScenario + { + $app = $this->createApp(); + + return new PreparedScenario( + 'http_repeated_warm', + 'prepared_repeated_warm_requests', + 1, + function () use ($app, $requestCount): array { + $last = null; + + for ($i = 0; $i < max(1, $requestCount); ++$i) { + $last = $this->handle($app, '/benchmark'); + } + + $last ??= []; + $last['request_count'] = max(1, $requestCount); + $last['bootstrap_count'] = 1; + $last['framework_constructed_in_subject'] = false; + $last['normal_framework_path_executed'] = true; + $last['prepared_framework_instance_reused'] = true; + + return $last; + }, + ); + } - $last ??= []; - $last['request_count'] = $requestCount; - $last['bootstrap_count'] = 1; + /** + * @param array $result + * @return array + */ + private function withPreparedHttpMetadata(array $result): array + { + $result['framework_constructed_in_subject'] = false; + $result['normal_framework_path_executed'] = true; - return $last; + return $result; } private function createApp(?object $middlewareState = null): App diff --git a/benchmarks/comparators/symfony/src/SymfonyComparatorFixture.php b/benchmarks/comparators/symfony/src/SymfonyComparatorFixture.php index 6d70dab..b136646 100644 --- a/benchmarks/comparators/symfony/src/SymfonyComparatorFixture.php +++ b/benchmarks/comparators/symfony/src/SymfonyComparatorFixture.php @@ -4,7 +4,8 @@ namespace Benchmark\Symfony; -use Evolve\Benchmarks\Comparator\ComparatorFixture; +use Evolve\Benchmarks\Comparator\PreparedComparatorFixture; +use Evolve\Benchmarks\Comparator\PreparedScenario; use Symfony\Component\EventDispatcher\EventDispatcher; use Symfony\Component\HttpFoundation\Request; use Symfony\Component\HttpFoundation\RequestStack; @@ -23,7 +24,7 @@ use Symfony\Component\Routing\Route; use Symfony\Component\Routing\RouteCollection; -final class SymfonyComparatorFixture implements ComparatorFixture +final class SymfonyComparatorFixture implements PreparedComparatorFixture { public function id(): string { @@ -37,52 +38,136 @@ public function availability(): array public function applicationBoot(): array { - $this->createKernel(); - - return ['scenario_id' => 'application_boot', 'status' => 'ok']; + return $this->prepareScenario('application_boot')->runOnce(); } public function httpStatic(): array { - return $this->handle($this->createKernel(), '/benchmark'); + return $this->prepareScenario('http_static')->runOnce(); } public function httpParameterized(string $id): array { - $result = $this->handle($this->createKernel(), '/benchmark/' . $id); - $result['parameters'] = ['id' => $id]; - - return $result; + return $this->prepareScenario('http_parameterized', ['id' => $id])->runOnce(); } public function httpMiddleware(): array { - $state = (object) ['order' => []]; - $result = $this->handle($this->createKernel($state), '/benchmark-middleware'); - $result['middleware_order'] = $state->order; - - return $result; + return $this->prepareScenario('http_middleware')->runOnce(); } public function httpNotFound(): array { - return $this->handle($this->createKernel(), '/benchmark-missing'); + return $this->prepareScenario('http_not_found')->runOnce(); } public function httpRepeatedWarm(int $requestCount): array + { + return $this->prepareScenario('http_repeated_warm', ['request_count' => $requestCount])->runOnce(); + } + + public function prepareScenario(string $scenarioId, array $options = []): PreparedScenario + { + return match ($scenarioId) { + 'application_boot' => new PreparedScenario( + 'application_boot', + 'application_boot_constructs_framework', + 0, + function (): array { + $this->createKernel(); + + return [ + 'scenario_id' => 'application_boot', + 'status' => 'ok', + 'framework_constructed_in_subject' => true, + 'framework_prepared_outside_subject' => false, + ]; + }, + ), + 'http_static' => $this->warmHttpSubject('http_static', '/benchmark'), + 'http_parameterized' => $this->warmHttpSubject( + 'http_parameterized', + '/benchmark/' . (string) ($options['id'] ?? '123'), + ['parameters' => ['id' => (string) ($options['id'] ?? '123')]], + ), + 'http_middleware' => $this->middlewareSubject(), + 'http_not_found' => $this->warmHttpSubject('http_not_found', '/benchmark-missing'), + 'http_repeated_warm' => $this->repeatedWarmSubject((int) ($options['request_count'] ?? 25)), + default => throw new \InvalidArgumentException("Unknown comparator scenario '{$scenarioId}'."), + }; + } + + /** + * @param array $extra + */ + private function warmHttpSubject(string $scenarioId, string $path, array $extra = []): PreparedScenario { $kernel = $this->createKernel(); - $last = null; - for ($i = 0; $i < $requestCount; ++$i) { - $last = $this->handle($kernel, '/benchmark'); - } + return new PreparedScenario( + $scenarioId, + 'prepared_warm_http_request', + 1, + fn(): array => $this->withPreparedHttpMetadata($this->handle($kernel, $path) + $extra), + ); + } + + private function middlewareSubject(): PreparedScenario + { + $state = (object) ['order' => []]; + $kernel = $this->createKernel($state); + + return new PreparedScenario( + 'http_middleware', + 'prepared_warm_http_request', + 1, + function () use ($kernel, $state): array { + $state->order = []; + $result = $this->handle($kernel, '/benchmark-middleware'); + $result['middleware_order'] = $state->order; + + return $this->withPreparedHttpMetadata($result); + }, + ); + } + + private function repeatedWarmSubject(int $requestCount): PreparedScenario + { + $kernel = $this->createKernel(); + + return new PreparedScenario( + 'http_repeated_warm', + 'prepared_repeated_warm_requests', + 1, + function () use ($kernel, $requestCount): array { + $last = null; + + for ($i = 0; $i < max(1, $requestCount); ++$i) { + $last = $this->handle($kernel, '/benchmark'); + } + + $last ??= []; + $last['request_count'] = max(1, $requestCount); + $last['bootstrap_count'] = 1; + $last['framework_constructed_in_subject'] = false; + $last['normal_framework_path_executed'] = true; + $last['prepared_framework_instance_reused'] = true; + + return $last; + }, + ); + } - $last ??= []; - $last['request_count'] = $requestCount; - $last['bootstrap_count'] = 1; + /** + * @param array $result + * @return array + */ + private function withPreparedHttpMetadata(array $result): array + { + $result['framework_constructed_in_subject'] = false; + $result['normal_framework_path_executed'] = true; - return $last; + return $result; } private function createKernel(?object $middlewareState = null): HttpKernel diff --git a/benchmarks/composer.json b/benchmarks/composer.json index b97de14..7ff25d8 100644 --- a/benchmarks/composer.json +++ b/benchmarks/composer.json @@ -49,6 +49,8 @@ }, "scripts": { "bench:local": "@php vendor/bin/phpbench run --config=phpbench.json --report=aggregate", + "comparator:preflight": "@php bin/comparator-preflight.php", + "comparator:run": "@php bin/comparator-run.php", "capture-environment": "@php bin/capture-environment.php", "smoke": "@php bin/benchmark-smoke.php", "syntax": "@php bin/check-syntax.php", diff --git a/benchmarks/results/README.md b/benchmarks/results/README.md index 5d70de2..17dfe17 100644 --- a/benchmarks/results/README.md +++ b/benchmarks/results/README.md @@ -10,3 +10,28 @@ Local and disposable output belongs under ignored paths: - `.phpbench/` A reference baseline should be recorded only after the documented protocol is run in a controlled, low-noise PHP 8.4 environment with the matching OPcache/JIT policy and environment fingerprint. + +Controlled comparator execution writes this local structure when `benchmarks/bin/comparator-run.php` is used: + +```text +results/local/comparator-candidate/ + manifest.json + raw/ + -.json + normalized/ + -.json +``` + +`manifest.json` is the evidence index. It records source SHA and dirty state, matrix hash, execution order, subprocess command lines, fixture identity, comparator lock hash, execution environment identity/fingerprint, worker runtime identity, availability, raw result hashes, per-sample raw hashes, normalized result hashes, boot protocol metadata, and actual worker-order provenance. + +Each controlled comparator run requires a fresh output directory. The path must be absent or empty before execution starts; existing files or directories cause the runner to fail before new evidence is written, and stale output is never deleted implicitly. + +If preflight rejects the controlled lane on a fresh path, the output directory contains `preflight.json` and `manifest.json` only. Raw and normalized timing directories are not written for a rejected controlled run. + +Accepted comparator runs use one subprocess per measured sample. The aggregate raw scenario file embeds independently hashable raw sample records for every accepted sample. The raw files preserve batch durations for repeated-warm scenarios, while normalized files report per-operation latency and throughput using the recorded `operations_per_sample`. + +For `application_boot`, the manifest records one separate discarded application_boot worker before each measured boot worker, rotating round-robin scheduling across selected comparators, zero in-process warmups in measured boot workers, retained measured samples, and p50 as the primary central statistic. p50 is the primary central comparison statistic for application_boot. The discarded workers are excluded from statistics and are recorded separately from `raw_samples`, so normalized sample counts and statistics include only measured workers. + +The `application_boot` normalized output keeps p50 available while mean, p95, p99 and relative standard deviation remain visible as tail and noise evidence. No measured sample is removed as an outlier. The measured process is fresh and performs its first framework construction inside the measured subject; only host-level paths may have been primed by the separate discarded process. + +Candidate output from a dirty or uncommitted worktree is not canonical reference evidence. Canonical reference evidence must be regenerated from the exact committed implementation ref in the controlled PHP 8.4.25 lane described in the benchmark README. diff --git a/benchmarks/src/Comparator/ComparatorExecutionRunner.php b/benchmarks/src/Comparator/ComparatorExecutionRunner.php new file mode 100644 index 0000000..16d824f --- /dev/null +++ b/benchmarks/src/Comparator/ComparatorExecutionRunner.php @@ -0,0 +1,1021 @@ + $options + * @return array + */ + public function run(string $matrixPath, string $outputDir, array $options = []): array + { + $matrixPath = realpath($matrixPath) ?: $matrixPath; + $matrix = ComparatorMatrix::fromJsonFile($matrixPath); + $benchmarkRoot = dirname($matrixPath, 2); + $repositoryRoot = dirname($benchmarkRoot); + $outputDir = rtrim($outputDir, DIRECTORY_SEPARATOR); + $workerPath = (string) ($options['worker_path'] ?? ($benchmarkRoot . DIRECTORY_SEPARATOR . 'bin' . DIRECTORY_SEPARATOR . 'comparator-worker.php')); + + $this->ensureFreshOutputDirectory($outputDir); + $this->ensureDirectory($outputDir); + + $preflight = is_array($options['preflight'] ?? null) + ? $options['preflight'] + : ComparatorPreflight::current($repositoryRoot, $matrixPath); + + $this->writeJson($outputDir . DIRECTORY_SEPARATOR . 'preflight.json', $preflight); + + if (($preflight['status'] ?? 'mismatched') !== 'matched') { + $manifest = $this->manifest( + 'preflight_rejected', + $repositoryRoot, + $matrixPath, + $preflight, + max(1, (int) ($options['samples'] ?? 100)), + max(0, (int) ($options['warmups'] ?? 5)), + max(1, (int) ($options['request_count'] ?? 25)), + [], + [], + $workerPath, + [], + ); + $this->writeJson($outputDir . DIRECTORY_SEPARATOR . 'manifest.json', $manifest); + + return $manifest; + } + + $rawDir = $outputDir . DIRECTORY_SEPARATOR . 'raw'; + $normalizedDir = $outputDir . DIRECTORY_SEPARATOR . 'normalized'; + $this->ensureDirectory($rawDir); + $this->ensureDirectory($normalizedDir); + + $selectedComparators = $this->selectComparators($matrix, $options['comparators'] ?? null); + $selectedScenarios = $this->selectScenarios($matrix, $options['scenarios'] ?? null); + $samples = max(1, (int) ($options['samples'] ?? 100)); + $warmups = max(0, (int) ($options['warmups'] ?? 5)); + $requestCount = max(1, (int) ($options['request_count'] ?? 25)); + $matrixHash = (string) hash_file('sha256', $matrixPath); + $results = []; + $executionOrder = []; + $bootWorkerExecutionSequence = []; + $executionIndex = 0; + $overallStatus = 'completed'; + + if (in_array('application_boot', $selectedScenarios, true)) { + $bootRawResults = $this->runApplicationBootScenarioSamples( + $benchmarkRoot, + $matrixPath, + $workerPath, + $selectedComparators, + $samples, + $requestCount, + $preflight, + $bootWorkerExecutionSequence, + ); + + foreach ($selectedComparators as $comparatorId => $definition) { + if (!isset($bootRawResults[$comparatorId])) { + continue; + } + + ++$executionIndex; + $executionOrder[] = $comparatorId . ':application_boot'; + $result = $this->writeScenarioResult( + $outputDir, + 'raw/' . $comparatorId . '-application_boot.json', + 'normalized/' . $comparatorId . '-application_boot.json', + $bootRawResults[$comparatorId], + $definition, + $preflight, + $matrixHash, + $executionIndex, + ); + + if (($result['availability'] ?? null) === 'failed') { + $overallStatus = 'failed'; + } + + $results[] = $result; + } + } + + foreach ($selectedComparators as $comparatorId => $definition) { + foreach ($selectedScenarios as $scenarioId) { + if ($scenarioId === 'application_boot') { + continue; + } + + if (!in_array($scenarioId, $definition['scenarios'], true)) { + continue; + } + + ++$executionIndex; + $executionOrder[] = $comparatorId . ':' . $scenarioId; + $rawRelativePath = 'raw/' . $comparatorId . '-' . $scenarioId . '.json'; + $normalizedRelativePath = 'normalized/' . $comparatorId . '-' . $scenarioId . '.json'; + $raw = $this->runScenarioSamples($benchmarkRoot, $matrixPath, $workerPath, $comparatorId, $scenarioId, $samples, $warmups, $requestCount, $preflight); + $result = $this->writeScenarioResult( + $outputDir, + $rawRelativePath, + $normalizedRelativePath, + $raw, + $definition, + $preflight, + $matrixHash, + $executionIndex, + ); + + if (($result['availability'] ?? null) === 'failed') { + $overallStatus = 'failed'; + } + + $results[] = $result; + } + } + + $manifest = $this->manifest($overallStatus, $repositoryRoot, $matrixPath, $preflight, $samples, $warmups, $requestCount, $executionOrder, $results, $workerPath, $bootWorkerExecutionSequence); + $this->writeJson($outputDir . DIRECTORY_SEPARATOR . 'manifest.json', $manifest); + + return $manifest; + } + + /** + * @param array> $selectedComparators + * @param array $preflight + * @param list> $bootWorkerExecutionSequence + * @return array> + */ + private function runApplicationBootScenarioSamples( + string $benchmarkRoot, + string $matrixPath, + string $workerPath, + array $selectedComparators, + int $samples, + int $requestCount, + array $preflight, + array &$bootWorkerExecutionSequence, + ): array { + $comparatorIds = []; + $rawByComparator = []; + $terminalComparators = []; + + foreach ($selectedComparators as $comparatorId => $definition) { + if (!in_array('application_boot', $definition['scenarios'], true)) { + continue; + } + + $comparatorIds[] = $comparatorId; + $rawByComparator[$comparatorId] = $this->initialApplicationBootRaw($comparatorId, $requestCount); + } + + for ($sampleIndex = 1; $sampleIndex <= $samples; ++$sampleIndex) { + foreach ($this->rotatedComparatorIds($comparatorIds, $sampleIndex) as $comparatorId) { + if (($terminalComparators[$comparatorId] ?? false) === true) { + continue; + } + + $discardCommand = $this->workerCommand($workerPath, $matrixPath, $comparatorId, 'application_boot', 0, $requestCount, $sampleIndex); + $discardProcess = $this->runProcess($discardCommand, $benchmarkRoot); + $discardRaw = $this->decodeRawOutput($discardProcess['stdout'], $discardProcess['stderr'], $discardProcess['exit_code'], $comparatorId, 'application_boot'); + $discardRaw['exit_code'] = $discardProcess['exit_code']; + $discardEvidence = $this->workerExecutionEvidence(count($bootWorkerExecutionSequence) + 1, 'discarded', $comparatorId, 'application_boot', $sampleIndex, $discardRaw, $discardProcess['exit_code']); + $bootWorkerExecutionSequence[] = $discardEvidence; + $rawByComparator[$comparatorId]['discarded_worker_provenance'][] = $this->discardedWorkerProvenance($discardEvidence, $discardCommand, $discardRaw); + + $discardFailure = $this->bootWorkerFailure($discardRaw, $discardProcess['exit_code'], $preflight, 'discarded'); + + if ($discardFailure !== null) { + $rawByComparator[$comparatorId] = $this->failedApplicationBootRaw($comparatorId, [], max(1, $discardProcess['exit_code']), $discardFailure, $discardRaw, $rawByComparator[$comparatorId], $bootWorkerExecutionSequence); + $terminalComparators[$comparatorId] = true; + continue; + } + + if (($discardRaw['availability'] ?? null) === 'unavailable') { + if (count($rawByComparator[$comparatorId]['samples'] ?? []) > 0) { + $rawByComparator[$comparatorId] = $this->failedApplicationBootRaw( + $comparatorId, + [], + max(1, $discardProcess['exit_code']), + 'application_boot availability changed during controlled boot sampling after accepted measured sample', + $discardRaw, + $rawByComparator[$comparatorId], + $bootWorkerExecutionSequence, + ); + } else { + $discardRaw['commands'] = []; + $discardRaw['exit_code'] = 0; + $rawByComparator[$comparatorId] = $this->unavailableApplicationBootRaw($discardRaw, $rawByComparator[$comparatorId], $bootWorkerExecutionSequence); + } + $terminalComparators[$comparatorId] = true; + continue; + } + + $measuredCommand = $this->workerCommand($workerPath, $matrixPath, $comparatorId, 'application_boot', 0, $requestCount, $sampleIndex); + $measuredProcess = $this->runProcess($measuredCommand, $benchmarkRoot); + $measuredRaw = $this->decodeRawOutput($measuredProcess['stdout'], $measuredProcess['stderr'], $measuredProcess['exit_code'], $comparatorId, 'application_boot'); + $measuredRaw['exit_code'] = $measuredProcess['exit_code']; + $measuredEvidence = $this->workerExecutionEvidence(count($bootWorkerExecutionSequence) + 1, 'measured', $comparatorId, 'application_boot', $sampleIndex, $measuredRaw, $measuredProcess['exit_code']); + $bootWorkerExecutionSequence[] = $measuredEvidence; + + $measuredFailure = $this->bootWorkerFailure($measuredRaw, $measuredProcess['exit_code'], $preflight, 'measured'); + + if ($measuredFailure !== null) { + $rawByComparator[$comparatorId] = $this->failedApplicationBootRaw($comparatorId, [$measuredCommand], max(1, $measuredProcess['exit_code']), $measuredFailure, $measuredRaw, $rawByComparator[$comparatorId], $bootWorkerExecutionSequence); + $terminalComparators[$comparatorId] = true; + continue; + } + + if (($measuredRaw['availability'] ?? null) === 'unavailable') { + $rawByComparator[$comparatorId] = $this->failedApplicationBootRaw( + $comparatorId, + [], + max(1, $measuredProcess['exit_code']), + 'application_boot inconsistent availability within measured boot slot after discarded worker was available', + $measuredRaw, + $rawByComparator[$comparatorId], + $bootWorkerExecutionSequence, + ); + $terminalComparators[$comparatorId] = true; + continue; + } + + $this->recordMeasuredApplicationBootSample($rawByComparator[$comparatorId], $measuredRaw, $measuredCommand); + } + } + + foreach ($rawByComparator as $comparatorId => $raw) { + if (($raw['availability'] ?? null) === 'available') { + $rawByComparator[$comparatorId]['sample_count'] = count($raw['samples']); + $rawByComparator[$comparatorId]['boot_worker_execution_sequence'] = $this->filterBootWorkerSequence($bootWorkerExecutionSequence, $comparatorId); + } + } + + return $rawByComparator; + } + + /** + * @return array + */ + private function initialApplicationBootRaw(string $comparatorId, int $requestCount): array + { + return [ + 'schema_version' => ComparatorScenarioExecutor::SCHEMA_VERSION, + 'comparator_id' => $comparatorId, + 'scenario_id' => 'application_boot', + 'availability' => 'available', + 'availability_status' => 'available', + 'sample_count' => 0, + 'samples' => [], + 'unit' => 'microseconds', + 'memory' => [], + 'warmups' => 0, + 'request_count' => $requestCount, + 'operations_per_sample' => 1, + 'samples_metadata' => [], + 'raw_samples' => [], + 'commands' => [], + 'exit_code' => 0, + 'boot_protocol' => $this->bootProtocol(), + 'discarded_worker_provenance' => [], + ]; + } + + /** + * @param list $comparatorIds + * @return list + */ + private function rotatedComparatorIds(array $comparatorIds, int $sampleIndex): array + { + $count = count($comparatorIds); + + if ($count === 0) { + return []; + } + + $start = ($sampleIndex - 1) % $count; + + return array_merge(array_slice($comparatorIds, $start), array_slice($comparatorIds, 0, $start)); + } + + /** + * @param array $raw + * @param array $preflight + */ + private function bootWorkerFailure(array $raw, int $exitCode, array $preflight, string $role): ?string + { + $availability = $raw['availability'] ?? null; + $expectedWorkerIdentityHash = $preflight['worker_environment_identity']['hash'] ?? null; + $workerIdentityHash = $raw['worker_environment_identity']['hash'] ?? null; + + if ( + in_array($availability, ['available', 'unavailable'], true) + && $expectedWorkerIdentityHash !== null + && $workerIdentityHash !== $expectedWorkerIdentityHash + ) { + return $role === 'discarded' + ? 'discarded boot worker runtime identity did not match accepted preflight lane' + : 'measurement worker runtime identity did not match accepted preflight lane'; + } + + if ($availability !== 'available' && $availability !== 'unavailable') { + return $role === 'discarded' + ? (string) ($raw['reason'] ?? 'discarded boot worker failed before measured sample') + : (string) ($raw['reason'] ?? 'comparator worker failed'); + } + + if ($exitCode !== 0) { + return $role === 'discarded' + ? (string) ($raw['reason'] ?? 'discarded boot worker failed before measured sample') + : (string) ($raw['reason'] ?? 'comparator worker failed'); + } + + return null; + } + + /** + * @param array $raw + * @return array + */ + private function workerExecutionEvidence( + int $sequenceIndex, + string $role, + string $comparatorId, + string $scenarioId, + int $sampleIndex, + array $raw, + int $exitCode, + ): array { + return [ + 'sequence_index' => $sequenceIndex, + 'role' => $role, + 'comparator_id' => $comparatorId, + 'scenario_id' => $scenarioId, + 'sample_index' => $sampleIndex, + 'excluded_from_statistics' => $role === 'discarded', + 'pid' => $raw['process']['pid'] ?? null, + 'worker_environment_identity_hash' => $raw['worker_environment_identity']['hash'] ?? null, + 'exit_code' => $exitCode, + 'availability' => $raw['availability'] ?? 'failed', + 'availability_status' => $raw['availability_status'] ?? null, + 'reason' => $raw['reason'] ?? null, + ]; + } + + /** + * @param array $evidence + * @param list $command + * @param array $raw + * @return array + */ + private function discardedWorkerProvenance(array $evidence, array $command, array $raw): array + { + return [ + 'sequence_index' => $evidence['sequence_index'], + 'role' => 'discarded', + 'comparator_id' => $evidence['comparator_id'], + 'scenario_id' => $evidence['scenario_id'], + 'sample_index' => $evidence['sample_index'], + 'pid' => $evidence['pid'], + 'worker_environment_identity_hash' => $evidence['worker_environment_identity_hash'], + 'exit_code' => $evidence['exit_code'], + 'availability' => $evidence['availability'], + 'availability_status' => $evidence['availability_status'], + 'duration_microseconds' => isset($raw['samples'][0]) ? (float) $raw['samples'][0] : null, + 'excluded_from_statistics' => true, + 'command' => $command, + 'reason' => $evidence['reason'], + ]; + } + + /** + * @param list> $commands + * @param array $raw + * @param array $previousRaw + * @param list> $bootWorkerExecutionSequence + * @return array + */ + private function failedApplicationBootRaw( + string $comparatorId, + array $commands, + int $exitCode, + string $reason, + array $raw, + array $previousRaw, + array $bootWorkerExecutionSequence, + ): array { + $failed = $previousRaw; + $failed['schema_version'] = ComparatorScenarioExecutor::SCHEMA_VERSION; + $failed['comparator_id'] = $comparatorId; + $failed['scenario_id'] = 'application_boot'; + $failed['availability'] = 'failed'; + $failed['availability_status'] = 'failed'; + $failed['reason'] = $reason; + $failed['sample_count'] = count(is_array($previousRaw['samples'] ?? null) ? $previousRaw['samples'] : []); + $failed['samples'] = is_array($previousRaw['samples'] ?? null) ? array_values($previousRaw['samples']) : []; + $failed['unit'] = 'microseconds'; + $failed['memory'] = is_array($previousRaw['memory'] ?? null) ? $previousRaw['memory'] : []; + $failed['warmups'] = 0; + $failed['request_count'] = $previousRaw['request_count'] ?? 1; + $failed['operations_per_sample'] = 1; + $failed['worker_environment_identity'] = $previousRaw['worker_environment_identity'] ?? ($raw['worker_environment_identity'] ?? null); + $failed['dependency_context'] = $previousRaw['dependency_context'] ?? ($raw['dependency_context'] ?? null); + $failed['samples_metadata'] = is_array($previousRaw['samples_metadata'] ?? null) ? $previousRaw['samples_metadata'] : []; + $acceptedCommands = is_array($previousRaw['commands'] ?? null) ? $previousRaw['commands'] : []; + $failed['commands'] = $acceptedCommands === [] ? $commands : $acceptedCommands; + $failed['raw_samples'] = is_array($previousRaw['raw_samples'] ?? null) ? $previousRaw['raw_samples'] : []; + $failed['exit_code'] = $exitCode; + $failed['boot_protocol'] = $this->bootProtocol(); + $failed['discarded_worker_provenance'] = $previousRaw['discarded_worker_provenance'] ?? []; + $failed['boot_worker_execution_sequence'] = $this->filterBootWorkerSequence($bootWorkerExecutionSequence, $comparatorId); + + return $failed; + } + + /** + * @param array $raw + * @param array $previousRaw + * @param list> $bootWorkerExecutionSequence + * @return array + */ + private function unavailableApplicationBootRaw(array $raw, array $previousRaw, array $bootWorkerExecutionSequence): array + { + $raw['boot_protocol'] = $this->bootProtocol(); + $raw['raw_samples'] = $raw['raw_samples'] ?? []; + $raw['discarded_worker_provenance'] = $previousRaw['discarded_worker_provenance'] ?? []; + $raw['boot_worker_execution_sequence'] = $this->filterBootWorkerSequence($bootWorkerExecutionSequence, (string) ($raw['comparator_id'] ?? '')); + + return $raw; + } + + /** + * @param array $aggregate + * @param array $raw + * @param list $command + */ + private function recordMeasuredApplicationBootSample(array &$aggregate, array $raw, array $command): void + { + $sampleValue = (float) (($raw['samples'][0] ?? null) ?? 0.0); + $sampleRecord = $raw; + $rawSampleHash = $this->hashJson($sampleRecord); + $aggregate['samples'][] = $sampleValue; + $aggregate['timing_boundary'] = $raw['subject_metadata']['timing_boundary'] ?? ($raw['timing_boundary'] ?? null); + $aggregate['prepared_framework_instance_count'] = $raw['subject_metadata']['prepared_framework_instance_count'] ?? ($raw['prepared_framework_instance_count'] ?? null); + $aggregate['worker_environment_identity'] = $raw['worker_environment_identity'] ?? ($aggregate['worker_environment_identity'] ?? null); + $aggregate['dependency_context'] = $raw['dependency_context'] ?? ($aggregate['dependency_context'] ?? null); + $aggregate['commands'][] = $command; + $aggregate['raw_samples'][] = [ + 'sample_index' => $raw['sample_index'] ?? (count($aggregate['samples']) + 1), + 'sha256' => $rawSampleHash, + 'record' => $sampleRecord, + ]; + $aggregate['samples_metadata'][] = [ + 'sample_index' => $raw['sample_index'] ?? count($aggregate['samples']), + 'pid' => $raw['process']['pid'] ?? null, + 'environment_identity_hash' => $raw['worker_environment_identity']['hash'] ?? null, + 'worker_environment_identity_hash' => $raw['worker_environment_identity']['hash'] ?? null, + 'raw_sample_sha256' => $rawSampleHash, + 'duration_microseconds' => $sampleValue, + ]; + $aggregate['last_result'] = $raw['last_result'] ?? ($aggregate['last_result'] ?? []); + + foreach (is_array($raw['memory'] ?? null) ? $raw['memory'] : [] as $key => $value) { + $aggregate['memory'][$key] = max((int) ($aggregate['memory'][$key] ?? 0), (int) $value); + } + } + + /** + * @param list> $bootWorkerExecutionSequence + * @return list> + */ + private function filterBootWorkerSequence(array $bootWorkerExecutionSequence, string $comparatorId): array + { + return array_values(array_filter( + $bootWorkerExecutionSequence, + static fn(array $entry): bool => ($entry['comparator_id'] ?? null) === $comparatorId, + )); + } + + + /** + * @param array $preflight + * @return array + */ + private function runScenarioSamples( + string $benchmarkRoot, + string $matrixPath, + string $workerPath, + string $comparatorId, + string $scenarioId, + int $samples, + int $warmups, + int $requestCount, + array $preflight, + ): array { + $sampleValues = []; + $samplesMetadata = []; + $commands = []; + $rawSamples = []; + $memory = []; + $lastResult = []; + $subjectMetadata = []; + $workerEnvironmentIdentity = null; + $dependencyContext = null; + $expectedWorkerIdentityHash = $preflight['worker_environment_identity']['hash'] ?? null; + + for ($sampleIndex = 1; $sampleIndex <= $samples; ++$sampleIndex) { + $command = $this->workerCommand($workerPath, $matrixPath, $comparatorId, $scenarioId, $warmups, $requestCount, $sampleIndex); + $commands[] = $command; + $processResult = $this->runProcess($command, $benchmarkRoot); + $raw = $this->decodeRawOutput($processResult['stdout'], $processResult['stderr'], $processResult['exit_code'], $comparatorId, $scenarioId); + $raw['exit_code'] = $processResult['exit_code']; + $availability = $raw['availability'] ?? null; + $workerIdentityHash = $raw['worker_environment_identity']['hash'] ?? null; + + if ( + in_array($availability, ['available', 'unavailable'], true) + && $expectedWorkerIdentityHash !== null + && $workerIdentityHash !== $expectedWorkerIdentityHash + ) { + return $this->failedRaw( + $comparatorId, + $scenarioId, + $commands, + 1, + 'measurement worker runtime identity did not match accepted preflight lane', + $raw, + ); + } + + if ($availability === 'unavailable') { + $raw['commands'] = $commands; + $raw['exit_code'] = 0; + + return $raw; + } + + if ($availability !== 'available' || $processResult['exit_code'] !== 0) { + return $this->failedRaw($comparatorId, $scenarioId, $commands, $processResult['exit_code'], (string) ($raw['reason'] ?? 'comparator worker failed'), $raw); + } + + $sampleValues[] = (float) (($raw['samples'][0] ?? null) ?? 0.0); + $workerEnvironmentIdentity = $raw['worker_environment_identity'] ?? $workerEnvironmentIdentity; + $dependencyContext = $raw['dependency_context'] ?? $dependencyContext; + $subjectMetadata = $raw['subject_metadata'] ?? $subjectMetadata; + $lastResult = $raw['last_result'] ?? $lastResult; + $sampleRecord = $raw; + $rawSampleHash = $this->hashJson($sampleRecord); + $rawSamples[] = [ + 'sample_index' => $sampleIndex, + 'sha256' => $rawSampleHash, + 'record' => $sampleRecord, + ]; + + foreach (is_array($raw['memory'] ?? null) ? $raw['memory'] : [] as $key => $value) { + $memory[$key] = max((int) ($memory[$key] ?? 0), (int) $value); + } + + $samplesMetadata[] = [ + 'sample_index' => $sampleIndex, + 'pid' => $raw['process']['pid'] ?? null, + 'environment_identity_hash' => $workerIdentityHash, + 'worker_environment_identity_hash' => $workerIdentityHash, + 'raw_sample_sha256' => $rawSampleHash, + 'duration_microseconds' => (float) end($sampleValues), + ]; + } + + return [ + 'schema_version' => ComparatorScenarioExecutor::SCHEMA_VERSION, + 'comparator_id' => $comparatorId, + 'scenario_id' => $scenarioId, + 'availability' => 'available', + 'availability_status' => 'available', + 'timing_boundary' => $subjectMetadata['timing_boundary'] ?? null, + 'prepared_framework_instance_count' => $subjectMetadata['prepared_framework_instance_count'] ?? null, + 'sample_count' => count($sampleValues), + 'samples' => $sampleValues, + 'unit' => 'microseconds', + 'memory' => $memory, + 'warmups' => $warmups, + 'request_count' => $requestCount, + 'operations_per_sample' => $scenarioId === 'http_repeated_warm' ? $requestCount : 1, + 'worker_environment_identity' => $workerEnvironmentIdentity, + 'dependency_context' => $dependencyContext, + 'samples_metadata' => $samplesMetadata, + 'raw_samples' => $rawSamples, + 'commands' => $commands, + 'exit_code' => 0, + 'last_result' => $lastResult, + ]; + } + + /** + * @param array $raw + * @param array $definition + * @param array $preflight + * @return array + */ + private function writeScenarioResult( + string $outputDir, + string $rawRelativePath, + string $normalizedRelativePath, + array $raw, + array $definition, + array $preflight, + string $matrixHash, + int $executionIndex, + ): array { + $this->writeJson($outputDir . DIRECTORY_SEPARATOR . $rawRelativePath, $raw); + $rawHash = (string) hash_file('sha256', $outputDir . DIRECTORY_SEPARATOR . $rawRelativePath); + + $normalized = $this->normalizeRawResult($raw, $definition, $preflight, $matrixHash, $rawHash); + $this->writeJson($outputDir . DIRECTORY_SEPARATOR . $normalizedRelativePath, $normalized); + $normalizedHash = (string) hash_file('sha256', $outputDir . DIRECTORY_SEPARATOR . $normalizedRelativePath); + + $result = [ + 'execution_index' => $executionIndex, + 'comparator_id' => $definition['id'], + 'framework_name' => $definition['name'], + 'framework_version' => $definition['framework_version'], + 'scenario_id' => $raw['scenario_id'] ?? null, + 'implementation_model' => $definition['implementation_model'], + 'availability' => $raw['availability'] ?? 'failed', + 'reason' => $raw['reason'] ?? null, + 'timing_boundary' => $raw['timing_boundary'] ?? null, + 'sample_count' => (int) ($raw['sample_count'] ?? 0), + 'operations_per_sample' => (int) ($raw['operations_per_sample'] ?? 1), + 'samples_metadata' => $raw['samples_metadata'] ?? [], + 'raw_samples' => $raw['raw_samples'] ?? [], + 'execution_environment_identity' => $this->executionEnvironmentIdentity($preflight), + 'execution_environment_fingerprint' => $this->executionEnvironmentIdentity($preflight)['hash'], + 'worker_environment_identity' => $raw['worker_environment_identity'] ?? null, + 'dependency_context' => $raw['dependency_context'] ?? null, + 'command' => $raw['commands'][0] ?? [], + 'commands' => $raw['commands'] ?? [], + 'exit_code' => (int) ($raw['exit_code'] ?? 1), + 'raw_result' => [ + 'path' => $rawRelativePath, + 'sha256' => $rawHash, + ], + 'normalized_result' => [ + 'path' => $normalizedRelativePath, + 'sha256' => $normalizedHash, + ], + 'fixture_identity_hash' => $normalized['fixture_identity_hash'] ?? null, + 'comparator_lock_sha256' => $definition['lock_sha256'] ?? null, + ]; + + foreach (['boot_protocol', 'boot_worker_execution_sequence', 'discarded_worker_provenance'] as $key) { + if (array_key_exists($key, $raw)) { + $result[$key] = $raw[$key]; + } + } + + return $result; + } + + /** + * @param list> $commands + * @param array $raw + * @return array + */ + private function failedRaw(string $comparatorId, string $scenarioId, array $commands, int $exitCode, string $reason, array $raw = []): array + { + return [ + 'schema_version' => ComparatorScenarioExecutor::SCHEMA_VERSION, + 'comparator_id' => $comparatorId, + 'scenario_id' => $scenarioId, + 'availability' => 'failed', + 'availability_status' => 'failed', + 'reason' => $reason, + 'sample_count' => 0, + 'samples' => [], + 'unit' => 'microseconds', + 'worker_environment_identity' => $raw['worker_environment_identity'] ?? null, + 'dependency_context' => $raw['dependency_context'] ?? null, + 'samples_metadata' => [], + 'commands' => $commands, + 'exit_code' => $exitCode, + ]; + } + + /** + * @param array $preflight + * @param list $executionOrder + * @param list> $results + * @param list> $bootWorkerExecutionSequence + * @return array + */ + private function manifest( + string $status, + string $repositoryRoot, + string $matrixPath, + array $preflight, + int $samples, + int $warmups, + int $requestCount, + array $executionOrder, + array $results, + string $workerPath, + array $bootWorkerExecutionSequence, + ): array { + return [ + 'schema_version' => self::MANIFEST_SCHEMA_VERSION, + 'status' => $status, + 'generated_at' => (new DateTimeImmutable('now', new DateTimeZone('UTC')))->format(DATE_ATOM), + 'source' => $preflight['environment']['source'] ?? null, + 'matrix' => [ + 'path' => $this->relativePath($repositoryRoot, $matrixPath), + 'sha256' => hash_file('sha256', $matrixPath), + ], + 'preflight' => $preflight, + 'environment_identity' => $this->executionEnvironmentIdentity($preflight), + 'execution_environment_identity' => $this->executionEnvironmentIdentity($preflight), + 'execution_environment_fingerprint' => $this->executionEnvironmentIdentity($preflight)['hash'], + 'worker_environment_identity' => $preflight['worker_environment_identity'] ?? null, + 'process_isolation' => [ + 'model' => 'subprocess_per_measured_sample', + 'worker' => $this->relativePath($repositoryRoot, $workerPath), + 'php_binary' => $this->phpBinary, + ], + 'samples' => $samples, + 'warmups' => $warmups, + 'request_count' => $requestCount, + 'boot_protocol' => $this->bootProtocol(), + 'boot_worker_execution_sequence' => $bootWorkerExecutionSequence, + 'execution_order' => $executionOrder, + 'results' => $results, + ]; + } + + /** + * @param mixed $requested + * @return array> + */ + private function selectComparators(ComparatorMatrix $matrix, mixed $requested): array + { + $comparators = $matrix->comparators(); + + if ($requested === null || $requested === [] || $requested === 'all') { + return $comparators; + } + + $ids = is_array($requested) ? $requested : explode(',', (string) $requested); + $selected = []; + + foreach ($ids as $id) { + $id = trim((string) $id); + + if ($id === '') { + continue; + } + + if (!isset($comparators[$id])) { + throw new RuntimeException("Unknown comparator '{$id}'."); + } + + $selected[$id] = $comparators[$id]; + } + + return $selected; + } + + /** + * @param mixed $requested + * @return list + */ + private function selectScenarios(ComparatorMatrix $matrix, mixed $requested): array + { + $scenarios = array_keys($matrix->commonScenarios()); + + if ($requested === null || $requested === [] || $requested === 'all') { + return $scenarios; + } + + $requestedIds = is_array($requested) ? $requested : explode(',', (string) $requested); + $selected = []; + + foreach ($scenarios as $scenarioId) { + if (in_array($scenarioId, $requestedIds, true)) { + $selected[] = $scenarioId; + } + } + + foreach ($requestedIds as $scenarioId) { + if (!in_array($scenarioId, $scenarios, true)) { + throw new RuntimeException("Unknown scenario '{$scenarioId}'."); + } + } + + return $selected; + } + + /** + * @return list + */ + private function workerCommand(string $workerPath, string $matrixPath, string $comparatorId, string $scenarioId, int $warmups, int $requestCount, int $sampleIndex): array + { + return [ + $this->phpBinary, + $workerPath, + '--matrix=' . $matrixPath, + '--comparator=' . $comparatorId, + '--scenario=' . $scenarioId, + '--warmups=' . $warmups, + '--request-count=' . $requestCount, + '--sample-index=' . $sampleIndex, + ]; + } + + /** + * @param list $command + * @return array{stdout: string, stderr: string, exit_code: int} + */ + private function runProcess(array $command, string $cwd): array + { + $process = proc_open($command, [1 => ['pipe', 'w'], 2 => ['pipe', 'w']], $pipes, $cwd); + + if (!is_resource($process)) { + throw new RuntimeException('Unable to start comparator worker process.'); + } + + $stdout = (string) stream_get_contents($pipes[1]); + $stderr = (string) stream_get_contents($pipes[2]); + fclose($pipes[1]); + fclose($pipes[2]); + + return [ + 'stdout' => $stdout, + 'stderr' => $stderr, + 'exit_code' => proc_close($process), + ]; + } + + /** + * @return array + */ + private function decodeRawOutput(string $stdout, string $stderr, int $exitCode, string $comparatorId, string $scenarioId): array + { + try { + $decoded = json_decode($stdout, true, flags: JSON_THROW_ON_ERROR); + } catch (\JsonException) { + $decoded = null; + } + + if (is_array($decoded)) { + return $decoded; + } + + return [ + 'schema_version' => ComparatorScenarioExecutor::SCHEMA_VERSION, + 'comparator_id' => $comparatorId, + 'scenario_id' => $scenarioId, + 'availability' => 'failed', + 'availability_status' => 'failed', + 'reason' => trim($stderr) !== '' ? trim($stderr) : 'comparator worker failed without JSON output', + 'exit_code' => $exitCode, + 'sample_count' => 0, + 'samples' => [], + 'unit' => 'microseconds', + ]; + } + + /** + * @param array $raw + * @param array $definition + * @param array $preflight + * @return array + */ + private function normalizeRawResult(array $raw, array $definition, array $preflight, string $matrixHash, string $rawHash): array + { + $fixtureIdentity = FixtureIdentity::fromArray([ + 'comparator_id' => $definition['id'], + 'framework_name' => $definition['name'], + 'framework_version' => $definition['framework_version'], + 'fixture_version' => $definition['fixture_version'] ?? null, + 'lock_hash' => $definition['lock_sha256'] ?? null, + 'configuration' => $definition['configuration'] ?? [], + ]); + $metadata = [ + 'id' => $definition['id'], + 'name' => $definition['name'], + 'framework_version' => $definition['framework_version'], + 'fixture_identity' => $fixtureIdentity, + 'execution_environment_identity' => $this->executionEnvironmentIdentity($preflight), + 'execution_environment_fingerprint' => $this->executionEnvironmentIdentity($preflight)['hash'], + 'worker_environment_identity' => $raw['worker_environment_identity'] ?? ($preflight['worker_environment_identity'] ?? null), + 'source_evolvephp_sha' => $preflight['environment']['source']['git_sha'] ?? null, + 'source_dirty' => $preflight['environment']['source']['dirty'] ?? null, + 'scenario_id' => $raw['scenario_id'] ?? null, + 'availability' => $raw['availability'] ?? 'failed', + 'availability_status' => $raw['availability_status'] ?? null, + 'availability_reason' => $raw['reason'] ?? null, + 'failure_reason' => $raw['reason'] ?? null, + 'implementation_model' => $definition['implementation_model'] ?? null, + 'matrix_sha256' => $matrixHash, + 'comparator_lock_sha256' => $definition['lock_sha256'] ?? null, + 'raw_evidence_sha256' => $rawHash, + 'operations_per_sample' => $raw['operations_per_sample'] ?? 1, + ]; + + if (($raw['availability'] ?? null) === 'unavailable') { + return ComparatorResultNormalizer::unavailable($metadata); + } + + if (($raw['availability'] ?? null) !== 'available') { + return ComparatorResultNormalizer::failed($metadata); + } + + $normalized = ComparatorResultNormalizer::normalizeRawScenario($raw, [ + 'source' => $preflight['environment']['source'] ?? [], + 'fingerprint' => $metadata['execution_environment_identity'], + ]); + + return ComparatorResultNormalizer::withComparatorMetadata($normalized, $metadata); + } + + /** + * @param array $preflight + * @return array{hash: string, fields: array} + */ + private function executionEnvironmentIdentity(array $preflight): array + { + if (isset($preflight['execution_environment_identity']) && is_array($preflight['execution_environment_identity'])) { + return $preflight['execution_environment_identity']; + } + + return ExecutionEnvironmentFingerprint::fromEnvironment(is_array($preflight['environment'] ?? null) ? $preflight['environment'] : []); + } + + /** + * @return array + */ + private function bootProtocol(): array + { + return [ + 'discarded_worker_processes_per_measured_sample' => 1, + 'sample_order' => 'rotating_round_robin', + 'measured_worker_in_process_warmups' => 0, + 'outlier_policy' => 'retain_all_measured_samples', + 'primary_central_statistic' => 'p50', + ]; + } + + private function ensureDirectory(string $path): void + { + if (!is_dir($path) && !mkdir($path, 0777, true) && !is_dir($path)) { + throw new RuntimeException("Unable to create directory: {$path}"); + } + } + + private function ensureFreshOutputDirectory(string $path): void + { + if (!is_dir($path)) { + return; + } + + $files = array_diff(scandir($path) ?: [], ['.', '..']); + + if ($files !== []) { + throw new RuntimeException('Comparator output directory must be absent or empty before a run begins.'); + } + } + + /** + * @param array $data + */ + private function writeJson(string $path, array $data): void + { + $this->ensureDirectory(dirname($path)); + file_put_contents($path, json_encode($data, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR) . PHP_EOL); + } + + /** + * @param array $data + */ + private function hashJson(array $data): string + { + return hash('sha256', json_encode($data, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR) . PHP_EOL); + } + + private function relativePath(string $root, string $path): string + { + $root = rtrim(str_replace('\\', '/', $root), '/'); + $path = str_replace('\\', '/', $path); + + return str_starts_with($path, $root . '/') ? substr($path, strlen($root) + 1) : $path; + } +} diff --git a/benchmarks/src/Comparator/ComparatorMatrix.php b/benchmarks/src/Comparator/ComparatorMatrix.php index 54269cf..ed51baa 100644 --- a/benchmarks/src/Comparator/ComparatorMatrix.php +++ b/benchmarks/src/Comparator/ComparatorMatrix.php @@ -19,7 +19,6 @@ final class ComparatorMatrix */ private function __construct( private readonly array $data, - private readonly string $baseDir = '', ) {} public static function fromJsonFile(string $path): self @@ -49,7 +48,7 @@ public static function fromArray(array $data, string $baseDir = ''): self // Validate schema version if (($data['schema_version'] ?? null) !== self::SCHEMA_VERSION) { throw new ComparatorMatrixException( - 'Invalid schema version. Expected: ' . self::SCHEMA_VERSION . ', Got: ' . ($data['schema_version'] ?? 'null') + 'Invalid schema version. Expected: ' . self::SCHEMA_VERSION . ', Got: ' . ($data['schema_version'] ?? 'null'), ); } @@ -132,27 +131,27 @@ public static function fromArray(array $data, string $baseDir = ''): self $fixturePath = $baseDir . DIRECTORY_SEPARATOR . trim((string) $comparator['fixture_path'], '/\\'); if (!is_dir($fixturePath)) { throw new ComparatorMatrixException( - "Fixture directory not found for comparator '{$id}': {$fixturePath}" + "Fixture directory not found for comparator '{$id}': {$fixturePath}", ); } $bootstrapPath = $baseDir . DIRECTORY_SEPARATOR . trim((string) $comparator['fixture_bootstrap'], '/\\'); if (!is_file($bootstrapPath)) { throw new ComparatorMatrixException( - "Fixture bootstrap not found for comparator '{$id}': {$bootstrapPath}" + "Fixture bootstrap not found for comparator '{$id}': {$bootstrapPath}", ); } $lockPath = $baseDir . DIRECTORY_SEPARATOR . trim((string) $comparator['lock_path'], '/\\'); if (!is_file($lockPath)) { throw new ComparatorMatrixException( - "Lockfile not found for comparator '{$id}': {$lockPath}" + "Lockfile not found for comparator '{$id}': {$lockPath}", ); } } // Validate scenarios - $scenarios = $comparator['scenarios'] ?? []; + $scenarios = $comparator['scenarios']; if (!is_array($scenarios)) { throw new ComparatorMatrixException("Scenarios for comparator '{$id}' must be an array"); } @@ -164,7 +163,7 @@ public static function fromArray(array $data, string $baseDir = ''): self } } - return new self($data, $baseDir); + return new self($data); } /** @@ -178,7 +177,7 @@ public function comparators(): array // Ensure stable ordering usort($comparators, static fn(array $a, array $b): int => strcmp( (string) $a['id'], - (string) $b['id'] + (string) $b['id'], )); foreach ($comparators as $comparator) { diff --git a/benchmarks/src/Comparator/ComparatorMatrixException.php b/benchmarks/src/Comparator/ComparatorMatrixException.php index 2cbffbe..70136cc 100644 --- a/benchmarks/src/Comparator/ComparatorMatrixException.php +++ b/benchmarks/src/Comparator/ComparatorMatrixException.php @@ -6,6 +6,4 @@ use InvalidArgumentException; -final class ComparatorMatrixException extends InvalidArgumentException -{ -} +final class ComparatorMatrixException extends InvalidArgumentException {} diff --git a/benchmarks/src/Comparator/ComparatorPreflight.php b/benchmarks/src/Comparator/ComparatorPreflight.php new file mode 100644 index 0000000..16862c3 --- /dev/null +++ b/benchmarks/src/Comparator/ComparatorPreflight.php @@ -0,0 +1,78 @@ + + */ + public static function current(string $repositoryRoot, string $matrixPath): array + { + return self::fromEnvironment(BenchmarkEnvironment::capture($repositoryRoot), $matrixPath, self::canonicalRequirements()); + } + + /** + * @param array $environment + * @param array $requirements + * @return array + */ + public static function fromEnvironment(array $environment, string $matrixPath, array $requirements): array + { + $runtimeIdentity = ComparatorRuntimeIdentity::fromCapturedEnvironment($environment); + $executionEnvironmentIdentity = ExecutionEnvironmentFingerprint::fromEnvironment($environment); + $mismatches = []; + + if (($runtimeIdentity['runtime']['php_version'] ?? null) !== $requirements['php_version']) { + $mismatches[] = 'PHP version is not exactly ' . $requirements['php_version'] . '.'; + } + + if (($runtimeIdentity['opcache']['enabled'] ?? null) !== $requirements['opcache_cli_enabled']) { + $mismatches[] = 'OPcache is not enabled for CLI.'; + } + + if (($runtimeIdentity['jit']['enabled'] ?? null) !== $requirements['jit_enabled']) { + $mismatches[] = 'JIT is not disabled.'; + } + + if (($runtimeIdentity['phalcon']['version'] ?? null) !== $requirements['phalcon_extension_version']) { + $mismatches[] = 'ext-phalcon ' . $requirements['phalcon_extension_version'] . ' is not loaded.'; + } + + return [ + 'schema_version' => self::SCHEMA_VERSION, + 'status' => $mismatches === [] ? 'matched' : 'mismatched', + 'canonical_requirements' => $requirements, + 'mismatches' => $mismatches, + 'environment' => $environment, + 'execution_environment_identity' => $executionEnvironmentIdentity, + 'execution_environment_fingerprint' => $executionEnvironmentIdentity['hash'], + 'worker_environment_identity' => $runtimeIdentity, + 'matrix' => [ + 'path' => 'benchmarks/comparators/matrix.json', + 'sha256' => is_file($matrixPath) ? hash_file('sha256', $matrixPath) : null, + ], + ]; + } + + /** + * @return array + */ + public static function canonicalRequirements(): array + { + return [ + 'php_version' => '8.4.25', + 'opcache_cli_enabled' => true, + 'jit_enabled' => false, + 'phalcon_extension_version' => '5.20.3', + 'same_extension_lane_for_all_comparator_processes' => true, + ]; + } +} diff --git a/benchmarks/src/Comparator/ComparatorResultNormalizer.php b/benchmarks/src/Comparator/ComparatorResultNormalizer.php index 5f82ffa..70b1731 100644 --- a/benchmarks/src/Comparator/ComparatorResultNormalizer.php +++ b/benchmarks/src/Comparator/ComparatorResultNormalizer.php @@ -40,7 +40,12 @@ public static function withComparatorMetadata(array $normalizedResult, array $co 'fixture_identity_hash' => $comparatorMetadata['fixture_identity_hash'] ?? ($comparatorMetadata['fixture_identity']['hash'] ?? null), 'execution_environment_identity' => $comparatorMetadata['execution_environment_identity'] ?? null, 'execution_environment_fingerprint' => $comparatorMetadata['execution_environment_fingerprint'] ?? ($comparatorMetadata['execution_environment_identity']['hash'] ?? null), + 'worker_environment_identity' => $comparatorMetadata['worker_environment_identity'] ?? null, 'source_evolvephp_sha' => $comparatorMetadata['source_evolvephp_sha'] ?? ($normalizedResult['source_sha'] ?? null), + 'source_dirty' => $comparatorMetadata['source_dirty'] ?? null, + 'matrix_sha256' => $comparatorMetadata['matrix_sha256'] ?? null, + 'comparator_lock_sha256' => $comparatorMetadata['comparator_lock_sha256'] ?? null, + 'raw_evidence_sha256' => $comparatorMetadata['raw_evidence_sha256'] ?? null, 'scenario_id' => $comparatorMetadata['scenario_id'] ?? null, 'availability' => $comparatorMetadata['availability'] ?? 'available', 'availability_status' => $comparatorMetadata['availability_status'] ?? null, @@ -63,6 +68,18 @@ public static function unavailable(array $comparatorMetadata): array 'comparator_id' => $comparatorMetadata['id'] ?? null, 'framework_name' => $comparatorMetadata['name'] ?? null, 'framework_version' => $comparatorMetadata['framework_version'] ?? $comparatorMetadata['version'] ?? null, + 'fixture_identity' => $comparatorMetadata['fixture_identity'] ?? null, + 'fixture_identity_hash' => $comparatorMetadata['fixture_identity_hash'] ?? ($comparatorMetadata['fixture_identity']['hash'] ?? null), + 'execution_environment_identity' => $comparatorMetadata['execution_environment_identity'] ?? null, + 'execution_environment_fingerprint' => $comparatorMetadata['execution_environment_fingerprint'] ?? ($comparatorMetadata['execution_environment_identity']['hash'] ?? null), + 'worker_environment_identity' => $comparatorMetadata['worker_environment_identity'] ?? null, + 'source_evolvephp_sha' => $comparatorMetadata['source_evolvephp_sha'] ?? null, + 'source_dirty' => $comparatorMetadata['source_dirty'] ?? null, + 'scenario_id' => $comparatorMetadata['scenario_id'] ?? null, + 'implementation_model' => $comparatorMetadata['implementation_model'] ?? null, + 'matrix_sha256' => $comparatorMetadata['matrix_sha256'] ?? null, + 'comparator_lock_sha256' => $comparatorMetadata['comparator_lock_sha256'] ?? null, + 'raw_evidence_sha256' => $comparatorMetadata['raw_evidence_sha256'] ?? null, 'availability' => 'unavailable', 'availability_status' => $comparatorMetadata['availability_status'] ?? 'unknown', 'availability_reason' => $comparatorMetadata['availability_reason'] ?? null, @@ -70,4 +87,100 @@ public static function unavailable(array $comparatorMetadata): array 'baseline_result' => null, ]; } + + /** + * Create result for an available comparator that failed before producing accepted timing. + * + * @param array $comparatorMetadata + * @return array + */ + public static function failed(array $comparatorMetadata): array + { + return [ + 'comparator_schema_version' => self::SCHEMA_VERSION, + 'comparator_id' => $comparatorMetadata['id'] ?? null, + 'framework_name' => $comparatorMetadata['name'] ?? null, + 'framework_version' => $comparatorMetadata['framework_version'] ?? $comparatorMetadata['version'] ?? null, + 'fixture_identity' => $comparatorMetadata['fixture_identity'] ?? null, + 'fixture_identity_hash' => $comparatorMetadata['fixture_identity_hash'] ?? ($comparatorMetadata['fixture_identity']['hash'] ?? null), + 'execution_environment_identity' => $comparatorMetadata['execution_environment_identity'] ?? null, + 'execution_environment_fingerprint' => $comparatorMetadata['execution_environment_fingerprint'] ?? ($comparatorMetadata['execution_environment_identity']['hash'] ?? null), + 'worker_environment_identity' => $comparatorMetadata['worker_environment_identity'] ?? null, + 'source_evolvephp_sha' => $comparatorMetadata['source_evolvephp_sha'] ?? null, + 'source_dirty' => $comparatorMetadata['source_dirty'] ?? null, + 'scenario_id' => $comparatorMetadata['scenario_id'] ?? null, + 'implementation_model' => $comparatorMetadata['implementation_model'] ?? null, + 'matrix_sha256' => $comparatorMetadata['matrix_sha256'] ?? null, + 'comparator_lock_sha256' => $comparatorMetadata['comparator_lock_sha256'] ?? null, + 'raw_evidence_sha256' => $comparatorMetadata['raw_evidence_sha256'] ?? null, + 'availability' => 'failed', + 'availability_status' => 'failed', + 'failure_reason' => $comparatorMetadata['failure_reason'] ?? $comparatorMetadata['availability_reason'] ?? null, + 'notes' => $comparatorMetadata['notes'] ?? [], + 'baseline_result' => null, + ]; + } + + /** + * @param array $raw + * @param array $environment + * @return array + */ + public static function normalizeRawScenario(array $raw, array $environment): array + { + $operationsPerSample = max(1, (int) ($raw['operations_per_sample'] ?? 1)); + $rawSampleRecords = self::rawSampleRecords($raw); + $samples = array_map( + static fn(float|int|string $sample): float => ((float) $sample) / $operationsPerSample, + $rawSampleRecords !== [] ? $rawSampleRecords : (is_array($raw['samples'] ?? null) ? $raw['samples'] : []), + ); + $unit = $operationsPerSample > 1 ? 'per_operation_microseconds' : (string) ($raw['unit'] ?? 'microseconds'); + $result = \Evolve\Benchmarks\Support\ResultNormalizer::normalize([ + 'environment' => $environment, + 'scenarios' => [ + [ + 'id' => (string) ($raw['scenario_id'] ?? 'unknown'), + 'samples' => $samples, + 'unit' => $unit, + 'memory' => is_array($raw['memory'] ?? null) ? $raw['memory'] : [], + ], + ], + ]); + + if (isset($result['scenarios'][0]) && is_array($result['scenarios'][0])) { + $result['scenarios'][0]['operations_per_sample'] = $operationsPerSample; + $result['scenarios'][0]['raw_sample_unit'] = (string) ($raw['unit'] ?? 'microseconds'); + } + + return $result; + } + + /** + * @param array $raw + * @return list + */ + private static function rawSampleRecords(array $raw): array + { + if (!is_array($raw['raw_samples'] ?? null)) { + return []; + } + + $samples = []; + + foreach ($raw['raw_samples'] as $rawSample) { + if (!is_array($rawSample) || !is_array($rawSample['record'] ?? null)) { + continue; + } + + $recordSamples = $rawSample['record']['samples'] ?? null; + + if (!is_array($recordSamples) || !array_key_exists(0, $recordSamples)) { + continue; + } + + $samples[] = $recordSamples[0]; + } + + return $samples; + } } diff --git a/benchmarks/src/Comparator/ComparatorRuntimeIdentity.php b/benchmarks/src/Comparator/ComparatorRuntimeIdentity.php new file mode 100644 index 0000000..530f3ce --- /dev/null +++ b/benchmarks/src/Comparator/ComparatorRuntimeIdentity.php @@ -0,0 +1,128 @@ + + */ + public static function capture(): array + { + $extensions = get_loaded_extensions(); + sort($extensions, SORT_STRING); + + return self::withFingerprint([ + 'schema_version' => self::SCHEMA_VERSION, + 'runtime' => [ + 'php_version' => PHP_VERSION, + 'php_binary' => PHP_BINARY, + 'php_sapi' => PHP_SAPI, + 'php_ini_loaded_file' => php_ini_loaded_file() ?: null, + ], + 'opcache' => self::opcache(), + 'jit' => self::jit(), + 'extensions' => $extensions, + 'phalcon' => [ + 'loaded' => extension_loaded('phalcon'), + 'version' => extension_loaded('phalcon') ? phpversion('phalcon') : null, + ], + ]); + } + + /** + * @param array $environment + * @return array + */ + public static function fromCapturedEnvironment(array $environment): array + { + $extensions = array_values(array_map('strval', $environment['extensions'] ?? [])); + sort($extensions, SORT_STRING); + $phalconLoaded = in_array('phalcon', array_map('strtolower', $extensions), true); + $extensionVersions = is_array($environment['extension_versions'] ?? null) ? $environment['extension_versions'] : []; + $phalconVersion = $extensionVersions['phalcon'] ?? null; + + return self::withFingerprint([ + 'schema_version' => self::SCHEMA_VERSION, + 'runtime' => [ + 'php_version' => $environment['runtime']['php_version'] ?? null, + 'php_binary' => $environment['runtime']['php_binary'] ?? null, + 'php_sapi' => $environment['runtime']['php_sapi'] ?? null, + 'php_ini_loaded_file' => $environment['runtime']['php_ini_loaded_file'] ?? null, + ], + 'opcache' => $environment['opcache'] ?? [], + 'jit' => $environment['jit'] ?? [], + 'extensions' => $extensions, + 'phalcon' => [ + 'loaded' => $phalconLoaded, + 'version' => $phalconLoaded && is_string($phalconVersion) ? $phalconVersion : null, + ], + ]); + } + + /** + * @param array $environment + * @return array + */ + public static function withFingerprint(array $environment): array + { + self::sortRecursive($environment); + $environment['hash'] = hash('sha256', json_encode($environment, JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES)); + + return $environment; + } + + /** + * @return array{enabled: bool, configuration: array} + */ + private static function opcache(): array + { + return [ + 'enabled' => filter_var(ini_get('opcache.enable_cli'), FILTER_VALIDATE_BOOLEAN), + 'configuration' => [ + 'opcache.enable' => ini_get('opcache.enable'), + 'opcache.enable_cli' => ini_get('opcache.enable_cli'), + 'opcache.validate_timestamps' => ini_get('opcache.validate_timestamps'), + 'opcache.memory_consumption' => ini_get('opcache.memory_consumption'), + 'opcache.interned_strings_buffer' => ini_get('opcache.interned_strings_buffer'), + 'opcache.max_accelerated_files' => ini_get('opcache.max_accelerated_files'), + ], + ]; + } + + /** + * @return array{enabled: bool, configuration: array} + */ + private static function jit(): array + { + $bufferSize = ini_get('opcache.jit_buffer_size'); + + return [ + 'enabled' => $bufferSize !== false && $bufferSize !== '' && $bufferSize !== '0', + 'configuration' => [ + 'opcache.jit' => ini_get('opcache.jit'), + 'opcache.jit_buffer_size' => $bufferSize, + ], + ]; + } + + /** + * @param array $value + */ + private static function sortRecursive(array &$value): void + { + foreach ($value as &$entry) { + if (is_array($entry)) { + self::sortRecursive($entry); + } + } + + if (!array_is_list($value)) { + ksort($value); + } + } +} diff --git a/benchmarks/src/Comparator/ComparatorScenarioExecutor.php b/benchmarks/src/Comparator/ComparatorScenarioExecutor.php new file mode 100644 index 0000000..2f2879c --- /dev/null +++ b/benchmarks/src/Comparator/ComparatorScenarioExecutor.php @@ -0,0 +1,182 @@ + + */ + public static function run( + string $matrixPath, + string $comparatorId, + string $scenarioId, + int $warmups, + int $requestCount, + int $sampleIndex = 1, + ): array { + $workerEnvironment = ComparatorRuntimeIdentity::capture(); + $matrix = ComparatorMatrix::fromJsonFile($matrixPath); + $definition = $matrix->comparator($comparatorId); + + if ($definition === null) { + throw new RuntimeException("Unknown comparator '{$comparatorId}'."); + } + + if (!in_array($scenarioId, $definition['scenarios'], true)) { + throw new RuntimeException("Comparator '{$comparatorId}' does not define scenario '{$scenarioId}'."); + } + + $fixture = self::loadFixture(dirname($matrixPath), $definition); + $availability = $fixture->availability(); + $dependencyContext = self::dependencyContext(dirname($matrixPath), $definition); + + if (($availability['available'] ?? false) !== true) { + return [ + 'schema_version' => self::SCHEMA_VERSION, + 'comparator_id' => $comparatorId, + 'scenario_id' => $scenarioId, + 'availability' => 'unavailable', + 'availability_status' => (string) ($availability['status'] ?? 'unavailable'), + 'reason' => (string) ($availability['reason'] ?? 'comparator unavailable'), + 'sample_count' => 0, + 'samples' => [], + 'unit' => 'microseconds', + 'worker_environment_identity' => $workerEnvironment, + 'dependency_context' => $dependencyContext, + 'process' => self::processMetadata(), + ]; + } + + if (!$fixture instanceof PreparedComparatorFixture) { + throw new RuntimeException("Comparator '{$comparatorId}' must implement PreparedComparatorFixture for controlled execution."); + } + + $warmups = $scenarioId === 'application_boot' ? 0 : max(0, $warmups); + $requestCount = max(1, $requestCount); + $subject = $fixture->prepareScenario($scenarioId, [ + 'id' => '123', + 'request_count' => $requestCount, + ]); + $subjectMetadata = $subject->metadata(); + $dependencyContext = self::dependencyContext(dirname($matrixPath), $definition); + + for ($warmup = 0; $warmup < $warmups; ++$warmup) { + $subject->runOnce(); + } + + $measurement = $subject->measureOnce(); + $operationsPerSample = $scenarioId === 'http_repeated_warm' ? $requestCount : 1; + + return [ + 'schema_version' => self::SCHEMA_VERSION, + 'comparator_id' => $comparatorId, + 'scenario_id' => $scenarioId, + 'availability' => 'available', + 'availability_status' => 'available', + 'timing_boundary' => $subjectMetadata['timing_boundary'] ?? null, + 'prepared_framework_instance_count' => $subjectMetadata['prepared_framework_instance_count'] ?? null, + 'sample_count' => 1, + 'sample_index' => max(1, $sampleIndex), + 'samples' => [$measurement['duration_microseconds']], + 'unit' => 'microseconds', + 'memory' => $measurement['memory'], + 'warmups' => $warmups, + 'request_count' => $requestCount, + 'operations_per_sample' => $operationsPerSample, + 'worker_environment_identity' => $workerEnvironment, + 'dependency_context' => $dependencyContext, + 'process' => self::processMetadata(), + 'subject_metadata' => $subjectMetadata, + 'last_result' => $measurement['result'], + ]; + } + + /** + * @param array $definition + */ + private static function loadFixture(string $baseDir, array $definition): ComparatorFixture + { + $fixtureRoot = $baseDir . DIRECTORY_SEPARATOR . trim((string) $definition['fixture_path'], '/\\'); + $autoloadPath = $fixtureRoot . DIRECTORY_SEPARATOR . 'vendor' . DIRECTORY_SEPARATOR . 'autoload.php'; + + if (is_file($autoloadPath)) { + require_once $autoloadPath; + } + + $path = $baseDir . DIRECTORY_SEPARATOR . trim((string) $definition['fixture_bootstrap'], '/\\'); + $fixture = require $path; + + if (!$fixture instanceof ComparatorFixture) { + throw new RuntimeException("Comparator '{$definition['id']}' bootstrap must return a ComparatorFixture."); + } + + return $fixture; + } + + /** + * @param array $definition + * @return array + */ + private static function dependencyContext(string $baseDir, array $definition): array + { + $fixtureRoot = $baseDir . DIRECTORY_SEPARATOR . trim((string) $definition['fixture_path'], '/\\'); + $benchmarkRoot = dirname($baseDir); + $included = array_map(static fn(string $path): string => str_replace('\\', '/', $path), get_included_files()); + $benchmarkAutoload = str_replace('\\', '/', $benchmarkRoot . DIRECTORY_SEPARATOR . 'vendor' . DIRECTORY_SEPARATOR . 'autoload.php'); + $frameworkClasses = [ + 'evolvephp' => 'Evolve\Http\HttpKernel', + 'laravel' => 'Illuminate\Routing\Router', + 'symfony' => 'Symfony\Component\HttpKernel\HttpKernel', + 'slim' => 'Slim\App', + 'phalcon' => 'Phalcon\Mvc\Micro', + ]; + $loadedFrameworks = []; + + foreach ($frameworkClasses as $id => $class) { + $loadedFrameworks[$id] = class_exists($class, false); + } + + $selectedId = (string) $definition['id']; + $unrelatedFrameworks = $loadedFrameworks; + unset($unrelatedFrameworks[$selectedId]); + + return [ + 'selected_comparator_root' => self::relativePath(dirname($benchmarkRoot), $fixtureRoot), + 'selected_comparator_autoload' => self::relativePath(dirname($benchmarkRoot), $fixtureRoot . DIRECTORY_SEPARATOR . 'vendor' . DIRECTORY_SEPARATOR . 'autoload.php'), + 'benchmark_root_autoload_loaded' => in_array($benchmarkAutoload, $included, true), + 'selected_framework_class' => $frameworkClasses[$selectedId] ?? null, + 'selected_framework_loaded' => $loadedFrameworks[$selectedId] ?? false, + 'unrelated_framework_classes_loaded' => $unrelatedFrameworks, + 'unrelated_framework_classes_preloaded' => $unrelatedFrameworks, + ]; + } + + /** + * @return array + */ + private static function processMetadata(): array + { + return [ + 'pid' => getmypid(), + 'php_binary' => PHP_BINARY, + 'php_version' => PHP_VERSION, + 'sapi' => PHP_SAPI, + 'cwd' => getcwd(), + ]; + } + + private static function relativePath(string $root, string $path): string + { + $root = rtrim(str_replace('\\', '/', $root), '/'); + $path = str_replace('\\', '/', $path); + + return str_starts_with($path, $root . '/') ? substr($path, strlen($root) + 1) : $path; + } +} diff --git a/benchmarks/src/Comparator/PreparedComparatorFixture.php b/benchmarks/src/Comparator/PreparedComparatorFixture.php new file mode 100644 index 0000000..e0983be --- /dev/null +++ b/benchmarks/src/Comparator/PreparedComparatorFixture.php @@ -0,0 +1,13 @@ + $options + */ + public function prepareScenario(string $scenarioId, array $options = []): PreparedScenario; +} diff --git a/benchmarks/src/Comparator/PreparedScenario.php b/benchmarks/src/Comparator/PreparedScenario.php new file mode 100644 index 0000000..fdf5c7b --- /dev/null +++ b/benchmarks/src/Comparator/PreparedScenario.php @@ -0,0 +1,79 @@ + $subject + * @param array $metadata + */ + public function __construct( + private readonly string $scenarioId, + private readonly string $timingBoundary, + private readonly int $preparedFrameworkInstanceCount, + private readonly Closure $subject, + private readonly array $metadata = [], + ) {} + + public function scenarioId(): string + { + return $this->scenarioId; + } + + public function timingBoundary(): string + { + return $this->timingBoundary; + } + + public function preparedFrameworkInstanceCount(): int + { + return $this->preparedFrameworkInstanceCount; + } + + /** + * @return array + */ + public function metadata(): array + { + return [ + 'scenario_id' => $this->scenarioId, + 'timing_boundary' => $this->timingBoundary, + 'prepared_framework_instance_count' => $this->preparedFrameworkInstanceCount, + ] + $this->metadata; + } + + /** + * @return array + */ + public function runOnce(): array + { + $result = ($this->subject)(); + $result['scenario_id'] ??= $this->scenarioId; + + return $result; + } + + /** + * @return array{duration_microseconds: float, memory: array, result: array} + */ + public function measureOnce(): array + { + $start = hrtime(true); + $result = $this->runOnce(); + $duration = (hrtime(true) - $start) / 1000; + + return [ + 'duration_microseconds' => $duration, + 'memory' => [ + 'current_bytes' => memory_get_usage(true), + 'peak_bytes' => memory_get_peak_usage(true), + ], + 'result' => $result, + ]; + } +} diff --git a/benchmarks/src/Support/BenchmarkEnvironment.php b/benchmarks/src/Support/BenchmarkEnvironment.php index df3acd2..efc591d 100644 --- a/benchmarks/src/Support/BenchmarkEnvironment.php +++ b/benchmarks/src/Support/BenchmarkEnvironment.php @@ -52,6 +52,7 @@ public static function capture(string $repositoryRoot): array 'opcache' => self::opcache(), 'jit' => self::jit(), 'extensions' => $extensions, + 'extension_versions' => self::extensionVersions($extensions), 'lock' => self::lockState($repositoryRoot), ]; @@ -95,6 +96,26 @@ private static function jit(): array ]; } + /** + * @param list $extensions + * @return array + */ + private static function extensionVersions(array $extensions): array + { + $versions = []; + + foreach (['phalcon'] as $extension) { + if (!in_array($extension, array_map('strtolower', $extensions), true)) { + continue; + } + + $version = phpversion($extension); + $versions[$extension] = is_string($version) ? $version : null; + } + + return $versions; + } + /** * @return array{path: string, exists: bool, hash: string|null} */ @@ -122,7 +143,7 @@ private static function composerVersion(string $repositoryRoot): ?string { $binary = getenv('COMPOSER_BINARY') ?: null; - if ($binary !== null && $binary !== '') { + if ($binary !== null) { return self::command($repositoryRoot, [PHP_BINARY, $binary, '--version']); } @@ -228,7 +249,7 @@ private static function command(string $cwd, array $command): ?string } /** - * @param array $value + * @param array $value */ private static function sortRecursive(array &$value): void { diff --git a/benchmarks/src/Support/ExecutionEnvironmentFingerprint.php b/benchmarks/src/Support/ExecutionEnvironmentFingerprint.php index 50a8572..e1bab3a 100644 --- a/benchmarks/src/Support/ExecutionEnvironmentFingerprint.php +++ b/benchmarks/src/Support/ExecutionEnvironmentFingerprint.php @@ -45,7 +45,7 @@ public static function fromEnvironment(array $environment): array } /** - * @param array $value + * @param array $value */ private static function sortRecursive(array &$value): void { diff --git a/benchmarks/tests/Comparator/ComparatorExecutionRunnerTest.php b/benchmarks/tests/Comparator/ComparatorExecutionRunnerTest.php new file mode 100644 index 0000000..f742918 --- /dev/null +++ b/benchmarks/tests/Comparator/ComparatorExecutionRunnerTest.php @@ -0,0 +1,960 @@ +outputDir = sys_get_temp_dir() . DIRECTORY_SEPARATOR . 'evolvephp-comparator-runner-' . bin2hex(random_bytes(4)); + $this->workerDir = sys_get_temp_dir() . DIRECTORY_SEPARATOR . 'evolvephp-comparator-worker-' . bin2hex(random_bytes(4)); + mkdir($this->outputDir, 0777, true); + mkdir($this->workerDir, 0777, true); + } + + protected function tearDown(): void + { + $this->removeDirectory($this->outputDir); + $this->removeDirectory($this->workerDir); + } + + public function testRunnerUsesIsolatedSubprocessesAndWritesEvidenceManifest(): void + { + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['http_static', 'http_not_found'], + 'samples' => 2, + 'warmups' => 1, + 'request_count' => 3, + 'preflight' => $this->matchedCurrentPreflight(), + ], + ); + + self::assertSame('evolvephp.comparator.evidence-manifest.v1', $manifest['schema_version']); + self::assertSame('completed', $manifest['status']); + self::assertSame('subprocess_per_measured_sample', $manifest['process_isolation']['model']); + self::assertSame( + ['evolvephp:http_static', 'evolvephp:http_not_found'], + $manifest['execution_order'], + ); + self::assertCount(2, $manifest['results']); + + foreach ($manifest['results'] as $result) { + self::assertSame('available', $result['availability']); + self::assertSame(0, $result['exit_code']); + self::assertStringNotContainsString('comparator-smoke.php', implode(' ', $result['command'])); + self::assertFileExists($this->outputDir . DIRECTORY_SEPARATOR . $result['raw_result']['path']); + self::assertFileExists($this->outputDir . DIRECTORY_SEPARATOR . $result['normalized_result']['path']); + self::assertMatchesRegularExpression('/^[a-f0-9]{64}$/', $result['raw_result']['sha256']); + self::assertMatchesRegularExpression('/^[a-f0-9]{64}$/', $result['normalized_result']['sha256']); + self::assertSame('prepared_warm_http_request', $result['timing_boundary']); + self::assertSame(2, $result['sample_count']); + self::assertCount(2, $result['samples_metadata']); + self::assertCount(2, array_unique(array_column($result['samples_metadata'], 'pid'))); + self::assertSame($manifest['worker_environment_identity']['hash'], $result['worker_environment_identity']['hash']); + self::assertSame($manifest['execution_environment_identity']['hash'], $result['execution_environment_identity']['hash']); + self::assertNotSame($manifest['execution_environment_identity']['hash'], $result['worker_environment_identity']['hash']); + self::assertSame('benchmarks/comparators/evolvephp', $result['dependency_context']['selected_comparator_root']); + self::assertFalse($result['dependency_context']['benchmark_root_autoload_loaded']); + self::assertSame('Evolve\Http\HttpKernel', $result['dependency_context']['selected_framework_class']); + self::assertTrue($result['dependency_context']['selected_framework_loaded']); + self::assertFalse($result['dependency_context']['unrelated_framework_classes_loaded']['laravel']); + self::assertFalse($result['dependency_context']['unrelated_framework_classes_loaded']['symfony']); + self::assertFalse($result['dependency_context']['unrelated_framework_classes_loaded']['slim']); + + $raw = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $result['raw_result']['path']), true, flags: JSON_THROW_ON_ERROR); + self::assertCount(2, $raw['raw_samples']); + + foreach ($raw['raw_samples'] as $rawSample) { + self::assertMatchesRegularExpression('/^[a-f0-9]{64}$/', $rawSample['sha256']); + self::assertSame($rawSample['sha256'], $this->hashJson($rawSample['record'])); + self::assertSame($rawSample['record']['sample_index'], $rawSample['sample_index']); + self::assertArrayHasKey('worker_environment_identity', $rawSample['record']); + self::assertArrayHasKey('dependency_context', $rawSample['record']); + self::assertArrayHasKey('subject_metadata', $rawSample['record']); + self::assertArrayHasKey('last_result', $rawSample['record']); + self::assertArrayHasKey('process', $rawSample['record']); + } + + self::assertSame( + array_column($result['samples_metadata'], 'raw_sample_sha256'), + array_column($raw['raw_samples'], 'sha256'), + ); + + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $result['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + self::assertSame($manifest['execution_environment_identity']['hash'], $normalized['execution_environment_identity']['hash']); + self::assertSame($manifest['execution_environment_identity']['hash'], $normalized['execution_environment_fingerprint']); + } + } + + public function testExternalComparatorDependencyContextIsCapturedAfterSelectedFrameworkLoads(): void + { + if (!is_file(dirname(__DIR__, 2) . '/comparators/laravel/vendor/autoload.php')) { + self::markTestSkipped('Laravel comparator dependency root is not installed.'); + } + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['laravel'], + 'scenarios' => ['http_static'], + 'samples' => 1, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $this->matchedCurrentPreflight(), + ], + ); + + $context = $manifest['results'][0]['dependency_context']; + + self::assertSame('benchmarks/comparators/laravel', $context['selected_comparator_root']); + self::assertSame('Illuminate\Routing\Router', $context['selected_framework_class']); + self::assertTrue($context['selected_framework_loaded']); + self::assertFalse($context['benchmark_root_autoload_loaded']); + self::assertFalse($context['unrelated_framework_classes_loaded']['evolvephp']); + self::assertFalse($context['unrelated_framework_classes_loaded']['symfony']); + self::assertFalse($context['unrelated_framework_classes_loaded']['slim']); + } + + public function testMismatchedPreflightBlocksTimingAndCreatesNoRawOrNormalizedArtifacts(): void + { + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['http_static'], + 'samples' => 2, + 'preflight' => [ + 'schema_version' => 'evolvephp.comparator.preflight.v1', + 'status' => 'mismatched', + 'mismatches' => ['PHP version is not exactly 8.4.25.'], + 'environment' => BenchmarkEnvironment::capture(dirname(__DIR__, 3)), + 'environment_identity' => null, + ], + ], + ); + + self::assertSame('preflight_rejected', $manifest['status']); + self::assertSame(['PHP version is not exactly 8.4.25.'], $manifest['preflight']['mismatches']); + self::assertSame([], $manifest['results']); + self::assertFileExists($this->outputDir . DIRECTORY_SEPARATOR . 'preflight.json'); + self::assertFileExists($this->outputDir . DIRECTORY_SEPARATOR . 'manifest.json'); + self::assertDirectoryDoesNotExist($this->outputDir . DIRECTORY_SEPARATOR . 'raw'); + self::assertDirectoryDoesNotExist($this->outputDir . DIRECTORY_SEPARATOR . 'normalized'); + self::assertSame(['manifest.json', 'preflight.json'], $this->outputDirectoryFiles()); + } + + public function testUnavailableComparatorDoesNotInventTimingSamples(): void + { + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['phalcon'], + 'scenarios' => ['http_static'], + 'samples' => 2, + 'preflight' => $this->matchedCurrentPreflight(), + ], + ); + + if ($manifest['results'][0]['availability'] === 'available') { + self::markTestSkipped('ext-phalcon is available in this environment; unavailable path cannot be asserted.'); + } + + self::assertSame('unavailable', $manifest['results'][0]['availability']); + self::assertSame(0, $manifest['results'][0]['sample_count']); + self::assertArrayHasKey('reason', $manifest['results'][0]); + + $normalizedPath = $this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']; + $normalized = json_decode((string) file_get_contents($normalizedPath), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame('unavailable', $normalized['availability']); + self::assertNull($normalized['baseline_result']); + self::assertArrayNotHasKey('timing', $normalized); + } + + public function testBrokenAvailableComparatorFailureRemainsFailed(): void + { + $brokenWorker = $this->workerDir . DIRECTORY_SEPARATOR . 'broken-worker.php'; + file_put_contents($brokenWorker, "run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['http_static'], + 'samples' => 1, + 'preflight' => $this->matchedCurrentPreflight(), + 'worker_path' => $brokenWorker, + ], + ); + + self::assertSame('failed', $manifest['status']); + self::assertSame('failed', $manifest['results'][0]['availability']); + self::assertSame(0, $manifest['results'][0]['sample_count']); + self::assertSame(1, $manifest['results'][0]['exit_code']); + + $normalizedPath = $this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']; + $normalized = json_decode((string) file_get_contents($normalizedPath), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame('failed', $normalized['availability']); + self::assertNull($normalized['baseline_result']); + self::assertArrayNotHasKey('timing', $normalized); + } + + public function testWrongWorkerRuntimeIdentityRejectsValidLookingAvailableSample(): void + { + $mismatchedWorker = $this->workerDir . DIRECTORY_SEPARATOR . 'mismatched-worker.php'; + file_put_contents($mismatchedWorker, <<<'PHP' + 'evolvephp.comparator.raw-result.v1', + 'comparator_id' => 'evolvephp', + 'scenario_id' => 'http_static', + 'availability' => 'available', + 'availability_status' => 'available', + 'timing_boundary' => 'prepared_warm_http_request', + 'sample_count' => 1, + 'sample_index' => 1, + 'samples' => [123.0], + 'unit' => 'microseconds', + 'memory' => ['peak_bytes' => 1], + 'warmups' => 0, + 'request_count' => 1, + 'operations_per_sample' => 1, + 'worker_environment_identity' => [ + 'schema_version' => 'evolvephp.comparator.runtime-identity.v1', + 'hash' => '0000000000000000000000000000000000000000000000000000000000000000', + ], + 'dependency_context' => [], + 'process' => ['pid' => getmypid()], + 'subject_metadata' => [], + 'last_result' => ['status_code' => 200], +], JSON_THROW_ON_ERROR); +PHP); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['http_static'], + 'samples' => 1, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $this->matchedCurrentPreflight(), + 'worker_path' => $mismatchedWorker, + ], + ); + + self::assertSame('failed', $manifest['status']); + self::assertSame('failed', $manifest['results'][0]['availability']); + self::assertSame('measurement worker runtime identity did not match accepted preflight lane', $manifest['results'][0]['reason']); + + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame('failed', $normalized['availability']); + self::assertSame('measurement worker runtime identity did not match accepted preflight lane', $normalized['failure_reason']); + self::assertNull($normalized['baseline_result']); + } + + public function testWrongWorkerRuntimeIdentityRejectsUnavailableSampleFromMatchedPreflight(): void + { + $mismatchedUnavailableWorker = $this->workerDir . DIRECTORY_SEPARATOR . 'mismatched-unavailable-worker.php'; + file_put_contents($mismatchedUnavailableWorker, <<<'PHP' + 'evolvephp.comparator.raw-result.v1', + 'comparator_id' => 'phalcon', + 'scenario_id' => 'http_static', + 'availability' => 'unavailable', + 'availability_status' => 'unavailable', + 'reason' => 'Phalcon extension is not loaded in worker', + 'sample_count' => 0, + 'samples' => [], + 'unit' => 'microseconds', + 'worker_environment_identity' => [ + 'schema_version' => 'evolvephp.comparator.runtime-identity.v1', + 'hash' => 'ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff', + ], + 'dependency_context' => [], + 'process' => ['pid' => getmypid()], +], JSON_THROW_ON_ERROR); +PHP); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['phalcon'], + 'scenarios' => ['http_static'], + 'samples' => 1, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $this->matchedCurrentPreflight(), + 'worker_path' => $mismatchedUnavailableWorker, + ], + ); + + self::assertSame('failed', $manifest['status']); + self::assertSame('failed', $manifest['results'][0]['availability']); + self::assertSame('measurement worker runtime identity did not match accepted preflight lane', $manifest['results'][0]['reason']); + self::assertSame(1, $manifest['results'][0]['exit_code']); + + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame('failed', $normalized['availability']); + self::assertSame('measurement worker runtime identity did not match accepted preflight lane', $normalized['failure_reason']); + self::assertNull($normalized['baseline_result']); + } + + public function testApplicationBootRunsOneDiscardedWorkerBeforeEveryMeasuredSampleAndExcludesDiscardedTiming(): void + { + $preflight = $this->matchedCurrentPreflight(); + $worker = $this->writeCountingWorker((string) $preflight['worker_environment_identity']['hash']); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['application_boot'], + 'samples' => 2, + 'warmups' => 5, + 'request_count' => 1, + 'preflight' => $preflight, + 'worker_path' => $worker, + ], + ); + + self::assertSame([ + 'discarded_worker_processes_per_measured_sample' => 1, + 'sample_order' => 'rotating_round_robin', + 'measured_worker_in_process_warmups' => 0, + 'outlier_policy' => 'retain_all_measured_samples', + 'primary_central_statistic' => 'p50', + ], $manifest['boot_protocol']); + self::assertSame('completed', $manifest['status']); + self::assertCount(4, $manifest['boot_worker_execution_sequence']); + self::assertSame(['discarded', 'measured', 'discarded', 'measured'], array_column($manifest['boot_worker_execution_sequence'], 'role')); + self::assertSame([true, false, true, false], array_column($manifest['boot_worker_execution_sequence'], 'excluded_from_statistics')); + self::assertSame([1, 1, 2, 2], array_column($manifest['boot_worker_execution_sequence'], 'sample_index')); + self::assertSame(['available', 'available', 'available', 'available'], array_column($manifest['boot_worker_execution_sequence'], 'availability')); + self::assertSame( + array_column($manifest['boot_worker_execution_sequence'], 'worker_environment_identity_hash'), + array_fill(0, 4, $preflight['worker_environment_identity']['hash']), + ); + + foreach (array_chunk($manifest['boot_worker_execution_sequence'], 2) as $pair) { + self::assertSame('discarded', $pair[0]['role']); + self::assertSame('measured', $pair[1]['role']); + self::assertTrue($pair[0]['excluded_from_statistics']); + self::assertFalse($pair[1]['excluded_from_statistics']); + self::assertSame($pair[0]['comparator_id'], $pair[1]['comparator_id']); + self::assertSame($pair[0]['sample_index'], $pair[1]['sample_index']); + self::assertNotSame($pair[0]['pid'], $pair[1]['pid']); + } + + $result = $manifest['results'][0]; + self::assertSame('application_boot', $result['scenario_id']); + self::assertSame(2, $result['sample_count']); + self::assertCount(2, $result['commands']); + self::assertCount(2, $result['raw_samples']); + self::assertCount(2, $result['discarded_worker_provenance']); + + $raw = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $result['raw_result']['path']), true, flags: JSON_THROW_ON_ERROR); + self::assertSame(0, $raw['warmups']); + self::assertSame(2, $raw['sample_count']); + self::assertSame([2.0, 4.0], array_map('floatval', $raw['samples'])); + self::assertCount(2, $raw['raw_samples']); + self::assertCount(2, $raw['discarded_worker_provenance']); + self::assertSame([1.0, 3.0], array_map('floatval', array_column($raw['discarded_worker_provenance'], 'duration_microseconds'))); + self::assertSame([true, true], array_column($raw['discarded_worker_provenance'], 'excluded_from_statistics')); + + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $result['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + self::assertSame(2, $normalized['baseline_result']['scenarios'][0]['sample_count']); + self::assertSame(3.0, (float) $normalized['baseline_result']['scenarios'][0]['p50']); + } + + public function testApplicationBootUsesDeterministicRotatingRoundRobinAcrossComparators(): void + { + $preflight = $this->matchedCurrentPreflight(); + $worker = $this->writeCountingWorker((string) $preflight['worker_environment_identity']['hash']); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp', 'slim', 'symfony'], + 'scenarios' => ['application_boot'], + 'samples' => 3, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $preflight, + 'worker_path' => $worker, + ], + ); + + $slots = []; + + foreach (array_chunk($manifest['boot_worker_execution_sequence'], 2) as $pair) { + self::assertSame('discarded', $pair[0]['role']); + self::assertSame('measured', $pair[1]['role']); + self::assertTrue($pair[0]['excluded_from_statistics']); + self::assertFalse($pair[1]['excluded_from_statistics']); + self::assertSame($pair[0]['comparator_id'], $pair[1]['comparator_id']); + self::assertSame($pair[0]['sample_index'], $pair[1]['sample_index']); + $slots[] = $pair[1]['comparator_id'] . ':' . $pair[1]['sample_index']; + } + + self::assertSame([ + 'evolvephp:1', + 'slim:1', + 'symfony:1', + 'slim:2', + 'symfony:2', + 'evolvephp:2', + 'symfony:3', + 'evolvephp:3', + 'slim:3', + ], $slots); + self::assertSame([ + 'evolvephp:application_boot', + 'slim:application_boot', + 'symfony:application_boot', + ], $manifest['execution_order']); + } + + public function testDiscardedApplicationBootWorkerRuntimeMismatchFailsClosed(): void + { + $preflight = $this->matchedCurrentPreflight(); + $worker = $this->writeCountingWorker( + (string) $preflight['worker_environment_identity']['hash'], + '1111111111111111111111111111111111111111111111111111111111111111', + [1], + ); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['application_boot'], + 'samples' => 1, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $preflight, + 'worker_path' => $worker, + ], + ); + + self::assertSame('failed', $manifest['status']); + self::assertSame('failed', $manifest['results'][0]['availability']); + self::assertSame('discarded boot worker runtime identity did not match accepted preflight lane', $manifest['results'][0]['reason']); + self::assertSame(1, $manifest['results'][0]['exit_code']); + self::assertSame(0, $manifest['results'][0]['sample_count']); + self::assertSame([], $manifest['results'][0]['raw_samples']); + self::assertCount(1, $manifest['boot_worker_execution_sequence']); + self::assertSame('discarded', $manifest['boot_worker_execution_sequence'][0]['role']); + self::assertSame(0, $manifest['boot_worker_execution_sequence'][0]['exit_code']); + self::assertTrue($manifest['boot_worker_execution_sequence'][0]['excluded_from_statistics']); + + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame('failed', $normalized['availability']); + self::assertNull($normalized['baseline_result']); + } + + public function testMeasuredApplicationBootWorkerRuntimeMismatchFailsClosedWithNonZeroAggregateExitCode(): void + { + $preflight = $this->matchedCurrentPreflight(); + $worker = $this->writeCountingWorker( + (string) $preflight['worker_environment_identity']['hash'], + '2222222222222222222222222222222222222222222222222222222222222222', + [2], + ); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['application_boot'], + 'samples' => 1, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $preflight, + 'worker_path' => $worker, + ], + ); + + self::assertSame('failed', $manifest['status']); + self::assertSame('failed', $manifest['results'][0]['availability']); + self::assertSame('measurement worker runtime identity did not match accepted preflight lane', $manifest['results'][0]['reason']); + self::assertSame(1, $manifest['results'][0]['exit_code']); + self::assertSame(0, $manifest['results'][0]['sample_count']); + self::assertSame([], $manifest['results'][0]['raw_samples']); + self::assertCount(2, $manifest['boot_worker_execution_sequence']); + self::assertSame(['discarded', 'measured'], array_column($manifest['boot_worker_execution_sequence'], 'role')); + self::assertSame([true, false], array_column($manifest['boot_worker_execution_sequence'], 'excluded_from_statistics')); + self::assertSame([0, 0], array_column($manifest['boot_worker_execution_sequence'], 'exit_code')); + + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame('failed', $normalized['availability']); + self::assertSame('measurement worker runtime identity did not match accepted preflight lane', $normalized['failure_reason']); + self::assertNull($normalized['baseline_result']); + } + + public function testDiscardedApplicationBootWorkerProcessFailureStopsBeforeMeasuredWorkerAndNormalizesNoTiming(): void + { + $preflight = $this->matchedCurrentPreflight(); + $worker = $this->writeCountingWorker( + (string) $preflight['worker_environment_identity']['hash'], + failedInvocations: [1], + ); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['application_boot'], + 'samples' => 1, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $preflight, + 'worker_path' => $worker, + ], + ); + + self::assertSame('failed', $manifest['status']); + self::assertSame('failed', $manifest['results'][0]['availability']); + self::assertNotSame(0, $manifest['results'][0]['exit_code']); + self::assertSame(0, $manifest['results'][0]['sample_count']); + self::assertSame([], $manifest['results'][0]['raw_samples']); + self::assertCount(1, $manifest['boot_worker_execution_sequence']); + self::assertSame('discarded', $manifest['boot_worker_execution_sequence'][0]['role']); + self::assertSame(7, $manifest['boot_worker_execution_sequence'][0]['exit_code']); + self::assertTrue($manifest['boot_worker_execution_sequence'][0]['excluded_from_statistics']); + + $raw = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['raw_result']['path']), true, flags: JSON_THROW_ON_ERROR); + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame([], $raw['samples']); + self::assertSame([], $raw['raw_samples']); + self::assertSame('failed', $normalized['availability']); + self::assertNull($normalized['baseline_result']); + } + + public function testLaterDiscardedApplicationBootWorkerProcessFailurePreservesEarlierMeasuredEvidence(): void + { + $preflight = $this->matchedCurrentPreflight(); + $worker = $this->writeCountingWorker( + (string) $preflight['worker_environment_identity']['hash'], + failedInvocations: [3], + ); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['application_boot'], + 'samples' => 2, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $preflight, + 'worker_path' => $worker, + ], + ); + + self::assertSame('failed', $manifest['status']); + self::assertSame('failed', $manifest['results'][0]['availability']); + self::assertNotSame(0, $manifest['results'][0]['exit_code']); + self::assertSame(1, $manifest['results'][0]['sample_count']); + self::assertCount(1, $manifest['results'][0]['raw_samples']); + self::assertCount(1, $manifest['results'][0]['samples_metadata']); + self::assertCount(1, $manifest['results'][0]['commands']); + self::assertCount(3, $manifest['boot_worker_execution_sequence']); + self::assertSame(['discarded', 'measured', 'discarded'], array_column($manifest['boot_worker_execution_sequence'], 'role')); + self::assertSame(7, $manifest['boot_worker_execution_sequence'][2]['exit_code']); + self::assertTrue($manifest['boot_worker_execution_sequence'][2]['excluded_from_statistics']); + + $raw = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['raw_result']['path']), true, flags: JSON_THROW_ON_ERROR); + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame([2.0], array_map('floatval', $raw['samples'])); + self::assertSame(1, $raw['sample_count']); + self::assertCount(1, $raw['raw_samples']); + self::assertArrayHasKey('sha256', $raw['raw_samples'][0]); + self::assertSame(2.0, (float) $raw['raw_samples'][0]['record']['samples'][0]); + self::assertCount(1, $raw['samples_metadata']); + self::assertCount(1, $raw['commands']); + self::assertSame('ok', $raw['last_result']['status']); + self::assertSame(2, $raw['memory']['peak_bytes']); + self::assertSame('failed', $normalized['availability']); + self::assertNull($normalized['baseline_result']); + } + + public function testLaterDiscardedApplicationBootWorkerUnavailableAfterAcceptedSampleFailsAndPreservesEvidence(): void + { + $preflight = $this->matchedCurrentPreflight(); + $worker = $this->writeCountingWorker( + (string) $preflight['worker_environment_identity']['hash'], + unavailableInvocations: [3], + ); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['application_boot'], + 'samples' => 2, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $preflight, + 'worker_path' => $worker, + ], + ); + + self::assertSame('failed', $manifest['status']); + self::assertSame('failed', $manifest['results'][0]['availability']); + self::assertSame(1, $manifest['results'][0]['exit_code']); + self::assertSame(1, $manifest['results'][0]['sample_count']); + self::assertStringContainsString('availability changed during controlled boot sampling', (string) $manifest['results'][0]['reason']); + self::assertCount(1, $manifest['results'][0]['raw_samples']); + self::assertCount(3, $manifest['boot_worker_execution_sequence']); + self::assertSame(['available', 'available', 'unavailable'], array_column($manifest['boot_worker_execution_sequence'], 'availability')); + + $raw = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['raw_result']['path']), true, flags: JSON_THROW_ON_ERROR); + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame([2.0], array_map('floatval', $raw['samples'])); + self::assertCount(1, $raw['raw_samples']); + self::assertCount(1, $raw['samples_metadata']); + self::assertCount(1, $raw['commands']); + self::assertSame('failed', $normalized['availability']); + self::assertNull($normalized['baseline_result']); + } + + public function testMeasuredApplicationBootWorkerUnavailableAfterAvailableDiscardFailsWithoutFabricatedTiming(): void + { + $preflight = $this->matchedCurrentPreflight(); + $worker = $this->writeCountingWorker( + (string) $preflight['worker_environment_identity']['hash'], + unavailableInvocations: [2], + ); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['application_boot'], + 'samples' => 1, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $preflight, + 'worker_path' => $worker, + ], + ); + + self::assertSame('failed', $manifest['status']); + self::assertSame('failed', $manifest['results'][0]['availability']); + self::assertSame(1, $manifest['results'][0]['exit_code']); + self::assertSame(0, $manifest['results'][0]['sample_count']); + self::assertSame([], $manifest['results'][0]['raw_samples']); + self::assertStringContainsString('inconsistent availability within measured boot slot', (string) $manifest['results'][0]['reason']); + self::assertCount(2, $manifest['boot_worker_execution_sequence']); + self::assertSame(['discarded', 'measured'], array_column($manifest['boot_worker_execution_sequence'], 'role')); + self::assertSame(['available', 'unavailable'], array_column($manifest['boot_worker_execution_sequence'], 'availability')); + self::assertSame([true, false], array_column($manifest['boot_worker_execution_sequence'], 'excluded_from_statistics')); + + $raw = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['raw_result']['path']), true, flags: JSON_THROW_ON_ERROR); + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame([], $raw['samples']); + self::assertSame([], $raw['raw_samples']); + self::assertSame([], $raw['samples_metadata']); + self::assertSame('failed', $normalized['availability']); + self::assertNull($normalized['baseline_result']); + } + + public function testDiscardedApplicationBootWorkerUnavailableStopsBeforeMeasuredWorkerAndFabricatesNoTiming(): void + { + $preflight = $this->matchedCurrentPreflight(); + $worker = $this->writeCountingWorker( + (string) $preflight['worker_environment_identity']['hash'], + unavailableInvocations: [1], + ); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['application_boot'], + 'samples' => 1, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $preflight, + 'worker_path' => $worker, + ], + ); + + self::assertSame('completed', $manifest['status']); + self::assertSame('unavailable', $manifest['results'][0]['availability']); + self::assertSame(0, $manifest['results'][0]['exit_code']); + self::assertSame(0, $manifest['results'][0]['sample_count']); + self::assertSame([], $manifest['results'][0]['raw_samples']); + self::assertCount(1, $manifest['boot_worker_execution_sequence']); + self::assertSame('discarded', $manifest['boot_worker_execution_sequence'][0]['role']); + self::assertSame(0, $manifest['boot_worker_execution_sequence'][0]['exit_code']); + self::assertSame('unavailable', $manifest['boot_worker_execution_sequence'][0]['availability']); + self::assertTrue($manifest['boot_worker_execution_sequence'][0]['excluded_from_statistics']); + + $raw = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['raw_result']['path']), true, flags: JSON_THROW_ON_ERROR); + $normalized = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['normalized_result']['path']), true, flags: JSON_THROW_ON_ERROR); + + self::assertSame([], $raw['raw_samples']); + self::assertSame('unavailable', $normalized['availability']); + self::assertNull($normalized['baseline_result']); + } + + public function testWarmHttpScenariosDoNotReceiveDiscardedWorkers(): void + { + $preflight = $this->matchedCurrentPreflight(); + $worker = $this->writeCountingWorker((string) $preflight['worker_environment_identity']['hash']); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + $manifest = $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['http_static'], + 'samples' => 2, + 'warmups' => 5, + 'request_count' => 1, + 'preflight' => $preflight, + 'worker_path' => $worker, + ], + ); + + self::assertSame([], $manifest['boot_worker_execution_sequence']); + self::assertSame('completed', $manifest['status']); + self::assertSame(2, $manifest['results'][0]['sample_count']); + self::assertCount(2, $manifest['results'][0]['commands']); + self::assertArrayNotHasKey('discarded_worker_provenance', $manifest['results'][0]); + + $raw = json_decode((string) file_get_contents($this->outputDir . DIRECTORY_SEPARATOR . $manifest['results'][0]['raw_result']['path']), true, flags: JSON_THROW_ON_ERROR); + self::assertSame(5, $raw['warmups']); + self::assertCount(2, $raw['raw_samples']); + self::assertArrayNotHasKey('discarded_worker_provenance', $raw); + } + + public function testPrePopulatedOutputDirectoryIsRejectedBeforeNewEvidenceIsWritten(): void + { + file_put_contents($this->outputDir . DIRECTORY_SEPARATOR . 'stale-evidence.json', '{}'); + + $runner = new ComparatorExecutionRunner(PHP_BINARY); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('Comparator output directory must be absent or empty before a run begins.'); + + try { + $runner->run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + $this->outputDir, + [ + 'comparators' => ['evolvephp'], + 'scenarios' => ['http_static'], + 'samples' => 1, + 'warmups' => 0, + 'request_count' => 1, + 'preflight' => $this->matchedCurrentPreflight(), + ], + ); + } finally { + self::assertSame(['stale-evidence.json'], $this->outputDirectoryFiles()); + self::assertFileDoesNotExist($this->outputDir . DIRECTORY_SEPARATOR . 'preflight.json'); + self::assertFileDoesNotExist($this->outputDir . DIRECTORY_SEPARATOR . 'manifest.json'); + self::assertDirectoryDoesNotExist($this->outputDir . DIRECTORY_SEPARATOR . 'raw'); + self::assertDirectoryDoesNotExist($this->outputDir . DIRECTORY_SEPARATOR . 'normalized'); + } + } + + private function removeDirectory(string $path): void + { + if (!is_dir($path)) { + return; + } + + $iterator = new \RecursiveIteratorIterator( + new \RecursiveDirectoryIterator($path, \FilesystemIterator::SKIP_DOTS), + \RecursiveIteratorIterator::CHILD_FIRST, + ); + + foreach ($iterator as $file) { + if ($file->isDir()) { + rmdir($file->getPathname()); + continue; + } + + unlink($file->getPathname()); + } + + rmdir($path); + } + + /** + * @param array $data + */ + private function hashJson(array $data): string + { + return hash('sha256', json_encode($data, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR) . PHP_EOL); + } + + /** + * @return list + */ + private function outputDirectoryFiles(): array + { + $files = array_values(array_diff(scandir($this->outputDir) ?: [], ['.', '..'])); + sort($files); + + return $files; + } + + /** + * @param list $mismatchedInvocations + * @param list $failedInvocations + * @param list $unavailableInvocations + */ + private function writeCountingWorker( + string $identityHash, + ?string $mismatchedIdentityHash = null, + array $mismatchedInvocations = [], + array $failedInvocations = [], + array $unavailableInvocations = [], + ): string { + $worker = $this->workerDir . DIRECTORY_SEPARATOR . 'counting-worker-' . bin2hex(random_bytes(4)) . '.php'; + $state = $this->workerDir . DIRECTORY_SEPARATOR . 'counting-worker-state-' . bin2hex(random_bytes(4)) . '.txt'; + $mismatchedIdentityHash ??= $identityHash; + $mismatchedInvocationsExport = var_export($mismatchedInvocations, true); + $failedInvocationsExport = var_export($failedInvocations, true); + $unavailableInvocationsExport = var_export($unavailableInvocations, true); + + file_put_contents($worker, << 'evolvephp.comparator.raw-result.v1', + 'comparator_id' => (string) (\$options['comparator'] ?? 'evolvephp'), + 'scenario_id' => \$scenarioId, + 'availability' => \$availability, + 'availability_status' => \$availability, + 'reason' => \$availability === 'unavailable' ? 'deterministic comparator unavailable' : null, + 'timing_boundary' => \$timingBoundary, + 'prepared_framework_instance_count' => \$preparedCount, + 'sample_count' => count(\$samples), + 'sample_index' => (int) (\$options['sample-index'] ?? 1), + 'samples' => \$samples, + 'unit' => 'microseconds', + 'memory' => ['current_bytes' => \$invocation, 'peak_bytes' => \$invocation], + 'warmups' => (int) (\$options['warmups'] ?? 0), + 'request_count' => (int) (\$options['request-count'] ?? 1), + 'operations_per_sample' => 1, + 'worker_environment_identity' => [ + 'schema_version' => 'evolvephp.comparator.runtime-identity.v1', + 'hash' => \$identityHash, + ], + 'dependency_context' => [], + 'process' => ['pid' => getmypid()], + 'subject_metadata' => [ + 'scenario_id' => \$scenarioId, + 'timing_boundary' => \$timingBoundary, + 'prepared_framework_instance_count' => \$preparedCount, + ], + 'last_result' => [ + 'status' => 'ok', + 'framework_constructed_in_subject' => \$scenarioId === 'application_boot', + 'framework_prepared_outside_subject' => \$scenarioId !== 'application_boot', + 'normal_framework_path_executed' => \$scenarioId !== 'application_boot', + ], +], JSON_THROW_ON_ERROR); +PHP); + + return $worker; + } + + /** + * @return array + */ + private function matchedCurrentPreflight(): array + { + return ComparatorPreflight::fromEnvironment( + BenchmarkEnvironment::capture(dirname(__DIR__, 3)), + dirname(__DIR__, 2) . '/comparators/matrix.json', + [ + 'php_version' => PHP_VERSION, + 'opcache_cli_enabled' => filter_var(ini_get('opcache.enable_cli'), FILTER_VALIDATE_BOOLEAN), + 'jit_enabled' => false, + 'phalcon_extension_version' => phpversion('phalcon') ?: null, + ], + ); + } +} diff --git a/benchmarks/tests/Comparator/ComparatorMatrixTest.php b/benchmarks/tests/Comparator/ComparatorMatrixTest.php index f9cd45a..cc86dbe 100644 --- a/benchmarks/tests/Comparator/ComparatorMatrixTest.php +++ b/benchmarks/tests/Comparator/ComparatorMatrixTest.php @@ -42,7 +42,7 @@ public function testComparatorMatrixEnforcesStableOrdering(): void $this->assertSame( array_keys($matrix1->comparators()), array_keys($matrix2->comparators()), - 'Comparator ordering must be deterministic' + 'Comparator ordering must be deterministic', ); } @@ -131,7 +131,7 @@ public function testMatrixIncludesAllRequiredFrameworks(): void $this->assertArrayHasKey( $framework, $comparators, - "Required framework '{$framework}' must be in matrix" + "Required framework '{$framework}' must be in matrix", ); } } @@ -181,11 +181,25 @@ public function testMatrixUsesExactSelectedVersions(): void $version = (string) $comparator['framework_version']; $this->assertDoesNotMatchRegularExpression('/\blatest\b/i', $version, "Comparator '{$id}' must not use latest wording"); - $this->assertDoesNotMatchRegularExpression('/\.x\b/i', $version, "Comparator '{$id}' must record an exact version"); $this->assertDoesNotMatchRegularExpression('/\^|~|\*/', $version, "Comparator '{$id}' must not use a Composer constraint as its selected version"); + + if ($id !== 'evolvephp') { + $this->assertDoesNotMatchRegularExpression('/\.x\b/i', $version, "Comparator '{$id}' must record an exact version"); + } } } + public function testEvolvePhpComparatorUsesStableDevelopmentIdentityWithoutCommitSuffix(): void + { + $matrixPath = $this->fixtureDir . DIRECTORY_SEPARATOR . 'matrix.json'; + $matrix = ComparatorMatrix::fromJsonFile($matrixPath); + $comparator = $matrix->comparator('evolvephp'); + + $this->assertIsArray($comparator); + $this->assertSame('2.0.x-dev', $comparator['framework_version']); + $this->assertDoesNotMatchRegularExpression('/\+[a-f0-9]{7,40}$/i', (string) $comparator['framework_version']); + } + public function testExternalComparatorPrincipalVersionsAreApprovedExactPins(): void { $matrixPath = $this->fixtureDir . DIRECTORY_SEPARATOR . 'matrix.json'; @@ -237,7 +251,7 @@ public function testMatrixLockHashesMatchLockfiles(): void $this->assertSame( strtolower((string) $comparator['lock_sha256']), hash_file('sha256', $lockPath), - "Comparator '{$id}' matrix lock hash must match its lockfile" + "Comparator '{$id}' matrix lock hash must match its lockfile", ); } } diff --git a/benchmarks/tests/Comparator/ComparatorResultNormalizerTest.php b/benchmarks/tests/Comparator/ComparatorResultNormalizerTest.php index a96de44..059a3bf 100644 --- a/benchmarks/tests/Comparator/ComparatorResultNormalizerTest.php +++ b/benchmarks/tests/Comparator/ComparatorResultNormalizerTest.php @@ -5,6 +5,7 @@ namespace Evolve\Benchmarks\Tests\Comparator; use Evolve\Benchmarks\Comparator\ComparatorResultNormalizer; +use Evolve\Benchmarks\Comparator\ComparatorRuntimeIdentity; use PHPUnit\Framework\TestCase; final class ComparatorResultNormalizerTest extends TestCase @@ -65,4 +66,96 @@ public function testUnavailableComparatorResultDoesNotInventTimingFields(): void self::assertArrayNotHasKey('p99', $result); self::assertArrayNotHasKey('timing', $result); } + + public function testFailedComparatorResultDoesNotBecomeUnavailable(): void + { + $result = ComparatorResultNormalizer::failed([ + 'id' => 'evolvephp', + 'name' => 'EvolvePHP', + 'framework_version' => '2.0.x-dev', + 'failure_reason' => 'fixture exploded', + ]); + + self::assertSame('failed', $result['availability']); + self::assertSame('fixture exploded', $result['failure_reason']); + self::assertNull($result['baseline_result']); + self::assertArrayNotHasKey('timing', $result); + } + + public function testRepeatedWarmBatchSamplesNormalizeToPerRequestLatency(): void + { + $normalized = ComparatorResultNormalizer::normalizeRawScenario([ + 'scenario_id' => 'http_repeated_warm', + 'samples' => [1000.0, 2000.0], + 'operations_per_sample' => 10, + 'unit' => 'microseconds', + 'memory' => [], + ], [ + 'source' => ['git_sha' => str_repeat('a', 40)], + 'fingerprint' => ['hash' => str_repeat('b', 64)], + ]); + + $scenario = $normalized['scenarios'][0]; + + self::assertSame('http_repeated_warm', $scenario['id']); + self::assertSame(10, $scenario['operations_per_sample']); + self::assertSame('per_operation_microseconds', $scenario['unit']); + self::assertSame(150.0, $scenario['mean']); + self::assertSame(1_000_000 / 150.0, $scenario['throughput_per_second']); + } + + public function testRawSamplesAreTheAcceptedSourceForNormalization(): void + { + $normalized = ComparatorResultNormalizer::normalizeRawScenario([ + 'scenario_id' => 'http_repeated_warm', + 'samples' => [999_999.0], + 'raw_samples' => [ + [ + 'sample_index' => 1, + 'sha256' => str_repeat('a', 64), + 'record' => ['sample_index' => 1, 'samples' => [1000.0]], + ], + [ + 'sample_index' => 2, + 'sha256' => str_repeat('b', 64), + 'record' => ['sample_index' => 2, 'samples' => [2000.0]], + ], + ], + 'operations_per_sample' => 10, + 'unit' => 'microseconds', + 'memory' => [], + ], [ + 'source' => ['git_sha' => str_repeat('a', 40)], + 'fingerprint' => ['hash' => str_repeat('b', 64)], + ]); + + $scenario = $normalized['scenarios'][0]; + + self::assertSame(2, $scenario['sample_count']); + self::assertSame(150.0, $scenario['mean']); + } + + public function testRuntimeIdentityConversionUsesOnlySuppliedCapturedEnvironment(): void + { + $capturedEnvironment = [ + 'runtime' => [ + 'php_version' => '8.4.25', + 'php_binary' => 'php', + 'php_sapi' => 'cli', + 'php_ini_loaded_file' => '/tmp/php.ini', + ], + 'opcache' => ['enabled' => true], + 'jit' => ['enabled' => false], + 'extensions' => ['json', 'phalcon'], + 'extension_versions' => [ + 'phalcon' => '5.20.3', + ], + ]; + + $first = ComparatorRuntimeIdentity::fromCapturedEnvironment($capturedEnvironment); + $second = ComparatorRuntimeIdentity::fromCapturedEnvironment($capturedEnvironment); + + self::assertSame($first, $second); + self::assertSame('5.20.3', $first['phalcon']['version']); + } } diff --git a/benchmarks/tests/Comparator/EnvironmentAndFixtureIdentityTest.php b/benchmarks/tests/Comparator/EnvironmentAndFixtureIdentityTest.php index 1e45d1b..c38a36d 100644 --- a/benchmarks/tests/Comparator/EnvironmentAndFixtureIdentityTest.php +++ b/benchmarks/tests/Comparator/EnvironmentAndFixtureIdentityTest.php @@ -16,12 +16,16 @@ public function testExecutionEnvironmentFingerprintExcludesLockHash(): void 'schema_version' => 'evolvephp.benchmark.environment.v2', 'runtime' => ['php_version' => '8.4.0'], 'platform' => ['os' => 'Linux'], + 'composer' => ['version' => 'Composer version 2.10.2'], + 'phpbench' => ['version' => '1.7.0'], 'opcache' => ['enabled' => true], 'jit' => ['enabled' => false], 'extensions' => ['json', 'pdo'], + 'lock' => ['hash' => str_repeat('a', 64)], ]; $environment2 = $environment1; + $environment2['lock']['hash'] = str_repeat('b', 64); $fp1 = ExecutionEnvironmentFingerprint::fromEnvironment($environment1); $fp2 = ExecutionEnvironmentFingerprint::fromEnvironment($environment2); @@ -30,6 +34,27 @@ public function testExecutionEnvironmentFingerprintExcludesLockHash(): void $this->assertArrayNotHasKey('lock_hash', $fp1['fields'], 'Lock hash must not be in execution environment fingerprint'); } + public function testDifferentPlatformCpuProducesDifferentExecutionEnvironmentHash(): void + { + $environment1 = [ + 'runtime' => ['php_version' => '8.4.25', 'php_sapi' => 'cli'], + 'platform' => ['os' => 'Linux', 'cpu_model' => 'AMD EPYC 7763', 'logical_cpu_count' => 4], + 'composer' => ['version' => 'Composer version 2.10.2'], + 'phpbench' => ['version' => '1.7.0'], + 'opcache' => ['enabled' => true], + 'jit' => ['enabled' => false], + 'extensions' => ['Core', 'json', 'phalcon'], + 'lock' => ['hash' => str_repeat('a', 64)], + ]; + $environment2 = $environment1; + $environment2['platform']['cpu_model'] = 'Intel Xeon Platinum'; + + $this->assertNotSame( + ExecutionEnvironmentFingerprint::fromEnvironment($environment1)['hash'], + ExecutionEnvironmentFingerprint::fromEnvironment($environment2)['hash'], + ); + } + public function testDifferentPhpVersionsProduceDifferentExecutionEnvironmentHash(): void { $environment1 = [ diff --git a/benchmarks/tests/Comparator/PhalconAvailabilityTest.php b/benchmarks/tests/Comparator/PhalconAvailabilityTest.php index 844e3e7..739ad08 100644 --- a/benchmarks/tests/Comparator/PhalconAvailabilityTest.php +++ b/benchmarks/tests/Comparator/PhalconAvailabilityTest.php @@ -7,6 +7,8 @@ use Evolve\Benchmarks\Comparator\PhalconAvailability; use PHPUnit\Framework\TestCase; +require_once dirname(__DIR__, 2) . '/comparators/phalcon/src/PhalconComparatorFixture.php'; + final class PhalconAvailabilityTest extends TestCase { public function testPhalconAvailabilityDetectsLoadedExtension(): void @@ -25,13 +27,16 @@ public function testPhalconAvailabilityDetectsLoadedExtension(): void public function testPhalconUnavailableReturnsConsistentStructure(): void { - // If extension is not available, the structure must still be deterministic $availability = PhalconAvailability::detect(); - if (!$availability['available']) { - $this->assertFalse($availability['extension_present']); - $this->assertSame('unavailable', $availability['status']); + if ($availability['available']) { + $this->markTestSkipped( + 'Phalcon extension is loaded; unavailable-state assertions are not applicable.' + ); } + + $this->assertFalse($availability['extension_present']); + $this->assertSame('unavailable', $availability['status']); } public function testPhalconAvailabilityCanBeMarkedUnavailable(): void @@ -51,4 +56,29 @@ public function testPhalconFixtureDoesNotCrashMatrixWhenUnavailable(): void // The actual Phalcon fixture will handle unavailability gracefully $this->assertIsArray($availability, 'Availability detection must return array structure'); } + + public function testPhalconMiddlewareImplementationUsesMicroBeforeHooks(): void + { + $fixtureSource = (string) file_get_contents(dirname(__DIR__, 2) . '/comparators/phalcon/src/PhalconComparatorFixture.php'); + + $this->assertStringContainsString('->before(', $fixtureSource); + $this->assertStringNotContainsString('for ($i = 1; $i <= 5; ++$i) {', $fixtureSource); + } + + public function testPhalconMiddlewareRuntimeUsesBeforeHooksWhenExtensionIsAvailable(): void + { + $availability = PhalconAvailability::detect(); + + if (!$availability['available']) { + self::markTestSkipped((string) ($availability['reason'] ?? 'phalcon extension not loaded')); + } + + $fixture = new \Benchmark\Phalcon\PhalconComparatorFixture(); + $result = $fixture->httpMiddleware(); + + self::assertSame(200, $result['status_code']); + self::assertSame('phalcon:middleware', $result['body']); + self::assertSame([1, 2, 3, 4, 5], $result['middleware_order']); + self::assertSame('before', $result['middleware_model']); + } } diff --git a/benchmarks/tests/Comparator/PreparedScenarioTimingBoundaryTest.php b/benchmarks/tests/Comparator/PreparedScenarioTimingBoundaryTest.php new file mode 100644 index 0000000..a434194 --- /dev/null +++ b/benchmarks/tests/Comparator/PreparedScenarioTimingBoundaryTest.php @@ -0,0 +1,101 @@ +prepareScenario('application_boot'); + $result = $subject->runOnce(); + + self::assertSame('application_boot', $subject->scenarioId()); + self::assertSame('application_boot_constructs_framework', $subject->timingBoundary()); + self::assertSame(0, $subject->preparedFrameworkInstanceCount()); + self::assertSame('ok', $result['status']); + self::assertTrue($result['framework_constructed_in_subject']); + self::assertFalse($result['framework_prepared_outside_subject']); + } + + public function testApplicationBootExecutorOverridesConfiguredWarmupsToZero(): void + { + $raw = ComparatorScenarioExecutor::run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + 'evolvephp', + 'application_boot', + 5, + 1, + ); + + self::assertSame('available', $raw['availability']); + self::assertSame('application_boot_constructs_framework', $raw['timing_boundary']); + self::assertSame(0, $raw['prepared_framework_instance_count']); + self::assertSame(0, $raw['warmups']); + self::assertTrue($raw['last_result']['framework_constructed_in_subject']); + self::assertFalse($raw['last_result']['framework_prepared_outside_subject']); + } + + public function testWarmHttpSubjectExcludesFrameworkConstructionButStillRoutes(): void + { + $fixture = new EvolvePhpComparatorFixture(); + $subject = $fixture->prepareScenario('http_parameterized', ['id' => '123']); + + $result = $subject->runOnce(); + + self::assertSame('http_parameterized', $subject->scenarioId()); + self::assertSame('prepared_warm_http_request', $subject->timingBoundary()); + self::assertSame(1, $subject->preparedFrameworkInstanceCount()); + self::assertFalse($result['framework_constructed_in_subject']); + self::assertTrue($result['normal_framework_path_executed']); + self::assertSame(200, $result['status_code']); + self::assertSame('evolvephp:parameterized:123', $result['body']); + self::assertSame(['id' => '123'], $result['parameters']); + } + + public function testWarmHttpExecutorPreservesConfiguredWarmups(): void + { + $raw = ComparatorScenarioExecutor::run( + dirname(__DIR__, 2) . '/comparators/matrix.json', + 'evolvephp', + 'http_static', + 3, + 1, + ); + + self::assertSame('available', $raw['availability']); + self::assertSame('prepared_warm_http_request', $raw['timing_boundary']); + self::assertSame(1, $raw['prepared_framework_instance_count']); + self::assertSame(3, $raw['warmups']); + self::assertFalse($raw['last_result']['framework_constructed_in_subject']); + self::assertTrue($raw['last_result']['normal_framework_path_executed']); + } + + public function testRepeatedWarmSubjectReusesOnePreparedFrameworkInstance(): void + { + $fixture = new EvolvePhpComparatorFixture(); + $subject = $fixture->prepareScenario('http_repeated_warm', ['request_count' => 5]); + + $result = $subject->runOnce(); + + self::assertSame('http_repeated_warm', $subject->scenarioId()); + self::assertSame('prepared_repeated_warm_requests', $subject->timingBoundary()); + self::assertSame(1, $subject->preparedFrameworkInstanceCount()); + self::assertFalse($result['framework_constructed_in_subject']); + self::assertTrue($result['prepared_framework_instance_reused']); + self::assertSame(5, $result['request_count']); + self::assertSame(1, $result['bootstrap_count']); + } +} diff --git a/tests/Architecture/EvolvePhp2BenchmarkHarnessTest.php b/tests/Architecture/EvolvePhp2BenchmarkHarnessTest.php index 7709a5a..0290a66 100644 --- a/tests/Architecture/EvolvePhp2BenchmarkHarnessTest.php +++ b/tests/Architecture/EvolvePhp2BenchmarkHarnessTest.php @@ -16,6 +16,9 @@ public function testBenchmarkToolingIsIsolatedUnderBenchmarksComposerRoot(): voi $this->assertFileExists($this->projectPath('benchmarks/composer.json')); $this->assertFileExists($this->projectPath('benchmarks/phpbench.json')); $this->assertFileExists($this->projectPath('benchmarks/bin/benchmark-smoke.php')); + $this->assertFileExists($this->projectPath('benchmarks/bin/comparator-run.php')); + $this->assertFileExists($this->projectPath('benchmarks/bin/comparator-worker.php')); + $this->assertFileExists($this->projectPath('benchmarks/bin/comparator-preflight.php')); $this->assertFileExists($this->projectPath('benchmarks/bin/capture-environment.php')); $this->assertFileExists($this->projectPath('benchmarks/bin/normalize-results.php')); $this->assertFileExists($this->projectPath('benchmarks/results/README.md')); @@ -75,11 +78,45 @@ public function testBenchmarkDocumentationRecordsProtocolAndNoMarketingClaim(): 'Cross-framework comparison', 'no current fastest-framework claim', 'no current top-three performance claim', + 'controlled comparator execution', + 'subprocess per measured sample', + 'worker runtime identity only to prove child-process conformance', + 'fresh output directory', + 'without writing raw or normalized timing artifacts', + 'zero in-process subject warmups', + 'operations_per_sample', + 'per-sample raw hashes', + 'Candidate evidence', + 'Canonical reference evidence', + 'Phalcon Micro before handlers', + 'non-ranking policy', ] as $phrase) { $this->assertStringContainsString($phrase, $readme); } } + public function testBenchmarkPublicDocsDoNotContainInternalReviewProvenance(): void + { + foreach ([ + 'benchmarks/README.md', + 'benchmarks/results/README.md', + ] as $path) { + $content = $this->readProjectFile($path); + + foreach ([ + '/Codex/i', + '/ChatGPT/i', + '/AI-generated/i', + '/assistant review/i', + '/credits/i', + '/usage limit/i', + '/internal gate labels/i', + ] as $pattern) { + $this->assertDoesNotMatchRegularExpression($pattern, $content, $path . ' must not expose internal review provenance.'); + } + } + } + /** * @return list */ diff --git a/tests/Documentation/EvolvePhp2BenchmarkDocumentationTest.php b/tests/Documentation/EvolvePhp2BenchmarkDocumentationTest.php new file mode 100644 index 0000000..284d08a --- /dev/null +++ b/tests/Documentation/EvolvePhp2BenchmarkDocumentationTest.php @@ -0,0 +1,53 @@ +readProjectFile('benchmarks/results/README.md'); + + foreach ([ + 'results/local/comparator-candidate/', + 'manifest.json', + 'raw/', + 'normalized/', + 'source SHA and dirty state', + 'matrix hash', + 'subprocess command lines', + 'comparator lock hash', + 'execution environment identity/fingerprint', + 'worker runtime identity', + 'per-sample raw hashes', + 'fresh output directory', + 'stale output is never deleted implicitly', + 'preflight rejects the controlled lane on a fresh path', + 'one subprocess per measured sample', + 'one separate discarded application_boot worker', + 'rotating round-robin', + 'zero in-process warmups', + 'discarded workers are excluded from statistics', + 'p50 is the primary central comparison statistic for application_boot', + 'mean, p95, p99 and relative standard deviation remain visible', + 'No measured sample is removed as an outlier', + 'actual worker-order provenance', + 'operations_per_sample', + 'Candidate output from a dirty or uncommitted worktree is not canonical reference evidence', + 'controlled PHP 8.4.25 lane', + ] as $phrase) { + $this->assertStringContainsString($phrase, $content); + } + } + + private function readProjectFile(string $path): string + { + $fullPath = dirname(__DIR__, 2) . DIRECTORY_SEPARATOR . str_replace('/', DIRECTORY_SEPARATOR, $path); + $this->assertFileExists($fullPath, $path . ' should exist before it is read.'); + + $content = file_get_contents($fullPath); + $this->assertIsString($content); + + return $content; + } +}