Durable Workflow Server performance qualification

[rmcdaniel]

Objective And Priority

Determine which HTTP/PHP execution model gives Durable Workflow Server the most useful capacity per CPU and GB while preserving durability, namespace isolation, task leasing, fencing, and recovery behavior. Do not assume any proposed technology is faster.

Accepted, deferred until the current committed work is complete. This is not an immediate runtime migration or a new release gate.

Ownership And Scope

This issue owns public Server engineering: the published default image, generic HTTP/PHP configurations, application hotspots, long polling, portable SDK correctness, and reproducible backend-recovery tests useful to self-hosted users.

It does not qualify managed-service plans, provider-specific deployments, customer SLAs, pricing/margins, or Cloud production rollout. Those decisions and their private evidence are separate Cloud work; a Server benchmark pass does not authorize a managed-runtime change. Keep private infrastructure and customer information out of this issue and its artifacts.

Keep the published default Server configuration unchanged throughout experimentation. Any recommended default change needs a separately reviewed Server implementation and release, followed by independent deployment qualification by its consumers.

Baseline And Experimental Contract

• Inspect Dockerfile, docker/apache-mpm-prefork.conf, docker/php-custom.ini, docker/entrypoint.sh, app/Support/LongPoller.php, workflow/activity/query pollers, benchmarks/capacity, and existing performance/soak scripts. Document the actual execution model before changing anything: HTTP serving, PHP worker count, Laravel bootstrap lifecycle, waiting polls, Redis wake signals, and SQL/Redis work per DW Standard Workflow v1.
• Run the existing DW Standard Workflow v1 benchmark against the untouched published production image. Record the Server image digest and source SHA, bundled workflow package and SDK versions, benchmark-runner commit, PHP/extensions, host CPU/RAM, container CPU/memory/swap limits, OS/kernel, storage, MySQL/Redis configuration, worker counts, concurrency, payloads, warmup and measurement durations, and final backlog.
• Record offered and completed workflows/sec separately, p50/p95/p99 end-to-end latency, errors/timeouts/retries, CPU utilization, peak RSS and total container memory, and SQL/Redis counts where measurable. Define latency boundaries and whether draining is included; do not inflate sustained throughput with an unreported drain period.
• Freeze the workload and artifact tuple for the comparison. Keep hardware, database/cache placement and configuration, client/worker resources, network path, payloads, and measurement windows identical. Verify that load generation or SDK workers are not the bottleneck. Treat different hardware or PHP versions as separate experiments, not equivalent comparisons.
• Use warmup and at least three measured repetitions for the baseline and finalists, reporting variation rather than a single best run. Recheck the baseline during the series to detect host drift. Preserve commands and raw outputs, including failed runs.

The initial job is to measure, not to meet an invented throughput target. A low but correct result is valid evidence. State any latency, saturation, or memory stop conditions before each experiment, and distinguish measured capacity from an advertised service guarantee.

Runtime Candidates

Build isolated Docker/config variants without replacing the production Dockerfile. Initially use the same PHP version where supported.

Candidate	Qualification
Apache prefork + mod_php	Untouched baseline, then separately identified tuning
nginx + PHP-FPM	Same application and public protocol
Apache event MPM + PHP-FPM	Same application and public protocol
Laravel Octane + FrankenPHP	Persistent-worker compatibility and state reset required
Laravel Octane + Swoole or OpenSwoole	Only if functional without changing public Server semantics

• Attempt each candidate within a bounded setup effort. Record unsupported combinations and concrete failures; do not silently omit them or build a new compatibility framework to force them to work.
• Test a small worker-count matrix appropriate to the fixed CPU/RAM envelope. Record worker count, memory per worker, total memory, CPU, and throughput/latency. Stop tuning after a reasonable stable point; reject apparent gains caused by exceeding limits, swapping uncontrollably, accumulating backlog, or growing memory without bounds.
• Describe operational differences: process supervision, worker recycling, deployment/reload behavior, readiness, observability, additional services, and maintenance burden.

Long Polling

• Measure workflow-task, activity-task, and query polls separately and in a realistic mixture. Identify which endpoints actually wait and their timeout semantics. Increase concurrent idle polls to a defined resource/latency limit, recording memory per waiting poll, idle CPU, task-created-to-worker-wakeup latency, disconnect behavior, and ordinary API latency while polls wait.
• Determine whether current waiting polls occupy a full PHP process/thread, and whether a candidate's database/Redis clients or other blocking calls prevent the expected concurrency. Do not infer asynchronous behavior from the runtime name.
• Inspect LongPoller and Redis wakeups. Where practical, compare coroutine/event waiting, FrankenPHP-compatible alternatives, blocking/event Redis notifications, or a lightweight asynchronous poll gateway using established components.
• For every polling prototype explain and test wakeup races, timeout, cancellation/disconnect cleanup, task claiming, lease expiry, and fencing. Wake notifications must not become the authority for durable task ownership. Do not change admission limits just to increase the reported number without measuring the consequence.

Profiling And Independent PHP Tuning

• Profile standard workflows: SQL count/cumulative time and highest-frequency/slowest queries; Redis command count/cumulative time; Laravel request CPU; payload/serialization cost where measurable. Identify the top five measured server-side costs and record measurement limitations. Keep intrusive profiling separate from throughput runs and quantify its overhead where practical.
• Make application optimizations only for measured hotspots, with isolated before/after comparisons and unchanged correctness contracts.
• Independently test opcache.validate_timestamps=0 for immutable images, Composer authoritative classmaps, OPcache sizing, preload if practical, and a newer supported PHP version if dependencies allow. Keep these results separate from HTTP-runtime comparisons so improvements can be attributed correctly.

Correctness And Recovery

• Run the existing Server test/conformance suite against every materially faster finalist. Use published first-party PHP, Python, and Rust SDKs for relevant end-to-end protocol coverage; keep the performance workload itself fixed. Do not equate a PHP-only happy path with full portable correctness.
• Persistent workers must prove no cross-request mutable state or cross-namespace/authentication leakage, including alternating/concurrent authenticated requests, failures followed by new requests, and worker recycling.
• Cover workflow start, workflow-task poll/completion, activity poll/completion, signals, queries, updates, timers, worker registration, external payloads, failure/retry, cancellation, and application reload/restart. Check history/results, duplicate claims/completions, and final backlog, not only HTTP success rates.
• Test generic backend recovery in reproducible, non-production fixtures: MySQL connection loss/restart or primary change and Redis interruption/promotion. Existing MySQL Router and Redis Sentinel fixtures may be used where applicable; these are reusable Server tests, not a qualification of any particular managed topology. Verify polls during interruption, stale persistent connections being discarded, reconnect behavior, lease/fencing correctness, and readiness failing and recovering. A faster candidate that weakens these Server guarantees is not acceptable.
• Run finalists for at least two hours with representative short requests and long polls. Record initial/peak/final memory, slope per hour after warmup, worker recycling, errors, completions, final backlog, Redis key growth, and server-owned cache growth. Distinguish bounded cache warmup from a leak; inspect recycling behavior rather than concealing growth behind restarts.

Deliverables

• A readable report with this comparison table and definitions for the reported units, load level, and limits:

Candidate	WF/s	Delta baseline	p95	p99	CPU	Peak RAM	Idle polls	Poll wake latency	2h memory slope	Correctness	Operational complexity

• Include p50, repetitions/variation, exact commands, all Docker/config variants, hardware/environment identity, failures and rejected experiments, measured hotspots, and raw artifact links. Mark unmeasured fields as such rather than estimating them.
• Keep reproducible source/configuration in small isolated commits in this repository. Store sanitized raw run outputs in existing artifact storage with an explicit bounded retention period sufficient for review. Do not commit permanent generated per-run directories or create a new benchmark orchestration platform.
• Recommend one of: keep Apache/mod_php; move to nginx/PHP-FPM; move to Apache event/PHP-FPM; move to Octane/FrankenPHP; move to Octane/Swoole; retain the HTTP runtime but redesign long polling; combine a runtime change with a long-poll redesign. State the measured reason, correctness/recovery tradeoffs, operational cost, confidence, and any remaining qualification required before a rollout.

Constraints

• No changes to public API semantics, workflow durability, task leasing, fencing, namespace isolation, or failover guarantees.
• Use isolated fixture data only. No live customer workloads or production configuration changes during experiments.
• No successful benchmark claims from partial, failed, or correctness-unqualified runs. Preserve failures as findings, not passes.
• Do not optimize DW Standard Workflow v1 at the expense of timers, signals, queries, activities, or long polls; measure those separately.
• Reuse the existing benchmark/conformance commands and mature OSS. Keep automation small and approachable; do not build a new orchestration platform.

Related completed foundations: #104 (DW Standard Workflow v1 definition), #86 (disposable performance soak runner). This issue evaluates execution architecture; it does not reopen their completed work.

[rmcdaniel]

The September 3-9 daily soaks were audited from their raw time series, not just Action conclusions.

The old green results do not establish two-hour live-worker polling or memory-leak qualification. Every one of these seven runs had polling-cache activity in only 7 of 1,440 periodic samples. Synthetic workers registered once but never heartbeated. After registration freshness expired, Server returned HTTP 200 with a terminal stale-registration status before entering the polling path; the harness accepted that response and required cache activity only once.

Run	Poll requests	HTTP 429	Peak Server MiB	Reported slope MiB/h	Active polling samples
Sep 3	356,505	0	136.0	+0.07	7 / 1,440
Sep 4	318,127	522	136.6	-0.08	7 / 1,440
Sep 5	312,120	525	134.8	-0.14	7 / 1,440
Sep 6	303,483	502	135.5	-0.04	7 / 1,440
Sep 7	422,923	572	136.4	-0.01	7 / 1,440
Sep 8	288,280	473	135.6	-0.41	7 / 1,440
Sep 9	319,916	508	136.1	-0.31	7 / 1,440

All seven started zero workflows. Healthy resource samples and zero final cache keys cannot compensate for the missing sustained workload. The daily default also omitted health/readiness/cluster-info probes and CPU measurements.

September 2 failed before measurement on a GitHub dist-download HTTP 504 during Composer installation, not a Server runtime defect. September 6 finished its workload but failed provider cleanup; #128 fixed cleanup verification, and September 7/8 confirmed deletion.

The audit also found a product Compose defect: queue:work exits normally after its configured one-hour lifetime, but Docker had no restart policy. Tracked in #151.

Merged repair: #150

• Maintain one heartbeated registration per polling thread and reject stale HTTP-200 responses.
• Require polling activity throughout the measurement window, not only startup.
• Exercise the existing PHP standard-workflow binding with result, status, and ordered activity-history validation.
• Always probe health/readiness/cluster-info; report latency distributions, backpressure, and component CPU/memory.
• Use Docker's native restart policy for the queue worker.
• Respect the real five-minute activity replay-cache lease with a 310-second final drain; exclude idle drain from load CPU and memory-slope measurements.
• Publish an Action summary and retain diagnostic artifacts for 14 days. Provider credentials remain only in the protected main-only job environment.

Local 90-second qualification at 0997b9c completed 18/18 workflows, zero errors, active polling in all 18 samples, healthy probes, and zero final cache keys. Laravel's native queue:restart recovered automatically during load. The final measurement correction at 882628f has focused regression tests. This is short harness qualification, not two-hour, capacity, cross-SDK, or failover evidence.

#150 merged as 080464e after all checks passed. The final CI smoke completed 17/17 standard workflows during 120 seconds with zero errors, polling activity in all 24 samples, healthy resource sampling, and zero final Server cache keys.

September 9 finished at 14:26 UTC with the old harness. It is also unqualified for sustained live polling despite green assertions. Its controller verified VM deletion at 14:25:57 UTC; a separate read-only provider lookup returned HTTP 404 at 14:27 UTC. No paid host remains from this run.

The same scheduled host and two-hour load remain; no extra paid rerun was started. Follow-up: inspect the first scheduled run containing 080464e for actual sustained activity, real completions, resource trends, queue-worker hourly recovery, final cache drain, and confirmed host deletion. Short CI/local results do not replace that two-hour evidence. The broader runtime, long-poll, profiling, and recovery comparison in this issue remains open.

[rmcdaniel]

September 10 daily soak: failed, reviewed

Run and retained artifact, source 7a38887ae8f18da2051f2740adec245c40d9fe16, 2 vCPU / 4 GB disposable host, 7,200-second load, 24 synthetic polling workers, plus the PHP Standard Workflow canary.

Observation	Result
Periodic samples with active polling	1,440 / 1,440
Standard Workflow attempts	595
Validated completions	575
Client result-wait timeouts	20, at the existing 30-second limit
Readiness-probe timeouts	4
Server memory during load	163.2 MiB peak; 115.7 MiB at last load sample
Server post-warmup memory slope during load	+0.52 MiB/hour
Final drained cache/backlog and complete summary	Missing
Disposable host cleanup	Controller deleted it and verified provider absence

The heartbeat correction now exercises sustained polling throughout the run, unlike the earlier runs whose registrations went stale. However, poll-loop errors=0 does not include the separate readiness and workflow errors.

The SSH connection failed with Broken pipe roughly 261 seconds into the intentionally silent 310-second post-load drain. PR #159 adds native OpenSSH keepalives; it does not shorten the drain or change workload thresholds. Local transport/soak tests pass. The next scheduled run must verify actual end-to-end reporting.

The workload also failed independently of SSH. The 20 client timeouts do not prove terminal workflow failures or data loss; they prove results were not returned within 30 seconds. Terminal histories and worker logs were not retained before host deletion, so their later outcomes and the exact bottleneck are unknown. Completion-only latency was p50 7.19s / p95 12.50s / p99 23.95s, excluding those 20 timeouts, not a capacity or plan claim.

Next: investigate short-workflow progress while all 24 polling clients remain active, with bounded local reproduction and history/backend timing inspection. Do not raise the client deadline or report a pass merely to accommodate unexplained stalls. No additional paid run has been launched. No production runtime change is justified by this partial failure.

[rmcdaniel]

September 11 scheduled soak: failed, daily spending paused

Run 34597695626, runner commit 762f799e586487f2ad45c613f02959f8522c6047, failed. The retained artifact is server-perf-soak (14-day retention).

• Only 126 of 1,440 expected periodic samples were written, spanning about 625 seconds. mysql_counts() tried to parse Unavailable: as an integer and raised ValueError. There is no final drain sample or complete summary. This is not a bounded-memory pass.
• The independent standard-workflow client ran 7,200 seconds: 728 attempts, 722 validated completions, six 30-second result timeouts. Two readiness-request timeouts were also recorded. Fixing the sampler will not make these workload failures disappear; they still need investigation.
• The exception occurs inside the load executor context, so unwinding waits for the already-running load tasks; this explains why an early sampling failure can be reported only at the end. The raw failed command output is not retained, so its underlying cause is not yet established.
• Cleanup completed and the runner verified the disposable host absent at 14:21:21 UTC. No additional run was triggered.

I set repository variable DW_PERF_SOAK_ENABLED=false to avoid paying for the same incomplete measurement tomorrow. Next: add a focused parser/command-failure regression, retain an explicitly unhealthy sample instead of aborting collection, verify recovery on the next sample, and review the actual workload timeouts. Re-enable the existing schedule only after the instrumentation fix is checked. No timeout threshold or performance target is being relaxed.

[rmcdaniel]

PR160 is merged as a65c4e17cc1c9905f48d522740d5f5ff0a5e7314. All public checks passed, including the feature suite and bounded-growth smoke. Local sampler/cleanup regressions: 27 tests pass.

The daily paid schedule remains paused. This fixes evidence collection, not the six workflow result timeouts or two readiness timeouts from the two-hour run. Next is a bounded local reproduction of those failures with the same workload configuration, then a deliberate decision to resume the existing schedule. No new paid run or package release was started.

[rmcdaniel]

Follow-up from the retained September 11 soak artifact: Apache logged AH00161: server reached MaxRequestWorkers at 12:29:02 UTC; the configured pool is 24. All six timed-out canary runs continued receiving HTTP 200 workflow-status responses while waiting, and the retained worker/MySQL logs contain no matching exception. That does not yet distinguish dispatch starvation, SDK progress or ordinary request contention.

A non-billable, isolated 300-second reproduction is now using a65c4e17, 24 pollers, 8 namespaces, 16 queues and the unchanged standard workflow canary, followed by the normal 310-second drain. No production worker limit, admission setting or deadline is changed. This local machine differs from the paid host, so it is diagnostic reproduction, not a replacement two-hour performance claim. I will inspect live task/history state on a stall. The daily paid schedule remains disabled; no new paid run was triggered.

[rmcdaniel]

The 300-second local reproduction at a65c4e17cc1c9905f48d522740d5f5ff0a5e7314 completed at 2026-09-11 15:18 UTC, including the normal 310-second drain. It retained the 24 poll producers, 8 namespaces, 16 queues, published PHP SDK 2.0.1 canary, and existing deadlines/admission settings.

• 60/60 DW Standard Workflows completed, no canary errors; p50 2.633s, p95 3.749s, p99 3.993s.
• 61 samples, no failed MySQL/Redis/resource samplers; polling workload reported zero errors.
• Disposable containers, networks and volumes were removed by the normal cleanup.
• This used a local 4-CPU/approximately 15.5-GiB Docker host, not the paid 2-CPU/4-GiB host. It is neither a comparable capacity result nor a two-hour endurance pass. It did not reproduce the six prior timeouts.

A separate deterministic liveness defect is now confirmed in the PHP worker: sdk-php#66. Both installed 2.0.1 and 2.0.9 can indefinitely retry one capacity-refused poll kind while never servicing another. The SDK fix does not establish the full cause of the paid run's six failures.

Next: qualify and publish that SDK fix, update the pinned canary, then continue the constrained timeout investigation. The paid daily schedule remains disabled; sampler repair alone is not sufficient to resume it.

[rmcdaniel]

The PHP worker backpressure fix is published as SDK 2.0.10 and is now consumed by the standard-workflow soak via #161, merged 8439eefedf6031849295e4572b96853fba982374.

PR smoke: 120 seconds, 11/11 standard workflows completed, zero errors, 25 healthy samples and final tracked polling/server cache keys zero. The raw summary retains 22 ready synthetic tasks; this is not a claim of empty task backlog or two-hour qualification. Source-built Server, Workflow 2.0.12, installed PHP SDK 2.0.10.

The paid daily schedule remains disabled (DW_PERF_SOAK_ENABLED=false). The six earlier result timeouts are not yet reproduced or attributed to this SDK defect. Next investigation should use the corrected sampler and SDK while retaining the same deadlines and recording complete final results; do not resume paid daily runs simply because the short smoke passed.