Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/server-perf-soak.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ jobs:
DW_PERF_CONCURRENCY: ${{ github.event_name == 'workflow_dispatch' && inputs.concurrency || '24' }}
DW_PERF_NAMESPACES: "8"
DW_PERF_TASK_QUEUES: "16"
DW_PERF_STANDARD_WORKFLOWS: "true"
DW_PERF_REDIS_CACHE_DB: "1"
DW_PERF_MAX_SERVER_MEMORY_MB: "1024"
DW_PERF_MAX_POLLING_KEYS: "2048"
Expand All @@ -66,10 +67,20 @@ jobs:
DW_PERF_RUNNER_ENVIRONMENT: "self-hosted"
run: scripts/perf/run-vultr-soak.sh

- name: Summarize measured workload
if: always()
run: |
if [ -f build/perf/summary.md ]; then
cat build/perf/summary.md >> "$GITHUB_STEP_SUMMARY"
else
printf '## Server endurance soak\n\nNo complete measurement summary was produced. Inspect setup, timeout, and provisioning logs; this is not a pass.\n' >> "$GITHUB_STEP_SUMMARY"
fi

- name: Upload perf artifacts
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: server-perf-soak
path: build/perf/
if-no-files-found: warn
retention-days: 14
4 changes: 4 additions & 0 deletions benchmarks/capacity/v1/bindings/php/capacity_adapter.php
Original file line number Diff line number Diff line change
Expand Up @@ -623,6 +623,10 @@ function capacityConformanceEvidence(array $fixtures): array
];
}

if (realpath($_SERVER['SCRIPT_FILENAME'] ?? '') !== __FILE__) {
return;
}

$mode = $argv[1] ?? '';
if ($mode === 'describe') {
echo json_encode(capacityAdapterDescriptor(), JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES).PHP_EOL;
Expand Down
1 change: 1 addition & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ services:
WORKFLOW_PACKAGE_REF: ${WORKFLOW_PACKAGE_REF:-}
WORKFLOW_PACKAGE_COMMIT: ${WORKFLOW_PACKAGE_COMMIT:-}
command: php artisan queue:work --sleep=1 --tries=3 --max-time=3600
restart: unless-stopped
healthcheck:
test: ["CMD", "server-process-healthcheck", "worker"]
interval: 5s
Expand Down
44 changes: 43 additions & 1 deletion docs/perf-runner.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,49 @@
# Server Perf Runner

The server perf harness exercises the HTTP worker polling path for bounded
memory growth and polling-cache cleanup.
memory growth and polling-cache cleanup, with a low-rate PHP standard-workflow
canary alongside it. The daily Action summary exposes both results directly.

This is an endurance test, not a capacity claim: the canary uses the existing
`capacity.v1.one_activity` binding (one 1-KiB echo activity), one workflow at a
time, with at most one start every five seconds. It verifies each result,
completed status, and ordered activity history. It skips missed start slots
instead of accumulating a backlog. Its rate is not the capacity suite's
concurrency/warmup measurement protocol. A slow correct completion is measured,
not rejected against an invented throughput target; a result timeout is 30s.

Health, readiness, and cluster-info probes run even without workflow-row growth.
Artifacts include HTTP p50/p95/p99, backpressure separately from successful
polls, sampled CPU/memory for Server, its queue worker/scheduler, MySQL, Redis,
and the PHP SDK process container. The SDK client and worker share a 0.5-CPU,
256-MiB container on the existing host. A 200 with an invalid health payload,
missing probe results, or an entirely backpressured poll run fails validation.
Compose restarts the Server queue worker after its configured hourly recycle;
an exited queue worker must not disappear unnoticed for the second soak hour.
Each synthetic polling thread owns and heartbeats one registration, distributed
across the configured namespaces and queues (24 workers, not 128 Cartesian
registrations, in the daily default). HTTP success carrying a stale/rejected
worker status fails the run. Polling-cache activity must be present in at least
the configured minimum sample fraction, not merely once at startup.

The default post-load drain is 310 seconds. Real activity poll-result caches
retain replay evidence through their five-minute lease plus five seconds;
heartbeat retention throttles and wake signals also outlive the old 12-second
idle-only drain. No cache flush, shorter production TTL, or relaxed final-key
limit is used to obtain a pass. The load remains two hours; build/bootstrap,
bounded drain, and teardown are additional host time.

`DW_PERF_STANDARD_WORKFLOWS=false` disables only the SDK canary for a focused
polling reproduction. The Compose wrapper enables it by default. Raw
`server_soak.py` invocation does not build the SDK image; use the wrapper.

The soak still does not exercise timer/signal/query execution, recovery fault
injection, all SDK bindings, or HA. Those need the separate conformance and
performance qualification experiments. Zero cache keys for an unexercised
policy do not prove that feature works. Review failed setup and cleanup steps
as well as workload results; neither is a successful experiment. Scheduled
artifacts are retained for 14 days; record significant findings on the owning
issue rather than committing generated run directories.

## Runner Shape

Expand Down
14 changes: 14 additions & 0 deletions scripts/perf/Dockerfile.sdk
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
FROM composer:2 AS dependencies
WORKDIR /sdk
COPY benchmarks/capacity/v1/bindings/php/composer.json benchmarks/capacity/v1/bindings/php/composer.lock ./
RUN composer install --no-dev --no-interaction --no-progress --prefer-dist

FROM php:8.3-cli
RUN docker-php-ext-install pcntl
WORKDIR /app
COPY benchmarks/capacity/v1/bindings/php benchmarks/capacity/v1/bindings/php
COPY --from=dependencies /sdk/vendor benchmarks/capacity/v1/bindings/php/vendor
COPY scripts/perf/standard_workflow_soak.php scripts/perf/standard_workflow_soak.php
USER 1000:1000
ENTRYPOINT ["php"]
CMD ["benchmarks/capacity/v1/bindings/php/capacity_adapter.php", "worker"]
14 changes: 11 additions & 3 deletions scripts/perf/run-server-soak.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,11 @@ PROMETHEUS_CONTAINER="${PROJECT}-prometheus"
PROMETHEUS_CONFIG_DIR=""

mkdir -p "$ARTIFACT_DIR"
export DW_PERF_DRAIN_SECONDS="${DW_PERF_DRAIN_SECONDS:-310}"

if [ -z "$LOAD_TIMEOUT_SECONDS" ]; then
DURATION_SECONDS="${DW_PERF_DURATION_SECONDS:-120}"
DRAIN_SECONDS="${DW_PERF_DRAIN_SECONDS:-12}"
DRAIN_SECONDS="$DW_PERF_DRAIN_SECONDS"
LOAD_TIMEOUT_SECONDS=$((DURATION_SECONDS + DRAIN_SECONDS + 300))
if [ "$LOAD_TIMEOUT_SECONDS" -lt 300 ]; then
LOAD_TIMEOUT_SECONDS=300
Expand All @@ -57,6 +58,7 @@ export DW_WORKER_TOKEN="${DW_WORKER_TOKEN:-}"
export DW_OPERATOR_TOKEN="${DW_OPERATOR_TOKEN:-}"
export DW_ADMIN_TOKEN="${DW_ADMIN_TOKEN:-}"
export DW_AUTH_BACKWARD_COMPATIBLE="${DW_AUTH_BACKWARD_COMPATIBLE:-true}"
export DW_PERF_STANDARD_WORKFLOWS="${DW_PERF_STANDARD_WORKFLOWS:-true}"

OVERRIDE_FILE="$ARTIFACT_DIR/docker-compose.perf.yml"
cat > "$OVERRIDE_FILE" <<YAML
Expand Down Expand Up @@ -96,6 +98,8 @@ services:
ports: !override []
YAML

compose=(docker compose -p "$PROJECT" -f "$ROOT_DIR/docker-compose.yml" -f "$OVERRIDE_FILE" -f "$ROOT_DIR/scripts/perf/standard-workflow.compose.yml")

cleanup() {
local status=$?

Expand All @@ -104,13 +108,14 @@ cleanup() {
docker logs "${PROJECT}-scheduler-1" > "$ARTIFACT_DIR/scheduler.log" 2>&1 || true
docker logs "${PROJECT}-mysql-1" > "$ARTIFACT_DIR/mysql.log" 2>&1 || true
docker logs "${PROJECT}-redis-1" > "$ARTIFACT_DIR/redis.log" 2>&1 || true
docker logs "${PROJECT}-soak-sdk-1" > "$ARTIFACT_DIR/soak-sdk.log" 2>&1 || true

docker rm -f "$PROMETHEUS_CONTAINER" >/dev/null 2>&1 || true
if [ -n "$PROMETHEUS_CONFIG_DIR" ]; then
rm -rf "$PROMETHEUS_CONFIG_DIR"
fi

docker compose -p "$PROJECT" -f "$ROOT_DIR/docker-compose.yml" -f "$OVERRIDE_FILE" down -v --remove-orphans || true
"${compose[@]}" --profile standard-soak down -v --remove-orphans || true
exit "$status"
}
trap cleanup EXIT
Expand Down Expand Up @@ -224,7 +229,10 @@ else
echo "Starting perf stack with project ${PROJECT} on a dynamic host port"
fi
setup_status=0
docker compose -p "$PROJECT" -f "$ROOT_DIR/docker-compose.yml" -f "$OVERRIDE_FILE" up -d --build --wait || setup_status=$?
"${compose[@]}" up -d --build --wait || setup_status=$?
if [ "$setup_status" -eq 0 ] && [ "$DW_PERF_STANDARD_WORKFLOWS" = true ]; then
"${compose[@]}" build soak-sdk || setup_status=$?
fi
if [ "$setup_status" -ne 0 ]; then
echo "Perf environment setup failed before product smoke execution; docker compose could not build or start the stack." >&2
write_environment_setup_failure "$setup_status" "docker_compose_up" "docker compose failed before server_soak.py started"
Expand Down
1 change: 1 addition & 0 deletions scripts/perf/run-vultr-soak.sh
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,7 @@ for name in \
DW_PERF_CONCURRENCY \
DW_PERF_NAMESPACES \
DW_PERF_TASK_QUEUES \
DW_PERF_STANDARD_WORKFLOWS \
DW_PERF_REDIS_CACHE_DB \
DW_PERF_MAX_SERVER_MEMORY_MB \
DW_PERF_MAX_POLLING_KEYS \
Expand Down
Loading