Skip to content

Keep the soak SSH session alive during cache drain - #159

Merged
rmcdaniel merged 1 commit into
mainfrom
fix/soak-ssh-drain
Sep 10, 2026
Merged

Keep the soak SSH session alive during cache drain#159
rmcdaniel merged 1 commit into
mainfrom
fix/soak-ssh-drain

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Sep 10, 2026

Copy link
Copy Markdown
Member

Problem

The September 10 daily soak completed all 1,440 load samples, then its SSH connection failed with Broken pipe during the intentionally silent 310-second cache drain. That prevented the final sample and summary from being retained. The controller still deleted the disposable VM and verified that it was absent.

Change

Use OpenSSH's built-in ServerAliveInterval=30 and ServerAliveCountMax=6 for the existing SSH transport. No custom heartbeat, retry coordinator, additional paid run, or change to the workload, host size, drain duration, or acceptance thresholds.

The regression test invokes ssh -G using the actual shell options, without a connection or provider credentials.

Verification

  • 24 focused Python tests passed.
  • bash -n scripts/perf/run-vultr-soak.sh passed.
  • git diff --check passed.
  • Next scheduled run must confirm the quiet drain and final report complete over the real transport.

This only addresses report transport. The same run retained 575 validated workflow completions from 595 attempts, 20 client wait timeouts, and four readiness-probe timeouts. Those findings remain unresolved; this is not a passing endurance result.

Related: #137

@rmcdaniel
rmcdaniel marked this pull request as ready for review September 10, 2026 14:40
@rmcdaniel

Copy link
Copy Markdown
Member Author

Reviewed the two-file diff: this only sets native SSH liveness options and tests the effective OpenSSH configuration. Provisioning credentials, host-key handling, cleanup, workload, memory thresholds and 310-second drain are unchanged.

All public checks now pass, including PHP feature qualification and the real short polling/cache-drain smoke. Local focused tests: 24 passed. This does not validate the remote two-hour transport itself or resolve the workload timeouts in the daily-run review.

No package, SDK, image or production deployment is needed. The next ordinary scheduled soak uses this repository script; no extra paid run is being triggered.

@rmcdaniel
rmcdaniel merged commit 762f799 into main Sep 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants