ci: let Phoenix SLURM jobs wait for a node up to the job timeout (queue cap default 0)

[sbryngelson]

What

Restores the pre-#1763 behavior: a SLURM job waits for a compute node up to the job-level timeout-minutes (480m / 8h) instead of being cut off after 4h.

Changes only the default of SLURM_MAX_QUEUE_SECONDS in monitor_slurm_job.sh from 14400 (4h) to 0 (wait indefinitely — the 480m job timeout is the backstop). The enforcement loop already treats 0 as "no cap," so this is a one-value change; #1763's build+test-in-one-allocation improvement is untouched.

Why

On Phoenix's preemptible embers QOS, jobs routinely sit PENDING for hours and need most of the 8h window to backfill onto a free GPU node. The 4h cap turned ordinary queue pressure into red CI:

##[error]SLURM job … did not start within 14404s (SLURM_MAX_QUEUE_SECONDS)
QUEUE STARVATION → CANCELLED → exit 1

These aren't test or code failures — just the free QOS not landing a node in 4h — but they block PRs and have to be rerun repeatedly. A month ago, with only the 8h job timeout as the limit, Phoenix jobs usually got a node and ran. This brings that back.

SLURM_MAX_QUEUE_SECONDS remains available (set it >0) to opt into an earlier cutoff on non-preemptible schedulers where a long PENDING genuinely signals a problem.

[Copilot]

Pull request overview

This PR adjusts Phoenix CI’s SLURM monitoring behavior to restore the pre-#1763 default: do not cap how long a job may remain queued, and instead rely on the GitHub Actions job-level timeout-minutes as the backstop. This targets queue-heavy/preemptible Phoenix runs where a 4-hour queue cap turns normal PENDING delays into CI failures.

Changes:

• Change the default SLURM_MAX_QUEUE_SECONDS from 14400 (4h) to 0 (no queue-wait cap).
• Update the inline documentation to clarify the rationale and how to opt into an earlier cutoff via SLURM_MAX_QUEUE_SECONDS>0.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.67%. Comparing base (b2a01f9) to head (ee2c180).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1770   +/-   ##
=======================================
  Coverage   61.67%   61.67%           
=======================================
  Files          84       84           
  Lines       21619    21619           
  Branches     3196     3196           
=======================================
  Hits        13334    13334           
  Misses       6093     6093           
  Partials     2192     2192           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.