Repository navigation
ci(python): the GraalPy experimental leg loses its runner 18–25 minutes into its tests on every recent run #1740
Description
Describe the bug
In about an hour, the Python graalpy-25.4 (experimental) leg lost its runner three times out of three, each time partway through Run unit tests:
| Run | Job | Test step ran | Last lines |
|---|---|---|---|
#1734 (ba67ef70e) |
115975614204 | 18 min | The runner has received a shutdown signal / The operation was canceled. |
main e530f39ad |
116009492040 | 25 min 11 s | same |
main 34a197dd6 |
116013863229 | 21 min 12 s | same |
The step's cap is 25 minutes. Two of the three ended well before it, so the cap is not what ends them.
Expected behavior
The leg ends on its own result or on its cap, not on a lost runner.
Additional context
The cause is not measured yet. Running out of memory or disk on the runner is the leading guess, since #1372 once wrote 6 GB per run on another leg. Until #1732 merges, this leg sits in test, so every push to main fails Required checks passed (#1738). After #1732 it runs in its own job and gates nothing, but it may still never finish. The next step is to record free -m and df -h in the background during that job's test step.
Activity
JarryShaw commentedon Oct 14, 2026
A fourth case, the same way: main
d06701064, job 116021448652,Run unit testsran 18:21:32Z→18:47:07Z (25 min 35 s) and ended withThe runner has received a shutdown signal. The four step durations so far are 18, 25.2, 21.2 and 25.6 min.JarryShaw commentedon Oct 14, 2026
A fifth case, of a different shape, on main
4e1720c44: job 116014068117.Run unit testsstarted at 18:13:36Z and never reported a conclusion. The job was cancelled at 19:06:05Z, 55 min after it started, which is past both the step's 25-minute cap and the job's 45. No error line was logged. A runner that went silent, so that neither cap could fire, fits resource exhaustion better than an ordinary shutdown.JarryShaw commentedon Oct 14, 2026
A sixth case. This one rules out the cap: #1742's own Experimental Tests run 38623641661, job 116057255255. The tests step is capped at 120 minutes there, yet it ran only 30 min 12 s (19:30:01Z→20:00:13Z) before
The runner has received a shutdown signal. The losses so far came 18 to 30 minutes in, plus one silent cancellation at 55. The split works as designed: the workflow run concluded success, and nothing waited on it.JarryShaw commentedon Oct 14, 2026
#1742 has merged, so this is unblocked. A worker is adding a background resource monitor to the experimental tests step:
free -m,df -hand the top processes by RSS every 30 s, uploaded as an artifact even when the runner is lost partway. The goal is to measure what exhausts the runner rather than guess.JarryShaw commentedon Oct 14, 2026
The owner has set the bar: a non-blocking leg may fail only as a warning, and its check must stay green. Today the GraalPy check is the only red on
main, on0cdd3cdd1and65122bf89. The workflow run itself concludes success, but a lost runner fails the job, and no workflow setting can turn that job green. The fix therefore has to keep the runner alive, so that the tests fail inside a step thatcontinue-on-errorcontains. That means capping the test process's memory and file size, cutting GraalPy's parallelism, and ending it withtimeoutbefore the step cap. This goes into this issue's PR alongside the resource monitor, and the same containment goes onto the KTLO jobs in #1738.JarryShaw commentedon Oct 15, 2026
First run under the containment (#1749, job 116172356218): the runner survived, and the check is green with warnings.
MemoryErrorline.MemoryErrorwith the machine half free points at the 2 GB per-process heap (--vm.Xmx2g), most likely in the xdist controller, as the next thing to tune.ci(python): contain and sample the experimental legs so a lost runner…
JarryShaw commentedon Oct 15, 2026
Done by #1749, now on
mainasdd846e3cf. On its run (job 116172356218) the runner survived and the leg ended on its own result, green with warnings, which is this issue's expected behaviour. The early stop that remains, an xdist workerMemoryError, is a separate cause and is now #1753.