ci(python): exit and replace a GraalPy worker whose heap is spent (#1753)

[JarryShaw]

Please follow the guide below

• You will be asked some questions, please read them carefully and answer honestly

• Put an x into all the boxes [ ] relevant to your pull request (like that [x])

• Use Preview tab to see how your pull request will actually look like

• Searched for similar pull requests

• Followed the coding style (make pylint, make mypy, make isort)

• make test passes, and a test case covers the change. Narrow runs only, listed below.

• Added a changelog entry under docs/source/changelog/ and regenerated CHANGELOG.md, if the change is user-visible: N/A — centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657

What is the purpose of your pull request?

Tick the commit type your subject line carries.

• fix — corrects a defect
• feat — adds a feature
• perf — changes performance, not behaviour
• refactor — changes neither behaviour nor performance
• test — tests only
• docs — documentation only
• ci — workflows or build tooling
• release — bumps the version or rolls up a distribution
• chore — anything else

---

Description

Closes #1753. Since #1749, the GraalPy experimental leg keeps its runner, but its tests stopped at 16 minutes. On job 116172356218, each xdist worker's live heap grew with the tests it ran until it filled the 2 GiB --vm.Xmx cap. RSS reached 2.2-2.6 GB, a MemoryError hit pytest's own failure report, and the session ended in INTERNALERROR (exit 3) while 7.9 GB was still free. Locally, a bigger heap only delays the crash, so the fix replaces the spent worker instead of enlarging the heap.

• experimental.yml:
  • each GraalPy process gets --vm.XX:+ExitOnOutOfMemoryError, so a spent worker exits and xdist starts a fresh one;
  • pytest gets --max-worker-restart=30, which is one count across all workers;
  • tests-status counts the node down lines, and Containment summary warns tests lost an xdist worker;
  • the MemoryError count is word-bounded;
  • every ci(python): contain and sample the experimental legs so a lost runner becomes a warning (#1740) #1749 containment property stays: the 12G scope, timeout before the cap, prlimit --fsize and continue-on-error.
• workflows.rst and the workflow comments carry the measurements and their sources: GraalVM's Native Image memory docs, and ExceptionUtils.java at graal-25.4.4.
• Test: tests/project/test_graalpy_worker_heap_1753.py. 7 of its 8 tests fail on main's experimental.yml.

Proof is this PR's own run. If the fix works, the Python graalpy-25.4 (experimental) leg prints [gwN] node down lines, gets past 569 passed, and ends in a normal pytest summary or the 110-minute timeout. It has no INTERNALERROR, and its check is green with warnings.

Tests, on Python 3.14: tests/project gives 921 passed and 1 skipped.

[JarryShaw]

Verdict: GOOD TO GO at 068152ab1. A Sonnet cross-review checked 13eca2105 (author: Opus).

• Root cause, confirmed independently:
  • gw1's MemoryError at log line 1316 came while pytest was formatting a failure report.
  • All three workers' RSS grew steadily over the run.
  • The reviewer reproduced it locally with GraalPy 25.4.4 and xdist 3.8.0: without the flag, INTERNALERROR and exit 3; with it, 4 node down lines and all 40 tests accounted for.
• External claims, read from their sources: ExceptionUtils.java:313-316 at graal-25.4.4 turns Java's OutOfMemoryError into Python's MemoryError, and the GraalVM docs say the heap is not the upper limit on total memory.
• Arithmetic:
  • memory: 3 × ~2.65 GB plus a ~0.5 GB controller ≈ 8.4 GB, inside the 12G scope;
  • restarts: about 21 in 110 minutes, against the cap of 30.
• ci(python): contain and sample the experimental legs so a lost runner becomes a warning (#1740) #1749 containment: every property is intact.
• Test: 7 of 8 tests fail on main's workflow.
• Since that review: the rewording ("30 restarts in all", the default "4 a worker, 12 at -n 3") and a rebase onto 3946bf527. git range-diff shows only those two hunks, and tests/project gives 921 passed, 1 skipped on the new head.
• Non-blocking: under -q, xdist prints no "maximum crashed workers reached". If the cap is ever hit, the sign is crashed_workers ≥ 31 in Containment summary.

The GraalPy leg on this PR's own run is the real proof. I'll report what it shows when it finishes.

[github-actions]

Coverage: 89.59% (unit tier, Python 3.15, 068152ab1, Unit Tests run success)

Package	Statements	Missed	Branches	Partial	Cover
pcapkit (top level)	104	3	20	3	95.16%
pcapkit/const	18860	954	2368	868	91.06%
pcapkit/corekit	2457	69	780	35	96.54%
pcapkit/dumpkit	276	3	100	3	98.40%
pcapkit/foundation	2915	164	1016	34	93.59%
pcapkit/interface	138	7	52	5	93.68%
pcapkit/protocols	17186	224	4574	189	98.08%
pcapkit/toolkit	872	177	298	8	77.18%
pcapkit/utilities	512	2	158	4	99.10%
pcapkit/vendor	4417	2342	1010	157	43.38%

Per-file detail: the coverage-html artifact of this run.

[JarryShaw]

The fix works on this PR's own run (job 116215824320).

• The whole unit tier ran: 69 failed, 3298 passed, 153 skipped, 20064 subtests passed in 5509.22s (1:31:49). That finishes inside the 110-minute timeout. Before this, the run stopped at 569 passed after 16 minutes.
• Workers were replaced, not lost: 13 node down, against a cap of 30. There was no INTERNALERROR and only 1 MemoryError line.
• Memory: the last sample used 3344 of 15989 MiB, inside the 12G scope.
• The check is green, with the Containment summary warnings.

pytest exited 2, but not because of the heap. gw10 crashed in the middle of the tier guard's own xdist probe test. The session then reported that probe's capture read as a tier violation and interrupted. That is a separate defect, filed as #1756.

The 69 failures are GraalPy behaviour and are not classified here.