Skip to content

test(tiers): keep the xdist probe out of recursive collection (#1756) - #1757

Merged
JarryShaw merged 1 commit into
mainfrom
test/1756-tier-guard-probe-crash-safe
Oct 15, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
test/1756-tier-guard-probe-crash-safe

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

Please follow the guide below

What is the purpose of your pull request?

Tick the commit type your subject line carries.

  • fix — corrects a defect
  • feat — adds a feature
  • perf — changes performance, not behaviour
  • refactor — changes neither behaviour nor performance
  • test — tests only
  • docs — documentation only
  • ci — workflows or build tooling
  • release — bumps the version or rolls up a distribution
  • chore — anything else

Description

Closes #1756.

On #1755's GraalPy run, a crashed worker turned the whole session into pytest exit 2. The mechanism is now measured, and it corrects the issue's guess:

  • Nothing is aggregated across workers. XdistSubprocessTests.setUp writes the probe tests/protocols/test_zzz_xdist_guard_probe_unit.py and removes it only through addCleanup.
  • When gw10 died mid-test, the probe stayed on disk. xdist's replacement worker collected tests/ again, and the collection-time audit in tests/conftest.py found the probe's read and set session.shouldfail.
  • The controller then raised Interrupted once the other workers drained.

Fix: tests/conftest.py adds collect_ignore = [f'protocols/{PROBE_MODULE_NAME}']. pytest consults collect_ignore only while it recurses into a directory, never for a path named on the command line. So:

  • no recursive session, including a replacement worker, collects a stale probe;
  • the probe's own nested run, which names the probe's path, still audits it;
  • real unit-tier reads are guarded as before, and SuiteIsCleanTests still scans the path on disk.

Test: ProbeWorkerCrashTests, in tests/test_tier_guard_xdist.py.

  • It runs the real probe test in a throwaway pytest -n 2 session, where the worker calls os._exit once the probe is on disk.
  • It asserts that the crash happened, that the probe was left behind, and that there is no Interrupted and the exit is 1.
  • On main's conftest.py it fails, with the CI-identical message. Without xdist it skips, like its neighbours.

Tests: the two tier-guard modules plus test_unit_tests_queue.py give 152 passed, 3 skipped (the xdist classes skip in the repo venv). In an xdist venv, both tier-guard modules under -n 2 give 117 passed.

A worker that dies mid-way through XdistSubprocessTests never runs its
cleanup, so tests/protocols/test_zzz_xdist_guard_probe_unit.py stays on
disk. xdist starts a replacement worker, which collects tests/ afresh;
its pytest_collection_modifyitems audit finds the probe, sets
session.shouldfail, and the controller ends the session with
xdist.dsession.Interrupted, exit 2 (GraalPy job 116215824320, gw10).

- tests/conftest.py: collect_ignore the probe's reserved path. pytest
  checks collect_ignore only while recursing, never for an initial
  path, so the probe's own nested run, which names it explicitly,
  still collects and audits it. SuiteIsCleanTests still scans it.
- tests/test_tier_guard_xdist.py: ProbeWorkerCrashTests runs the real
  probe test under -n 2 with a -p plugin that kills the worker after
  the probe is written. It holds the exclusive lock throughout, then
  removes the stale probe. Skips without xdist, like its neighbours.

Fails on main (Interrupted, exit 2), passes here; tier-guard, queue
and docs-claims tests pass in the repo venv and under -n 2.

Closes #1756
@JarryShaw JarryShaw added the test Pull requests that add or correct tests (test: subject prefix) label Oct 15, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 15, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Verdict: GOOD TO GO at eb3253d34. A Sonnet cross-review checked it (author: Opus).

  • Mechanism, re-derived and reproduced without a crash: a hand-placed stale probe plus pytest -n 2 --dist load tests/protocols on main's conftest.py exits 2 with the CI-identical Interrupted line. The CI log fits: gw10's nested probe run is in the monitor at 04:04:48, node down follows at 04:04:50, and the interrupt comes when the session drains at 04:08:20.
  • Reproduced independently with SIGKILL on the worker: on main, exit 2 with Interrupted; with the fix, exit 1 and only the crashed test fails.
  • The guard still bites: a real unit-tier module reading a generated capture still errors serially (exit 4) and still interrupts under -n 2 (exit 2).
  • The new test fails on main for the intended reason ('Interrupted' unexpectedly found), after its crash and stale-probe assertions pass.
  • Gates pass, including test_tier_guard_xdist.py at -n 3, repeated six times with no flake.
  • Non-blocking: XdistSubprocessTests.setUp checks probe_path.exists() before taking the lock (:176-179 versus :189). That pre-existing race is filed as test(tiers): XdistSubprocessTests checks for a leftover probe before taking the lock #1758.
  • My own check: both tier-guard modules plus test_unit_tests_queue.py give 152 passed and 3 skipped, with pcapkit.__file__ printed inside the PR's worktree.

@JarryShaw JarryShaw added the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage: 89.59% (unit tier, Python 3.15, eb3253d34, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 3 20 3 95.16%
pcapkit/const 18862 954 2368 868 91.06%
pcapkit/corekit 2461 69 780 35 96.54%
pcapkit/dumpkit 276 3 100 3 98.40%
pcapkit/foundation 2915 164 1016 34 93.59%
pcapkit/interface 138 7 52 5 93.68%
pcapkit/protocols 17184 224 4574 189 98.08%
pcapkit/toolkit 872 177 298 8 77.18%
pcapkit/utilities 512 2 158 4 99.10%
pcapkit/vendor 4417 2342 1010 157 43.38%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw
JarryShaw merged commit 1f5766c into main Oct 15, 2026
56 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 15, 2026
@JarryShaw
JarryShaw deleted the test/1756-tier-guard-probe-crash-safe branch October 15, 2026 13:41
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test Pull requests that add or correct tests (test: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

test(tiers): a worker crash in the tier guard's xdist probe test interrupts the whole session

1 participant