[CI] Add semaphore isolation helper to avoid hangs#2308
Merged
mhalk merged 1 commit intoJul 1, 2026
Conversation
mhalk
requested review from
carlobertolli,
estewart08,
gregrodgers,
ronlieb and
zGoldthorpe
as code owners
Kewen12
approved these changes
Jun 29, 2026
Kewen12
left a comment
Kewen12
left a comment
Contributor
There was a problem hiding this comment.
do we need to apply this to other scripts?
Contributor
|
The lint seems failed due to double quote. |
Collaborator
I think this is the only spot needed. The other check_* scripts should symlink to |
Collaborator
Author
|
Thanks for the feedback and clarification! FYI in coordination with JP: I'll open a NFC(I) PR which fixes the shellcheck issues of |
1 task
jplehr
reviewed
Jul 1, 2026
mhalk
force-pushed
the
amd/dev/mhalkenh/fix/ci-smoke-semaphore-hang
branch
from
34cdb90 to
34d68fb
Compare
jplehr
approved these changes
Jul 1, 2026
Fix possible stale semaphores during parallel runs - isolate semaphores per test run - steal semaphore after timeout Issue: Semaphores including hostnames, which in turn include uppercase letters, are not reaped correctly. This lead to hangs of CI runs. Very recent parallel versions 20260522 and newer handle this case by converting hostname to lowercase: https://cgit.git.savannah.gnu.org/cgit/parallel.git/tree/src/parallel?h=20260522#n7565
mhalk
force-pushed
the
amd/dev/mhalkenh/fix/ci-smoke-semaphore-hang
branch
from
34d68fb to
3aab18f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Observed hangs during smoke suite on CI machine (with hostname being all uppercase letters).
Traced issue to stale semaphores created by previous run's
parallelinvocation.Probable cause could be a reboot of the machine during run or
TERM/KILLof corresponding process(es).Technical Details
Semaphores including hostnames, which in turn include uppercase letters, are not reaped correctly.
Stale semaphores can block indefinitely, until cleaned (e.g. manually).
Approach avoids hanging of CI runs by:
Very recent parallel versions 20260522 and newer handle this case by converting hostname to lowercase:
https://cgit.git.savannah.gnu.org/cgit/parallel.git/tree/src/parallel?h=20260522#n7565
Test Plan
Ran corresponding CI script for smoke tests.
Test Result
No hangs observed, no stealing (since nothing was aborted).
Run completed successfully.
Submission Checklist