Repository navigation
pack install: release the lock only if this run owns it (flaky 'rmdir ... No such file or directory') - #192
Conversation
`pack install (0.82-0.86)` fails on roughly one run in four with
rmdir: failed to remove '/tmp/bugsee-pack-install.lock.d': No such file or directory
after the pack it guarded had finished. The lock (added in #163, first failure the
next day) was released with a bare `rmdir "$lock"` under `set -e`, which removes
whatever sits at that path and fails the whole run when nothing does.
The release is now ownership-checked and idempotent (scripts/pack-lock.sh): a holder
writes a token into the lock, and release removes the lock only if the token is still
its own. A lock that has gone, or that another run now holds, is reported on stderr and
left alone instead of failing this run or deleting the other run's lock. The EXIT trap
and the explicit release are the same function and can both run.
scripts/__tests__/pack-lock.test.ts runs the library as real concurrent bash
processes: mutual exclusion over 12 contenders, a lock removed while held (the old
rmdir died here), a lock taken by someone else, idempotent release, release from the
EXIT trap, the timeout message. It also pins check-pack-install.sh to the library.
What is NOT established: what removes the lock directory in CI. It did not reproduce
in 25 cold-cache iterations of the real lock section on bash 5.2 / node 22.18, so this
makes the run survive that event and stops it spreading, but does not stop the event.
Human guided, AI implemented
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 43 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
Claude finished @krassx's task in 1m 14s —— View job Review of #192: pack-install lock releaseI read Findings (inline comments on the relevant lines)
Verified OK
Not fixed by this PR (acknowledged in the description)
Overall risk: Low. It only touches a CI helper script, and the change is strictly safer than the bare Merge recommendation: Approve once finding 1 is handled, which is a small change plus a test. Findings 2 and 3 are optional but cheap. Most important to fix: make a failed owner-file write fail the acquire and undo it, with a test. Positives
|
| if mkdir "$lock" 2>/dev/null; then | ||
| PACK_LOCK_DIR="$lock" | ||
| PACK_LOCK_TOKEN="$$.$RANDOM$RANDOM" | ||
| printf '%s\n' "$PACK_LOCK_TOKEN" > "$lock/owner" |
There was a problem hiding this comment.
Severity: P2 Medium
Problem: A failed token write is swallowed, so the run reports success and the lock directory is leaked. check-pack-install.sh calls pack_lock_acquire ... || exit 1, which disables set -e inside the function. A failing printf ... > "$lock/owner" therefore only prints an error, and the function still returns 0 with PACK_LOCK_TOKEN set. At release owner is empty, which differs from the token. The function takes the "held by another run" branch and leaves the directory in place. rmdir ... || true hides any similar failure.
Impact: The old rmdir would have cleaned this up. Now every later run waits the full 300 s and fails with "stale lock? remove it", and the warning blames "another run".
Scenario: I reproduced it on this checkout with umask 0222, so the lock dir is created read-only. The output was:
line 33: /tmp/t1/l.d/owner: Permission denied
acquired rc=0
warning: /tmp/t1/l.d is held by another run now; left it alone
ls then showed l.d still present. The same happens with ENOSPC or EROFS on $TMPDIR. A SIGKILL between mkdir and this write leaves an ownerless lock the same way, as it did before. The write is only a fresh failure source.
Fix: Treat an unwritable token as a failed acquire and undo it:
if ! printf '%s\n' "$PACK_LOCK_TOKEN" > "$lock/owner"; then
rmdir "$lock" 2>/dev/null || true
PACK_LOCK_DIR="" PACK_LOCK_TOKEN=""
echo "FAIL: could not record the owner in $lock" >&2
return 1
fiAdd a test for it. None of the 13 covers a failed write.
There was a problem hiding this comment.
Fixed in the latest push, as suggested: a failed token write removes the (empty or partial) owner, removes the lock directory, clears the state, prints FAIL: could not record the owner in <lock> and returns 1, so the acquire fails instead of leaking an ownerless lock. The test overrides printf to fail (the redirect still creates owner, so it also checks that it is cleaned up) rather than relying on file permissions, which root ignores; it asserts the lock is gone, the acquire returned 1 and the lock can be taken again, and a second test that a later release does not claim 'held by another run'. Both fail against the previous version.
| pack_lock_acquire "${TMPDIR:-/tmp}/bugsee-pack-install.lock.d" 300 || exit 1 | ||
| # A kill between taking the lock and releasing it must not leave it behind. The | ||
| # release is idempotent, so this and the explicit one below can both run. | ||
| trap 'pack_lock_release; [[ -n "${PACK_INSTALL_DIR:-}" ]] || rm -rf "$WORK"' EXIT |
There was a problem hiding this comment.
Severity: P3 Low
Problem: The EXIT trap is installed after pack_lock_acquire. A SIGINT or SIGTERM between mkdir and this line leaves the lock behind, and the new token write widens that window slightly. The pre-existing code had the same gap.
Impact: The leaked lock makes every other run wait 300 s and then fail. This matters mainly on a developer machine or a cancelled job on a reused host. Hosted runners are ephemeral.
Fix: pack_lock_release is now idempotent and does nothing without a token. The trap can therefore be set before the acquire, with the same body:
trap 'pack_lock_release; [[ -n "${PACK_INSTALL_DIR:-}" ]] || rm -rf "$WORK"' EXIT
pack_lock_acquire ... || exit 1This is cheap. The pin test trap 'pack_lock_release;[^']*' EXIT still matches.
There was a problem hiding this comment.
Fixed: the EXIT trap is now installed before pack_lock_acquire, with the same body (the release does nothing without a token, so it is safe before one exists). A test pins that the trap comes before the acquire, and fails against the previous order.
| if [[ -e "$lock" ]]; then | ||
| echo "warning: $lock is held by another run now; left it alone" >&2 | ||
| else | ||
| echo "warning: $lock was removed while it was held, so the pack was not serialized" >&2 |
There was a problem hiding this comment.
Severity: P3 Low
Problem: The only signal that serialization failed is a plain stderr line. ci-each-version.sh writes each run's output to a log and prints it in a collapsed ::group:: only after all runs finish. A run that hit this branch exits 0, so nobody sees the line.
Impact: The fix turns a loud failure into a silent pass. The overlapping npm pack the lock exists to prevent can still happen. If it does, the next failure is probably a confusing missing native-versions.json in a later step, with the cause buried in a green job's log. The unexplained directory removal is also the only clue to the real root cause, and a quiet line makes it likely to be missed.
Fix: Emit it as an annotation, for example echo "::warning::$lock was removed while it was held..." when GITHUB_ACTIONS is set. Alternatively ci-each-version.sh could grep the logs for the warning and surface it. Without this the root cause stays hard to catch.
There was a problem hiding this comment.
Agreed, this was the weak point: the fix turned a loud failure into a quiet pass and buried the only clue to the real cause. pack_lock_warn now also emits ::warning title=pack-install lock::... on stdout when GITHUB_ACTIONS is set, for both the lost-lock and the taken-by-another-run cases (the runner parses it when ci-each-version.sh cats the log, so it lands on the run summary), and the text says nothing in this repository removes the directory and asks for the run to be reported. The stderr line stays for reads outside Actions. Tests cover both cases on Actions and no annotation off Actions.
|
Note: the review fixes on this PR (failed token write, trap before acquire, Human guided, AI implemented |
Symptom
pack install (0.82-0.86)fails intermittently with exactly one version failing and the other four passing:It is not caused by any change under review. It predates the current PRs: the lock was added in #163 (2026-10-22) and the job failed the next day on an unrelated Dependabot branch.
examples/bare/react-native/eslint-config-0.88.0node-forge-1.4.3feat/launch-options-parity/feat/native-api-parityRoughly one run in four on the recent runs, and it does not follow the code: re-running the same commit sometimes passes.
What goes wrong
The lock was released with a bare
rmdir "$PACK_LOCK"underset -e. That removes whatever sits at the path, and kills the whole run (exit 1) when nothing does, even though the pack it guarded had already finished. If the directory is gone while it is held, the failure also cascades: the next waiter'smkdirsucceeds while the first holder is still packing, and each later release removes the lock under the next holder, so the last releaser is the one that dies. That matches the logs: exactly one failing version per run, and in the bad run three.Reproduction, and what I could not establish
Reproduced (bash 5.2.21, the CI runner's major version):
So the failing line and its message follow directly from "the lock directory was already gone at release".
Not reproduced: what removes the directory. I ran the real lock section of
check-pack-install.sh(realnpm packx2 with prepack/postpack, real coldnpx --yes @react-native-community/cli) with the CI invocation (xargs -P 3, five versions) innode:22.18/ bash 5.2.15, 25 cold-cache iterations: 0 failures. A stub with the same lock/trap structure ran 300 iterations on bash 5.2.21: 0 failures. Ruled out by test: a bash subshell or command substitution does not run the parent's EXIT trap (bash 3.2 and 5.2); the prepack/postpack scripts do not touch/tmp; nothing else in the repo references the path. By reading, no interleaving of the script's ownmkdir/rmdir/trap lines lets one run remove another's lock. So the trigger is something about the hosted runner that I could not reproduce in a container. I am not claiming to have found it.Fix
The release is now safe against that event, and the event no longer spreads (
scripts/pack-lock.sh, sourced bycheck-pack-install.sh):warning: ... was removed while it was held, so the pack was not serialized) and left alone: not fatal, and another run's lock is never deleted.mkdirlock (noflock, so macOS works), 300 s wait, the stale-lock message.Remaining honest limit: if the directory really is removed externally while held, two runs can overlap in
npm pack, which is the hazard the lock exists for. That window is no worse than before, and a failed pack would still fail loudly through the existingnpm packerror path.Tests
scripts/__tests__/pack-lock.test.ts(13 tests) runs the library as real bash processes, withset -euo pipefailas the script has: mutual exclusion across 12 concurrent contenders, a lock removed while held (the case the oldrmdirdied on), a lock taken by another run, idempotent release, release from the EXIT trap, the timeout message, and a pin thatcheck-pack-install.shreleases only through the library. Also checked by hand on bash 5.2: 400 contenders, 40 at a time, no overlap, lock cleaned up. The modified real lock section passed 3 cold iterations innode:22.18.yarn lint,yarn typecheck,yarn jest: all pass (5,273 tests). shellcheck is unchanged from baseline (the existing SC2016 note).Human guided, AI implemented