Skip to content

pack install: release the lock only if this run owns it (flaky 'rmdir ... No such file or directory') - #192

Merged
krassx merged 1 commit into
mainfrom
fix/pack-install-lock-race
Nov 16, 2026
Merged

krassx merged 1 commit into
mainfrom
fix/pack-install-lock-race

Conversation

@krassx

@krassx krassx commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Symptom

pack install (0.82-0.86) fails intermittently with exactly one version failing and the other four passing:

##[group]0.84 (exit 1)
--- packing (npm pack, prepack/postpack as on publish)
rmdir: failed to remove '/tmp/bugsee-pack-install.lock.d': No such file or directory

It is not caused by any change under review. It predates the current PRs: the lock was added in #163 (2026-10-22) and the job failed the next day on an unrelated Dependabot branch.

When Branch Job
2026-10-23 07:49 dependabot examples/bare/react-native/eslint-config-0.88.0 failure
2026-11-15 20:01 dependabot node-forge-1.4.3 failure
2026-11-16 06:04, 08:46, 09:11 (+ re-runs) feat/launch-options-parity / feat/native-api-parity failure; the version that failed was 0.82, 0.84, 0.82, then 0.84 again; one run had three versions fail

Roughly one run in four on the recent runs, and it does not follow the code: re-running the same commit sometimes passes.

What goes wrong

The lock was released with a bare rmdir "$PACK_LOCK" under set -e. That removes whatever sits at the path, and kills the whole run (exit 1) when nothing does, even though the pack it guarded had already finished. If the directory is gone while it is held, the failure also cascades: the next waiter's mkdir succeeds while the first holder is still packing, and each later release removes the lock under the next holder, so the last releaser is the one that dies. That matches the logs: exactly one failing version per run, and in the bad run three.

Reproduction, and what I could not establish

Reproduced (bash 5.2.21, the CI runner's major version):

$ bash -c 'set -e; L=/tmp/old.lock.d; mkdir $L; rm -rf $L; rmdir $L; echo survived'
rmdir: failed to remove '/tmp/old.lock.d': No such file or directory      # exit 1

So the failing line and its message follow directly from "the lock directory was already gone at release".

Not reproduced: what removes the directory. I ran the real lock section of check-pack-install.sh (real npm pack x2 with prepack/postpack, real cold npx --yes @react-native-community/cli) with the CI invocation (xargs -P 3, five versions) in node:22.18 / bash 5.2.15, 25 cold-cache iterations: 0 failures. A stub with the same lock/trap structure ran 300 iterations on bash 5.2.21: 0 failures. Ruled out by test: a bash subshell or command substitution does not run the parent's EXIT trap (bash 3.2 and 5.2); the prepack/postpack scripts do not touch /tmp; nothing else in the repo references the path. By reading, no interleaving of the script's own mkdir/rmdir/trap lines lets one run remove another's lock. So the trigger is something about the hosted runner that I could not reproduce in a container. I am not claiming to have found it.

Fix

The release is now safe against that event, and the event no longer spreads (scripts/pack-lock.sh, sourced by check-pack-install.sh):

  • A holder writes a token into the lock; release removes the lock only if the token is still its own.
  • A lock that is gone, or held by another run now, is reported on stderr (warning: ... was removed while it was held, so the pack was not serialized) and left alone: not fatal, and another run's lock is never deleted.
  • The EXIT trap and the explicit release are the same idempotent function.
  • Unchanged: mkdir lock (no flock, so macOS works), 300 s wait, the stale-lock message.

Remaining honest limit: if the directory really is removed externally while held, two runs can overlap in npm pack, which is the hazard the lock exists for. That window is no worse than before, and a failed pack would still fail loudly through the existing npm pack error path.

Tests

scripts/__tests__/pack-lock.test.ts (13 tests) runs the library as real bash processes, with set -euo pipefail as the script has: mutual exclusion across 12 concurrent contenders, a lock removed while held (the case the old rmdir died on), a lock taken by another run, idempotent release, release from the EXIT trap, the timeout message, and a pin that check-pack-install.sh releases only through the library. Also checked by hand on bash 5.2: 400 contenders, 40 at a time, no overlap, lock cleaned up. The modified real lock section passed 3 cold iterations in node:22.18.

yarn lint, yarn typecheck, yarn jest: all pass (5,273 tests). shellcheck is unchanged from baseline (the existing SC2016 note).

Human guided, AI implemented

`pack install (0.82-0.86)` fails on roughly one run in four with

    rmdir: failed to remove '/tmp/bugsee-pack-install.lock.d': No such file or directory

after the pack it guarded had finished. The lock (added in #163, first failure the
next day) was released with a bare `rmdir "$lock"` under `set -e`, which removes
whatever sits at that path and fails the whole run when nothing does.

The release is now ownership-checked and idempotent (scripts/pack-lock.sh): a holder
writes a token into the lock, and release removes the lock only if the token is still
its own. A lock that has gone, or that another run now holds, is reported on stderr and
left alone instead of failing this run or deleting the other run's lock. The EXIT trap
and the explicit release are the same function and can both run.

scripts/__tests__/pack-lock.test.ts runs the library as real concurrent bash
processes: mutual exclusion over 12 contenders, a lock removed while held (the old
rmdir died here), a lock taken by someone else, idempotent release, release from the
EXIT trap, the timeout message. It also pins check-pack-install.sh to the library.

What is NOT established: what removes the lock directory in CI. It did not reproduce
in 25 cold-cache iterations of the real lock section on bash 5.2 / node 22.18, so this
makes the run survive that event and stops it spreading, but does not stop the event.

Human guided, AI implemented
@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 43 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: fe5bca91-af30-4648-a62e-a8dff21a28ac
Commits

Reviewing files that changed from the base of the PR and between 115e53c and 68ab2ca.

Files selected for processing: 3

  • scripts/__tests__/pack-lock.test.ts
  • scripts/check-pack-install.sh
  • scripts/pack-lock.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@claude

claude Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Claude finished @krassx's task in 1m 14s —— View job


Review of #192: pack-install lock release

I read scripts/pack-lock.sh, the changed lock section of scripts/check-pack-install.sh, scripts/ci-each-version.sh, the workflow invocations and the new tests. I did not run the jest suite. I did run one bash repro against this checkout, described in finding 1.

Findings (inline comments on the relevant lines)

Sev Where Summary
P2 pack-lock.sh:33 A failed token write is swallowed because the function runs under || exit 1, where set -e is off. The lock is held with no owner, and release leaves it forever, so every later run waits 300 s and fails. I reproduced it with umask 0222. The old rmdir cleaned this case up.
P3 check-pack-install.sh:59 The EXIT trap is installed after the acquire, so a signal in between leaks the lock. Release is now idempotent, so the trap can go before the acquire.
P3 pack-lock.sh:58 The "removed while held" warning is plain stderr in a collapsed group of a green run. Emit it as a ::warning:: annotation so the unexplained directory removal is noticed.

Verified OK

  • The cascade scenario in the PR description now behaves correctly. A holds, the directory is removed, B takes the lock, A releases and leaves B's lock alone, and B releases cleanly.
  • The EXIT trap and the explicit release are safe together. The state is cleared before the checks, and the trap is replaced afterwards.
  • rm -f owner followed by rmdir has a tiny check-then-act window if the lock changes hands between the cat and the rm. I judged it not worth reporting.

Not fixed by this PR (acknowledged in the description)

  • The root cause, what removes the directory, is still unknown. Mutual exclusion is still lost if it recurs. This is a mitigation, not a cure.

Overall risk: Low. It only touches a CI helper script, and the change is strictly safer than the bare rmdir under set -e.

Merge recommendation: Approve once finding 1 is handled, which is a small change plus a test. Findings 2 and 3 are optional but cheap.

Most important to fix: make a failed owner-file write fail the acquire and undo it, with a test.

Positives

  • The PR is honest about what was and was not reproduced.
  • The tests run the library as real concurrent bash processes under set -euo pipefail.
  • The mkdir lock keeps working on macOS.
  • It adds a pin test that the script releases only through the library.

Comment thread scripts/pack-lock.sh
if mkdir "$lock" 2>/dev/null; then
PACK_LOCK_DIR="$lock"
PACK_LOCK_TOKEN="$$.$RANDOM$RANDOM"
printf '%s\n' "$PACK_LOCK_TOKEN" > "$lock/owner"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: P2 Medium

Problem: A failed token write is swallowed, so the run reports success and the lock directory is leaked. check-pack-install.sh calls pack_lock_acquire ... || exit 1, which disables set -e inside the function. A failing printf ... > "$lock/owner" therefore only prints an error, and the function still returns 0 with PACK_LOCK_TOKEN set. At release owner is empty, which differs from the token. The function takes the "held by another run" branch and leaves the directory in place. rmdir ... || true hides any similar failure.

Impact: The old rmdir would have cleaned this up. Now every later run waits the full 300 s and fails with "stale lock? remove it", and the warning blames "another run".

Scenario: I reproduced it on this checkout with umask 0222, so the lock dir is created read-only. The output was:

line 33: /tmp/t1/l.d/owner: Permission denied
acquired rc=0
warning: /tmp/t1/l.d is held by another run now; left it alone

ls then showed l.d still present. The same happens with ENOSPC or EROFS on $TMPDIR. A SIGKILL between mkdir and this write leaves an ownerless lock the same way, as it did before. The write is only a fresh failure source.

Fix: Treat an unwritable token as a failed acquire and undo it:

if ! printf '%s\n' "$PACK_LOCK_TOKEN" > "$lock/owner"; then
  rmdir "$lock" 2>/dev/null || true
  PACK_LOCK_DIR="" PACK_LOCK_TOKEN=""
  echo "FAIL: could not record the owner in $lock" >&2
  return 1
fi

Add a test for it. None of the 13 covers a failed write.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the latest push, as suggested: a failed token write removes the (empty or partial) owner, removes the lock directory, clears the state, prints FAIL: could not record the owner in <lock> and returns 1, so the acquire fails instead of leaking an ownerless lock. The test overrides printf to fail (the redirect still creates owner, so it also checks that it is cleaned up) rather than relying on file permissions, which root ignores; it asserts the lock is gone, the acquire returned 1 and the lock can be taken again, and a second test that a later release does not claim 'held by another run'. Both fail against the previous version.

pack_lock_acquire "${TMPDIR:-/tmp}/bugsee-pack-install.lock.d" 300 || exit 1
# A kill between taking the lock and releasing it must not leave it behind. The
# release is idempotent, so this and the explicit one below can both run.
trap 'pack_lock_release; [[ -n "${PACK_INSTALL_DIR:-}" ]] || rm -rf "$WORK"' EXIT

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: P3 Low

Problem: The EXIT trap is installed after pack_lock_acquire. A SIGINT or SIGTERM between mkdir and this line leaves the lock behind, and the new token write widens that window slightly. The pre-existing code had the same gap.

Impact: The leaked lock makes every other run wait 300 s and then fail. This matters mainly on a developer machine or a cancelled job on a reused host. Hosted runners are ephemeral.

Fix: pack_lock_release is now idempotent and does nothing without a token. The trap can therefore be set before the acquire, with the same body:

trap 'pack_lock_release; [[ -n "${PACK_INSTALL_DIR:-}" ]] || rm -rf "$WORK"' EXIT
pack_lock_acquire ... || exit 1

This is cheap. The pin test trap 'pack_lock_release;[^']*' EXIT still matches.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: the EXIT trap is now installed before pack_lock_acquire, with the same body (the release does nothing without a token, so it is safe before one exists). A test pins that the trap comes before the acquire, and fails against the previous order.

Comment thread scripts/pack-lock.sh
if [[ -e "$lock" ]]; then
echo "warning: $lock is held by another run now; left it alone" >&2
else
echo "warning: $lock was removed while it was held, so the pack was not serialized" >&2

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: P3 Low

Problem: The only signal that serialization failed is a plain stderr line. ci-each-version.sh writes each run's output to a log and prints it in a collapsed ::group:: only after all runs finish. A run that hit this branch exits 0, so nobody sees the line.

Impact: The fix turns a loud failure into a silent pass. The overlapping npm pack the lock exists to prevent can still happen. If it does, the next failure is probably a confusing missing native-versions.json in a later step, with the cause buried in a green job's log. The unexplained directory removal is also the only clue to the real root cause, and a quiet line makes it likely to be missed.

Fix: Emit it as an annotation, for example echo "::warning::$lock was removed while it was held..." when GITHUB_ACTIONS is set. Alternatively ci-each-version.sh could grep the logs for the warning and surface it. Without this the root cause stays hard to catch.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this was the weak point: the fix turned a loud failure into a quiet pass and buried the only clue to the real cause. pack_lock_warn now also emits ::warning title=pack-install lock::... on stdout when GITHUB_ACTIONS is set, for both the lost-lock and the taken-by-another-run cases (the runner parses it when ci-each-version.sh cats the log, so it lands on the run summary), and the text says nothing in this repository removes the directory and asks for the run to be reported. The stderr line stays for reads outside Actions. Tests cover both cases on Actions and no annotation off Actions.

@krassx
krassx merged commit bad582d into main Nov 16, 2026
29 checks passed
@krassx
krassx deleted the fix/pack-install-lock-race branch November 16, 2026 12:21
@krassx

krassx commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Note: the review fixes on this PR (failed token write, trap before acquire, ::warning annotation) were pushed after it merged, so they are not in main. They are in the follow-up #193.

Human guided, AI implemented

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant