Skip to content

Mark dead runs as failed; atomic, Windows-safe write-run guard - #1

Merged
rtravellin merged 4 commits into
rtravellin:mainfrom
webandseo:fix/retire-dead-runs
Sep 28, 2026
Merged

rtravellin merged 4 commits into
rtravellin:mainfrom
webandseo:fix/retire-dead-runs

Conversation

@webandseo

Copy link
Copy Markdown
Contributor

Three fixes to run tracking, found on Windows. One commit per fix, reviewable separately; each builds on the one before, so they come as one PR.

1. Refuse concurrent write-capable delegate runs; fix the Windows liveness check

The muse-delegate subagent ran run --write in the foreground. Claude Code's Bash tool moved it to the background after its 120 s default timeout, and the subagent then called run again, so two write-capable Muse runs edited the same checkout at once.

  • process.mjs: isProcessAlive() asked ps -o stat= after signal 0 to spot zombies. On Windows ps is either missing or Git Bash's (no -o), so every live process looked dead. Windows now trusts signal 0 (ESRCH = dead, EPERM = alive); POSIX keeps the zombie check.
  • muse-bridge.mjs: run --write refuses to start while another write-capable run in the workspace is alive, names its id with the /muse:runs <id> --wait and /muse:stop <id> follow-ups, and exits non-zero. Stale records are marked failed instead of blocking. --allow-concurrent overrides. Foreground runs print their run id to stderr so a backgrounded call can still be followed.
  • muse-delegate agent, muse-delegate-runtime skill, /muse:delegate: the single Bash call sets timeout 600000; if it is backgrounded or times out, the subagent returns the notice and run id and never calls run again.

2. Make the write-run guard atomic

The guard read the run list without a lock, and the new run's record was written later, so run --write calls started together could all pass the check. claimWriteSlot now checks for a live write run and records the new one as queued inside a single state lock.

3. Mark runs whose processes are gone as failed everywhere

A run whose bridge and muse processes had exited (an interrupted foreground review or delegate, a killed worker, a closed terminal) stayed running forever:

  • runs <id> --wait sat out its whole timeout and returned running.
  • stop answered "process may still be running (kill not delivered)" for a process that had died long before.
  • --resume refused with "still running", and run-resume-candidate did not offer the interrupted run.

retireDeadRuns claims each dead queued or running run as failed under the state lock and logs it only when the claim wins, so a run that completed in the meantime is left alone. runs, show, stop, resume and run-resume-candidate call it first, and runs --wait calls it on every poll. The write guard uses the same helper. Runs with no pid yet keep a 60 s grace.

README and CHANGELOG (Unreleased) are updated.

Tests

  • process.test.mjs: live and exited children, signal 0 on win32 without ps, POSIX zombies still dead.
  • job-control.test.mjs (new): live/stale partitioning with the no-pid grace, sequential claimWriteSlot, a four-process race on claimWriteSlot, and retireDeadRuns (dead review, dead read-only run, live run, finished run).
  • runtime.test.mjs: write-run refusal and --allow-concurrent; a CLI race of three concurrent run --write --background calls; a dead write run retired by the guard; the foreground run id notice; runs, runs --wait and show on a dead run; a run that dies while --wait is polling; stop on a dead run; --resume-last and run-resume-candidate after a run died.
  • commands.test.mjs: the delegate forwarding contract (long Bash timeout, no second run, --wait kept on the Claude side).
  • Both race tests fail against a check-then-record guard, and the mid-wait test times out without the per-poll retire.
  • Local suite on Windows 11, Node 24: 104/104 pass.

🤖 Generated with Claude Code

webandseo and others added 3 commits September 24, 2026 22:31
…heck

The muse-delegate subagent ran `run --write` in the foreground; Claude
Code's Bash tool moved it to the background after its 120s default
timeout, and the subagent then called `run` again, so two write-capable
Muse runs edited the same checkout at once.

- process.mjs: export isProcessAlive(). It asked `ps -o stat=` after
  signal 0 to spot zombies, but on Windows `ps` is either missing or Git
  Bash's (no `-o`), so every live process looked dead. Windows now trusts
  signal 0 (ESRCH = dead, EPERM = alive); POSIX keeps the zombie check.
- job-control.mjs: partitionActiveWriteRuns() splits queued/running
  write-capable task runs into live and stale by their recorded pids,
  with a 60s grace for a run that has no pid yet.
- muse-bridge.mjs: `run --write` refuses to start while another
  write-capable run in the workspace is alive, naming its id and the
  `/muse:runs <id> --wait` and `/muse:stop <id>` follow-ups, and exits
  non-zero. Stale records are marked failed instead of blocking.
  `--allow-concurrent` overrides. Foreground runs print their run id to
  stderr so a backgrounded call can still be followed.
- agents/muse-delegate.md, skills/muse-delegate-runtime/SKILL.md,
  commands/delegate.md: the single Bash call sets `timeout` 600000; if it
  is backgrounded or times out the subagent returns the notice and run id
  and never calls `run` again. `--wait` is never forwarded; `--background`
  only as the bridge's own `run --background`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guard read the run list without a lock and the new run's record was
written later, so run --write calls started together could all pass the
check. claimWriteSlot now checks for a live write run and records the new
one as queued inside a single state lock.

Tests: a sequential claim test, a four-process race on claimWriteSlot, and
a CLI race of three concurrent run --write --background calls. Both race
tests fail against a check-then-record version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A run whose bridge and muse processes had exited (an interrupted foreground
review or delegate, a killed worker, a closed terminal) stayed "running"
forever: `runs --wait` sat out its whole timeout, `stop` reported "process
may still be running", and `--resume` refused with "still running". Only
the write-run guard retired dead records, and only write-capable ones.

- lib/job-control.mjs: partitionActiveRuns covers every queued or running
  run; partitionActiveWriteRuns filters it to write-capable delegate runs.
  retireDeadRuns claims each dead run as failed under the state lock and
  logs it only when the claim wins, so a run that completed meanwhile is
  left alone.
- muse-bridge.mjs: runs, show, stop, resume and run-resume-candidate call
  retireDeadRuns first; runs --wait also calls it on every poll. stop on a
  run that had just been retired says it had already ended. The write guard
  uses the same helper.
- Tests: unit test for retireDeadRuns; runtime tests for runs/--wait/show,
  a run that dies mid-wait (mutation-checked: without the per-poll retire
  --wait times out), stop, and --resume-last / run-resume-candidate.
- README and CHANGELOG (Unreleased).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@rtravellin rtravellin left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, really solid work. The write-run guard, the atomic claim, and cleaning up dead runs all fix real problems, and the tests are great, especially the race tests.

Three things I'd like changed before merging:

1. Live runs get marked failed when ps isn't available. isZombieProcess returns true (dead) when ps can't run or exits non-zero. That was harmless when only stop used it, but retireDeadRuns now calls it on every runs/show/stop/resume and every --wait poll. On a box without ps (e.g. slim Docker images), /muse:runs marks every active run failed while it's still working, and the real result is thrown away when it finishes. Since signal 0 has already shown the process exists, can we count it as alive when ps itself fails? It's the same thing you fixed for Windows, just on the POSIX side.

2. The guard blocks --worktree runs. Worktree runs edit their own checkout under .muse/worktrees/, so they can't clash with the live checkout or with each other. Right now run --write --worktree is refused while any other write run is going, and a worktree run blocks a normal one. Could you record the worktree flag on the job and leave worktree runs out of partitionActiveWriteRuns?

3. (minor) A stop prefix can match the wrong run. handleCancel looks at the just-retired runs first, so if a partial id matches both a dead run and a live one, it prints "had already ended… Nothing to stop" and the live run keeps going. I'd expect the usual "ambiguous reference" error there.

Happy to merge once 1 and 2 are in. 3 can go in this PR or a follow-up, whichever you prefer.

- On POSIX, count a pid that signal 0 found as alive unless ps reports a
  zombie; a missing or failing ps no longer marks live runs failed.
- Record the worktree flag on task jobs and leave worktree runs out of the
  one-write-run guard both ways, since they edit their own checkout.
- stop resolves a reference against just-retired and live runs together,
  so a prefix matching both is ambiguous instead of picking the ended one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@webandseo

Copy link
Copy Markdown
Contributor Author

All three addressed in b8b5801:

  1. isZombieProcess only reports dead when ps shows a zombie state; a missing/failing ps or empty output keeps the pid alive since signal 0 already found it. Test covers ENOENT, non-zero exit, empty stdout and a throw.
  2. Task jobs record worktree; partitionActiveWriteRuns excludes them and run --write --worktree skips the guard, so worktree runs neither wait for nor block other write runs. The flag is also kept on the foreground record path in runTrackedJob.
  3. stop resolves the reference against just-retired and live runs together, so a prefix matching both gets the usual "ambiguous" error.

@rtravellin rtravellin left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, all three look good. Merging.

@rtravellin
rtravellin merged commit 511b6f7 into rtravellin:main Sep 28, 2026
6 checks passed
@webandseo
webandseo deleted the fix/retire-dead-runs branch September 30, 2026 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants