Skip to content

fix(ops): stop VMs before recreating them, with the real grace period - #17

Merged
psyb0t merged 1 commit into
psyb0t:masterfrom
Marinski:fix/recreate-vm-stop-timeout
Aug 25, 2026
Merged

fix(ops): stop VMs before recreating them, with the real grace period#17
psyb0t merged 1 commit into
psyb0t:masterfrom
Marinski:fix/recreate-vm-stop-timeout

Conversation

@Marinski

Copy link
Copy Markdown
Contributor

recreate-vm.sh recreates with docker compose up -d --force-recreate, whose implicit stop uses compose's own --timeout10 seconds by default — rather than the service's declared stop_grace_period. A dockurr/windows guest cannot shut down in ten seconds, so compose stops waiting and goes straight to removing a container that is still running:

Error response from daemon: cannot remove container "1d5e3c2f...":
container is running: stop the container before removing or force remove

The script exits 1 and the VM is left unhealthy with its network_mode sidecar stranded on a dead netns — the exact outcome recreate-vm.sh exists to prevent.

It is timing-dependent, which is why it can look fine for a while. On the deployment where I hit this, the script recreated two VMs successfully three times inside one hour, then failed on the fourth attempt when the guest took longer than ten seconds to go down.

The change

Targets are stopped explicitly first with a timeout matching the grace period, and the same value is passed to up so its implicit stop cannot fall back to 10s:

docker compose stop -t "$STOP_TIMEOUT" "${targets[@]}"
docker compose up -d --force-recreate --no-deps -t "$STOP_TIMEOUT" "${targets[@]}"

RECREATE_STOP_TIMEOUT overrides the 120s default. Anything invoking this script under a timeout of its own should stay above it.

Tests

The script had no direct coverage, which is how this reached production. tests/test_recreate_vm_script.py covers it through --dry-run, so it needs no Docker daemon: stop-before-up ordering, the timeout default and override, and the sidecar expansion that is the reason the script exists.

Four of the seven fail against master and pass with this change — they pin the bug, not just the new code.

Dockerfile.test copies a named subset of scripts/ and recreate-vm.sh was not in it, so the tests could not see the script. It is added to that COPY line; nothing else about the image changes.

make test-unit: 423 passed, 2 skipped (416 on master plus the 7 new). make verify-binaries: OK.

Relationship to #15

That PR's watchdog delegates recovery to this script, per your review. Merging it without this fix ships an automated recovery path that hits the failure above — which is how I found it. This PR is independent and can land first.

recreate-vm.sh recreates with `docker compose up -d --force-recreate`, whose
implicit stop uses compose's own --timeout -- 10 SECONDS by default -- rather
than the service's declared stop_grace_period. A dockurr/windows guest cannot
shut down in ten seconds, so compose stops waiting and goes straight to
removing a container that is still running:

  Error response from daemon: cannot remove container "1d5e3c2f...":
  container is running: stop the container before removing or force remove

The script then exits 1 and the VM is left unhealthy with its network_mode
sidecar stranded on a dead netns -- the exact outcome recreate-vm.sh exists to
prevent.

It is timing-dependent, which is why it can look fine for a while. On the
deployment where this was found the script recreated two VMs successfully three
times inside one hour, then failed on the fourth attempt when the guest took
longer than ten seconds to go down.

The targets are now stopped explicitly first with a timeout that matches the
grace period, and the same value is passed to `up` so its implicit stop cannot
fall back to 10s. RECREATE_STOP_TIMEOUT overrides the 120s default; anything
calling this script on a timeout of its own should stay above it.

The script had no direct test coverage. tests/test_recreate_vm_script.py covers
it through --dry-run, so it needs no Docker daemon: stop-before-up ordering, the
timeout default and its override, and the sidecar expansion that is the reason
the script exists. Four of the seven fail against the current version.

Worth noting for psyb0t#15: that watchdog delegates recovery to this script, so
merging it without this fix ships an automated recovery path that hits the
failure above.

Dockerfile.test copies a named subset of scripts/ and recreate-vm.sh was not in
it, so the new tests could not see the script. It is added to that COPY line;
nothing else about the image changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants