fix(ops): stop VMs before recreating them, with the real grace period - #17
Merged
Merged
Conversation
recreate-vm.sh recreates with `docker compose up -d --force-recreate`, whose implicit stop uses compose's own --timeout -- 10 SECONDS by default -- rather than the service's declared stop_grace_period. A dockurr/windows guest cannot shut down in ten seconds, so compose stops waiting and goes straight to removing a container that is still running: Error response from daemon: cannot remove container "1d5e3c2f...": container is running: stop the container before removing or force remove The script then exits 1 and the VM is left unhealthy with its network_mode sidecar stranded on a dead netns -- the exact outcome recreate-vm.sh exists to prevent. It is timing-dependent, which is why it can look fine for a while. On the deployment where this was found the script recreated two VMs successfully three times inside one hour, then failed on the fourth attempt when the guest took longer than ten seconds to go down. The targets are now stopped explicitly first with a timeout that matches the grace period, and the same value is passed to `up` so its implicit stop cannot fall back to 10s. RECREATE_STOP_TIMEOUT overrides the 120s default; anything calling this script on a timeout of its own should stay above it. The script had no direct test coverage. tests/test_recreate_vm_script.py covers it through --dry-run, so it needs no Docker daemon: stop-before-up ordering, the timeout default and its override, and the sidecar expansion that is the reason the script exists. Four of the seven fail against the current version. Worth noting for psyb0t#15: that watchdog delegates recovery to this script, so merging it without this fix ships an automated recovery path that hits the failure above. Dockerfile.test copies a named subset of scripts/ and recreate-vm.sh was not in it, so the new tests could not see the script. It is added to that COPY line; nothing else about the image changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
recreate-vm.shrecreates withdocker compose up -d --force-recreate, whose implicit stop uses compose's own--timeout— 10 seconds by default — rather than the service's declaredstop_grace_period. Adockurr/windowsguest cannot shut down in ten seconds, so compose stops waiting and goes straight to removing a container that is still running:The script exits 1 and the VM is left unhealthy with its
network_modesidecar stranded on a dead netns — the exact outcomerecreate-vm.shexists to prevent.It is timing-dependent, which is why it can look fine for a while. On the deployment where I hit this, the script recreated two VMs successfully three times inside one hour, then failed on the fourth attempt when the guest took longer than ten seconds to go down.
The change
Targets are stopped explicitly first with a timeout matching the grace period, and the same value is passed to
upso its implicit stop cannot fall back to 10s:RECREATE_STOP_TIMEOUToverrides the 120s default. Anything invoking this script under a timeout of its own should stay above it.Tests
The script had no direct coverage, which is how this reached production.
tests/test_recreate_vm_script.pycovers it through--dry-run, so it needs no Docker daemon: stop-before-up ordering, the timeout default and override, and the sidecar expansion that is the reason the script exists.Four of the seven fail against
masterand pass with this change — they pin the bug, not just the new code.Dockerfile.testcopies a named subset ofscripts/andrecreate-vm.shwas not in it, so the tests could not see the script. It is added to thatCOPYline; nothing else about the image changes.make test-unit: 423 passed, 2 skipped (416 onmasterplus the 7 new).make verify-binaries: OK.Relationship to #15
That PR's watchdog delegates recovery to this script, per your review. Merging it without this fix ships an automated recovery path that hits the failure above — which is how I found it. This PR is independent and can land first.