Repository navigation
feat(sandbox-providers): end one slot of a shared VM, and never end the shared VM from one task's close - #829
mandykwok-scale wants to merge 3 commits into
Conversation
tear_down_slot ends one slot's stack, leaving the VM and every other slot running: it removes the containers, volumes and networks its compose project labels, then its dir and scratch files. It needs no compose file, so it also clears a slot whose deploy failed early, and it is idempotent, so a retry is safe. It refuses the default stack and any layout not made by for_slot, so its file glob can't reach another stack's files. Nothing calls it yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| for layout in (GONE, KEPT): | ||
| _docker("compose", "-p", layout.compose_project, "-f", str(compose_file), "up", "-d", "--quiet-pull") | ||
| try: |
There was a problem hiding this comment.
Failed setup leaves containers running. The
compose up calls run before the try. If either call fails after creating resources, two_slots exits without running down -v. This leaves containers, volumes, and networks on the developer’s Docker daemon. Move the setup loop inside the try so partial setup is cleaned up too.
Prompt To Fix With AI
This is a comment left during a code review.
Path: tst/integration/providers/env_providers/slot_teardown_test.py
Line: 45-47
Comment:
**Failed setup leaves containers running.** The `compose up` calls run before the `try`. If either call fails after creating resources, `two_slots` exits without running `down -v`. This leaves containers, volumes, and networks on the developer’s Docker daemon. Move the setup loop inside the `try` so partial setup is cleaned up too.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| volumes: | ||
| data: {{}} | ||
| """ | ||
| GONE, KEPT = StackLayout.for_slot(43), StackLayout.for_slot(44) |
There was a problem hiding this comment.
Concurrent tests delete shared stacks. Every run uses Docker projects
s43 and s44 on the host daemon. Two copies of this test, such as runs from separate checkouts, therefore create and delete the same resources. The unique CLI container name does not isolate those projects. Use a private Docker daemon or a lock shared across test runs before creating either stack.
Prompt To Fix With AI
This is a comment left during a code review.
Path: tst/integration/providers/env_providers/slot_teardown_test.py
Line: 27
Comment:
**Concurrent tests delete shared stacks.** Every run uses Docker projects `s43` and `s44` on the host daemon. Two copies of this test, such as runs from separate checkouts, therefore create and delete the same resources. The unique CLI container name does not isolate those projects. Use a private Docker daemon or a lock shared across test runs before creating either stack.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.…_shared_box Several tasks' slots will share one Modal VM, a shared box, so no single task's close may end it. Every Modal terminate now goes through one check: a handle this process created terminates as before; one reattached by id, and terminate_sandbox, first read the sandbox's tags and refuse, with SharedBoxTerminateRefused, to end one tagged agent_env_shared_box. A tag read that keeps failing raises and terminates nothing. create_shared_box tags the VM at create, and terminate_shared_box ends a box only when its tag names that box. Callers already log a failed terminate, and a teardown reports the box as not terminated. Nothing creates boxes yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dal SDK module for stray terminates reset_vm refuses a handle known to be a shared box, whose reboot would end every slot on it. The source check now covers every agent_env module that uses the Modal SDK and fails on a direct SandboxTerminate RPC anywhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| *, | ||
| cpu: float = 1.0, | ||
| memory: int = 8192, | ||
| timeout: int = 3600 * 2, |
There was a problem hiding this comment.
Shared timeout repeats unnamed values. The new
create_shared_box methods both use 3600 * 2 for their default timeout. The repository requires magic numbers to be stored in descriptively named class or instance variables. Define a shared named default and use it here and in ModalVmSandboxProvider.create_shared_box so the two defaults stay together.
Rule Used: Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. (source)
Learned From
scaleapi/scaleapi#126388
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/providers/sandbox_providers/sandbox_provider.py
Line: 106
Comment:
**Shared timeout repeats unnamed values.** The new `create_shared_box` methods both use `3600 * 2` for their default timeout. The repository requires magic numbers to be stored in descriptively named class or instance variables. Define a shared named default and use it here and in `ModalVmSandboxProvider.create_shared_box` so the two defaults stay together.
**Rule Used:** Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. ([source](https://app.greptile.com/-/switch-tenant?tenant=scale-ai&next=%2Fcustom-context%3Fmemory%3D002e0051-41ad-46c1-9098-47433c580150))
**Learned From**
[scaleapi/scaleapi#126388](https://github.com/scaleapi/scaleapi/pull/126388)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Why
Several tasks will share one Modal VM to cut cost, each in its own slot (#775 gives each slot its own compose project, dir, ports and scratch files). Today every way an env closes ends its whole VM, so once a VM holds several tasks, one task's close would kill the others. This PR adds the two framework pieces that stop that: ending a single slot, and one central guard so a shared VM is ended only by the step that ends the whole box. Nothing creates shared VMs yet.
What
tear_down_slot(sandbox, layout)removes every container, volume and network labelled with the slot's compose project, then its dir and scratch files, and leaves the VM, its images and every other slot alone. It needs no compose file, is idempotent, and refuses the default stack and any layout not made byStackLayout.for_slot. The caller must stop the slot's deploy first.ModalVmSandboxProvider.create_shared_box(box_id, ...)tags the VMagent_env_shared_box=<box_id>at create. Every Modal terminate (both handles'terminate(), everyterminate_sandbox) goes through one check: a handle created as a normal sandbox terminates exactly as today; a handle reattached by id reads the sandbox's tags first and refuses a box withSharedBoxTerminateRefused; onlyterminate_shared_box(sandbox_id, box_id)ends a box.reset_vmrefuses a known box. A source test fails if any module that uses the Modal SDK terminates a sandbox another way.One change to today's behaviour: a terminate by id (teardown, reattached handles) now makes one extra
get_tagsread. If that read keeps failing (3 tries, about 3 s), the sandbox is not terminated and teardown reports it, so it runs until its Modal timeout. That fails toward cost, never toward ending a shared box.Validation
5,271 provider, env and task unit tests pass (the one failure, the Windows disk-recipe test, also fails on
main). The new guard tests and every close path of a slot env fail with the guard removed. A real-Docker test tears one of two slots down twice and checks the other is untouched. On real Modal, gVisor and VM sandboxes: tags set at create come back exactly from a freshfrom_idhandle, and stay readable after the sandbox exits or is terminated, so teardown of finished sandboxes is unchanged. Not yet run: a full box on a Modal VM.🤖 Generated with Claude Code
Meet the repository’s named-default requirement before merging; the two earlier test-cleanup issues also remain.
Fix with agent prompt
Summary
Adds slot-only cleanup and protects shared Modal VMs from ordinary shutdown calls.
Reviews (2) · Last reviewed commit: "fix(sandbox-providers): never reboot a s..." · Reviewed by Greptile