Skip to content

feat(sandbox-providers): end one slot of a shared VM, and never end the shared VM from one task's close - #829

Open
mandykwok-scale wants to merge 3 commits into
mainfrom
mandy/shared-slot-teardown
Open

mandykwok-scale wants to merge 3 commits into
mainfrom
mandy/shared-slot-teardown

Conversation

@mandykwok-scale

@mandykwok-scale mandykwok-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

Several tasks will share one Modal VM to cut cost, each in its own slot (#775 gives each slot its own compose project, dir, ports and scratch files). Today every way an env closes ends its whole VM, so once a VM holds several tasks, one task's close would kill the others. This PR adds the two framework pieces that stop that: ending a single slot, and one central guard so a shared VM is ended only by the step that ends the whole box. Nothing creates shared VMs yet.

What

  • tear_down_slot(sandbox, layout) removes every container, volume and network labelled with the slot's compose project, then its dir and scratch files, and leaves the VM, its images and every other slot alone. It needs no compose file, is idempotent, and refuses the default stack and any layout not made by StackLayout.for_slot. The caller must stop the slot's deploy first.
  • ModalVmSandboxProvider.create_shared_box(box_id, ...) tags the VM agent_env_shared_box=<box_id> at create. Every Modal terminate (both handles' terminate(), every terminate_sandbox) goes through one check: a handle created as a normal sandbox terminates exactly as today; a handle reattached by id reads the sandbox's tags first and refuses a box with SharedBoxTerminateRefused; only terminate_shared_box(sandbox_id, box_id) ends a box. reset_vm refuses a known box. A source test fails if any module that uses the Modal SDK terminates a sandbox another way.

One change to today's behaviour: a terminate by id (teardown, reattached handles) now makes one extra get_tags read. If that read keeps failing (3 tries, about 3 s), the sandbox is not terminated and teardown reports it, so it runs until its Modal timeout. That fails toward cost, never toward ending a shared box.

Validation

5,271 provider, env and task unit tests pass (the one failure, the Windows disk-recipe test, also fails on main). The new guard tests and every close path of a slot env fail with the guard removed. A real-Docker test tears one of two slots down twice and checks the other is untouched. On real Modal, gVisor and VM sandboxes: tags set at create come back exactly from a fresh from_id handle, and stay readable after the sandbox exits or is terminated, so teardown of finished sandboxes is unchanged. Not yet run: a full box on a Modal VM.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 4/5

Meet the repository’s named-default requirement before merging; the two earlier test-cleanup issues also remain.

Fix All in CursorFindings

  1. P2 Shared timeout repeats unnamed values ▶
  2. P2 Failed setup leaves containers running. ▶
  3. P2 Concurrent tests delete shared stacks. ▶
Fix with agent prompt
### Issue 1
src/agent_env/providers/sandbox_providers/sandbox_provider.py:106
**Shared timeout repeats unnamed values.** The new `create_shared_box` methods both use `3600 * 2` for their default timeout. The repository requires magic numbers to be stored in descriptively named class or instance variables. Define a shared named default and use it here and in `ModalVmSandboxProvider.create_shared_box` so the two defaults stay together.

### Issue 2
tst/integration/providers/env_providers/slot_teardown_test.py:45-47
The `compose up` calls run before the `try`. If either call fails after creating resources, `two_slots` exits without running `down -v`. This leaves containers, volumes, and networks on the developer’s Docker daemon. Move the setup loop inside the `try` so partial setup is cleaned up too.

### Issue 3
tst/integration/providers/env_providers/slot_teardown_test.py:undefined-27
Every run uses Docker projects `s43` and `s44` on the host daemon. Two copies of this test, such as runs from separate checkouts, therefore create and delete the same resources. The unique CLI container name does not isolate those projects. Use a private Docker daemon or a lock shared across test runs before creating either stack.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds slot-only cleanup and protects shared Modal VMs from ordinary shutdown calls.

  • Callers can remove one slot without ending its shared VM.
  • Only the matching box owner can end a shared Modal VM.

Reviews (2) · Last reviewed commit: "fix(sandbox-providers): never reboot a s..." · Reviewed by Greptile

tear_down_slot ends one slot's stack, leaving the VM and every other slot
running: it removes the containers, volumes and networks its compose project
labels, then its dir and scratch files. It needs no compose file, so it also
clears a slot whose deploy failed early, and it is idempotent, so a retry is
safe. It refuses the default stack and any layout not made by for_slot, so
its file glob can't reach another stack's files. Nothing calls it yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mandykwok-scale
mandykwok-scale requested a review from a team as a code owner November 19, 2026 23:06
Comment on lines +45 to +47
for layout in (GONE, KEPT):
_docker("compose", "-p", layout.compose_project, "-f", str(compose_file), "up", "-d", "--quiet-pull")
try:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Failed setup leaves containers running. The compose up calls run before the try. If either call fails after creating resources, two_slots exits without running down -v. This leaves containers, volumes, and networks on the developer’s Docker daemon. Move the setup loop inside the try so partial setup is cleaned up too.

Prompt To Fix With AI
This is a comment left during a code review.
Path: tst/integration/providers/env_providers/slot_teardown_test.py
Line: 45-47

Comment:
**Failed setup leaves containers running.** The `compose up` calls run before the `try`. If either call fails after creating resources, `two_slots` exits without running `down -v`. This leaves containers, volumes, and networks on the developer’s Docker daemon. Move the setup loop inside the `try` so partial setup is cleaned up too.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

volumes:
data: {{}}
"""
GONE, KEPT = StackLayout.for_slot(43), StackLayout.for_slot(44)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Concurrent tests delete shared stacks. Every run uses Docker projects s43 and s44 on the host daemon. Two copies of this test, such as runs from separate checkouts, therefore create and delete the same resources. The unique CLI container name does not isolate those projects. Use a private Docker daemon or a lock shared across test runs before creating either stack.

Prompt To Fix With AI
This is a comment left during a code review.
Path: tst/integration/providers/env_providers/slot_teardown_test.py
Line: 27

Comment:
**Concurrent tests delete shared stacks.** Every run uses Docker projects `s43` and `s44` on the host daemon. Two copies of this test, such as runs from separate checkouts, therefore create and delete the same resources. The unique CLI container name does not isolate those projects. Use a private Docker daemon or a lock shared across test runs before creating either stack.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

mandykwok-scale and others added 2 commits November 19, 2026 18:39
…_shared_box

Several tasks' slots will share one Modal VM, a shared box, so no single task's
close may end it. Every Modal terminate now goes through one check: a handle this
process created terminates as before; one reattached by id, and terminate_sandbox,
first read the sandbox's tags and refuse, with SharedBoxTerminateRefused, to end one
tagged agent_env_shared_box. A tag read that keeps failing raises and terminates
nothing. create_shared_box tags the VM at create, and terminate_shared_box ends a box
only when its tag names that box. Callers already log a failed terminate, and a
teardown reports the box as not terminated. Nothing creates boxes yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dal SDK module for stray terminates

reset_vm refuses a handle known to be a shared box, whose reboot would end every
slot on it. The source check now covers every agent_env module that uses the
Modal SDK and fails on a direct SandboxTerminate RPC anywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mandykwok-scale mandykwok-scale changed the title feat(env-providers): tear down one slot of a shared VM feat(sandbox-providers): end one slot of a shared VM, and never end the shared VM from one task's close Nov 19, 2026
*,
cpu: float = 1.0,
memory: int = 8192,
timeout: int = 3600 * 2,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Shared timeout repeats unnamed values. The new create_shared_box methods both use 3600 * 2 for their default timeout. The repository requires magic numbers to be stored in descriptively named class or instance variables. Define a shared named default and use it here and in ModalVmSandboxProvider.create_shared_box so the two defaults stay together.

Rule Used: Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. (source)

Learned From
scaleapi/scaleapi#126388

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/providers/sandbox_providers/sandbox_provider.py
Line: 106

Comment:
**Shared timeout repeats unnamed values.** The new `create_shared_box` methods both use `3600 * 2` for their default timeout. The repository requires magic numbers to be stored in descriptively named class or instance variables. Define a shared named default and use it here and in `ModalVmSandboxProvider.create_shared_box` so the two defaults stay together.

**Rule Used:** Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. ([source](https://app.greptile.com/-/switch-tenant?tenant=scale-ai&next=%2Fcustom-context%3Fmemory%3D002e0051-41ad-46c1-9098-47433c580150))

**Learned From**
[scaleapi/scaleapi#126388](https://github.com/scaleapi/scaleapi/pull/126388)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Claude Code Fix in Codex

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant