Skip to content

feat(task): agent-env task teardown finishes the teardown a run left, which the run names - #786

Merged
earakely-scale merged 4 commits into
mainfrom
edgararakelyan/fd-6255-a-run-whose-teardown-was-stopped
Nov 18, 2026
Merged

earakely-scale merged 4 commits into
mainfrom
edgararakelyan/fd-6255-a-run-whose-teardown-was-stopped

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

agent-env task teardown --instance <id> finishes the teardown a run left. agent-env run names it whenever something is still up.

When a second Ctrl-C or SIGTERM stopped a run's teardown, or a sandbox couldn't be torn down, the run listed what was still up and exited. No command removed it: the user had to docker rm -f the container and delete its folder by hand. Two gaps sat behind that:

  • Nothing finished a teardown later.
  • A cancelled run's record didn't hold everything to tear down. record_task_cancelled writes no context, so a sandbox recorded by a step the signal cut short was in the run's memory but not in the store. Nor did the store hold which sandboxes the teardown had already taken down.

This PR adds:

  • Every teardown is recorded (record_teardown in task/teardown.py). It keeps on the instance the context the teardown ended with: what the run deployed, an interrupted step's records included; what it took down (torn_down_sandbox_ids); and what it left up (left_up_sandbox_ids, a new key, covering a sandbox terminated whose folder couldn't be removed).
    • The write is fenced by the instance's attempt epoch as the teardown began, as the store's other writes are, so it never overwrites an attempt that took the instance over meanwhile.
    • It runs in the run's namespace (run_scope(instance_id)), since instance ids derive from task ids.
    • Where it's called: run_bundle and BundleRun.teardown call it once their loop has closed, as _mark_cancelled does; task run --keep after its own teardown of kept runs, stopped or not; and the command.
    • An instance that can't be read before a teardown (attempt_epoch returns None) leaves that teardown unrecorded, never stopped.
  • The runs name the command. After "Still up", agent-env run and task run --keep print:
    Tear down what's left with:
      agent-env task teardown --instance <id>
    
  • agent-env task teardown --instance <id> tears down what the record still has up (recorded_still_up), through the run's own teardown (teardown_run), in the run's namespace:
    • The recordings it left open end, each env still up closes, and each sandbox is terminated by id, a local one's folder removed.
    • It skips what's down and retries only the folder of a sandbox terminated before.
    • A local sandbox whose folder is gone counts as gone, as agent-env run already lists it (gone() in cli/teardown_output.py, now shared). That covers instances recorded before this.
    • It records what it reached, keeps that record even through Ctrl-C, and exits 1 while anything is still up. Running it again goes on from there.
    • It leaves an instance recorded running alone, since its run may still be using the sandboxes, unless --force, as for a run killed outright. It rejects an unknown instance.
  • BundleRun.teardown turns namespace routing on, as run_bundle does, so a script's --keep teardown finds its @local envs and record.
  • AGENTS.md says the above where it describes the teardown core, and where local teardown says its folder is kept "so teardown can be retried".

FD-6255, under FD-6253.

Not here:

  • FD-3093's sweep of instances a hard kill left running; --force covers those by hand.
  • run_bundle's own teardown of a run (settle) still runs outside the run's scope, as before this PR. With a configured remote object store, a recording it ends or registers may use the configured store rather than the run's own. That's worth a look separately.

For the docs: the CUA docs' "Deploying your computer environment" says a stopped teardown "prints what is still up". It can now add that the run names agent-env task teardown --instance <id>, which finishes it.

Proof

  • Tests (tst/unit/bundle/run_teardown_test.py, through the CLI with the fake backend):
    • A teardown a second signal stopped: the run exits 130 and ends with the command, naming its instance. The command tears down (exit 0), and run again says Nothing instance … recorded is still up. With the record off, the command finds nothing, since the interrupted step never completed.
    • A finished run whose teardown failed: the run records left_up_sandbox_ids == ["sb-ok"]. The command, still failing, exits 1. Once the backend recovers, it tears down, and the record shows torn_down_sandbox_ids == ["sb-ok"].
    • A finished run torn down cleanly is recorded (torn_down == ["sb-ok"], left_up == []), and the command says nothing is up and terminates nothing again.
    • A sandbox terminated whose folder wasn't removed is still up, and the command removes only its folder, terminating nothing again.
    • A local sandbox whose folder is gone, on an instance recorded before this, counts as gone. Without that rule the command exits 1.
    • The command tears down inside the run's namespace (local).
    • A record from an older attempt doesn't overwrite one that took the instance over.
    • An instance recorded running is refused without --force and torn down with it. An unknown instance is refused.
  • Tests (tst/unit/cli/task_run_teardown_test.py):
    • task run --keep, whose teardown can't take a sandbox down, names the command with its instance and keeps the context the teardown ended with.
    • Its kept teardown goes ahead though the instance can't be read to record it. That test fails without the fix.
  • Suites: tst/unit/bundle, cli, task, runner and task_step gave 5,609 passed, 14 skipped. Ruff is clean.
  • An independent review found the same gaps Greptile did, plus Ctrl-C during the command's record and library use of BundleRun.teardown; all are fixed above.
  • Live, on an agent-environments-admin VM (amd64, local Docker), with the final code before the last commit, which only guards the epoch read: the shipped desktop example's probe task.
    • It was sent SIGTERM mid-probe, and a second once the run printed Cancelling. GNU timeout sent only one there, and two sent at once are taken as one.
    • The run: exit 143, then:
      Still up, 1 sandbox:
        local-271a9906  local  /home/ubuntu/.agent-env-sandboxes/agent-env-local-271a9906-6d77ibom
      Tear down what's left with:
        agent-env task teardown --instance @local/~/tf/run-lUUIwb/proof/desktop/probe-r9a1lmw5
      
      Its container was still running, and its folder was there.
    • That command: Tore down 1 sandbox, exit 0. No container was left, and the folder was gone.
    • Run again: Nothing … recorded is still up., exit 0.
    • The warm-up run, which passed and tore down cleanly: the command on its instance says Nothing … recorded is still up.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The follow-up changes appear safe to merge.

Summary

Adds a command to finish cleanup that a run left behind, saves cleanup progress, and prints the recovery command when sandboxes remain up.

  • Task instances keep the final teardown context and which sandboxes may still be up.
  • Operators can finish cleanup from a task instance.
  • Runs show the command for unfinished cleanup.

The previous findings are addressed. No new actionable issues were found in the follow-up changes.

Reviews (8) · Last reviewed commit: "fix(task): reading an instance to record..." · Reviewed by Greptile

earakely-scale and others added 2 commits November 18, 2026 12:51
… which the run names

A run whose teardown a second signal stopped, or that couldn't tear a sandbox down, listed what was still up and left
it there: no command removed it, and a cancelled run's record didn't hold what a step the signal cut short had
deployed. Such a run now keeps on its instance the context its teardown ended with (record_teardown), and agent-env
run and task run --keep name agent-env task teardown --instance <id>, which tears down what the instance records still
up as the run's own teardown does, skipping what it took down, and exits 1 while any is still up. An instance recorded
running is left alone unless --force, as for a run killed outright.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… does; AGENTS.md names agent-env task teardown

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 18, 2026 21:01
Comment thread src/agent_env/cli/task/teardown.py Resolved
Comment thread src/agent_env/cli/task/teardown.py Resolved
Comment thread src/agent_env/task/store.py Resolved
Comment thread src/agent_env/bundle/run.py Resolved
… command counts what the run lists

Greptile and an independent review found four gaps:
- The command tore down outside the run's namespace, so a recording it ended or registered used the configured
  stores. It now runs in the instance's run scope.
- A record written as a teardown ended could overwrite an attempt that took the instance over meanwhile. It's now
  fenced by the instance's attempt epoch as the teardown began.
- A sandbox terminated whose folder wasn't removed was marked torn down, so the command found nothing to do. A run now
  keeps what its teardown left up (left_up_sandbox_ids), and the command takes those too.
- Only a teardown that left something up was recorded, so a finished one still listed its sandboxes up. Every
  teardown is recorded now, and a local sandbox whose folder is gone counts as gone, as agent-env run lists it.

The command also keeps its record inside its signal handling, and BundleRun.teardown routes namespaces, as run_bundle
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/agent_env/cli/task/run.py Resolved
… teardown

attempt_epoch now returns None, logged, for an instance it can't read, and a teardown it can't fence goes unrecorded,
so a store error before task run --keep's teardown no longer leaves every kept sandbox up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale enabled auto-merge (squash) November 18, 2026 22:03
@earakely-scale
earakely-scale merged commit aeedf50 into main Nov 18, 2026
36 of 37 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-6255-a-run-whose-teardown-was-stopped branch November 18, 2026 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant