Skip to content

feat(task-steps): computer_exec, a command run inside a computer through its driver - #749

Merged
earakely-scale merged 5 commits into
mainfrom
wloo/computer-exec-step
Nov 19, 2026
Merged

earakely-scale merged 5 commits into
mainfrom
wloo/computer-exec-step

Conversation

@williamloo-scale

@williamloo-scale williamloo-scale commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

First PR of a stack that replaces osworld_initialize with native agent-env steps. This PR adds computer_exec, which covers OSWorld's execute and command setup types and, with background, launch and open.

What it does

computer_exec runs one command inside a deployed computer as its session user, through the driver's computer-exec/v1 run. That's a control route behind the env's control token, so the agent can't call it.

  • Command: a program and its arguments (["thunar", "Desktop"]), or a string run by sh -c.
  • Where: cwd is a folder relative to the session user's home (the home by default). env adds variables.
  • Timeout: timeout_s defaults to 120 s. A value above the card's max_timeout_s is refused before anything runs.
  • Result: recorded in context.metadata["computer_exec"][step_id]: argv, exit_code, timed_out, truncated, and the last 4,000 characters of stdout and stderr.
  • Failure: a non-zero exit or a timeout fails the step, with the end of its stderr in the message. check: false records it and goes on.
  • background: starts the command detached from the run's pipes and leaves it running, as a setup does when it opens an app for the agent. The driver kills a run's process group only on timeout, and already tolerates a background child, so the app outlives the run. The program is checked first: one that isn't on the PATH, or a path that isn't an executable file, exits 127, so a misspelled app fails the step instead of looking started.
  • Multi env: computer names the computer by its environment name. It can be left out when there's only one. The multi env's website browser computer (browser_envs) counts as one of its computers.

inside.run now passes the cwd that computer-exec/v1 already accepts.

  • Root (scope: "system"): runs the command as root through the computer's sandbox, the same RootChannel (docker exec -u 0) site routing and the clock config use, rather than through the driver. It runs in the session user's folder with its HOME, so cwd and ~ mean the same as in session scope, under timeout, and answers like run: exit code, output, timed_out. Its output is capped in the computer (the last 64 KiB of each stream, base64'd so any bytes decode), and it counts as timed out when timeout stopped or killed it past the deadline. It's refused with background and on a computer booted from a disk.
  • Windows: a string command or a background start, which both need sh, is refused on a Windows computer before anything runs. OSWorld setups that sudo (apt-get installs, a mkdir, an su) need it; 8 of its 369 tasks do.

Example

{"id": "open", "type": "computer_exec", "env_id": "desk", "command": ["thunar", "Desktop"], "background": true}

Testing

  • Unit: tst/unit/task_step/computer_exec_test.py, 25 tests against the fake computer in tst/unit/computer_home.py, which now honours cwd.

    • Covers: argv and sh -c, cwd, env, the token, recording, failure and check, timeout, the timeout limit, background, a background program that isn't there, multi-env lookup, preflight with a browser computer, the config checks, and the registry round-trip.
    • tst/unit/task and tst/unit/task_step: 3,715 pass, including the check that every step type is classified.
  • E2E: tst/integration/computer/computer_exec_test.py passes locally (38 s) on a real ubuntu-24.04 computer with the reference driver. It:

    • writes a file and records its output;
    • starts Thunar with background;
    • finds Thunar's window on the display in a later step;
    • records a failing command without failing the task, with check: false;
    • records a background start of a program that isn't there as exit 127.
  • test_disk_recipe.py::test_a_windows_install_boots_its_iso_first_… fails locally, on main too, so it isn't from this PR.

  • Root and chaos, on an agent-environments-admin VM (ubuntu-24.04 computer, reference driver):

    • The e2e test adds a root command (uid 0, the session home for both pwd and ~, a root-owned file in /etc) and the session user refused the same write.
    • A chaos run (not committed): a root command past its timeout leaves nothing running; an app that dies after the start window counts as started; an app writing stderr without end isn't blocked and leaves /tmp at 32 KB; an opener exiting 0 counts as started; two starts at once both start; a driver killed with SIGKILL mid-command fails that step, without resending, while the command finishes once and a background app started earlier survives.
    • Root chaos: a root command ignoring SIGTERM is killed and recorded as timed out; one printing 110 MB finishes with its tail recorded and truncated set; a chatty background app leaves no temp files.
    • The unit tests for steps, tasks and envs pass, apart from 4 that need unzip and fail the same on main on that VM.

The stack

  1. This PR: computer_exec.
  2. test(computer): load files into a multi env's computer as an environment named after it #767: load_artifact with computer, so download files reach a multi env's computer.
  3. A task importer that emits these steps for a task's init_config instead of osworld_initialize.

The rubric-judge verifier, cua_verifier, is #772, which is also stacked on this PR.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 4/5

Fix the wait on child output before merging so successful root setup commands cannot stall and fail.

Fix All in CursorFindings

  1. P1 Root commands can hang ▶
Fix with agent prompt
### Issue 1
src/agent_env/task_step/task_steps/computer/computer_exec.py:67-69
**Root commands can hang.** The new `tail -c` pipes wait until every child closes its output, not just until the requested command exits. A root setup command such as `sleep 60 & echo started` exits successfully but keeps `_CAPPED` waiting. The inner `timeout` has already exited, so it cannot stop that wait. Once `asyncio.wait_for` expires, the step fails without recording a result, even with `check: false`.

Stop reading after a bounded grace period once the command exits, while keeping the output cap.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds computer_exec to run setup commands inside a deployed computer, record their results, and optionally leave an app running.

  • Tasks can run commands inside a deployed computer and record the results.
  • Tasks can run system setup commands as root inside a computer.

Reviews (9) · Last reviewed commit: "fix(task-steps): computer_exec caps root..." · Reviewed by Greptile

@williamloo-scale
williamloo-scale requested a review from a team as a code owner November 17, 2026 21:48
…ugh its driver

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale williamloo-scale changed the title feat(task-steps): computer_exec, a command run inside a computer through its driver (FD-5492) feat(task-steps): computer_exec, a command run inside a computer through its driver Nov 19, 2026
raise ValueError(f"{self.type} {self.id!r}: timeout_s {self.timeout_s} is above the {limit} s the computer "
f"of env {self.env_id!r} lets a command run")
argv = self.argv()
answer = await inside.run(deployed, argv, timeout_s=self.timeout_s, env=self.env, cwd=self.cwd)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This always runs as the session user, and the session user can't sudo in our computers. Once the importer (step 3 of the stack) maps OSWorld setups onto computer_exec, the ones that need root will fail. 8 of OSWorld's 369 tasks run sudo in their setup:

  • mostly apt-get install (unzip, jq, sysstat, xsel), in chrome/3299584d, chrome/9656a811, chrome/6766f2b8, multi_apps/a74b607e, multi_apps/2373b66a and multi_apps/716a6079;
  • os/e0df059f, a sudo mkdir;
  • os/a462a795, a sudo su - charles.

Core already has a root channel into the computer, which it uses to write the clock's config. Could computer_exec take a user: "root" that runs through that channel? If we'd rather not give tasks root, the importer could refuse those tasks with a clear message, and we'd preinstall the packages they need in the base. Either is fine for this PR, but it'd be good to decide before the importer lands.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in c716164: scope: "system" runs the command as root through the computer's sandbox, the same RootChannel (docker exec -u 0) that site routing and the clock config use. It runs in the session user's folder with its HOME, so cwd and ~ mean the same as in session scope, under timeout. It's refused with background and on a computer booted from a disk. On a real ubuntu-24.04 computer, the e2e test checks a root command gets uid 0 and the session home, leaves a root-owned file, and that the session user is refused the same write; a chaos run checks a root command past its timeout leaves nothing running.

"""What the driver runs: the command, through ``sh -c`` when it's a string, started detached with
``background``."""
argv = ["sh", "-c", self.command] if isinstance(self.command, str) else list(self.command)
return ["sh", "-c", _BACKGROUND, "computer_exec", *argv] if self.background else argv

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

background can report success for an app that never started:

  • String commands skip the program check. The launcher checks $1, which for a string command is sh. So {"command": "thunarr Desktop", "background": true} passes and nothing opens. 216 of OSWorld's exec-like setup commands are strings, so the importer will produce these.
  • An app that exits right after starting passes too, for example one given a bad flag or with no display to open on. Its output goes to /dev/null, and the step returns as soon as it has started. In the e2e test, only the later window step would catch it.

Suggestion: after starting the app, wait a second or two. If it has already exited non-zero, fail with the start of its stderr, which would need writing to a temp file instead of /dev/null for that window. Treat an exit of 0 as fine, because xdg-open and single-instance apps hand off to another process and exit. That also covers string commands, without parsing the string.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, both cases were real. Fixed in 9836edb, roughly as you suggested:

  • Watched start: after starting the app, the launcher watches it for START_WINDOW_S (2 s). If it exits non-zero in that window, the step fails with its exit status and the end of its stderr. Stderr goes through tail -c 4096, not /dev/null, so a long-running app's output doesn't pile up on disk.
  • String commands: these are covered without parsing the string. sh -c "thunarr Desktop" exits 127 right away, so it fails with sh: thunarr: not found.
  • Exit 0: still counts as started, for xdg-open and single-instance apps. The e2e hit exactly this: a second thunar hands off to the running one and exits 0.
  • Messages: failures now show the task's command rather than the launcher script. The full launcher is still in the record's argv.
  • timeout_s: a background step's timeout_s must be at least 5 s, so the window fits.

Tests: unit tests cover the string-command case, an app that exits non-zero with stderr, and one that hands off with exit 0. The e2e on a real ubuntu-24.04 computer now also runs "thunarr Desktop" (127) and ls --no-such-flag in the background, and both fail as expected.

…ne that exits non-zero right after

A background command is now watched for START_WINDOW_S (2 s) after it starts:
if it exits non-zero in that window, the step fails with its exit status and
the end of its stderr. That catches a string command naming a program that
isn't there (its program is sh, which passed the old check) and an app given a
bad flag or with nowhere to open. An exit of 0 counts as started, since
xdg-open and single-instance apps hand off and exit. Failures name the task's
command, not the launcher around it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
earakely-scale and others added 2 commits November 19, 2026 10:44
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scope: "system" runs the command as root through the computer's sandbox (RootChannel, docker exec -u 0), as site routing
and the clock follower's config already do, in the session user's folder and HOME, under timeout; it isn't offered with
background, nor on a computer booted from a disk, which core can't reach as root. OSWorld's setups that sudo (apt-get
installs, a mkdir, an su) need it. RootChannel gains exec, which answers the exit code and output instead of raising.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dline, refuses sh on Windows, leaves no temp files

- A system command's streams are capped in the computer (tail -c, SYSTEM_OUTPUT_BYTES) and base64'd, so a command that
  prints without end can't fill the task runner's memory and any bytes decode; the record says when it was cut.
- It timed out when timeout stopped or killed it past the deadline (124, or 137 for one that ignored the stop signal),
  not when it exits with that status of its own, earlier.
- On a Windows computer (its card's computer/v1 platform), a string or background command, which needs sh, is refused
  before anything runs.
- A background start still running when its window ends removes the launcher's folder; the app keeps the stderr file it
  writes open. The launcher's polling interval and stderr wait are named, and its loops derive from them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 61e4e5b into main Nov 19, 2026
36 checks passed
@earakely-scale
earakely-scale deleted the wloo/computer-exec-step branch November 19, 2026 19:29
earakely-scale added a commit that referenced this pull request Nov 19, 2026
Resolve the conflicts with the squash-merged #749: main's computer_exec and AGENTS.md, with this
PR's computer_of split, computer_problem and its AGENTS.md text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment on lines +67 to +69
{ { "$@"; echo $? >"$d/rc"; } 2>&1 >&3 3>&- | tail -c "$cap" >"$d/err"; } 3>&1 | tail -c "$cap" | base64 -w0
base64 -w0 "$d/err" >&2
rc=$(cat "$d/rc" 2>/dev/null)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Root commands can hang. The new tail -c pipes wait until every child closes its output, not just until the requested command exits. A root setup command such as sleep 60 & echo started exits successfully but keeps _CAPPED waiting. The inner timeout has already exited, so it cannot stop that wait. Once asyncio.wait_for expires, the step fails without recording a result, even with check: false.

Stop reading after a bounded grace period once the command exits, while keeping the output cap.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/computer/computer_exec.py
Line: 67-69

Comment:
**Root commands can hang.** The new `tail -c` pipes wait until every child closes its output, not just until the requested command exits. A root setup command such as `sleep 60 & echo started` exits successfully but keeps `_CAPPED` waiting. The inner `timeout` has already exited, so it cannot stop that wait. Once `asyncio.wait_for` expires, the step fails without recording a result, even with `check: false`.

Stop reading after a bounded grace period once the command exits, while keeping the output cap.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants