Repository navigation
feat(task-steps): computer_exec, a command run inside a computer through its driver - #749
Conversation
…ugh its driver Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
c1759be to
3d3ffd8
Compare
| raise ValueError(f"{self.type} {self.id!r}: timeout_s {self.timeout_s} is above the {limit} s the computer " | ||
| f"of env {self.env_id!r} lets a command run") | ||
| argv = self.argv() | ||
| answer = await inside.run(deployed, argv, timeout_s=self.timeout_s, env=self.env, cwd=self.cwd) |
There was a problem hiding this comment.
This always runs as the session user, and the session user can't sudo in our computers. Once the importer (step 3 of the stack) maps OSWorld setups onto computer_exec, the ones that need root will fail. 8 of OSWorld's 369 tasks run sudo in their setup:
- mostly
apt-get install(unzip, jq, sysstat, xsel), inchrome/3299584d,chrome/9656a811,chrome/6766f2b8,multi_apps/a74b607e,multi_apps/2373b66aandmulti_apps/716a6079; os/e0df059f, asudo mkdir;os/a462a795, asudo su - charles.
Core already has a root channel into the computer, which it uses to write the clock's config. Could computer_exec take a user: "root" that runs through that channel? If we'd rather not give tasks root, the importer could refuse those tasks with a clear message, and we'd preinstall the packages they need in the base. Either is fine for this PR, but it'd be good to decide before the importer lands.
There was a problem hiding this comment.
Added in c716164: scope: "system" runs the command as root through the computer's sandbox, the same RootChannel (docker exec -u 0) that site routing and the clock config use. It runs in the session user's folder with its HOME, so cwd and ~ mean the same as in session scope, under timeout. It's refused with background and on a computer booted from a disk. On a real ubuntu-24.04 computer, the e2e test checks a root command gets uid 0 and the session home, leaves a root-owned file, and that the session user is refused the same write; a chaos run checks a root command past its timeout leaves nothing running.
| """What the driver runs: the command, through ``sh -c`` when it's a string, started detached with | ||
| ``background``.""" | ||
| argv = ["sh", "-c", self.command] if isinstance(self.command, str) else list(self.command) | ||
| return ["sh", "-c", _BACKGROUND, "computer_exec", *argv] if self.background else argv |
There was a problem hiding this comment.
background can report success for an app that never started:
- String commands skip the program check. The launcher checks
$1, which for a string command issh. So{"command": "thunarr Desktop", "background": true}passes and nothing opens. 216 of OSWorld's exec-like setup commands are strings, so the importer will produce these. - An app that exits right after starting passes too, for example one given a bad flag or with no display to open on. Its output goes to
/dev/null, and the step returns as soon as it has started. In the e2e test, only the laterwindowstep would catch it.
Suggestion: after starting the app, wait a second or two. If it has already exited non-zero, fail with the start of its stderr, which would need writing to a temp file instead of /dev/null for that window. Treat an exit of 0 as fine, because xdg-open and single-instance apps hand off to another process and exit. That also covers string commands, without parsing the string.
There was a problem hiding this comment.
Thanks, both cases were real. Fixed in 9836edb, roughly as you suggested:
- Watched start: after starting the app, the launcher watches it for
START_WINDOW_S(2 s). If it exits non-zero in that window, the step fails with its exit status and the end of its stderr. Stderr goes throughtail -c 4096, not/dev/null, so a long-running app's output doesn't pile up on disk. - String commands: these are covered without parsing the string.
sh -c "thunarr Desktop"exits 127 right away, so it fails withsh: thunarr: not found. - Exit 0: still counts as started, for
xdg-openand single-instance apps. The e2e hit exactly this: a secondthunarhands off to the running one and exits 0. - Messages: failures now show the task's command rather than the launcher script. The full launcher is still in the record's
argv. timeout_s: a background step'stimeout_smust be at least 5 s, so the window fits.
Tests: unit tests cover the string-command case, an app that exits non-zero with stderr, and one that hands off with exit 0. The e2e on a real ubuntu-24.04 computer now also runs "thunarr Desktop" (127) and ls --no-such-flag in the background, and both fail as expected.
…ne that exits non-zero right after A background command is now watched for START_WINDOW_S (2 s) after it starts: if it exits non-zero in that window, the step fails with its exit status and the end of its stderr. That catches a string command naming a program that isn't there (its program is sh, which passed the old check) and an app given a bad flag or with nowhere to open. An exit of 0 counts as started, since xdg-open and single-instance apps hand off and exit. Failures name the task's command, not the launcher around it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scope: "system" runs the command as root through the computer's sandbox (RootChannel, docker exec -u 0), as site routing and the clock follower's config already do, in the session user's folder and HOME, under timeout; it isn't offered with background, nor on a computer booted from a disk, which core can't reach as root. OSWorld's setups that sudo (apt-get installs, a mkdir, an su) need it. RootChannel gains exec, which answers the exit code and output instead of raising. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dline, refuses sh on Windows, leaves no temp files - A system command's streams are capped in the computer (tail -c, SYSTEM_OUTPUT_BYTES) and base64'd, so a command that prints without end can't fill the task runner's memory and any bytes decode; the record says when it was cut. - It timed out when timeout stopped or killed it past the deadline (124, or 137 for one that ignored the stop signal), not when it exits with that status of its own, earlier. - On a Windows computer (its card's computer/v1 platform), a string or background command, which needs sh, is refused before anything runs. - A background start still running when its window ends removes the launcher's folder; the app keeps the stderr file it writes open. The launcher's polling interval and stderr wait are named, and its loops derive from them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the conflicts with the squash-merged #749: main's computer_exec and AGENTS.md, with this PR's computer_of split, computer_problem and its AGENTS.md text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| { { "$@"; echo $? >"$d/rc"; } 2>&1 >&3 3>&- | tail -c "$cap" >"$d/err"; } 3>&1 | tail -c "$cap" | base64 -w0 | ||
| base64 -w0 "$d/err" >&2 | ||
| rc=$(cat "$d/rc" 2>/dev/null) |
There was a problem hiding this comment.
Root commands can hang. The new
tail -c pipes wait until every child closes its output, not just until the requested command exits. A root setup command such as sleep 60 & echo started exits successfully but keeps _CAPPED waiting. The inner timeout has already exited, so it cannot stop that wait. Once asyncio.wait_for expires, the step fails without recording a result, even with check: false.
Stop reading after a bounded grace period once the command exits, while keeping the output cap.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/computer/computer_exec.py
Line: 67-69
Comment:
**Root commands can hang.** The new `tail -c` pipes wait until every child closes its output, not just until the requested command exits. A root setup command such as `sleep 60 & echo started` exits successfully but keeps `_CAPPED` waiting. The inner `timeout` has already exited, so it cannot stop that wait. Once `asyncio.wait_for` expires, the step fails without recording a result, even with `check: false`.
Stop reading after a bounded grace period once the command exits, while keeping the output cap.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
First PR of a stack that replaces
osworld_initializewith native agent-env steps. This PR addscomputer_exec, which covers OSWorld'sexecuteandcommandsetup types and, withbackground,launchandopen.What it does
computer_execruns one command inside a deployed computer as its session user, through the driver's computer-exec/v1run. That's a control route behind the env's control token, so the agent can't call it.["thunar", "Desktop"]), or a string run bysh -c.cwdis a folder relative to the session user's home (the home by default).envadds variables.timeout_sdefaults to 120 s. A value above the card'smax_timeout_sis refused before anything runs.context.metadata["computer_exec"][step_id]:argv,exit_code,timed_out,truncated, and the last 4,000 characters of stdout and stderr.check: falserecords it and goes on.background: starts the command detached from the run's pipes and leaves it running, as a setup does when it opens an app for the agent. The driver kills a run's process group only on timeout, and already tolerates a background child, so the app outlives the run. The program is checked first: one that isn't on thePATH, or a path that isn't an executable file, exits 127, so a misspelled app fails the step instead of looking started.computernames the computer by its environment name. It can be left out when there's only one. The multi env's website browser computer (browser_envs) counts as one of its computers.inside.runnow passes thecwdthat computer-exec/v1 already accepts.scope: "system"): runs the command as root through the computer's sandbox, the sameRootChannel(docker exec -u 0) site routing and the clock config use, rather than through the driver. It runs in the session user's folder with itsHOME, socwdand~mean the same as in session scope, undertimeout, and answers likerun: exit code, output,timed_out. Its output is capped in the computer (the last 64 KiB of each stream, base64'd so any bytes decode), and it counts as timed out whentimeoutstopped or killed it past the deadline. It's refused withbackgroundand on a computer booted from a disk.backgroundstart, which both needsh, is refused on a Windows computer before anything runs. OSWorld setups thatsudo(apt-get installs, amkdir, ansu) need it; 8 of its 369 tasks do.Example
{"id": "open", "type": "computer_exec", "env_id": "desk", "command": ["thunar", "Desktop"], "background": true}Testing
Unit:
tst/unit/task_step/computer_exec_test.py, 25 tests against the fake computer intst/unit/computer_home.py, which now honourscwd.sh -c,cwd,env, the token, recording, failure andcheck, timeout, the timeout limit, background, a background program that isn't there, multi-env lookup, preflight with a browser computer, the config checks, and the registry round-trip.tst/unit/taskandtst/unit/task_step: 3,715 pass, including the check that every step type is classified.E2E:
tst/integration/computer/computer_exec_test.pypasses locally (38 s) on a realubuntu-24.04computer with the reference driver. It:background;check: false;test_disk_recipe.py::test_a_windows_install_boots_its_iso_first_…fails locally, onmaintoo, so it isn't from this PR.Root and chaos, on an agent-environments-admin VM (ubuntu-24.04 computer, reference driver):
pwdand~, a root-owned file in/etc) and the session user refused the same write./tmpat 32 KB; an opener exiting 0 counts as started; two starts at once both start; a driver killed with SIGKILL mid-command fails that step, without resending, while the command finishes once and a background app started earlier survives.truncatedset; a chatty background app leaves no temp files.unzipand fail the same on main on that VM.The stack
computer_exec.load_artifactwithcomputer, sodownloadfiles reach a multi env's computer.init_configinstead ofosworld_initialize.The rubric-judge verifier,
cua_verifier, is #772, which is also stacked on this PR.🤖 Generated with Claude Code
Fix the wait on child output before merging so successful root setup commands cannot stall and fail.
Fix with agent prompt
Summary
Adds
computer_execto run setup commands inside a deployed computer, record their results, and optionally leave an app running.Reviews (9) · Last reviewed commit: "fix(task-steps): computer_exec caps root..." · Reviewed by Greptile