Repository navigation
feat(task-steps): OSWorld initialize and evaluate on computer envs, graded in the worker - #606
williamloo-scale wants to merge 2 commits into
Conversation
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
b1e420c to
c3a2f60
Compare
3505480 to
3fb6169
Compare
2b14616 to
8a09fac
Compare
8a09fac to
1d26cc5
Compare
05d371c to
edee857
Compare
edee857 to
e762abc
Compare
| guest_url, headers = await guest.guest_access(deployed) | ||
| await self._stage_result_files(guest_url, headers, context) | ||
| prompt_response = context.prompt_responses[-1] if context.prompt_responses else None |
There was a problem hiding this comment.
execute contacts the guest and stages files before checking whether the agent's turn should be skipped. If an infrastructure error leaves the computer unable to serve its card, guest_access raises instead of recording the promised zero-score skip. A staging failure has the same effect, leaving later steps without a verification result.
Check the turn's error and budget status before contacting or changing the computer.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/computer/osworld/evaluate.py
Line: 166-168
Comment:
**Failed turns lose their grade**
`execute` contacts the guest and stages files before checking whether the agent's turn should be skipped. If an infrastructure error leaves the computer unable to serve its card, `guest_access` raises instead of recording the promised zero-score skip. A staging failure has the same effect, leaving later steps without a verification result.
Check the turn's error and budget status before contacting or changing the computer.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in 4bf2f43: execute now checks the turn's error and budget status before guest_access and any staging. A failed turn on an unreachable computer gets its zero-score skip, and a test covers it.
…raded in the worker Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
e762abc to
17e274a
Compare
… contacting the computer Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Note
Reworked: OSWorld setup and grading now run in the worker, with no controller sidecar. Every check and every judge runs in the worker, the model key never enters a sandbox, and nothing the agent can reach holds a grader or a key.
osworld_deploy_controller, theosworld-controllerimage and its GHCR publish, the controller token and theEntityRef(default=...)resolver change are all gone. The branch is now one commit on top ofmain.Summary
This PR adds two task steps to core,
osworld_initializeandosworld_evaluate, plus theosworldbundle with itsosworld-servercomputer app (all insrc/agent_env/task_step/task_steps/computer/osworld/). Together they let a classic OSWorld task (init_configplusevaluator) run on a frameworkcomputerenv. No plugin is needed.Design
Grading runs in the worker.
SetupController, getters and metrics, from xlang-ai/OSWorld at the pinned commit5123bfdb(Apache-2.0), unmodified, in the worker process (runtime.py).asyncio.to_thread). Each call builds its own controllers, forwarders and cache folder, so calls for different computers run side by side.Upstream reaches the computer through loopback forwarders (
forwarder.py).http://{vm_ip}:{port}, so each setup or check runs against ephemeral127.0.0.1ports of its own, open only for that call./osworldendpoint, path, query and body unchanged, with the env's control token.DeployedEnv.control_headers_for).Upstream's source and dependencies.
~/.cache/agentenv/osworld/<commit>. Only upstream'sdesktop_envcontrollers, evaluators and the modules they import are kept, with the LICENSE (upstream.py).python -m agent_env.task_step.task_steps.computer.osworld.upstream <folder>and setAGENT_ENV_OSWORLD_SOURCE=<folder>.requirements.txt, which a worker that runs these steps installs (pip install -r ...).agentenv-frameworkextra. The repo's guard tests require an extra's packages to be imported by library code, and only the fetched upstream code imports these.devcarries the same lines, so CI runs the e2e test, and a unit test keeps the two lists in step.compare_image_textfails, saying why.The computer runs only the OSWorld guest server, as the
osworld-serverapp./osworld/<route>to127.0.0.1:5000, behind the env's control token, which it removes before forwarding.urn:agentenv:osworld-guest/v1at that prefix. The steps find the guest server by that URI in the computer's card, from the stored card first and then the live card for up to 2 minutes.LLM judges.
evaluate.register_judge(func, judge), either foragent_judge(all variants) or for a single variant.rubric,parameters,instructionand the judge model: the run'suser_overrides.judge_model, else the func's ownmodeloption, else the step'sjudge_model. The models used are recorded.Not supported (each fails with a clear message):
osworld_v2_task_path). Their setup and evaluate are the task's own code.Step reference
osworld_initializeenv_id,init_config,osworld_v2_task_path, and the legacyosworld_v2_task_url. A V2 path in either field fails the step.init_configthrough upstream'sSetupController, at the guest's screen size. Upstream's/terminalretries are reduced from 20 to 3, so a dead guest fails within the step's window.osworld_evaluateenv_id,evaluator,verifier_id(default: a uuid hex),osworld_v2_task_path/osworld_v2_task_url(either fails the step), andjudge_model.resultgetter, against itsexpectedgetter when it has one, with itsoptions. Results are combined byconjasDesktopEnv.evaluatedoes: forand, 0 if any func scores 0, else the mean; foror, the max.debug.error. The other funcs are still graded.infeasibleis graded on the agent's answer (the[INFEASIBLE]tag), ascua_evaluatedoes.stage_files(fromstep_overrides) is downloaded onto the computer before anything reads it. The step refuses to grade if staging fails.postconfig: runs once before grading. A failure is logged (bounded) and grading continues.error_typeis set is recorded as a skip with score 0.partial_credit. The check matches each shipped agent's stop message.Result shape (same as
cua_evaluate)context.metadata["verifications"][verifier_id] = {"results": [{"func", "score", "result", "results": [{"id", "score", "result", "message"}]}], "score", "debug"}. Each row's ownscoreandresult(passed at 1.0) are whataggregate_verifiersreads.Testing
tst/unit/task_step/osworld/.tst/unit/{task_step,bundle,task}pass.tst/integration/computer/osworld_test.pypasses locally on the final commit 17e274a, the same tree as before the history rewrite (1m59s, from an empty upstream cache, so the GitHub fetch and sha256 check ran too). It deploys a realubuntu-24.04computer with theosworld-serverapp. In the worker it runs a classicinit_configexecutethrough upstream'sSetupController, then grades with upstream's owncheck_include_excludeovervm_command_line(1.0 and 0.0) andfile_containsovervm_file(1.0). It also checks that no controller sandbox was created.🤖 Generated with Claude Code


Confidence Score: 5/5
The follow-up changes appear safe to merge; the previous finding is fixed.Summary
Adds OSWorld task setup and grading in the worker, with a guest-server app on the computer.
The previous numbered finding is fixed. No new issues found in the follow-up changes.
Reviews (35) · Last reviewed commit: "fix(task-steps): osworld_evaluate record..." · Reviewed by Greptile