Skip to content

feat(task-steps): OSWorld initialize and evaluate on computer envs, graded in the worker - #606

Closed
williamloo-scale wants to merge 2 commits into
mainfrom
wloo/osworld-native-computer-steps
Closed

williamloo-scale wants to merge 2 commits into
mainfrom
wloo/osworld-native-computer-steps

Conversation

@williamloo-scale

@williamloo-scale williamloo-scale commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Note

Reworked: OSWorld setup and grading now run in the worker, with no controller sidecar. Every check and every judge runs in the worker, the model key never enters a sandbox, and nothing the agent can reach holds a grader or a key. osworld_deploy_controller, the osworld-controller image and its GHCR publish, the controller token and the EntityRef(default=...) resolver change are all gone. The branch is now one commit on top of main.

Summary

This PR adds two task steps to core, osworld_initialize and osworld_evaluate, plus the osworld bundle with its osworld-server computer app (all in src/agent_env/task_step/task_steps/computer/osworld/). Together they let a classic OSWorld task (init_config plus evaluator) run on a framework computer env. No plugin is needed.

Design

Grading runs in the worker.

  • The steps run upstream OSWorld's own SetupController, getters and metrics, from xlang-ai/OSWorld at the pinned commit 5123bfdb (Apache-2.0), unmodified, in the worker process (runtime.py).
  • The worker holds the env's control token and any judge's model key. The agent holds neither.
  • Upstream's code blocks, so each call runs off the event loop (asyncio.to_thread). Each call builds its own controllers, forwarders and cache folder, so calls for different computers run side by side.

Upstream reaches the computer through loopback forwarders (forwarder.py).

  • Upstream builds every URL as http://{vm_ip}:{port}, so each setup or check runs against ephemeral 127.0.0.1 ports of its own, open only for that call.
  • The forwarder sends each call to the computer's /osworld endpoint, path, query and body unchanged, with the env's control token.
  • The token is sent only when the guest URL is under the env's own address (DeployedEnv.control_headers_for).
  • The Chrome DevTools (9222) and VLC (8080) ports, which the app doesn't expose, get refusers. A setup or check that calls them fails and says which port it needed.

Upstream's source and dependencies.

  • Source: fetched on first use from GitHub's archive of the commit, checked by sha256, and cached under ~/.cache/agentenv/osworld/<commit>. Only upstream's desktop_env controllers, evaluators and the modules they import are kept, with the LICENSE (upstream.py).
    • An image can bake the source in with python -m agent_env.task_step.task_steps.computer.osworld.upstream <folder> and set AGENT_ENV_OSWORLD_SOURCE=<folder>.
  • Dependencies: what upstream imports is listed in the bundle's requirements.txt, which a worker that runs these steps installs (pip install -r ...).
    • It isn't an agentenv-framework extra. The repo's guard tests require an extra's packages to be imported by library code, and only the fetched upstream code imports these.
    • dev carries the same lines, so CI runs the e2e test, and a unit test keeps the two lists in step.
    • The lines are floors, not upstream's tight pins, so main's numpy, pandas and Pillow are unchanged.
    • easyocr is left out (it pulls in torch for one metric) and replaced by a stub, so compare_image_text fails, saying why.

The computer runs only the OSWorld guest server, as the osworld-server app.

  • The app's own server forwards /osworld/<route> to 127.0.0.1:5000, behind the env's control token, which it removes before forwarding.
  • The app's card advertises urn:agentenv:osworld-guest/v1 at that prefix. The steps find the guest server by that URI in the computer's card, from the stored card first and then the live card for up to 2 minutes.
  • A computer without the app fails, saying so.

LLM judges.

  • agent-env ships no judge. A deployment registers one in the worker with evaluate.register_judge(func, judge), either for agent_judge (all variants) or for a single variant.
  • The judge gets the func's getters, rubric, parameters, instruction and the judge model: the run's user_overrides.judge_model, else the func's own model option, else the step's judge_model. The models used are recorded.
  • A judge func with no judge registered scores 0, saying so. No judge implementation ships in this repo.

Not supported (each fails with a clear message):

  • OSWorld-V2 tasks (osworld_v2_task_path). Their setup and evaluate are the task's own code.
  • Setups or checks that need Chrome DevTools or VLC.
  • Google Drive setups and upstream's proxy.

Step reference

osworld_initialize

  • Fields: env_id, init_config, osworld_v2_task_path, and the legacy osworld_v2_task_url. A V2 path in either field fails the step.
  • Runs the whole init_config through upstream's SetupController, at the guest's screen size. Upstream's /terminal retries are reduced from 20 to 3, so a dead guest fails within the step's window.
  • A failure raises with the error type plus at most 300 characters.

osworld_evaluate

  • Fields: env_id, evaluator, verifier_id (default: a uuid hex), osworld_v2_task_path / osworld_v2_task_url (either fails the step), and judge_model.
  • Grading: each func is upstream's metric over its result getter, against its expected getter when it has one, with its options. Results are combined by conj as DesktopEnv.evaluate does: for and, 0 if any func scores 0, else the mean; for or, the max.
    • A result file the guest doesn't have scores 0.
    • A func or getter that upstream doesn't have, or any other failure, scores that func 0 with the reason and marks debug.error. The other funcs are still graded.
    • infeasible is graded on the agent's answer (the [INFEASIBLE] tag), as cua_evaluate does.
    • A per-func list whose length doesn't match the number of funcs is an error, not silently cut short.
  • Re-grades: stage_files (from step_overrides) is downloaded onto the computer before anything reads it. The step refuses to grade if staging fails.
  • postconfig: runs once before grading. A failure is logged (bounded) and grading continues.
  • Skips: a turn whose error_type is set is recorded as a skip with score 0.
  • Partial credit: an agent that ran out of turns or time is graded anyway and marked partial_credit. The check matches each shipped agent's stop message.

Result shape (same as cua_evaluate)

context.metadata["verifications"][verifier_id] = {"results": [{"func", "score", "result", "results": [{"id", "score", "result", "message"}]}], "score", "debug"}. Each row's own score and result (passed at 1.0) are what aggregate_verifiers reads.

Testing

  • Unit: tst/unit/task_step/osworld/.
    • The steps run against a stand-in runtime, covering conj, judges, skips, partial credit, staging, V2 refusal and token scoping.
    • The forwarder runs on real loopback sockets, covering forwarding, 502s, refusers and port close.
    • The runtime runs with upstream stood in, covering getters and metrics, missing files, unknown checks, setup and unexposed ports.
    • The upstream fetch covers sha256, member filtering, the cache path and the baked-folder override.
    • Also the app and its card, plus the bundle and alias tests. 5,088 tests in tst/unit/{task_step,bundle,task} pass.
  • E2E: tst/integration/computer/osworld_test.py passes locally on the final commit 17e274a, the same tree as before the history rewrite (1m59s, from an empty upstream cache, so the GitHub fetch and sha256 check ran too). It deploys a real ubuntu-24.04 computer with the osworld-server app. In the worker it runs a classic init_config execute through upstream's SetupController, then grades with upstream's own check_include_exclude over vm_command_line (1.0 and 0.0) and file_contains over vm_file (1.0). It also checks that no controller sandbox was created.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The follow-up changes appear safe to merge; the previous finding is fixed.

Summary

Adds OSWorld task setup and grading in the worker, with a guest-server app on the computer.

  • OSWorld setup runs in the worker and reaches the computer through its guest server.
  • OSWorld checks produce scores in the worker using the task's evaluator.
  • The OSWorld bundle and its task steps are available in framework installs.

The previous numbered finding is fixed. No new issues found in the follow-up changes.

Reviews (35) · Last reviewed commit: "fix(task-steps): osworld_evaluate record..." · Reviewed by Greptile

@socket-security

socket-security Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Priority Alert  (click "▶" to expand/collapse) Action
High priority
High CVE: Soup Sieve: Exponential ReDoS in selector compilation via \9-run in CSS_ESCAPES/IDENTIFIER in pypi soupsieve

CVE: GHSA-6655-6qjp-jx7x Soup Sieve: Exponential ReDoS in selector compilation via \9-run in CSS_ESCAPES/IDENTIFIER (HIGH)

Affected versions: < 3.0.2

Patched version: 3.0.2

From: uv.lock → pypi/beautifulsoup4@4.15.0 → pypi/soupsieve@2.10

ℹ Read more on: This package | This alert | What is a CVE?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/soupsieve@2.10. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
High priority
High CVE: Soup Sieve: Polynomial-time denial of service in the descendant combinator in pypi soupsieve

CVE: GHSA-mmhr-3c5m-f64f Soup Sieve: Polynomial-time denial of service in the descendant combinator (HIGH)

Affected versions: < 4.0.0

Patched version: 4.0.0

From: uv.lock → pypi/beautifulsoup4@4.15.0 → pypi/soupsieve@2.10

ℹ Read more on: This package | This alert | What is a CVE?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/soupsieve@2.10. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
Low priority
Potential vulnerability: pypi numpy with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server/requirements.txt → pypi/numpy@2.5.5

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/numpy@2.5.5. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn
See 57 more rows in the dashboard

View full report

@williamloo-scale williamloo-scale changed the title feat(task-steps): native OSWorld initialize and evaluate on computer envs feat(task-steps): native OSWorld initialize and evaluate on computer envs (FD-5492) Nov 11, 2026
@williamloo-scale
williamloo-scale marked this pull request as ready for review November 12, 2026 19:57
@williamloo-scale
williamloo-scale requested a review from a team as a code owner November 12, 2026 19:57
@williamloo-scale
williamloo-scale force-pushed the wloo/osworld-native-computer-steps branch from b1e420c to c3a2f60 Compare November 12, 2026 23:47

@earakely-scale earakely-scale left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 7ee469d. The two commits since c3a2f60 only move files and change import paths. Comment 1 is the question I'd settle first; 2 to 4 are correctness; 5 collects small things.

@williamloo-scale
williamloo-scale force-pushed the wloo/osworld-native-computer-steps branch from 3505480 to 3fb6169 Compare November 17, 2026 00:42
@williamloo-scale
williamloo-scale force-pushed the wloo/osworld-native-computer-steps branch 2 times, most recently from 2b14616 to 8a09fac Compare November 17, 2026 10:07
@williamloo-scale williamloo-scale changed the title feat(task-steps): native OSWorld initialize and evaluate on computer envs (FD-5492) feat(task-steps): OSWorld initialize and evaluate on computer envs, graded in the worker (FD-5492) Nov 17, 2026
@williamloo-scale
williamloo-scale force-pushed the wloo/osworld-native-computer-steps branch from 8a09fac to 1d26cc5 Compare November 17, 2026 10:14
@williamloo-scale
williamloo-scale force-pushed the wloo/osworld-native-computer-steps branch 2 times, most recently from 05d371c to edee857 Compare November 17, 2026 10:30
@williamloo-scale
williamloo-scale force-pushed the wloo/osworld-native-computer-steps branch from edee857 to e762abc Compare November 17, 2026 10:36
Comment on lines +166 to +168
guest_url, headers = await guest.guest_access(deployed)
await self._stage_result_files(guest_url, headers, context)
prompt_response = context.prompt_responses[-1] if context.prompt_responses else None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Failed turns lose their grade

execute contacts the guest and stages files before checking whether the agent's turn should be skipped. If an infrastructure error leaves the computer unable to serve its card, guest_access raises instead of recording the promised zero-score skip. A staging failure has the same effect, leaving later steps without a verification result.

Check the turn's error and budget status before contacting or changing the computer.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/computer/osworld/evaluate.py
Line: 166-168

Comment:
**Failed turns lose their grade**

`execute` contacts the guest and stages files before checking whether the agent's turn should be skipped. If an infrastructure error leaves the computer unable to serve its card, `guest_access` raises instead of recording the promised zero-score skip. A staging failure has the same effect, leaving later steps without a verification result.

Check the turn's error and budget status before contacting or changing the computer.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4bf2f43: execute now checks the turn's error and budget status before guest_access and any staging. A failed turn on an unreachable computer gets its zero-score skip, and a test covers it.

…raded in the worker

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale
williamloo-scale force-pushed the wloo/osworld-native-computer-steps branch from e762abc to 17e274a Compare November 19, 2026 00:11
@williamloo-scale williamloo-scale changed the title feat(task-steps): OSWorld initialize and evaluate on computer envs, graded in the worker (FD-5492) feat(task-steps): OSWorld initialize and evaluate on computer envs, graded in the worker Nov 19, 2026
… contacting the computer

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale

Copy link
Copy Markdown
Contributor Author

Closing: the native steps in #749, #767 and #772 replace this approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants