Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/scripts/shipped_bundles.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
_PINNED_KINDS = ("envs", "artifacts")
# The shipped env folders whose images a release publishes, as agent_env.env.published_images.EXAMPLE_IMAGES lists them.
PUBLISHED_ENVS = ("src/agent_env/examples/mail-mcp/envs/mail",)
# The shipped folders whose requirements don't install the protocol, so a release has nothing to pin in them: the
# osworld-server app's, which runs upstream OSWorld's guest server.
PROTOCOL_FREE = ("src/agent_env/task_step/task_steps/computer/osworld/envs/osworld-server",)
# A requirement on the protocol, with its extras, and not on a package whose name starts with the protocol's.
_PROTOCOL_REQUIREMENT = re.compile(rf"^{PROTOCOL}(\[[^\]]*\])?(?![\w.-])")
# The quoted requirement in pyproject.toml, with its extras: the directory name in the sdist include list and the uv
Expand All @@ -42,9 +45,9 @@ def bundle_roots(repo: Path = REPO) -> dict[str, Path]:


def requirements_files(repo: Path = REPO) -> list[Path]:
"""Every ``<bundle>/<envs|artifacts>/<name>/requirements*.txt`` a shipped bundle holds but a ``PUBLISHED_ENVS``
folder's: those a release pins."""
published = {repo / folder for folder in PUBLISHED_ENVS}
"""Every ``<bundle>/<envs|artifacts>/<name>/requirements*.txt`` a shipped bundle holds but a ``PUBLISHED_ENVS`` or
``PROTOCOL_FREE`` folder's: those a release pins."""
published = {repo / folder for folder in (*PUBLISHED_ENVS, *PROTOCOL_FREE)}
return sorted(path for root in bundle_roots(repo).values() for kind in _PINNED_KINDS
for path in root.glob(f"{kind}/*/requirements*.txt") if path.parent not in published)

Expand Down
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ and deserialized through a registry.
| `agents/` | The `agents` bundle: the agents agent-env ships, one folder each under `agents/agents/`, with a smoke task each under `agents/tasks/`. Its Claude Code, `@local/agentenv-framework/agents/claude-code`, is the default agent. An agent's Python packages are its folder's `requirements.txt`, installed in its image, never in agent-env's own environment. A run that pulls one of its agents reads the whole bundle, so `tst/unit/bundle/shipped_bundles_test.py` parses and checks every shipped bundle. |
| `computers/` | The `computers` bundle: the base computers agent-env ships as `computer` artifacts, `ubuntu-24.04` (XFCE on Xvfb, settings as `AGENTENV_SETTING_<NAME>`; it starts as root, its container form's `start_as`, whose supervisord, `root.conf`, runs the clock follower as `agentenv-clock`, the egress proxy as `agentenv-egress` (`egress/`, a Go program its Dockerfile builds and tests, on real time, through which every session program goes out, so a live site loads at any universe date: it passes a site's TLS through when the universe is inside the site's certificate, else re-signs it with the computer's own CA, minted at boot and valid from 1950 to 9999, handshaking with the site as the client did, and refuses any destination that isn't global) and the session's supervisord as the session user, which runs every session program; `root/` installs them at the build, `install.sh`, and starts them at boot, `init`; `ubuntu-24.04-browser`, the base of agent-env's website browsers, has copies of `clock/`, `egress/`, `root/` and `root.conf` that a unit test keeps identical), and `omarchy-4.5`, a disk alone built on the composing machine from Omarchy's own VM image (its `[forms.disk]` `source`) by `disk/omarchy/`'s recipe, its services systemd user units of the graphical session that start through `with-seed`, booted by the launcher as a `linux-desktop` guest (`env/envs/computer/utils/omarchy_disk.py`, on `thin_disk.py`, which Windows' disks use too), aliased through `[artifacts]`. It has no tasks either. `disk/linux/` is the Linux disk form's recipe: `boot/`, whose Dockerfile makes a disk form (a GPT disk of an EFI system partition holding one UKI, Ubuntu's kernel with a busybox initramfs whose init reads a NoCloud seed and `switch_root`s into the image's entrypoint, and an ext4 root filesystem that is a container image's, plus `/etc/agentenv/vm/image.env` from its OCI config), `tools/`, the disk tools that make it unprivileged on the builder's own platform (`mkfs.ext4 -d`), and `launcher/`, the QEMU launcher that boots one under KVM. A base's `[forms.disk]` with no image is made from its container form; the composer makes each arch's disk from its composed container, keeping the base disk's EFI system partition, keyed by the container's key, the base disk and the recipe; gate 3 probes it where a provider boots disks natively (`disk_image_platforms()`), and a deploy's first boot with KVM probes one left `unverified`; a deploy naming `form = "disk"` boots it through `create_vm(image=..., env=...)`, its settings in the seed. `artifacts/ubuntu-24.04/clock/` is the computer's clock, in the base's folder since an image builds only from its own: agent-env's build of libfaketime 0.9.12 (`libfaketime.patch` and the two `agentenv_clock_*.c` files it includes, GPL-2.0-only as libfaketime is, built by `build.sh` from upstream's sha256-pinned tarball, in a stage of the base's Dockerfile of its own) and the TLS shim, which `/etc/ld.so.preload` loads into every process and which fake only the session user's: clock reads and waits for a set clock time follow the page the follower (`clock-follower`) writes (`clockpage.py`), monotonic and relative waits staying real, with its default settings (`clock.conf`); `tst/clock` gates it. The follower long-polls the gateway's `/clock/watch` under the config core writes through the root channel when a multi env deploys the computer and after a reset (`env/envs/computer/utils/clock.py`), each time waiting on the computer's `sync_time` until it follows, a warning naming why when it doesn't (arming the clock refuses such a computer), and the driver serves `clock/v1` `sync_time` where the follower is (`agentenv_protocol.computer.clock`), not on a disk guest, whose follower core can't reach. |
| `browsers/` | The `browsers` bundle: the website browsers agent-env ships, today `chromium`, a `computer_app` env (Chrome for Testing at the build Playwright pins, for amd64 and arm64, set up for unattended use by managed policies in `/etc/opt/chrome_for_testing/policies/managed`, opened at boot on its `start_urls`, its profile its data). All of it, its pin, install step, policies and command, is in `envs/chromium/chromium.py`; `tst/integration/computer/chromium_app_test.py` runs it. Like `drivers`, it has no tasks. |
| `task_step/task_steps/computer/osworld/` | The `osworld_*` steps, and the `osworld` bundle: the `osworld-server` computer app (`envs/`), upstream OSWorld's guest server fetched at `OSWORLD_COMMIT` and checked by sha256, run on loopback and forwarded under `/osworld/` behind the computer's control token. `osworld_initialize` and `osworld_evaluate` run upstream OSWorld's own setup, getters and metrics at that commit in the worker (`runtime.py`), through loopback forwarders to that endpoint with the env's control token (`forwarder.py`); upstream's source is fetched on first use, checked by sha256, or read from `AGENT_ENV_OSWORLD_SOURCE` (`upstream.py`), and what it imports is the folder's `requirements.txt`, installed where the steps run (and in `dev`, for the e2e test). agent-env ships no LLM judge: a deployment registers one in the worker (`evaluate.register_judge`). No OSWorld-V2 tasks, and no getter needing Chrome's DevTools or VLC's HTTP port, which the app doesn't expose. `tst/integration/computer/osworld_test.py` runs it. Like `drivers`, it has no tasks. |
| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`, the page driver aliased `page`, a browser computer's, whose screen is the browser's active tab and which serves Playwright MCP's `browser_*` tools beside `computer_*` from a Playwright MCP it runs attached to the browser over a CDP address the relay issues it (`browser_evaluate` and `browser_run_code_unsafe` hidden by default; each call an entry in the browser's log, begun and ended with `browser/v1` `calls/begin` and `calls/end`, that its CDP commands name as `for`; `scripts/upgrade_playwright_mcp.py` pins its version), and trycua's cua-driver aliased `cua-driver` (its daemon from the pinned PyPI wheel, its own instructions, tools and skill resources served to agents through `@server_instructions`, `forwarded_tools` and `@resource`, its tools beside `computer_*`, `computer/v1` on its desktop-scope calls, its skill pack leading the computer skill; `tst/integration/computer/cua_driver_test.py` runs it). It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. |
| `examples/` | The example bundles agent-env ships (`hello`, and `echo-mcp`, which needs Docker, and `desktop`, a computer env of `ubuntu-24.04` and `reference` with a probe task and an agent task). Tests run them, but they are examples for users, not fixtures. Every shipped bundle is registered under `agent_env.bundles` in `pyproject.toml`, `<name> = "<package>"` shipping `src/<package as a path>/<name>/`. |

Expand Down
37 changes: 37 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,42 @@ dev = [
"pytest-timeout>=2.3.0",
"pytest-xdist>=3.0.0",
"ruff==0.16.10", # the lint gate; exact, since what it reports changes between releases
# What upstream OSWorld's checks import, which the osworld_* steps run in the worker and the osworld e2e test runs:
# task_step/task_steps/computer/osworld/requirements.txt, which a worker running them installs.
"requests>=2.31",
"requests-toolbelt>=1.0",
"websocket-client>=1.8.0",
"python-dotenv>=1.0",
"playwright>=1.40",
"pydrive>=1.3",
"numpy>=1.26",
"Pillow>=11",
"pandas>=2.2",
"pytz>=2024.1",
"opencv-python-headless>=4.8",
"openpyxl>=3.1",
"lxml>=5",
"cssselect>=1.2",
"xmltodict>=0.13",
"rapidfuzz>=3",
"python-docx>=1.1",
"python-pptx>=1.0",
"odfpy>=1.4",
"pypdf>=4",
"PyPDF2>=3",
"pdfplumber>=0.11",
"pymupdf>=1.24",
"borb<3",
"beautifulsoup4>=4.12",
"scikit-image>=0.22",
"scipy>=1.11",
"ImageHash>=4.3",
"librosa>=0.10",
"pyacoustid>=1.3",
"fastdtw>=0.3",
"mutagen>=1.47",
"formulas>=1.2",
"tldextract>=5",
]
smol = [
"smolmachines>=1.27.0",
Expand All @@ -185,6 +221,7 @@ agents = "agent_env"
browsers = "agent_env"
computers = "agent_env"
drivers = "agent_env"
osworld = "agent_env.task_step.task_steps.computer"
desktop = "agent_env.examples"
hello = "agent_env.examples"
echo-mcp = "agent_env.examples"
Expand Down
2 changes: 1 addition & 1 deletion src/agent_env/task/episode/rules.py
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ def _rubrics_verifier(step: RubricsVerifierTaskStep, outside: Outside) -> list[s
"browser_release_context", "browser_set_cookies", "browser_sign_in", "browser_set_local_storage", "build_mcp_cli",
"combine_universe_verdicts", "configure_scheduled_checks", "deploy_env", "deploy_sandbox", "end_browser_recording",
"end_computer_recording", "start_browser_recording", "start_computer_recording", "env_outcome_verifier",
"modify_env_tool_access", "reset_env", "review", "user_sim_judge",
"modify_env_tool_access", "osworld_evaluate", "osworld_initialize", "reset_env", "review", "user_sim_judge",
"run_container_unit_tests_verifier", "run_docker_container", "set_env_clock_rate", "snapshot_env", "snapshot_computer_env",
"sync_env_clock",
"validation_gate_aggregator", "verify_env_card", "verify_env_core_protocol", "verify_mcp_tool_schema",
Expand Down
4 changes: 4 additions & 0 deletions src/agent_env/task_step/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@ def _builtin_registry() -> dict[str, type["TaskStep"]]:
from agent_env.task_step.task_steps.snapshot_agent_state import SnapshotAgentStateTaskStep
from agent_env.task_step.task_steps.snapshot_env import SnapshotEnvTaskStep
from agent_env.task_step.task_steps.computer.snapshot_computer_env import SnapshotComputerEnvTaskStep
from agent_env.task_step.task_steps.computer.osworld.evaluate import OSWorldEvaluateTaskStep
from agent_env.task_step.task_steps.computer.osworld.initialize import OSWorldInitializeTaskStep
from agent_env.task_step.task_steps.peer_agents import PeerAgentsTaskStep
from agent_env.task_step.task_steps.verifiers.verify_sandbox import VerifySandboxTaskStep
from agent_env.task_step.task_steps.verifiers.aggregate_verifiers import AggregateVerifiersTaskStep
Expand Down Expand Up @@ -174,6 +176,8 @@ def _builtin_registry() -> dict[str, type["TaskStep"]]:
_get_type(SnapshotAgentStateTaskStep): SnapshotAgentStateTaskStep,
_get_type(SnapshotEnvTaskStep): SnapshotEnvTaskStep,
_get_type(SnapshotComputerEnvTaskStep): SnapshotComputerEnvTaskStep,
_get_type(OSWorldInitializeTaskStep): OSWorldInitializeTaskStep,
_get_type(OSWorldEvaluateTaskStep): OSWorldEvaluateTaskStep,
_get_type(RunCodeTaskStep): RunCodeTaskStep,
_get_type(PeerAgentsTaskStep): PeerAgentsTaskStep,
_get_type(VerifyA2ASnapshotStep): VerifyA2ASnapshotStep,
Expand Down
36 changes: 36 additions & 0 deletions src/agent_env/task_step/task_steps/computer/osworld/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
OSWorld tasks on agent-env computers, from upstream OSWorld ([xlang-ai/OSWorld](https://github.com/xlang-ai/OSWorld),
Apache-2.0) at commit `5123bfdbbc34b947bcffcff564abc62d0ab7072d` (`OSWORLD_COMMIT`). The `osworld_initialize` and
`osworld_evaluate` steps set a task up and grade it in the worker, never where the agent can reach it; this bundle ships
the computer app they reach the task's computer through. It has no tasks.

`envs/osworld-server/` is the `osworld-server` computer app, for an X11 desktop on Linux (`amd64` and `arm64`), such as
`ubuntu-24.04`: upstream's guest server (`desktop_env/server/main.py`, with `pyxcursor.py` and `screen_recording.py`),
downloaded when the computer installs the app and checked by sha256, run unmodified as the session user on
127.0.0.1:5000, without Werkzeug's debugger. The app's own server forwards `/osworld/<route>` to it, and its card
advertises that prefix as the endpoint of the `urn:agentenv:osworld-guest/v1` extension, which the computer serves
behind the env's control token. Name it in a computer env's `apps` as `osworld-server`, its alias, or by its id,
`@local/agentenv-framework/osworld/osworld-server`.

## Grading in the worker

The steps run upstream's own `SetupController`, getters and metrics at the same commit, unmodified, in the worker
(`runtime.py`). Upstream builds every URL as `http://{vm_ip}:{port}`, so each setup or check runs against loopback
ports of its own, where a forwarder sends every call on to the computer's `/osworld` endpoint with the env's control
token (`forwarder.py`). The worker holds that token and any judge's model key; the agent holds neither.

- **Upstream's source** is fetched on first use from GitHub's archive of the commit, checked by sha256, and kept under
the cache root (`~/.cache/agentenv/osworld/<commit>`). An image can bake it in instead: run
`python -m agent_env.task_step.task_steps.computer.osworld.upstream <folder>` when building it, and set
`AGENT_ENV_OSWORLD_SOURCE=<folder>` (`upstream.py`).
- **What upstream imports** is this folder's `requirements.txt`, which a worker running the steps installs:
`pip install -r <it>`. It isn't an extra, since agent-env's own code imports none of it. It leaves out easyocr,
which pulls torch in for `compare_image_text` alone, so that metric fails.
- **LLM-judge funcs** (`agent_judge` and its `agent_judge_<format>` variants): agent-env ships no judge. The worker's
deployment registers one with `evaluate.register_judge(func, judge)`, and it runs in the worker with the run's judge
model (`user_overrides.judge_model`, else the func's `model` option, else the step's `judge_model`). A judge func with
no judge registered scores 0, saying so.
- **Not supported:**
- OSWorld-V2 tasks (`osworld_v2_task_path`), whose setup and evaluate are the task's own code; the steps fail saying so.
- A setup or check that needs the computer's Chrome DevTools port (9222) or VLC's HTTP port (8080), which the app
doesn't expose; it fails saying which.
- Google Drive setups and upstream's proxy.
29 changes: 29 additions & 0 deletions src/agent_env/task_step/task_steps/computer/osworld/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
"""OSWorld tasks on agent-env computers, and the ``osworld`` bundle agentenv-framework ships.

``osworld_initialize`` sets a task up on a computer that runs the ``osworld-server`` app and ``osworld_evaluate`` grades
it, both in the worker, with upstream OSWorld's own setup, getters and metrics (:mod:`.runtime`, from :mod:`.upstream`)
reaching the computer's guest server through loopback forwarders (:mod:`.forwarder`) with the env's control token,
which the agent doesn't hold (:mod:`.guest`). What upstream imports is this
folder's ``requirements.txt``, installed where the steps run.

This folder is also the ``osworld`` bundle (``README.md``), whose ``osworld-server`` computer app (``envs/``), which
runs upstream OSWorld's guest server, fetched at ``OSWORLD_COMMIT`` and checked by sha256, a task names by its alias
(``aliases.toml``)."""
from __future__ import annotations

from pathlib import Path

#: The upstream OSWorld commit the guest server and the steps' checks are from (xlang-ai/OSWorld, Apache-2.0).
OSWORLD_COMMIT = "5123bfdbbc34b947bcffcff564abc62d0ab7072d"
#: The app's ``environment_name``.
APP_NAME = "osworld-server"
#: The extension the app's card advertises at the prefix its server forwards to the guest server under. A computer
#: nests the app's card under its ``children_environments`` with the endpoint rooted where it serves the app, so a step
#: finds the guest server by this URI in the computer's card.
GUEST_EXTENSION = "urn:agentenv:osworld-guest/v1"
#: The prefix the app's server forwards to the guest server under.
PROXY_PREFIX = "/osworld"
#: The bundle's folder, this package's.
BUNDLE_DIR = Path(__file__).parent
#: The app's folder.
APP_DIR = BUNDLE_DIR / "envs" / APP_NAME
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Ids a task can name this bundle's app by; a store's entity of the id wins over an alias.
[envs]
osworld-server = ["osworld-server"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
type = "computer_app"
environment_name = "osworld-server"
module = "osworld_server_app.py:OSWorldServer"
computer = "ubuntu-24.04"
driver = "reference"
Loading
Loading