Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ and deserialized through a registry.
| `computers/` | The `computers` bundle: the base computers agent-env ships as `computer` artifacts, `ubuntu-24.04` (XFCE on Xvfb, settings as `AGENTENV_SETTING_<NAME>`; it starts as root, its container form's `start_as`, whose supervisord, `root.conf`, runs the clock follower as `agentenv-clock`, the egress proxy as `agentenv-egress` (`utils/egress_proxy/`, a Go program its Dockerfile builds and tests, on real time, through which every session program goes out, so a live site loads at any universe date: it passes a site's TLS through when the universe is inside the site's certificate, else re-signs it with the computer's own CA, minted at boot and valid from 1950 to 9999, handshaking with the site as the client did, and refuses any destination that isn't global) and the session's supervisord as the session user, which runs every session program; `root/` installs them at the build, `install.sh`, and starts them at boot, `init`; `ubuntu-24.04-browser`, the base of agent-env's website browsers, which takes the same clock and egress proxy, and ubuntu-24.04's `root/` and `root.conf`, through its `shared_sources.toml`), and `omarchy-4.5`, a disk alone built on the composing machine from Omarchy's own VM image (its `[forms.disk]` `source`) by `disk/omarchy/`'s recipe, its services systemd user units of the graphical session that start through `with-seed`, booted by the launcher as a `linux-desktop` guest (`env/envs/computer/utils/omarchy_disk.py`, on `thin_disk.py`, which Windows' disks use too), aliased through `[artifacts]`. It has no tasks either. `disk/linux/` is the Linux disk form's recipe: `boot/`, whose Dockerfile makes a disk form (a GPT disk of an EFI system partition holding one UKI, Ubuntu's kernel with a busybox initramfs whose init reads a NoCloud seed and `switch_root`s into the image's entrypoint, and an ext4 root filesystem that is a container image's, plus `/etc/agentenv/vm/image.env` from its OCI config), `tools/`, the disk tools that make it unprivileged on the builder's own platform (`mkfs.ext4 -d`), and `launcher/`, the QEMU launcher that boots one under KVM. A base's `[forms.disk]` with no image is made from its container form; the composer makes each arch's disk from its composed container, keeping the base disk's EFI system partition, keyed by the container's key, the base disk and the recipe; gate 3 probes it where a provider boots disks natively (`disk_image_platforms()`), and a deploy's first boot with KVM probes one left `unverified`; a deploy naming `form = "disk"` boots it through `create_vm(image=..., env=...)`, its settings in the seed. `utils/faketime_clock/` is the computer's clock, which each base takes through its `shared_sources.toml`: agent-env's build of libfaketime 0.9.12 (`libfaketime.patch` and the two `agentenv_clock_*.c` files it includes, GPL-2.0-only as libfaketime is, built by `build.sh` from upstream's sha256-pinned tarball, in a stage of the base's Dockerfile of its own) and the TLS shim, which `/etc/ld.so.preload` loads into every process and which fake only the session user's: clock reads and waits for a set clock time follow the page the follower (`clock-follower`) writes (`clockpage.py`), monotonic and relative waits staying real, with its default settings (`clock.conf`); `tst/clock` gates it. The follower long-polls the gateway's `/clock/watch` under the config core writes through the root channel when a multi env deploys the computer and after a reset (`env/envs/computer/utils/clock.py`), each time waiting on the computer's `sync_time` until it follows, a warning naming why when it doesn't (arming the clock refuses such a computer), and the driver serves `clock/v1` `sync_time` where the follower is (`agentenv_protocol.computer.clock`), not on a disk guest, whose follower core can't reach. |
| `utils/faketime_clock/`, `utils/egress_proxy/` | Sources several shipped images share, each built into the image from its own context: `faketime_clock/` (agent-env's libfaketime, the TLS shim, the clock page and the clock follower) and `egress_proxy/` (the egress proxy). An image's folder names what it takes in its `shared_sources.toml` (`clock = "utils/faketime_clock"`, paths under `src/agent_env`, at the name its Dockerfile copies from), and its build context (`utils/docker/context.py`, and release CI's `infra_build.py`) holds each at that name, so the image still builds from one self-contained context and its source hash covers what it takes. A fix lands once, in every image that takes it. |
| `browsers/` | The `browsers` bundle: the website browsers agent-env ships, today `chromium`, a `computer_app` env (Chrome for Testing at the build Playwright pins, for amd64 and arm64, set up for unattended use by managed policies in `/etc/opt/chrome_for_testing/policies/managed`, opened at boot on its `start_urls`, its profile its data). All of it, its pin, install step, policies and command, is in `envs/chromium/chromium.py`; `tst/integration/computer/chromium_app_test.py` runs it. Like `drivers`, it has no tasks. |
| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`, the page driver aliased `page`, a browser computer's, whose screen is the browser's active tab and which serves Playwright MCP's `browser_*` tools beside `computer_*` from a Playwright MCP it runs attached to the browser over a CDP address the relay issues it (`browser_evaluate` and `browser_run_code_unsafe` hidden by default; each call an entry in the browser's log, begun and ended with `browser/v1` `calls/begin` and `calls/end`, that its CDP commands name as `for`; `scripts/upgrade_playwright_mcp.py` pins its version), and trycua's cua-driver aliased `cua-driver` (its daemon from the pinned PyPI wheel, its own instructions, tools and skill resources served to agents through `@server_instructions`, `forwarded_tools` and `@resource`, its tools beside `computer_*`, `computer/v1` on its desktop-scope calls, its skill pack leading the computer skill; `tst/integration/computer/cua_driver_test.py` runs it). It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. |
| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`, the page driver aliased `page`, a browser computer's, whose screen is the browser's active tab and which serves Playwright MCP's `browser_*` tools beside `computer_*` from a Playwright MCP it runs attached to the browser over a CDP address the relay issues it (`browser_evaluate` and `browser_run_code_unsafe` hidden by default; each call an entry in the browser's log, begun and ended with `browser/v1` `calls/begin` and `calls/end`, that its CDP commands name as `for`; `scripts/upgrade_playwright_mcp.py` pins its version), and trycua's cua-driver aliased `cua-driver` (its daemon from the pinned PyPI wheel, its own instructions, tools and skill resources served to agents through `@server_instructions`, `forwarded_tools` and `@resource`, its tools beside `computer_*`, `computer/v1` on its desktop-scope calls, its skill pack leading the computer skill; `tst/integration/computer/cua_driver_test.py` runs it). Hybrid Computer Use (HCU) 1.0.1, aliased `hcu`, supports Linux X11 on `amd64` only. Its AX/GUI tools run beside `computer_*` from `X11Driver`. The bundle has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. |
| `examples/` | The example bundles agent-env ships (`hello`, and `echo-mcp`, which needs Docker, and `desktop`, a computer env of `ubuntu-24.04` and `reference` with a probe task and an agent task). Tests run them, but they are examples for users, not fixtures. Every shipped bundle is registered under `agent_env.bundles` in `pyproject.toml`, `<name> = "<package>"` shipping `src/<package as a path>/<name>/`. |

### Training helpers
Expand Down
46 changes: 46 additions & 0 deletions scripts/dump_hcu_mcp.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
"""Write HCU's instructions and tools from a local release executable, without a desktop or network:

python scripts/dump_hcu_mcp.py /path/to/computer-use src/agent_env/drivers/envs/hcu/mcp.json

On Windows, run computer-use.exe with the Windows provider and input-acknowledgement DLL beside it, and write
mcp-windows.json. Linux uses mcp.json. Pass --version to refuse an executable from a different release.

Review the manifest alongside the release's URL and sha256 pins in env.toml and driver.py.
"""
from __future__ import annotations

import argparse
import asyncio
import json
from pathlib import Path

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client


async def dump(binary: Path, output: Path, version: str | None = None) -> None:
parameters = StdioServerParameters(command=str(binary.resolve()), args=["mcp"])
async with stdio_client(parameters) as (read, write), ClientSession(read, write) as session:
initialized = await session.initialize()
if version is not None and initialized.serverInfo.version != version:
raise ValueError(f"Expected HCU {version}, got {initialized.serverInfo.version}")
tools = (await session.list_tools()).tools
kept = ("name", "title", "description", "inputSchema", "annotations", "outputSchema")
manifest = {"instructions": initialized.instructions or "", "tools": [
{name: value for name, value in tool.model_dump(by_alias=True, exclude_none=True).items() if name in kept}
for tool in tools]}
output.write_text(json.dumps(manifest, indent=2, ensure_ascii=False) + "\n", encoding="utf-8", newline="\n")
print(f"{initialized.serverInfo.name} {initialized.serverInfo.version}: {', '.join(tool.name for tool in tools)}")


def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("binary", type=Path)
parser.add_argument("output", type=Path)
parser.add_argument("--version", help="Require this server version before writing the manifest")
arguments = parser.parse_args()
asyncio.run(dump(arguments.binary, arguments.output, arguments.version))


if __name__ == "__main__":
main()
46 changes: 46 additions & 0 deletions src/agent_env/drivers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,52 @@ server lists the tools `mcp.json` pins, and the TCC permissions on macOS, the in
pin each tarball's URL and sha256 in `env.toml` and the executables' in `driver.py`'s `EXECUTABLES`, and take `mcp.json`
again from what that version's server says at connect and lists. Name it as `agent-desktop`, the alias `aliases.toml` gives it.

`envs/hcu/` is [Hybrid Computer Use (HCU)](https://github.com/microsoft/Hybrid-Computer-Use) as a driver
for Linux X11 and Windows desktops (`amd64` only). Windows installation awaits a published, pinned archive.
The Linux install checks the pinned 1.0.1
[release tarball](https://github.com/microsoft/Hybrid-Computer-Use/releases/download/v1.0.1/computer-use-1.0.1-linux-amd64.tar.gz)
by sha256 and places the extracted executable at `bin/computer-use` beside the driver.
Agents get HCU's own accessibility (AX) and graphical user interface (GUI) tools, with their instructions and schemas
pinned in `mcp.json` on Linux and `mcp-windows.json` on Windows. The `computer_*` tools, display, input and recording
come from `X11Driver` on Linux and `WindowsDriver` on Windows. Each driver instance keeps one HCU stdio process and
serializes calls to that process.
`computer-ready` checks the executable's sha256 against `EXECUTABLES` and the server's instructions and tools against
the manifest, alongside the desktop driver's readiness checks. Name it as `hcu`, the alias `aliases.toml` gives it.

On Linux, the HCU process inherits only the variables in `driver.py`'s `KEPT` and receives a fixed `PATH`.
If `XDG_RUNTIME_DIR` is unset, the driver creates a private directory with mode `0700` for HCU.
If `DBUS_SESSION_BUS_ADDRESS` is unset and `/run/agentenv/bus` exists, HCU uses that session bus.
The install sets the dconf system default `org.gnome.desktop.interface toolkit-accessibility=true` so applications
started before the first HCU call can expose their accessibility trees.

On Windows, the adapter uses `computer = "windows-11"` and `driver = "hcu"`.
No public Windows download is configured yet. Development bundles must supply `computer-use.zip`
through their file collection, matching the installer's SHA256 pin; the installer refuses a missing or different archive.
`install-windows.ps1` extracts `computer-use.exe`, `computer-use-windows-provider.exe`,
`computer-use-input-ack.dll` and their licenses into `bin`. The three runtime files must stay together.
The provider includes its .NET runtime; HCU needs no Python provider.
The driver's venv installs the protocol's `computer` and `windows` extras.
`agentenv-session` starts the driver in the signed-in desktop session. HCU receives the session and system variables
in `WINDOWS_KEPT` and a fixed system `PATH`, without deployment credentials or provider overrides.
`computer-ready` checks all three runtime hashes, the Windows MCP contract and access to the session's input desktop.

To move it to another HCU version:

1. Set `VERSION` in `envs/hcu/driver.py` to the new version.
2. Pin each platform's release archive URL and sha256 in `envs/hcu/env.toml`.
3. Update `EXECUTABLES`, `PROVIDER_SHA256` and `INPUT_ACK_SHA256` in `envs/hcu/driver.py`.
Update the Windows archive pin in `install-windows.ps1` as well.
4. Check the manifest on each platform with `scripts/dump_hcu_mcp.py BINARY OUT --version VERSION`.
The helper starts the executable, sends MCP `initialize` and `tools/list`, and writes the manifest.
From the repository root, run:

```bash
.venv/bin/python scripts/dump_hcu_mcp.py /path/to/computer-use src/agent_env/drivers/envs/hcu/mcp.json --version 1.0.1
```

On Windows, use the driver's Python, pass `bin/computer-use.exe` and write `mcp-windows.json`.
The helper writes UTF-8 with LF line endings on both platforms and refuses a different server version.

`envs/macos-harness/` is browser-use's [macOS harness](https://github.com/browser-use/macos-harness) as a driver, for
the same macOS computers. macos-harness 0.2.1, from PyPI into the driver's venv, reads an app as numbered text and acts
on a number or a pixel, in the background, through eleven commands its skill teaches an agent to run in a shell,
Expand Down
1 change: 1 addition & 0 deletions src/agent_env/drivers/aliases.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ reference = ["reference"]
cua-driver = ["cua-driver"]
page = ["page"]
agent-desktop = ["agent-desktop"]
hcu = ["hcu"]
macos-harness = ["macos-harness"]
codex-computer-use = ["codex-computer-use"]
aside = ["aside"]
Loading