diff --git a/AGENTS.md b/AGENTS.md index eef7c07bb..d6bbf6649 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -123,7 +123,7 @@ and deserialized through a registry. | Package | Contents | |---|---| -| `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset`, `close` and `load_file_artifact_universe`; teardown closes each env a run deployed before terminating its sandboxes by id. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites and computers; its website browser is the one its `browser_envs` names, none for `[]`, else the run's when it has websites: an MCP server env such as `playwright-mcp` runs among the gateway's MCP servers, and a browser computer such as `chromium` on a sandbox of its own as one more computer child, sized by its `child_envs` entry, its tools under its name and `_`, routing the multi env's synthetic sites unless that entry's `site_routing` says, its network its own env's, as any computer child's; a website env deployed alone refuses a browser computer as the run's browser), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`, one site: a server image, `backend_image` in its env.toml, serving the pages and API, ready when its card's `website/v1` `ready` answers a `WebsiteReadiness` that says so, as `agentenv_protocol.website` (`website_readiness`, `env_readiness`) defines a website env's health for the deploy and `DeployedEnv.readiness()` alike, or, for a card without `ready`, when its `health` path answers 200, which is deprecated; its env.toml's `ready_timeout_s`, how long it may take from its server's container starting to ready, is that container's health check `start_period` and what the deploy waits, or its card's `ready` timeout if longer, and its `shm_mb` is the container's `shm_size`, each left to the gateway and Docker without them; the gateway container's `env/gateway/constants.py` mirrors the URI and op, a test keeping them equal; a separate frontend image is deprecated, warned of at put and optional in a bundle, and a stored `backend_docker_image_artifact` reads as the server image; a live one, `live = true`, is a real site whose server serves none of its pages, its `live`, `domains` and `allow_hosts` read at put from its server's `@website(...)` (`utils/card_naming.py` `WebsiteSource`) unless given, its `settings`' secrets only as references the gateway resolves as it writes the site's container (`WEBSITE_SETTINGS_ENV`); the gateway gives it no site route, a computer's site table leaves it out, a computer whose network is an allowlist is let reach its names (`MultiEnv.live_hosts`), a browser alone's forward port takes them straight to the real site (`site_proxy.py` `direct`), as the browser base's stand-in proxy takes any host, refusing nothing, each proxy logging each such connection as a JSON line, and a run that deploys one is marked live, `reproducible: false`), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, `env/envs/computer/driver_env.py`: a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed, with the `port` it serves on and its install by OS, `[install.]`, or for one distribution `[install."/"]`, which replaces its OS type's there, as its `known_failures` do (`env/envs/computer/utils/install.py`): files, apt `packages` with extra `sources`, a `python` requirements file for a venv, `downloads` by URL and sha256, `run` scripts, the services the base's init runs, and `ports`; it deploys only as a computer env's driver), `ComputerEnv` (`computer`, in `env/envs/computer/`: a base computer and a driver, matched at put by gate 1 before anything is stored, then composed by `composer.py` into an image for each form and arch on a builder of the put's sandbox provider, by default the env's (`utils/builder.py`; `--sandbox` on `env build`): this machine's Docker for `local`, else a VM of the provider, which fetches the put's stored files by signed URL, takes the rest inline and pushes to the image store itself, so it builds only with stores a remote VM reaches; each arch goes to the first link that runs it natively, else one that emulates it (a VM registers QEMU; `--no-emulate` leaves it `skipped` instead, for a later put), and its pair records the provider that built it (`builder`); a run or register that reuses a computer env composes what it has left (`pending`), a run, and a register given a `--sandbox`, only the forms its sandbox's deploys boot (`booted_forms`: on each link, the first form it boots natively, by the deploy's own rule, `boot.first_native`; a compose of a disk adds its container), and a deploy composes first what it boots that no compose started, and a container it runs natively that was built but not probed, as one whose probe a setting that didn't resolve stopped, booting it only once ready (`ComputerEnv.compose_unstarted`); tagged `agentenv-computers:` in the image store by a key of the image's Dockerfile, build context and lock so any env with the same inputs reuses it; every layer's packages, requirements and downloads are resolved at put in a container of the base for each arch (`utils/packages.py`, `utils/resolve_in_base.py`), stored in the object store by sha256 under `agentenv-packages/`, and listed in a lock stored by a digest of its inputs, which any env with the same inputs reuses, and builds install only the lock's files, every install step `RUN --network=none`; each pair's status goes into its metadata, `composed`, the arches built are written as its computer, a `computer` artifact over one index, and `agent-env env build` resumes what a put left; gate 3 (`utils/probing.py`) then boots each native build on the put's sandbox provider and probes it at `full`, making it `ready` or `failed`, and a build no provider runs natively stays `unverified`. Its deploy, through the `computer` env provider (`providers/env_providers/env_computer_provider.py`), boots its first `ready` build on a sandbox provider that runs its arch natively (`native_platforms()`), with no gateway in front and the driver's card as the env card, its `/dev/shm` the most its apps' Linux installs ask for (`shm_mb`, in MiB, which its composed computer and the deploy record keep; `--shm-size` on a container, `AGENTENV_SHM_MB` read by a Linux disk's init, passed only to a provider whose `SHARED_MEMORY` says it gives that much, else the sandbox's default, which the deploy logs), then checks its `display`; `deploy_env` `env_params` take a `form`, `computer_settings` (deploy-phase values of the base's settings; a secret one, here or on the env, only as a bare `secret:`/`env:` reference) and `driver_settings` (the driver's `ComputerSettings`, such as its default coordinate space); a `network` that allows nothing runs it on local Docker's internal network behind a relay (`providers/sandbox_providers/local_docker/relay/`); `reset` and `close` have the driver end its recordings first (`computer-recording/v1` `end_all`, each uploading what's left and its manifest under the grant it started with), and `reset` then boots a fresh container in its place (`reset_container`), keeping the computer's address but not an agent's MCP session to the old driver; its `load_file_artifact_universe` writes files under the session user's home through the driver's `data/add`, `Downloads` by default, each file's bytes through a read grant when the store's grants reach it and its network allows everything, else inline (`env/envs/computer/utils/inside.py`)), `ComputerAppEnv` (`computer_app`, `env/envs/computer/app_env.py`: an app on a computer, a `ComputerAppEnvironment` class its env.toml's `module` names, read at put by the same `_declare`, which declares each OS with its own decorator (`@linux(...)`, or `@install(os_type, spec)` for any OS type; `@linux(distribution=...)` or `@install("/", spec)` for one distribution, which replaces its OS type's there) holding its install, its launch and, optionally, a skill and settings and build settings that add to the class's there, and install steps and ready checks for every OS or those their `os=` names; everything that reads its settings, launch or steps asks for the computer's OS type; deployed alone it is a one-app computer env, a `ComputerEnv` of the computer and driver its env.toml names, the composer installing it under `/opt/agentenv/apps//` with its services as supervisord programs beside its own server (`agentenv_protocol.computer.serve_app`, in a venv with the protocol's `app` extra, on `.agentenv/server.sock` in its folder), which adds its launcher, launches a `boot` app once per boot and says through `computer-app/v1` whether it's ready, as the driver's `computer-ready` asks, its card named after it, and `env_params` `app_settings` handing it values of its `Settings` in `AGENTENV_APP_SETTINGS`, a secret one only as a bare reference, over its env.toml's `[settings]` and a computer env's `settings` beside it in `apps`, each by top-level field (`params.app_settings`; checked at put, a required one left to the deploy; the deploy record keeps the merged values as `app_settings`, secrets withheld); its env.toml's `build_settings`, values of its `BuildSettings` checked at put, are written to `.agentenv/build_settings.json` in its folder, so they're part of the image's key, its install steps read them, and a deploy can't change them). An env type whose services take environment data defines `load_environment_artifact` and `load_environment_universe_artifact`; `load_artifact` refuses an environment artifact aimed at a type that doesn't, at save and before it loads. An env built on the protocol SDK can serve agents a skill (`@agent_skill`, `skill/v1`), which `add_skills` `env_ids` installs under its environment name (behind a gateway, each server's under that server's name); a skill with files beside its SKILL.md goes as a bundle stored once per content under `agentenv-env-skills/` in the object store. `env/gateway/gateway.py` is the env-side gateway server, container-only code. Its site route, `/website//`, never forwards a site's data plane or card (`/agentenv/…`, `/.well-known/agent-env.json`), which agentenv reaches through `/svc//`; a site with no frontend image is served by its server image there. A request from a computer's site proxy names its computer (`AgentEnv-Computer`, sent from the proxy's private upstream headers, a page's own copy and any `AgentEnv-Role` it sends dropped) and takes the role driving that computer now (the gateway's computers being those `COMPUTERS` names, a multi env's, with or without a control token): the last to make an input call to its tools, as the call starts, through `/mcp`, `/step` or a trigger action under its `as` (`Gateway._drive`); without one, or from a computer the gateway doesn't front, it is `default`, the role read once per request. `changes/v1` (`ChangeAttribution`) counts no write window for a call to a computer's tools, a trigger action's included, so a write the computer's browser made goes to its request and one another party made during the call to that party. Its CDP route, `/browser//cdp//` (`env/gateway/browser_proxy.py`), reaches the website browser behind the `/svc` child ``, found from that child's card (it, or an app among its children, offering `browser/v1` `cdp` with `/cdp/` in its `computer-app/v1` `agent_routes`): it relays the browser's discovery and WebSocket both ways, each leg pinged every 20 s, and every CDP address the browser hands out, in `json/version`, `json/list` and its `cdp` answer through `/svc//`, comes back at the gateway, everything before `/cdp//` becoming the address the caller reached the gateway at and `/browser/`, keeping the https or wss of an address the browser minted under an origin its caller named, as `browser_connect_cdp` names the agent's, since a tunnel that ends TLS, as Modal's, may send no `X-Forwarded-Proto`; each connection is a `browser_connection` trajectory event, without its token. Its clock (`env/gateway/clock.py`) has one opaque version, a per-boot id and a count, that every `set_time`, `advance` and `clear` changes; `set_rate` (`/clock/set-rate`) changes an armed clock's rate from its time at the write, keeping its `t0` and generation as `advance` does, so time triggers keep their marks. Every clock change goes through `Gateway._change_clock`, which wakes the triggers when the change kept the clock's generation, answers the held watches and waits for the followers. Its followers, the computers on it (`CLOCK_FOLLOWER_TOKENS`, each child whose card serves `clock/v1` `sync_time`, by its read token, its control token for now), long-poll `/clock/watch` naming the version they show, each watch held up to the card's `hold_s` (`env/gateway/clock_followers.py`), and a write given `wait_followers_s` waits for each, answering 504 naming those behind, but not for one that hasn't watched for longer than a hold and a reconnect (`ClockFollowers.QUIET_S`), which it names at once with when each last watched; a restarted gateway is unarmed. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and also in the universe's zone, the one `sync_env_clock`'s `timezone` armed it in, or in an IANA `time_zone` it's given; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. | +| `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset`, `close` and `load_file_artifact_universe`; teardown closes each env a run deployed before terminating its sandboxes by id. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites and computers; its website browser is the one its `browser_envs` names, none for `[]`, else the run's when it has websites: an MCP server env such as `playwright-mcp` runs among the gateway's MCP servers, and a browser computer such as `chromium` on a sandbox of its own as one more computer child, sized by its `child_envs` entry, its tools under its name and `_`, routing the multi env's synthetic sites unless that entry's `site_routing` says, its network its own env's, as any computer child's; a website env deployed alone refuses a browser computer as the run's browser), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`, one site: a server image, `backend_image` in its env.toml, serving the pages and API, ready when its card's `website/v1` `ready` answers a `WebsiteReadiness` that says so, as `agentenv_protocol.website` (`website_readiness`, `env_readiness`) defines a website env's health for the deploy and `DeployedEnv.readiness()` alike, or, for a card without `ready`, when its `health` path answers 200, which is deprecated; its env.toml's `ready_timeout_s`, how long it may take from its server's container starting to ready, is that container's health check `start_period` and what the deploy waits, or its card's `ready` timeout if longer, and its `shm_mb` is the container's `shm_size`, each left to the gateway and Docker without them; the gateway container's `env/gateway/constants.py` mirrors the URI and op, a test keeping them equal; a separate frontend image is deprecated, warned of at put and optional in a bundle, and a stored `backend_docker_image_artifact` reads as the server image; a live one, `live = true`, is a real site whose server serves none of its pages, its `live`, `domains` and `allow_hosts` read at put from its server's `@website(...)` (`utils/card_naming.py` `WebsiteSource`) unless given, its `settings`' secrets only as references the gateway resolves as it writes the site's container (`WEBSITE_SETTINGS_ENV`); the gateway gives it no site route, a computer's site table leaves it out, a computer whose network is an allowlist is let reach its names (`MultiEnv.live_hosts`), a browser alone's forward port takes them straight to the real site (`site_proxy.py` `direct`), as the browser base's stand-in proxy takes any host, refusing nothing, each proxy logging each such connection as a JSON line, and a run that deploys one is marked live, `reproducible: false`), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, `env/envs/computer/driver_env.py`: a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed, with the `port` it serves on and its install by OS, `[install.]`, or for one distribution `[install."/"]`, which replaces its OS type's there, as its `known_failures` do (`env/envs/computer/utils/install.py`): files, apt `packages` with extra `sources`, a `python` requirements file for a venv, `downloads` by URL and sha256, `run` scripts, the services the base's init runs, and `ports`; it deploys only as a computer env's driver), `ComputerEnv` (`computer`, in `env/envs/computer/`: a base computer and a driver, matched at put by gate 1 before anything is stored, then composed by `composer.py` into an image for each form and arch on a builder of the put's sandbox provider, by default the env's (`utils/builder.py`; `--sandbox` on `env build`): this machine's Docker for `local`, else a VM of the provider, which fetches the put's stored files by signed URL, takes the rest inline and pushes to the image store itself, so it builds only with stores a remote VM reaches; each arch goes to the first link that runs it natively, else one that emulates it (a VM registers QEMU; `--no-emulate` leaves it `skipped` instead, for a later put), and its pair records the provider that built it (`builder`); a run or register composes only the computer envs its tasks deploy (`materialize._composed_envs`: each env a task's steps name, and each env those hold but a computer env's own apps and driver, which run inside it) and those it pulls in from an installed bundle, whose one write every bundle shares, every one for a plan with no tasks, any other composing at its first deploy; one that reuses a computer env composes what it has left (`pending`), a run, and a register given a `--sandbox`, only the forms its sandbox's deploys boot (`booted_forms`: on each link, the first form it boots natively, by the deploy's own rule, `boot.first_native`; a compose of a disk adds its container), and a deploy composes first what it boots that no compose started, and a container it runs natively that was built but not probed, as one whose probe a setting that didn't resolve stopped, booting it only once ready (`ComputerEnv.compose_unstarted`); tagged `agentenv-computers:` in the image store by a key of the image's Dockerfile, build context and lock so any env with the same inputs reuses it; every layer's packages, requirements and downloads are resolved at put in a container of the base for each arch (`utils/packages.py`, `utils/resolve_in_base.py`), stored in the object store by sha256 under `agentenv-packages/`, and listed in a lock stored by a digest of its inputs, which any env with the same inputs reuses, and builds install only the lock's files, every install step `RUN --network=none`; each pair's status goes into its metadata, `composed`, the arches built are written as its computer, a `computer` artifact over one index, and `agent-env env build` resumes what a put left; gate 3 (`utils/probing.py`) then boots each native build on the put's sandbox provider and probes it at `full`, making it `ready` or `failed`, and a build no provider runs natively stays `unverified`. Its deploy, through the `computer` env provider (`providers/env_providers/env_computer_provider.py`), boots its first `ready` build on a sandbox provider that runs its arch natively (`native_platforms()`), with no gateway in front and the driver's card as the env card, its `/dev/shm` the most its apps' Linux installs ask for (`shm_mb`, in MiB, which its composed computer and the deploy record keep; `--shm-size` on a container, `AGENTENV_SHM_MB` read by a Linux disk's init, passed only to a provider whose `SHARED_MEMORY` says it gives that much, else the sandbox's default, which the deploy logs), then checks its `display`; `deploy_env` `env_params` take a `form`, `computer_settings` (deploy-phase values of the base's settings; a secret one, here or on the env, only as a bare `secret:`/`env:` reference) and `driver_settings` (the driver's `ComputerSettings`, such as its default coordinate space); a `network` that allows nothing runs it on local Docker's internal network behind a relay (`providers/sandbox_providers/local_docker/relay/`); `reset` and `close` have the driver end its recordings first (`computer-recording/v1` `end_all`, each uploading what's left and its manifest under the grant it started with), and `reset` then boots a fresh container in its place (`reset_container`), keeping the computer's address but not an agent's MCP session to the old driver; its `load_file_artifact_universe` writes files under the session user's home through the driver's `data/add`, `Downloads` by default, each file's bytes through a read grant when the store's grants reach it and its network allows everything, else inline (`env/envs/computer/utils/inside.py`)), `ComputerAppEnv` (`computer_app`, `env/envs/computer/app_env.py`: an app on a computer, a `ComputerAppEnvironment` class its env.toml's `module` names, read at put by the same `_declare`, which declares each OS with its own decorator (`@linux(...)`, or `@install(os_type, spec)` for any OS type; `@linux(distribution=...)` or `@install("/", spec)` for one distribution, which replaces its OS type's there) holding its install, its launch and, optionally, a skill and settings and build settings that add to the class's there, and install steps and ready checks for every OS or those their `os=` names; everything that reads its settings, launch or steps asks for the computer's OS type; deployed alone it is a one-app computer env, a `ComputerEnv` of the computer and driver its env.toml names, the composer installing it under `/opt/agentenv/apps//` with its services as supervisord programs beside its own server (`agentenv_protocol.computer.serve_app`, in a venv with the protocol's `app` extra, on `.agentenv/server.sock` in its folder), which adds its launcher, launches a `boot` app once per boot and says through `computer-app/v1` whether it's ready, as the driver's `computer-ready` asks, its card named after it, and `env_params` `app_settings` handing it values of its `Settings` in `AGENTENV_APP_SETTINGS`, a secret one only as a bare reference, over its env.toml's `[settings]` and a computer env's `settings` beside it in `apps`, each by top-level field (`params.app_settings`; checked at put, a required one left to the deploy; the deploy record keeps the merged values as `app_settings`, secrets withheld); its env.toml's `build_settings`, values of its `BuildSettings` checked at put, are written to `.agentenv/build_settings.json` in its folder, so they're part of the image's key, its install steps read them, and a deploy can't change them). An env type whose services take environment data defines `load_environment_artifact` and `load_environment_universe_artifact`; `load_artifact` refuses an environment artifact aimed at a type that doesn't, at save and before it loads. An env built on the protocol SDK can serve agents a skill (`@agent_skill`, `skill/v1`), which `add_skills` `env_ids` installs under its environment name (behind a gateway, each server's under that server's name); a skill with files beside its SKILL.md goes as a bundle stored once per content under `agentenv-env-skills/` in the object store. `env/gateway/gateway.py` is the env-side gateway server, container-only code. Its site route, `/website//`, never forwards a site's data plane or card (`/agentenv/…`, `/.well-known/agent-env.json`), which agentenv reaches through `/svc//`; a site with no frontend image is served by its server image there. A request from a computer's site proxy names its computer (`AgentEnv-Computer`, sent from the proxy's private upstream headers, a page's own copy and any `AgentEnv-Role` it sends dropped) and takes the role driving that computer now (the gateway's computers being those `COMPUTERS` names, a multi env's, with or without a control token): the last to make an input call to its tools, as the call starts, through `/mcp`, `/step` or a trigger action under its `as` (`Gateway._drive`); without one, or from a computer the gateway doesn't front, it is `default`, the role read once per request. `changes/v1` (`ChangeAttribution`) counts no write window for a call to a computer's tools, a trigger action's included, so a write the computer's browser made goes to its request and one another party made during the call to that party. Its CDP route, `/browser//cdp//` (`env/gateway/browser_proxy.py`), reaches the website browser behind the `/svc` child ``, found from that child's card (it, or an app among its children, offering `browser/v1` `cdp` with `/cdp/` in its `computer-app/v1` `agent_routes`): it relays the browser's discovery and WebSocket both ways, each leg pinged every 20 s, and every CDP address the browser hands out, in `json/version`, `json/list` and its `cdp` answer through `/svc//`, comes back at the gateway, everything before `/cdp//` becoming the address the caller reached the gateway at and `/browser/`, keeping the https or wss of an address the browser minted under an origin its caller named, as `browser_connect_cdp` names the agent's, since a tunnel that ends TLS, as Modal's, may send no `X-Forwarded-Proto`; each connection is a `browser_connection` trajectory event, without its token. Its clock (`env/gateway/clock.py`) has one opaque version, a per-boot id and a count, that every `set_time`, `advance` and `clear` changes; `set_rate` (`/clock/set-rate`) changes an armed clock's rate from its time at the write, keeping its `t0` and generation as `advance` does, so time triggers keep their marks. Every clock change goes through `Gateway._change_clock`, which wakes the triggers when the change kept the clock's generation, answers the held watches and waits for the followers. Its followers, the computers on it (`CLOCK_FOLLOWER_TOKENS`, each child whose card serves `clock/v1` `sync_time`, by its read token, its control token for now), long-poll `/clock/watch` naming the version they show, each watch held up to the card's `hold_s` (`env/gateway/clock_followers.py`), and a write given `wait_followers_s` waits for each, answering 504 naming those behind, but not for one that hasn't watched for longer than a hold and a reconnect (`ClockFollowers.QUIET_S`), which it names at once with when each last watched; a restarted gateway is unarmed. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and also in the universe's zone, the one `sync_env_clock`'s `timezone` armed it in, or in an IANA `time_zone` it's given; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. | | `artifact/` | Immutable versioned artifacts; every `put()` creates a new version. The document goes to the document store, binary payloads to the object store. Every artifact but an environment universe, whose `metadata` key holds its legacy metadata file ids, carries a `metadata` dict outside its versioned fields, changed by compare-and-swap (`merge_metadata`). Built-ins: `CliArtifact` (`cli`), `ComputerArtifact` (`computer`, a base computer: its architectures, OS, session user, settings and container form, whose image `put` reads from its registry into `metadata.validated_images`), `FileArtifact` (`file`), `FileArtifactUniverse` (`file_artifact_universe`), `DockerImageArtifact` (`docker_image`), `EnvironmentArtifact` (`environment`: its environment's files by path, stored as the one wrapped `file_artifact_ref` every environment artifact had when it has one file at its filename, else as `file_artifact_refs`; `get_file_artifacts()` reads either, and a load hands each file to the env's `data/add` with its path, which a pre-v1 load, taking one file, refuses), `EnvironmentUniverseArtifact` (`environment_universe`), `SkillArtifact` (`skill`); `[artifacts] type_aliases` maps another stored spelling of a type onto one of them. A `DockerImageArtifact` holds a tar.gz of its image, or, written by `put_ref` (`--image-ref` on `env mcp-server put` and `a2a-agent put`), only a registry reference with its tag pinned to a digest, which sandboxes pull, or a published image's tag with its `image_digest` and its build context, which sandboxes pull by the digest and build from the context when they can't (the infra images, and a bundle agent built from a shipped agent's sources, `Plan.published`), or, written by `put_context`, only its build context, which a VM sandbox builds before running it (`VmSandbox.load_docker_images`) and Modal builds before creating its container (`ModalSandboxProvider.prepare_image`, outside a fallback chain's deadline); anything else that runs images by name (a container on the local provider, a lone server on any provider but Modal) refuses such an image. A build context (`utils/docker/context.py`) is what `docker build` would send, `.dockerignore` applied, written so equal files give equal bytes and an equal `source_digest`. | | `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. Its `headline_verifiers`, step ids saved only when set, name the grades that are its reward: one is the reward, several a multi-reward over them, none a multi-reward over every grade; it refuses one that names no step of the task, or one named twice. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, among them `start_computer_recording` and `end_computer_recording`, `start_browser_recording` and `end_browser_recording` (`browser-recording/v1`: one recording per context, a lease's or the default one's, with a video of the tab its agent is on, its context's log, the browser's network log as HAR and opt-in frames keyed to the action log, the contexts its agents make recorded with it, one universe per recording, anything missing failing the run), and more). Every built-in verifier is on the `VerifierTaskStep` base (`verifiers/verifier.py`): it implements `verify` and returns a `Verification`, and the base writes the step's named grade to `context.metadata["grades"][]` (score, `min_passing_score`, `passed`, `verifier_status` graded, skipped or failed, criteria rows, type, version, and `verifier_settings`, the values the run graded with, its overrides applied and named in `overridden`, with `scoring_rules`) beside the legacy `verifications` entries every reader still takes; an exception from `verify` or a `Verification` with an `error`, as a timed-out `run_container_unit_tests_verifier`, is a failed grade, and `rubrics_verifier` still writes its `-trajectory-mistakes` entry itself. On an env whose card serves computer-exec/v1 (a computer), `run_code` with `env_id` and `run_container_unit_tests_verifier` with `env_id` run inside it as its session user through the driver: files staged with `data/add` and read with `data/get`, commands run with `run`; `env_outcome_verifier` hands a `verify` that takes an `env_url` keyword the env's address. A computer's framework-only routes (everything but MCP and the card) take the per-deploy control token its provider boots it with (`DeployedEnv.control_token`, `AGENTENV_CONTROL_TOKEN`), a multi env's computers the multi env's own, which its gateway passes on only on their `/svc` routes and its clock watch (`CONTROL_TOKEN_CHILDREN`); framework calls send `deployed.control_headers`, `Task.run` runs every step under `protocol_v1.authorized` with each deployed env's (a `verify` included), and a `run_code` script inside the computer gets `input["env"]["headers"]`. `teardown_run` and the `teardown_sandboxes` step share one core (`task/teardown.py` `tear_down`): the recordings the run left open end with fresh grants, each env still up closes (`Env.close`; a computer env ends any recording still open), what stayed open is registered from its namespace, then every sandbox is terminated by id. A run whose teardown leaves sandboxes up, stopped by a second signal or failing, keeps on its instance the context it ended with (`record_teardown`), and `agent-env run` and `task run --keep` name `agent-env task teardown --instance `, which tears down what that context still records up, a running one only with `--force`. `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). | | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB, DynamoDB, Firestore with MongoDB compatibility), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. `store/routing.py` is the local namespaces, `@local` and `@hf` (`is_local_id`), each with a documents file of its own: while the CLI or a `build_app` server runs, a local id and what a local task's run records go to local stores beside the configured ones, and that run's writes to the configured stores are refused. A new backend must pass the conformance kits in `tst/store/`. | diff --git a/src/agent_env/bundle/authoring.py b/src/agent_env/bundle/authoring.py index deee849e2..bbb285088 100644 --- a/src/agent_env/bundle/authoring.py +++ b/src/agent_env/bundle/authoring.py @@ -51,6 +51,8 @@ class AuthoringContext: entry: BundleEntry #: How writing a computer env composes it. composing: Composing = Composing() + #: Whether writing a computer env composes it at all: not one no task of the run or registration deploys. + compose: bool = True @property def id(self) -> str: diff --git a/src/agent_env/bundle/materialize.py b/src/agent_env/bundle/materialize.py index 16211a6fe..d86ed9636 100644 --- a/src/agent_env/bundle/materialize.py +++ b/src/agent_env/bundle/materialize.py @@ -48,7 +48,7 @@ from .ledger import Ledger, locked_ids, materializing from .parse import BundleError, BundleKind from .plan import Plan, Write, artifact_writer, env_writer, unpinned_store_refs -from .resolve import BuiltImage, BundleImage, ReferencedImage, build_step +from .resolve import BuiltImage, BundleImage, ReferencedImage, ResolvedEntry, build_step #: How much of each build's reason the problem naming a computer env no deploy can boot quotes: a failed build's reason #: holds the build's error, which can run to pages. @@ -112,10 +112,11 @@ def materialize( Each write gets the version it would reuse or the store's next, which another run can take first. A step reading what the run would write first isn't preflighted, since the store doesn't hold it yet. - Each computer env written is composed as ``composing`` says, and so is what one reused has left, as ``agent-env - env build`` does; ``on_progress``, when given, is told how each goes, a line at a time with its write. With - ``bootable``, as a registration has it, a computer env written or reused whose compose left no build a deploy can - boot stops it once every entity is written, before any task or eval, raising BundleError naming each such env. + Each computer env written that the run or registration composes (``_composed_envs``) is composed as ``composing`` + says, and so is what one reused has left, as ``agent-env env build`` does; any other is written uncomposed and + composes at its first deploy. ``on_progress``, when given, is told how each goes, a line at a time with its write. + With ``bootable``, as a registration has it, a computer env written or reused whose compose left no build a deploy + can boot stops it once every entity is written, before any task or eval, raising BundleError naming each such env. ``ledger`` keeps the rows, this run's in the ``@local`` namespace's store unless given; it also says whether each task and eval names the versions it reads (``Ledger.pins_references``) and whether each image the bundle builds is @@ -130,6 +131,10 @@ def materialize( evals = [write for write in plan.writes if write.kind is BundleKind.EVAL] done: dict[tuple[str, str], Materialized] = {} left: list[str] = [] + composed = _composed_envs(plan) + + def composes(write: Write) -> bool: + return composed is None or write.id in composed unbootable: list[str] = [] def through_ledger(write: Write, write_fn: Callable[[], int]) -> None: @@ -154,13 +159,13 @@ def through_ledger(write: Write, write_fn: Callable[[], int]) -> None: through_ledger(write, lambda: _write_pushed_image(plan, write, on_build)) else: through_ledger(write, lambda: _write_entity(plan, write, here, on_build, - _composing(composing, on_progress, write))) + _composing(composing, on_progress, write), composes(write))) if here and done[_key(write)].reused: _build_reused(plan, write, done[_key(write)].version, on_build) - if write.kind is BundleKind.ENV and done[_key(write)].reused and _compose_reused( + if write.kind is BundleKind.ENV and composes(write) and done[_key(write)].reused and _compose_reused( write, done[_key(write)].version, _composing(composing, on_progress, write), dry_run): left.append(write.id) - if bootable and not dry_run and write.kind is BundleKind.ENV and ( + if bootable and not dry_run and write.kind is BundleKind.ENV and composes(write) and ( why := _unbootable(plan, write, done[_key(write)].version, composing)): unbootable.append(why) if unbootable: @@ -192,7 +197,7 @@ def _composing(composing: Composing, on_progress: Callable[[Write, str], None] | def _write_entity(plan: Plan, write: Write, here: bool, on_build: Callable[[Write], None] | None, - composing: Composing) -> int: + composing: Composing, compose: bool = True) -> int: if isinstance(write.source, ReferencedImage): return _write_referenced_image(plan, write) if isinstance(write.source, BuiltImage): @@ -200,7 +205,7 @@ def _write_entity(plan: Plan, write: Write, here: bool, on_build: Callable[[Writ return _write_published_image(plan, write) return _write_build_context(plan, write, here, on_build) if write.kind is BundleKind.ENV: - return _write_env(plan, write, composing) + return _write_env(plan, write, composing, compose) return _WRITERS[write.kind](plan, write) @@ -374,11 +379,34 @@ def _unbootable(plan: Plan, write: Write, version: int, composing: Composing) -> f"{write.id} --version {version}{how}, and then register again") -def _write_env(plan: Plan, write: Write, composing: Composing = Composing()) -> int: +def _write_env(plan: Plan, write: Write, composing: Composing = Composing(), compose: bool = True) -> int: entry = write.source.entry cls = get_env_registry()[entry.type] return cls.from_toml(_pinned(plan, write, cls.toml_refs), - AuthoringContext(plan.bundle_of(entry), entry, composing=composing)).version + AuthoringContext(plan.bundle_of(entry), entry, composing=composing, compose=compose)).version + + +def _composed_envs(plan: Plan) -> frozenset[str] | None: + """The envs a run or registration of ``plan`` composes: each its tasks deploy, and each those hold but not a + computer env's own apps and driver, which run inside it; and each it pulls in from an installed bundle, whose one + write every bundle shares. None, every env, for a plan with no tasks, which writes its envs for others to deploy, + each alone.""" + tasks = [write.source for write in plan.writes if write.kind is BundleKind.TASK] + if not tasks: + return None + envs = {write.id: write.source for write in plan.writes + if write.kind is BundleKind.ENV and isinstance(write.source, ResolvedEntry)} + named = [ref.id for task in tasks for ref in task.references if ref.kind is EntityKind.ENV] + deployed: set[str] = set() + while named: + if (env_id := named.pop()) in deployed: + continue + deployed.add(env_id) + source = envs.get(env_id) + cls = get_env_registry().get(source.entry.type) if source is not None else None + if source is not None and not (cls is not None and issubclass(cls, ComputerEnv)): + named += [ref.id for ref in source.references if ref.kind is EntityKind.ENV] + return frozenset(deployed | {env_id for env_id, source in envs.items() if source.entry in plan.origins}) def _pinned(plan: Plan, write: Write, refs: tuple[EntityRef, ...]) -> Any: diff --git a/src/agent_env/env/envs/computer/app_env.py b/src/agent_env/env/envs/computer/app_env.py index 52d059a3c..4144b054a 100644 --- a/src/agent_env/env/envs/computer/app_env.py +++ b/src/agent_env/env/envs/computer/app_env.py @@ -168,7 +168,7 @@ def from_toml(cls, data: dict, ctx: AuthoringContext) -> ComputerAppEnv: return cls.put(id=ctx.id, environment_name=fields["environment_name"], module=fields["module"], files=universe, declaration=declaration, computer=computer, driver=driver, build_settings=fields.get("build_settings"), settings=fields.get("settings"), - composing=ctx.composing) + composing=ctx.composing, compose=ctx.compose) except ValueError as e: ctx.refuse([str(e)]) diff --git a/src/agent_env/env/envs/computer/env.py b/src/agent_env/env/envs/computer/env.py index 2b9dd7fbf..dd79b4c67 100644 --- a/src/agent_env/env/envs/computer/env.py +++ b/src/agent_env/env/envs/computer/env.py @@ -165,7 +165,8 @@ def from_toml(cls, data: dict, ctx: AuthoringContext) -> ComputerEnv: ComputerDriverEnv) fields["apps"], fields["app_settings"] = _listed_apps(fields.get("apps", []), ctx) try: - return cls.put(id=ctx.id, computer=base, driver=driver, composing=ctx.composing, **fields) + return cls.put(id=ctx.id, computer=base, driver=driver, composing=ctx.composing, compose=ctx.compose, + **fields) except ValueError as e: ctx.refuse([str(e)]) diff --git a/tst/unit/bundle/computer_env_folder_test.py b/tst/unit/bundle/computer_env_folder_test.py index ea814bf53..60e2fa210 100644 --- a/tst/unit/bundle/computer_env_folder_test.py +++ b/tst/unit/bundle/computer_env_folder_test.py @@ -245,6 +245,64 @@ def test_a_computer_env_folder_that_declares_what_cant_compose_is_refused_naming assert composed == [] +#: The two app folders, each naming the computer and driver it deploys on alone. +APPS_ALONE = {path: text + 'computer = "ubuntu-24.04"\ndriver = "reference"\n' if path.endswith("env.toml") else text + for path, text in APPS.items()} + + +def _composed_ids(composed) -> list[str]: + return [call[0].rsplit("/", 1)[1] for call in composed] + + +@pytest.mark.parametrize("names, expected, apps", [ + ("desktop", ["desktop"], ("mail", "notes")), ("mail", ["mail"], ("mail",)), +], ids=["the-desktop", "an-app-alone"]) +def test_a_run_composes_only_the_computer_envs_its_tasks_deploy_not_the_apps_on_one(tmp_path, composed, names, + expected, apps): + """An app that names its own computer and driver is a computer too, which no task deploys while it's on a + desktop; a task that names it alone does deploy it.""" + root = layout(tmp_path / "desk", {"envs/desktop/env.toml": WITH_APPS, **APPS_ALONE, "tasks/t.json": json.dumps([ + {"id": "env", "type": "deploy_env", "env_id": names}])}) + + materialize(_plan(root), build_here=local_builds(_plan(root))) + + id_root = _plan(root).bundle.bundle.id_root + assert _composed_ids(composed) == expected + assert all(Env.get(f"{id_root}/{app}").computer is not None for app in apps) + + +def test_a_computer_a_multi_env_holds_is_deployed_with_it_but_not_that_computers_apps(tmp_path, composed): + root = layout(tmp_path / "desk", {"envs/desktop/env.toml": WITH_APPS, **APPS_ALONE, + "envs/office/env.toml": 'type = "multi"\ncomputer_envs = ["desktop"]\n', + "tasks/t.json": json.dumps([{"id": "env", "type": "deploy_env", + "env_id": "office"}])}) + + materialize(_plan(root), build_here=local_builds(_plan(root))) + + assert _composed_ids(composed) == ["desktop"] + + +def test_a_bundle_with_no_tasks_composes_every_computer_env_it_writes_each_deployable_alone(tmp_path, composed): + root = layout(tmp_path / "apps", APPS_ALONE) + resolved = resolve_bundle(parse_bundle(root)) + + materialize(plan_entities(resolved, [entry.id for entry in resolved.bundle.entries], installed=installed_bundles)) + + assert sorted(_composed_ids(composed)) == ["mail", "notes"] + + +def test_a_reused_app_no_task_deploys_is_not_composed_again_nor_said_to_be(tmp_path, composed): + root = layout(tmp_path / "desk", {"envs/desktop/env.toml": WITH_APPS, **APPS_ALONE, "tasks/t.json": json.dumps([ + {"id": "env", "type": "deploy_env", "env_id": "desktop"}])}) + materialize(_plan(root), build_here=local_builds(_plan(root))) + told = [] + + dry = materialize(_plan(root), dry_run=True, + on_progress=lambda write, line: told.append(write.id.rsplit("/", 1)[1])) + + assert set(told) <= {"desktop"} and all(not name.endswith(("mail", "notes")) for name in dry.composing) + + def _office(root, computers): """A bundle whose multi env ``office`` holds the computer env folders ``computers``, by folder.""" return layout(root, {**{f"envs/{folder}/env.toml": toml for folder, toml in computers.items()}, diff --git a/tst/unit/bundle/register_test.py b/tst/unit/bundle/register_test.py index 236893e80..4a45961b9 100644 --- a/tst/unit/bundle/register_test.py +++ b/tst/unit/bundle/register_test.py @@ -44,6 +44,8 @@ from agent_env.task import Task from agent_env.task_step.task_step import TaskStep from tst.unit.bundle._support import DIGEST, layout +from tst.unit.bundle.computer_app_env_test import APP, TOML as APP_TOML +from tst.unit.bundle.computer_driver_env_test import LINUX BOX = {"id": "box", "type": "deploy_sandbox", "sandbox_name": "box", "sandbox_mode": "vm"} # Not a credential: what a remote URL's user info holds in the tests that its name drops, joined in at run time. @@ -543,9 +545,9 @@ def test_the_installed_chromium_registers_over_its_own_driver_though_the_store_h assert Env.get("chromium", versions["chromium"]).driver.version == 3 -def _desktop(home, monkeypatch, compose): +def _desktop(home, monkeypatch, compose, extra=None): """A bundle of a computer env on the shipped base and driver, registered into a shared store as the hub is, under - their aliases, ``compose`` standing in for the composer.""" + their aliases, ``compose`` standing in for the composer; ``extra`` replaces or adds files.""" for module in (register_module, env_store_module): monkeypatch.setattr(module, "shared_store", lambda store: True) bases = {image.ref() for image in COMPUTER_IMAGES} @@ -554,10 +556,28 @@ def _desktop(home, monkeypatch, compose): lambda image, arches, read: computer_artifact_module._unread(arches, "no registry in a test")) monkeypatch.setattr(composer_module, "compose", compose) files = {"envs/desktop/env.toml": 'type = "computer"\ncomputer = "ubuntu-24.04"\ndriver = "reference"\n', - "tasks/t.json": json.dumps([{"id": "env", "type": "deploy_env", "env_id": "desktop"}])} + "tasks/t.json": json.dumps([{"id": "env", "type": "deploy_env", "env_id": "desktop"}]), **(extra or {})} return layout(home / "desk", files) +def test_a_registration_composes_the_desktop_its_task_deploys_and_not_each_app_on_it_alone(home, composed, monkeypatch): + """Each app names the computer and driver it deploys on alone, so each is a computer too, which no task deploys.""" + apps = {f"envs/{name}/{file}": text for name in ("mail", "notes") for file, text in ( + ("env.toml", APP_TOML + 'computer = "ubuntu-24.04"\ndriver = "reference"\n'), + ("app.py", APP.format(supports=LINUX, skill="skill.md", linux_skill="skill-linux.md")), + ("requirements.txt", "agentenv-framework-protocol\n"), ("skill.md", "# Mail\n"), + ("skill-linux.md", "# Mail on Linux\n"))} + composings = [] + root = _desktop(home, monkeypatch, lambda env, pairs=None, composing=None: composings.append(env.id), { + **apps, "envs/desktop/env.toml": 'type = "computer"\ncomputer = "ubuntu-24.04"\ndriver = "reference"\n' + 'apps = ["mail", "notes"]\n'}) + + registered = register_bundle(root) + + assert composings == ["desk-desktop"] + assert {done.write.id for done in registered.materialization.writes} >= {"desk-mail", "desk-notes", "desk-desktop"} + + def _link(containers=(), disks=()): """A sandbox provider that runs containers of ``containers`` and boots disks of ``disks`` natively.""" link = SimpleNamespace(native_platforms=lambda: frozenset(f"linux/{arch}" for arch in containers), @@ -697,3 +717,17 @@ def dies_after_writing_docs(self, write, needs): assert {done.write.id: (done.version, done.reused) for done in resumed.materialization.writes} == { "triage-docs": (1, True), "triage-greeting": (1, False), "triage-t": (1, False)} + + +def test_an_installed_app_a_registration_pulls_in_is_composed_though_no_task_deploys_it_alone(home, composed, + monkeypatch): + """Every bundle shares the shared store's one write of an installed bundle's env, and a local run can't compose the + store's env at its first deploy, so the registration that writes it leaves it deployable.""" + composings = [] + root = _desktop(home, monkeypatch, lambda env, pairs=None, composing=None: composings.append(env.id), { + "envs/desktop/env.toml": 'type = "computer"\ncomputer = "ubuntu-24.04"\ndriver = "reference"\n' + 'apps = ["chromium"]\n'}) + + register_bundle(root) + + assert sorted(composings) == ["chromium", "desk-desktop"]