Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ and deserialized through a registry.
| `openenv/` | `build_app(bundle)`: the [OpenEnv](https://github.com/openenv/OpenEnv) server for a bundle's tasks, each session playing one episode through the Episode API, with the envs' MCP tools, an added `end_turn` tool and the task's verifier score as the reward. It prepares bundles as `agent-env run` does (`prepare_bundle(played=True)`, without the agent the client plays) and keeps namespace routing on for the app's life. `openenv` comes with the `openenv` extra, a floor with no cap; this package and `training/` are the only ones that import it. Each task's description in the Task API names its `toolset` (the played agent's envs, role, and whether it gets `bash`). With `token`, every request and WebSocket but `/health` must carry `Authorization: Bearer <token>`, for a server others reach. |
| `training/` | One module per outside trainer, each behind its own extra, and the modules they share; [Training helpers](#training-helpers) lists them. |
| `runner/` | The `[runner]` seam: `Runner.submit()` returns a `RunHandle` (`run_id`, `instance_id`) before the run starts; `LocalRunner` is built in. `runner/store.py` is the run index every runner writes and the explorer reads. |
| `bundle/` | A bundle is a folder of entities (`envs/`, `agents/`, `artifacts/`, `skills/`, `tasks/<name>.json`, `evals/<name>.toml`) that runs without a registry: `agent-env run` plans it, writes it into the `@local` namespace (reusing each version whose inputs haven't changed) and runs its evals or tasks; `agent-env run hf://<org>/<repo>[@<revision>][/<path>]` (`bundle/hub.py`) runs one from a Hugging Face Hub dataset, picked from the commit's file list (the path, else `bundles/`, else the root) and downloaded alone with the card into a folder per commit, under ids rooted at `@hf/<org>/<repo>`, which only that path produces, each run's context recording the commit (`metadata["bundle_source"]`); `agent-env bundle init` writes a stored task or eval as one, with every env, agent and artifact it uses in its own folder (each image as its build context, pruned to what its Dockerfile reads and pinned to the Python and npm versions of the stored image, or, given `--image-ref ID=REF`, as that registry image pinned by digest with its context beside it, `image = { ref, dockerfile }`, which a run pulls and builds only when it can't; `task_images` lists the images an import writes, downloading nothing, so an export can push them first) so it runs with no store; `agent-env bundle register` writes a folder into the configured stores under registry ids (`<prefix><name>`), each task and eval naming the versions it reads and each image it builds pushed, through a ledger in the target store so registering again writes only what changed. |
| `bundle/` | A bundle is a folder of entities (`envs/`, `agents/`, `artifacts/`, `skills/`, `tasks/<name>.json`, `evals/<name>.toml`) that runs without a registry: `agent-env run` plans it, writes it into the `@local` namespace (reusing each version whose inputs haven't changed) and runs its evals or tasks; `agent-env run hf://<org>/<repo>[@<revision>][/<path>]` (`bundle/hub.py`) runs one from a Hugging Face Hub dataset, picked from the commit's file list (the path, else `bundles/`, else the root) and downloaded alone with the card into a folder per commit, under ids rooted at `@hf/<org>/<repo>`, which only that path produces, each run's context recording the commit (`metadata["bundle_source"]`); `agent-env bundle init` writes a stored task or eval as one, with every env, agent and artifact it uses in its own folder (each image as its build context, pruned to what its Dockerfile reads and pinned to the Python and npm versions of the stored image, or, given `--image-ref ID=REF`, as that registry image pinned by digest with its context beside it, `image = { ref, dockerfile }`, which a run pulls and builds only when it can't; `task_images` lists the images an import writes, downloading nothing, so an export can push them first) so it runs with no store; `agent-env bundle register` writes a folder into the configured stores under registry ids (`<prefix><name>`, and an installed bundle's agent, env or artifact a reference pulls in under its alias, an env only into a shared store), each task and eval naming the versions it reads and each image it builds pushed, through a ledger in the target store so registering again writes only what changed. |
| `eval/` | `Eval` (`eval`), a versioned list of task references, and its store. |
| `plugins/` | Entry-point plugins (see Extension points): the group names, `inventory()`, which `agent-env plugin list/show/check` report, `load_failures` and `settings`. The top level is public; the underscored modules are internal. |
| `explorer/` | Optional local web UI: `agent-env up`, binds loopback `:8234`. |
Expand Down Expand Up @@ -273,13 +273,13 @@ Out-of-tree code plugs in either through `config.toml`, as a `module:Class` poin
in a task's steps, so a MultiEnv's servers or a computer env's driver may name one, and writes the entry naming the
env's id, the entry pulled in from another bundle as well as the bundle's own. The env `Env.get` gives for an alias answers to the alias's
id, so a deployment records, and steps look it up by, the id the task names; its metadata writes go to the env the
alias names (`Env.stored_id`). An alias names its env only where namespace routing is on. There an env store refuses
to create an env under an installed env alias's id (`EnvStore.put_document`), so a deployment recorded under an alias
always reloads the alias's env; an env stored under such an id before keeps taking versions. A service, with routing
off, resolves no alias, so it may hold an env under an alias's id, as the hub holds `chromium` for the tasks that name
it. That env shadows the alias wherever that store is read, routing on or off (`Env.get` and the planner read the
store first), so a deployment recorded under the alias there before it was written reattaches to it from then on, or
isn't found when it lacks that deployment's version.
alias names (`Env.stored_id`). An alias names its env only where namespace routing is on. This machine's own env store
refuses to create an env under an installed env alias's id (`EnvStore.put_document`), so a deployment its runs record
under an alias always reloads the alias's env; an env stored under such an id before keeps taking versions. A shared
store (`store.routing.shared_store`), such as the hub's, may hold one, as the hub holds `chromium` for the tasks that
name it, which `bundle register` writes. That env shadows the alias wherever that store is read, routing on or off
(`Env.get` and the planner read the store first), so a deployment recorded under the alias there before it was written
reattaches to it from then on, or isn't found when it lacks that deployment's version.
- `aliases.toml` gives artifacts aliases too (`[artifacts] "<folder>" = ["<alias>", ...]`; the `computers` bundle
names its base `ubuntu-24.04`), resolved by the planner wherever env aliases are, as a computer env's `computer` names
its base. `Artifact.get` reads no alias: the planner rewrites every alias an entry or a task names to its id. An alias may
Expand Down
42 changes: 27 additions & 15 deletions src/agent_env/bundle/register.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,13 @@

from agent_env.config import get_config
from agent_env.entity.refs import RefRole, ref_sites
from agent_env.plugins._bundles import AGENT_ALIASES
from agent_env.plugins._bundles import AGENT_ALIASES, ARTIFACT_ALIASES, ENV_ALIASES
from agent_env.env.envs.computer.utils.composer import Composing
from agent_env.providers import build_sandbox_provider
from agent_env.store.document_store import DocumentStore, LocalSqliteDocumentStore
from agent_env.store.ids import LOCAL_PREFIX, is_local_id
from agent_env.store.image_store.oci_registry_credentials import is_loopback_host
from agent_env.store.routing import configured_store, namespace_routing
from agent_env.store.routing import configured_store, namespace_routing, shared_store
from agent_env.task_step.context import context_path_segments, renamed_context_path
from agent_env.task_step.registry import get_task_step_registry

Expand Down Expand Up @@ -106,7 +106,7 @@ def register_bundle(
shared = shared_store(target)
identity, commit, dirty = bundle_identity(bundle.root)
ledger = RegistrationLedger(target, registered, identity, commit, dirty)
problems = list(_unregistrable(registered, shared))
problems = [*_aliased_envs_refused(plan, ids, shared), *_unregistrable(registered, shared)]
if problems:
raise BundleError(problems)
requires = tuple(declare(resolved, declaring or declaring_for(bundle.root, installed=False, dry_run=dry_run)))
Expand All @@ -120,11 +120,6 @@ def register_bundle(
return Registration(registered, materialization, ids, tuple(_floating(registered)), identity, shared, requires)


def shared_store(store: DocumentStore) -> bool:
"""Whether ``store`` is other than this machine's own, so others read what's written to it."""
return not isinstance(store, LocalSqliteDocumentStore)


def registered_line(plan: Plan, done: Materialized) -> str:
"""A write, as a registration names it: where it's written from, the registry id and version, and why."""
why = "unchanged" if done.reused else "; ".join(done.reasons)
Expand Down Expand Up @@ -159,10 +154,10 @@ def registry_ids(plan: Plan, prefix: str) -> tuple[dict[str, str], list[str]]:
"""The registry id each ``@local`` id the plan writes takes, and each problem giving them.

The bundle's own entry is ``<prefix><name>``, its name lowercased with each run of what an id can't hold made
``-``. An installed bundle's agent pulled in by a reference takes the first alias its bundle gives it, which the
reference named it by; any other entity of an installed bundle has no registry id, and an env's alias can't be
one, since no store env may start on an installed env alias's id. An id derived from another (``<id>__env_image``) keeps its suffix after the
other's registry id, and a task's output is ``<task id>-<name>``."""
``-``. An installed bundle's agent, env or artifact pulled in by a reference takes the first alias its bundle gives
it, which the reference named it by; any other entity of an installed bundle has no registry id. An id derived
from another (``<id>__env_image``) keeps its suffix after the other's registry id, and a task's output is
``<task id>-<name>``."""
ids: dict[str, str] = {}
problems: list[str] = []
for write in plan.writes:
Expand All @@ -172,8 +167,8 @@ def registry_ids(plan: Plan, prefix: str) -> tuple[dict[str, str], list[str]]:
if entry in plan.origins:
alias = _alias(plan.origins[entry], entry)
if alias is None:
problems.append(f"{plan.path(entry)}: comes with an installed bundle, and only an agent with an alias "
"is registered from one; name an entity the store holds instead")
problems.append(f"{plan.path(entry)}: comes with an installed bundle, and only an agent, env or "
"artifact with an alias is registered from one; name an entity the store holds instead")
continue
ids[entry.id] = alias
elif name := _slug(entry.name):
Expand Down Expand Up @@ -207,10 +202,27 @@ def registry_ids(plan: Plan, prefix: str) -> tuple[dict[str, str], list[str]]:


def _alias(bundle: Bundle, entry: BundleEntry) -> str | None:
aliases = bundle.aliases.get(AGENT_ALIASES, {}).get(entry.name, ()) if entry.kind is BundleKind.AGENT else ()
table = _ALIAS_TABLES.get(entry.kind)
aliases = bundle.aliases.get(table, {}).get(entry.name, ()) if table else ()
return aliases[0] if aliases else None


def _aliased_envs_refused(plan: Plan, ids: Mapping[str, str], shared: bool) -> Iterator[str]:
"""For a store that isn't ``shared``, a problem for each installed bundle's env registered under its alias, which
this machine's own env store refuses (``EnvStore.put_document``), found before anything is written."""
if shared:
return
for entry in plan.origins:
if entry.kind is BundleKind.ENV and entry.id in ids:
yield (f"{plan.path(entry)}: comes with an installed bundle, so it registers under its alias "
f"{ids[entry.id]!r}, and this machine's own store holds no env under an alias's id; register it "
"into a shared store")


# The aliases.toml table an installed bundle's entity of each kind takes its registry id from.
_ALIAS_TABLES = {BundleKind.AGENT: AGENT_ALIASES, BundleKind.ENV: ENV_ALIASES, BundleKind.ARTIFACT: ARTIFACT_ALIASES}


def _derived(entity_id: str, ids: Mapping[str, str]) -> str | None:
"""The registry id of ``entity_id`` when it's derived from an id in ``ids``: that id's, with the same suffix."""
base = max((known for known in ids if entity_id.startswith(f"{known}__")), key=len, default=None)
Expand Down
9 changes: 4 additions & 5 deletions src/agent_env/env/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
VersionedEntityStoreCache,
)
from agent_env.store.query import QueryBuilder, to_document_query
from agent_env.store.routing import namespace_routing_enabled
from agent_env.store.routing import shared_store

if TYPE_CHECKING:
from agent_env.env.env import DeployedEnv, Env
Expand Down Expand Up @@ -102,10 +102,9 @@ def put_document(self, env: _E) -> _E:
)
if existing and existing.get("type") != env.type:
raise ValueError(f"Env id '{env.id}' already exists as type '{existing.get('type')}'; cannot put as type '{env.type}'.")
# Where aliases resolve, with namespace routing on, a deployment recorded under an alias reloads its env, so no
# env of the store's own starts on its id there. A service resolves none, so it may hold one, as the hub holds
# `chromium`.
if (not existing and namespace_routing_enabled()
# This machine's runs record a deployment under an alias, which reloads the alias's env, so no env of its own
# store starts on the alias's id. A shared store, such as the hub's, may hold one: a task there names it.
if (not existing and not shared_store(store)
and (named := installed_aliases(table=ENV_ALIASES).target(env.id)) is not None):
raise ValueError(f"env id {env.id!r} is an alias of {named.target_id}, which comes with "
f"{named.declared_by}, and a task naming it deploys that env; give this env another id")
Expand Down
6 changes: 6 additions & 0 deletions src/agent_env/store/routing.py
Original file line number Diff line number Diff line change
Expand Up @@ -579,6 +579,12 @@ def configured_store(store: Any) -> Any:
return store.configured if isinstance(store, (RoutingDocumentStore, LocalRunObjectStore, LocalRunImageStore)) else store


def shared_store(store: DocumentStore) -> bool:
"""Whether the configured store behind ``store`` is other than this machine's own, so others read what's written
to it."""
return not isinstance(configured_store(store), LocalSqliteDocumentStore)


def image_store_holding(ref: str, configured: ImageStore, local: ImageStore) -> ImageStore:
host = registry_host_from_ref(ref)
if isinstance(local, OciRegistryImageStore) and host == normalize_registry_host(local.registry_host):
Expand Down
32 changes: 29 additions & 3 deletions tst/unit/bundle/register_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@
from agent_env.bundle import BundleError
from agent_env.bundle import materialize as materialize_module
from agent_env.bundle import register as register_module
from agent_env.artifact.artifacts import computer as computer_artifact_module
from agent_env.env import store as env_store_module
from agent_env.env.published_images import COMPUTER_IMAGES
from agent_env.bundle.register import REGISTRATIONS, _remote_name, register_bundle
from agent_env.cli import cli
from agent_env.config import get_config
Expand Down Expand Up @@ -474,17 +477,40 @@ def raced(plan, write):
Artifact.get("triage-docs", 2)


def test_an_installed_env_a_task_names_by_its_alias_is_refused_before_anything_is_written(home):
def test_an_installed_env_a_task_names_by_its_alias_is_refused_this_machines_own_store_before_anything_is_written(
home):
files = {"tasks/t.json": json.dumps([{"id": "env", "type": "deploy_env", "env_id": "mail"}])}

with pytest.raises(BundleError) as refused:
register_bundle(layout(home / "triage", files))

assert "comes with an installed bundle, and only an agent with an alias is registered from one" in \
refused.value.problems[0]
assert ("comes with an installed bundle, so it registers under its alias 'mail', and this machine's own store "
"holds no env under an alias's id; register it into a shared store") in refused.value.problems[0]
assert get_config().get_document_store().count("envs", Filter()) == 0


def test_chromium_registers_into_a_shared_store_with_its_driver_and_base_computer_under_their_aliases(
home, composed, monkeypatch):
"""As promotion registers it into the hub, whose tasks name it `chromium`."""
for module in (register_module, env_store_module):
monkeypatch.setattr(module, "shared_store", lambda store: True)
bases = {image.ref() for image in COMPUTER_IMAGES}
monkeypatch.setattr("agent_env.env.published_images._registry_digest", lambda ref: DIGEST if ref in bases else None)
monkeypatch.setattr(computer_artifact_module, "_inspect",
lambda image, arches, read: computer_artifact_module._unread(arches, "no registry in a test"))
files = {"tasks/t.json": json.dumps([{"id": "env", "type": "deploy_env", "env_id": "chromium"}])}

registered = register_bundle(layout(home / "triage", files))

written = {done.write.id for done in registered.materialization.writes}
assert written == {"chromium", "page", "ubuntu-24.04-browser", "ubuntu-24.04-browser__forms_container_image",
"triage-t"}
step = _steps("triage-t")[0]
assert (step["env_id"], step["env_version"]) == ("chromium", 1)
chromium = Env.get("chromium")
assert (chromium.type, chromium.computer.id, chromium.driver.id) == ("computer_app", "ubuntu-24.04-browser", "page")


def test_a_write_that_fails_leaves_no_pending_row(home, monkeypatch):
def failing(*args, **kwargs):
raise RuntimeError("the store went away")
Expand Down
22 changes: 17 additions & 5 deletions tst/unit/env/alias_get_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -127,21 +127,33 @@ def test_an_env_the_store_held_under_an_aliass_id_before_keeps_taking_versions(a
assert _put("mail").version == 2


def test_a_service_holds_an_env_under_an_installed_aliass_id_and_a_task_naming_it_there_deploys_it(bundles):
"""A service runs with namespace routing off and resolves no alias, so it holds the env a task names by it."""
@pytest.fixture
def shared(monkeypatch):
"""The env store is a shared one, such as the hub's, rather than this machine's own."""
monkeypatch.setattr(env_store_module, "shared_store", lambda store: True)


def test_a_shared_store_holds_an_env_under_an_installed_aliass_id_and_a_task_naming_it_there_deploys_it(
bundles, shared):
"""The hub runs with namespace routing off and resolves no alias, so it holds the env a task names by it."""
held = _put("mail")

env = Env.get("mail")

assert (env.id, env.stored_id, env.version) == ("mail", "mail", held.version)


def test_where_aliases_resolve_the_env_a_service_holds_under_an_aliass_id_is_the_one_it_names(bundles):
"""The store's env of an id wins over an alias, so a CLI run against the service's store deploys the same env."""
held = _put("mail")
def test_where_aliases_resolve_the_env_a_shared_store_holds_under_an_aliass_id_is_the_one_it_names(bundles, shared):
"""The store's env of an id wins over an alias, so a CLI run against the hub's store deploys the same env."""
with namespace_routing():
held = _put("mail")
_put(SHIPPED)

env = Env.get("mail")

assert (env.stored_id, env.version) == ("mail", held.version)


def test_this_machines_own_store_refuses_an_aliass_id_with_namespace_routing_off_too(bundles):
with pytest.raises(ValueError, match="env id 'mail' is an alias of"):
_put("mail")
Loading