Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .github/workflows/tinker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: Tinker contract

# The Tinker helper's episodes on tinker-cookbook itself (tst/tinker): EpisodeEnv played as the cookbook's rollouts step
# a MessageEnv, on the Episode API's fake steps with a stand-in renderer, so nothing samples and no Tinker key is needed,
# on tinker-cookbook as released and on its main branch. tinker-cookbook and CPU torch are installed over the dev
# environment. It runs on pull requests that touch the helper, the Episode API or the tier, weekly, and on demand; it
# isn't a required check.

on:
pull_request:
paths:
- "src/agent_env/training/**"
- "src/agent_env/task/episode/**"
- "src/agent_env/openenv/reward.py"
- "tst/tinker/**"
- "tst/unit/training/**"
- "pyproject.toml"
- "uv.lock"
- ".github/workflows/tinker.yml"
schedule:
- cron: "17 8 * * 1"
workflow_dispatch:

permissions:
contents: read

concurrency:
group: tinker-${{ github.event_name == 'pull_request' && github.ref || github.sha }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

env:
PIP_DISABLE_PIP_VERSION_CHECK: "1"

jobs:
episodes:
name: episodes (tinker-cookbook ${{ matrix.cookbook }})
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
- cookbook: "released"
requirement: "tinker-cookbook"
- cookbook: "main"
requirement: "tinker-cookbook@git+https://github.com/thinking-machines-lab/tinker-cookbook@main"
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Install from uv.lock with the tinker extra
run: |
python -m pip install --quiet -r .github/ci-requirements.txt
uv sync --locked --extra dev --extra tinker --python "$(command -v python)"
- name: Install tinker-cookbook and CPU torch
run: |
uv pip install --python .venv/bin/python ${{ matrix.requirement }} torch \
--extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match
.venv/bin/python -c "from importlib.metadata import version; print('tinker-cookbook', version('tinker-cookbook'), 'tinker', version('tinker'), 'torch', version('torch'))"
- name: Tinker tier
run: .venv/bin/python -m pytest tst/tinker -rA --timeout=900 --junitxml=junit-tinker.xml
- name: No skips
if: always()
run: .venv/bin/python .github/scripts/check_skip_policy.py junit-tinker.xml
15 changes: 14 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,10 @@ Shared:
in-process `Server`.
- `_tools.py`: turns an env's tools into typed, documented Python methods from their JSON Schema, as transformers'
`get_json_schema` reads them.
- `_episodes.py`, what a helper needs without its trainer installed: `typed_arguments` types a tool call's arguments by
the tool's schema (a renderer that writes them as untyped text reads digits as numbers), `next_prompt` writes a later
turn's prompt as every helper's model reads it (a system prompt the turn changes, as text before it), and `RunGroup`
with `TrainingSteps` names the run group a helper files a group's episodes in, `<run>/train|eval/step-<n>`.
- `_chat.py`, the rows and episode the Miles and slime helpers share:
- `write_rows` and its `command` write a server's tasks as `<split>.jsonl` rows of chat `messages` and `metadata`
(the episode's split, index, task id and tools, and the server it plays on with its deadlines and turn cap).
Expand All @@ -170,6 +174,14 @@ Per trainer:
- `trl.py` (`trl` extra): `AgentEnv(url)` over a server `openenv/` makes, with `dataset` (a row per task) and `factory`
(TRL's `environment_factory`: a generated class per toolset whose methods are the tools, a background `reset` and an
owned `get_reward`); `AgentEnv.local(bundle)` serves the bundle in process.
- `tinker.py` (`tinker` extra, installed into tinker-cookbook's environment): `AgentEnvDatasetBuilder` is `rl.train`'s
dataset builder, each rollout an episode played in process on the Episode API rather than through a server, since
Tinker samples itself. The turn's tools are rendered by the cookbook's renderer, each sampled message is recorded before
its calls run, a reply without a call ends the turn, and the reward follows `openenv/reward.py`. A later turn's new
system prompt comes before its prompt (`next_prompt`), a turn that offers other tools than the first fails its
rollout, and a task that gives the agent skills is left out. A rollout the cookbook cuts
is scored as it is cut, which frees its slot under `max_concurrent_episodes`; a failed episode raises for the
cookbook's retry; each group is filed in the run index by `RunGroup`.
- `harness.py` (`harness` extra): trains a bundle's own agents with TRL's harness rollouts (`HarnessRolloutWorker` with
`harness_adapter=None`). `AgentEnvSessionFactory` runs each rollout's task as `agent-env run` does: the solver, the
agent a step prompts, reaches the trainer's vLLM through OpenEnv's capture proxy, which records the tokens TRL trains
Expand Down Expand Up @@ -378,7 +390,8 @@ orchestrators call it the same way. What callers rely on:
in the proxy, which records the turn and checks it against the model calls the caller records. Its
record is tagged `context.metadata["episode"]` and names the original task, so it isn't resumable
with `start_step`. The episode tears down what its run deployed, and an abandoned one is recorded
`cancelled`.
`cancelled`. It records itself in the run index as a runner would (runner `episode`, in the run group
`start(run_group_id=...)` names), so the explorer lists it.

## Tests

Expand Down
8 changes: 7 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help install lint unit-test int-test int-test-fast int-test-slow installer-test clock-test egress-test trl-test nemo-gym-test verl-test skyrl-test miles-test slime-test clean-install-test test clean
.PHONY: help install lint unit-test int-test int-test-fast int-test-slow installer-test clock-test egress-test trl-test nemo-gym-test verl-test skyrl-test miles-test slime-test tinker-test clean-install-test test clean

VENV := .venv
VENV_BIN := $(abspath $(VENV))/bin
Expand Down Expand Up @@ -111,6 +111,12 @@ miles-test: ## Play rows through Miles' rollout and session server (needs Miles,
slime-test: ## Play rows through the slime helper on slime's OpenAI adapter (needs slime, SGLang and Megatron-LM, CPU torch, and the Hub)
$(PYTHON) -m pytest tst/slime --timeout=900 $(PYTEST_ARGS)

# The Tinker tier plays episodes through agent_env.training.tinker on tinker-cookbook itself, with a stand-in renderer,
# so nothing samples. tinker-cookbook and torch aren't in the dev extra: install them first, as
# .github/workflows/tinker.yml does.
tinker-test: ## Play episodes through the Tinker helper on tinker-cookbook itself (needs tinker-cookbook and CPU torch)
$(PYTHON) -m pytest tst/tinker --timeout=900 $(PYTEST_ARGS)

# The clean-install gate CI runs, on the same Python. The script gives every step an allowlisted
# environment, so nothing in this shell reaches the install it checks. A failed run keeps its folder.
clean-install-test: ## Build both distributions, install them clean from public PyPI and run hello twice (needs python3.11, uv, git)
Expand Down
7 changes: 6 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,11 @@ miles = [
slime = [
"agentenv-framework[openenv]",
]
# agent_env.training.tinker: tinker-cookbook's rl.train on a bundle's tasks, each rollout an episode played in process;
# openenv for the reward rules build_app applies. tinker-cookbook, Tinker and torch are the trainer's to pin.
tinker = [
"agentenv-framework[openenv]",
]
# The AWS store backends (S3, Secrets Manager, DynamoDB, ECR), selected by impl pointer in config.toml.
aws = [
"boto3>=1.35.0",
Expand Down Expand Up @@ -152,7 +157,7 @@ coreweave = [
"cwsandbox>=1.19.6,<2", # Reject exec streams that close without an exit status.
]
dev = [
"agentenv-framework[aws,boat,boxd,coreweave,daytona,gcp,harness,openenv,sail,skyrl,tensorlake,tenki,trl,vercel,verl]", # exercise portable optional providers and stores in CI
"agentenv-framework[aws,boat,boxd,coreweave,daytona,gcp,harness,openenv,sail,skyrl,tensorlake,tenki,tinker,trl,vercel,verl]", # exercise portable optional providers and stores in CI
"fastmcp>=3.4,<4", # the standalone-fastmcp mount path of the protocol; <4 keeps mcp 1.x
"Pillow>=10.4", # the protocol's computer extra, whose tests the unit tier runs
"av>=17", # the protocol's computer extra, whose tests the unit tier runs and which the recording tests decode with
Expand Down
12 changes: 11 additions & 1 deletion src/agent_env/explorer/routers/runs.py
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,17 @@ async def start_runs(task_id: EntityId, body: StartRunsRequest | None = None) ->

@router.post("/{task_id}/cancel-run")
async def cancel_run(task_id: EntityId, workflow_id: str = Query(...)) -> dict:
canceled = await get_runner().cancel(workflow_id)
"""Cancel a run of the configured runner. A run another process records, such as an episode a trainer plays, is
refused with 409: only the process running it can stop it, so asking this runner would silently do nothing."""
from agent_env.runner import store as run_store

runner = get_runner()
record = run_store.get_run(workflow_id)
if record is not None and record.runner != runner.type:
raise HTTPException(status_code=409, detail=(
f"run {workflow_id} is run by {record.runner!r}, not by this explorer's {runner.type!r} runner: stop it "
"where it runs (an episode, from the trainer that plays it)"))
canceled = await runner.cancel(workflow_id)
return {"workflow_id": workflow_id, "canceled": canceled}


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1090,7 +1090,15 @@ export function TaskDetailPage({
)}/cancel-run?workflow_id=${encodeURIComponent(workflowId)}`,
{ method: 'POST' },
);
if (!res.ok) throw new Error(`Cancel failed (${res.status})`);
if (!res.ok) {
// A refusal says why (an episode is stopped by the trainer that plays it); show that, not just the status.
const body = (await res.json().catch(() => null)) as { detail?: unknown } | null;
throw new Error(
typeof body?.detail === 'string'
? body.detail
: `Cancel failed (${res.status})`,
);
}
// Cancel endpoint flips Mongo to `cancelled` synchronously; refresh
// so the row transitions without waiting for the next background poll.
fetchRunGroups();
Expand Down
48 changes: 47 additions & 1 deletion src/agent_env/task/episode/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@

import asyncio
import logging
import os
import socket
from collections import deque
from datetime import datetime, timezone
from typing import Any

import httpx
Expand All @@ -18,6 +21,8 @@
from agent_env.a2a_agent import protocol
from agent_env.a2a_agent.a2a_agent import A2AAgent
from agent_env.providers.sandbox_providers.sandbox_provider import sandbox_request_headers_for_url
from agent_env.runner import store as run_store
from agent_env.runner.runner import RunRecord, RunStatus, run_outcome
from agent_env.store.routing import namespace_routing, run_scope
from agent_env.task.episode.proxy import InProcessProxy, ProxyClient, ProxyRefused
from agent_env.task.store import record_task_cancelled
Expand Down Expand Up @@ -249,10 +254,17 @@ async def _drive(self, running: asyncio.Event) -> None:
resolves."""
running.set()
with namespace_routing():
failure: BaseException | None = None
try:
# Before any await, so start() returns with the run listed; a write that fails is the run's error, and
# the finally below still takes the episode out of _live.
_record_run(self._task, self.id, self._context.metadata)
await self._task.run(context=self._context, instance_id=self.id)
self._ran = True
self._notify()
except BaseException as e:
failure = e
raise
finally:
self._tearing_down = True
try:
Expand All @@ -264,7 +276,19 @@ async def _drive(self, running: asyncio.Event) -> None:
if self._abandoned is not None:
await asyncio.to_thread(_record_cancelled, self._task.id, self.id, self._abandoned)
finally:
del _live[self.id]
try:
await asyncio.to_thread(_record_ended, self._task.id, self.id, *self._outcome(failure))
finally:
del _live[self.id]

def _outcome(self, failure: BaseException | None) -> tuple[RunStatus, str | None]:
"""How the run ended, by the rule every runner records a run's end by (``run_outcome``: a step that failed
without failing the task fails the run too), but cancelled when it was abandoned or cancelled."""
if self._abandoned is not None:
return RunStatus.CANCELED, self._abandoned
if isinstance(failure, asyncio.CancelledError):
return RunStatus.CANCELED, "the episode was cancelled"
return run_outcome(self._context, failure)

async def _wait(self, changed: asyncio.Future[None]) -> None:
"""Until the run ends or what was ``changed`` when the caller last looked changes again."""
Expand Down Expand Up @@ -368,6 +392,28 @@ def _episode_of(context: TaskStepContext) -> Episode:
return _live[context.metadata["episode"]["id"]]


def _record_run(task: Task, instance_id: str, metadata: dict[str, Any]) -> None:
"""The episode in the run index as a runner records the runs it accepts, so the explorer lists it, in its run group
when ``start`` was given one; in the local stores for an ``@local`` task, as ``Task.run`` scopes its records."""
now = datetime.now(timezone.utc).isoformat()
with run_scope(task.id):
run_store.insert_run(RunRecord(
run_id=_run_id(instance_id), runner="episode", task_id=task.id, task_version=task.version,
instance_id=instance_id, status=RunStatus.RUNNING, started_at_utc=now,
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Resolved
owner={"host": socket.gethostname(), "pid": os.getpid(), "seen": now},
overrides={"metadata": {key: metadata[key] for key in ("episode", "run_group_id", "seed")
if key in metadata}}))


def _record_ended(task_id: str, instance_id: str, status: RunStatus, error: str | None) -> None:
with run_scope(task_id):
run_store.mark_terminal(_run_id(instance_id), status, error=error)


def _run_id(instance_id: str) -> str:
return f"episode-{instance_id}"


def _record_cancelled(task_id: str, instance_id: str, reason: str) -> None:
"""Marked where the run recorded its instance, as ``Task.run`` scopes it: an ``@local`` task's in the local
stores."""
Expand Down
7 changes: 5 additions & 2 deletions src/agent_env/task/episode/spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,17 @@ def prompts(self) -> tuple[tuple[str, str | None], ...]:
prompts.append((step.prompt, system_prompts[step.agent_name]))
return tuple(prompts)

async def start(self, *, sandbox: str | None = None) -> Episode:
async def start(self, *, sandbox: str | None = None, run_group_id: str | None = None) -> Episode:
"""Start one run of the task, a new episode each call. It returns once the run's record exists, before
anything is deployed; ``next_turn()`` is where the deploys and setup are waited for. ``sandbox`` is the
provider its envs, agents, sandboxes and judge deploy on, as ``agent-env run --sandbox`` sets it. Its id is
its run's, minted as ``agent-env run`` mints one, so what the run names after it stays in its task's
namespace."""
namespace. The run is in the run index, which the explorer lists; ``run_group_id`` groups it with the other
runs given the same id."""
episode_id = new_instance_id(self._copy.id)
metadata: dict[str, Any] = {"episode": {"id": episode_id, "trainer_agents": list(self.trainer_agents)}}
if run_group_id:
metadata["run_group_id"] = run_group_id
if self._seed:
metadata["seed"] = dict(self._seed)
if sandbox:
Expand Down
86 changes: 86 additions & 0 deletions src/agent_env/training/_episodes.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
"""What a trainer's adapter needs without the trainer installed: a tool call's arguments typed by its tool's schema,
a later turn's prompt as the model reads it, and the run group a trainer files its episodes in."""

from __future__ import annotations

import json
from collections.abc import Mapping
from dataclasses import dataclass
from typing import Any, Literal


_JSON_TYPES = {bool: "boolean", int: "integer", float: "number", type(None): "null"}


def typed_arguments(arguments: Any, schema: Mapping[str, Any]) -> Any:
"""``arguments`` with each scalar whose type the tool's schema doesn't allow, but a string, as the string it was
written as. Renderers that write parameters as untyped text (Qwen3.5's ``<parameter=zip>19122</parameter>``) parse
each with ``json.loads``, reading digits as a number; Qwen's own parser types values by the schema, as this does."""
if not isinstance(arguments, dict):
return arguments
properties = schema.get("properties") or {}
typed = dict(arguments)
for name, value in arguments.items():
allowed = _allowed_types(properties.get(name) or {})
kind = _JSON_TYPES.get(type(value))
widened = kind == "integer" and "number" in allowed
if kind is not None and "string" in allowed and kind not in allowed and not widened:
typed[name] = json.dumps(value)
return typed


def next_prompt(prompt: str, system_prompt: str | None, in_effect: str | None) -> tuple[str, str | None]:
"""A later turn's prompt as every trainer's model reads it, and the system prompt in effect after it. A trainer
plays an episode as one conversation, whose system prompt is written once, at its start; so a turn that changes the
system prompt has the new one written before its prompt, as text. A turn that sets none, or the one in effect,
is its prompt alone."""
if not system_prompt or system_prompt == in_effect:
return prompt, in_effect
return f"{system_prompt}\n\n{prompt}", system_prompt


def _allowed_types(schema: Mapping[str, Any]) -> set[str]:
declared = schema.get("type")
allowed = set(declared) if isinstance(declared, list) else {declared} if declared else set()
for option in schema.get("anyOf") or schema.get("oneOf") or []:
allowed |= _allowed_types(option)
return allowed


def run_group_id(run: str, phase: Literal["train", "eval"], step: int) -> str:
"""The run group a trainer files a group of episodes in: its training run, whether they were played to train or
to evaluate, and the training step whose policy played them. Every trainer names its groups so, and the explorer
lists a task's runs by them."""
return f"{run}/{phase}/step-{step:04d}"


class TrainingSteps:
"""The training step a trainer's batches are played at: a training batch's index, and for an evaluation the step
after the last training batch handed out, for a trainer that, as ``rl.train`` does, evaluates a step's policy
before it hands out that step's batch and once more after the last."""

def __init__(self) -> None:
self._last_trained: int | None = None

def trained(self, index: int) -> int:
self._last_trained = index
return index

def evaluated(self) -> int:
return 0 if self._last_trained is None else self._last_trained + 1


@dataclass(frozen=True)
class RunGroup:
"""Where a trainer files a group of episodes: under the training batch ``trained`` names, fixed when the batch is
handed out, or, for an evaluation, under the step it is played at, read each time it is played, since a trainer
may make its evaluation's groups once and play them at every evaluation, as ``rl.train`` does."""

run: str
steps: TrainingSteps
trained: int | None = None

def id(self) -> str:
if self.trained is not None:
return run_group_id(self.run, "train", self.trained)
return run_group_id(self.run, "eval", self.steps.evaluated())
Loading
Loading