Four kinds of agent share the loop, the models and the job folders: browser use over @playwright/mcp, computer use
over Cua Driver, games and embodied text, and rails that answer one question at a
hook of a running agent. The injection guard rail fails closed: a decision error quarantines the tool result too.
| agent | front | what it measures | run |
|---|---|---|---|
allrecipes |
browser | browser use against the chat model: WebVoyager task Allrecipes--0, a vegetarian lasagna with over 100 reviews, 4.5 stars or more, for 6 | s1a run allrecipes --model jev --batch on --headed |
flights |
browser | browser use: wall clock on a Google Flights search | s1a run flights --model jev --batch on --profile-out run.json |
desktop |
tool | computer use: clicks toward --goal until the window shows --expect; the Calculator on macOS or Windows is the example |
s1a run desktop --app Calculator --goal "compute 12 times 7" --expect 84 --execute --clear "All Clear" --model jev --rethink off --episodes 1 |
ticket_router |
tool | correct routes on a seeded batch of 30 labelled tickets, five queues | s1a run ticket_router --model jev --rethink off --episodes 1 |
alfworld |
tool | success on unseen household tasks in text | s1a run alfworld --model jev --rethink on --episodes 12 --stride 11 |
game2048 |
tool | score and largest tile at a move cap | s1a run game2048 --model jev --rethink on --episodes 10 |
millionaire |
tool | winnings on a 15-question quiz ladder | s1a run millionaire --model jev --rethink off --episodes 5 |
blackjack |
tool | payoff per hand (RLCard) | s1a run blackjack --model jev --rethink off --episodes 100 |
injection_guard |
rail | precision and recall on a labelled injection set | s1a run injection_guard |
Every tool agent takes --model jev|laya|cua|llm|random|rule, --rethink on|off, --episodes N, --seed S,
--max-steps and --timeout, and writes a Harbor-shaped job folder under evals/results/<agent>/. A browser agent
takes --model jev|laya|cua|llm and --goal. A rail takes --model jev|laya, the two models that answer noul.
uv run python -m evals.table evals/results aggregates every job folder per eval and model into one table.
Every run prints one JSON object on stdout and nothing else there; s1a-mcp serves the same agents over stdio
with list_agents, run_agent and decide. Flags, exit codes and the job-folder layout:
architecture.md. The extras each agent needs and the keys: CONTRIBUTING.md. The --model values and the
models behind them: architecture.md.
s1a run <agent> --help lists every flag with its default. Beyond the shared ones: flights and allrecipes take --goal,
--batch on|off, --prefetch on|off, --goal-values on|off, --profile-out and --logs-dir; desktop takes
--app, --goal, --expect, --execute, --plan and --clear; ticket_router takes --dataset and
--batch-size; injection_guard takes --labelled-set. The four games take no flag of their own.
allrecipes is the first Allrecipes task of the WebVoyager task set
(He et al., 2024, Apache License 2.0, attribution in NOTICE), verbatim: find a
vegetarian lasagna with more than 100 reviews, a rating of at least 4.5 stars, suitable for 6 people, and answer in
text. The run is headed
because the site answers a headless Chromium with a bot wall. scripts/browser_showcase.sh allrecipes 2 runs both models
twice with a frame after every browser call and renders the pair GIFs from the median run of each model; the numbers
and the replay are in benchmarks.md.
ticket_router routes locally labelled tickets to five queues: logistics, payment, returns, account and human. The
shipped set is 30 tickets in s1a/agents/_data/ticket_router_eval.jsonl, each with a title, a description and an
order status; the label is read by the scorer alone. --dataset points the agent at a
JSONL of your own with the same fields, and --batch-size caps the tickets per episode. The rules text the model
reads is RULES in s1a/agents/ticket_router.py.