Skip to content

feat(always_on_agent_verifier): with use_trajectory, the agent judge searches every wake staged as files - #680

Merged
sarah-chitty-1 merged 26 commits into
mainfrom
sarahchitty/fd-5840-agent-judge
Nov 16, 2026
Merged

sarah-chitty-1 merged 26 commits into
mainfrom
sarahchitty/fd-5840-agent-judge

Conversation

@sarah-chitty-1

@sarah-chitty-1 sarah-chitty-1 commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

Third of three stacked PRs (Linear FD-5840); it builds on #679. The always-on process judge can now read every tool call of every wake.

The problem.

  • Today always_on_agent_verifier grades from the user's chat and a summary of each wake, and refuses use_trajectory.
  • Agent judges on other long runs were given each turn as one line of up to 1.8 MB. The judge's Read tool refuses lines that long, so it wrote Python to read them, and timeouts were the most common failure.
  • On Claude Code trajectories the user's replies were missing entirely.

What this PR adds.

  • always_on_staging.py builds the run's files from its stored timeline and each wake's trajectory. Each trajectory is matched to its wake by the timeline's trajectory_url, never by list position. The files:
    • wakes/wake_NNN.jsonl, one event per line. First what woke the assistant, with the user's whole message. Then every tool call with its input and result, and the assistant's own text. Then the rest of the wake in time order: every app change, its closing reply, messages to the user, nudges, and the user's reads.
    • events.jsonl: every wake in one file.
    • triggers.jsonl: every trigger firing.
    • index.md: a block per wake with its time, what woke it, the user's message, its write actions and messages (each with file:line), and errors.
    • Any line too long for the Read tool keeps a preview and moves its long fields to wakes/results/. Lines longer than 1,800 characters there are split across lines, nothing dropped (a JSON file that needed it becomes .txt).
    • The agent's thinking, when its trajectory recorded any, is an assistant_thinking line in its wake.
    • A write action is a call of a tool that made one of that wake's app changes, from the apps' change logs. If the apps keep no change log, it falls back to a guess from the tool name, and index.md says which method it used. In a run with change logs, a write-sounding call of a tool no change names (an app without a log) is listed as a "possible action".
    • triggers.jsonl places each firing after the right wake by comparing parsed times; the staging log gives a count of untrimmed trajectories, not the list.
    • Missing or unreadable trajectories, and turns compaction didn't shrink, are recorded and logged.
  • always_on_agent_verifier:
    • Accepts use_trajectory with the agent judge only.
    • Writes the files into the judge's sandbox concurrently.
    • Adds a prompt note on reading them: start at index.md, search, then read with an offset; later actions replace earlier ones; where the user's messages are; cite wake and call numbers; what counts as the assistant's own act.
    • Records the staging and what each judge call read (judge_reads, counting files read through Read, Grep or Bash).
    • Gives the judge only Bash and Read (builtin_tools: {"only": ["Bash", "Read"]}) and no peer-agents tools (peer_tools: false): no helper agents, no web, no writes, no messaging other agents. Its CLI command is --tools Bash,Read --strict-mcp-config with no --mcp-config. The Bash/Read selection is sent only to a judge whose card names its built-in tools that way (Claude Code); a judge that names them otherwise (Codex, OpenCode) keeps its own tools, still with no peer tools, and the log says so.
    • Defaults: 2,400 s and 400 turns per judge call, one timeout retry, at most 3 batches at once. Batch size still comes from the task. The judge itself is given 60 s more than the verifier waits (feat(rubrics_verifier): opt-in judge timeout retries, a judge turn cap, and override points for staging and judge config #679), so the verifier's deadline is the one that ends a call.
    • Without a stored timeline, it falls back to rubrics_verifier's usual per-turn loading.

Re-grade of a real run

Silent Week (psi-ts-rw-silent-week-always-on-ab4d9621782ac6f6: 109 wakes, 3,586 app changes) was re-graded locally. The data was read-only from the hub. The judge was the Claude Code CLI on Opus 5.5 with 1M context, run with the agent's own command builder. The criteria were ML's 8 original process criteria.

  • Cost and time: 36 s, $0.71, one judge call, 8 turns, no timeouts or retries.
  • How it read the files:
    • It searched index.md, made one Python pass over every wake's tool calls, then opened wakes 9, 11, 18, 22, 59, 60, 71, 72, 75 and 80.
    • It didn't open events.jsonl or triggers.jsonl.
    • The judge's tool list had no Agent, Workflow or Monitor.
  • Verdicts: all 8 criteria passed, each with reasons citing wakes.
  • Spot-checks: two reasons were checked against the staged files and are right: W23 ("Mei Sato has approved the endoscopy") and W60 ("never send words as her").

Then the full path: the judge deployed the normal way (the Claude Code agent in local Docker, files staged into its sandbox, settings sent through agent-config and read back), ML's original criteria:

Run Result Cost
Silent Week (109 wakes) 8/8 passed, agreeing with the re-grade above $0.61–0.69
Holiday Rush (psi-ts-craftweb-holiday-rush-always-on-0dc41c6159628433), 4 batches of 4, 3 at once 9/15; the 6 fails are real (no forecast, extra pour days never asked, 4 boxes of wax, the Rae offer, "DONE" with Rae unanswered) $0.81
Fender Bender (psi-ts-rw-fender-bender-always-on-ef40b3a3fb2082d1, cut off at wake 80) 2/6; real fails (no footage, no rental, no repairs, agreed to a recorded statement) $0.49
Silent Week with a 20 s limit and 1 retry Timed out, retried once, both judge trajectories kept, failed with JudgeTimeoutError ~$0

The read-back config showed max_turns: 400 and timeout_seconds: 2400. Those runs exposed peer-agents, web and write tools on the judge, which this PR now removes (above).

Backward compatibility

  • An always_on_agent_verifier without use_trajectory grades from the same prompt as before, with these default changes:
    • 2,400 s and 400 turns per judge call;
    • one timeout retry;
    • at most 3 batches at once;
    • the judge gets only Bash and Read, and no peer-agents tools.
  • A task that sets these explicitly keeps its own values.
  • The converted always-on tasks store judge_timeout_seconds: 1000 and use_agent_judge: false, so they grade as today until their task files turn this on.

How it was tested

  • test_always_on_staging.py, on a redacted recorded timeline and recorded trajectories. It covers:
    • turns matched to wakes by URL;
    • a nudge in its wake;
    • missing and unreadable trajectories flagged;
    • the user's message first;
    • 250 app changes kept whole;
    • long results and long lines moved to side files;
    • decoded email;
    • index.md's file:line pointing at the right line;
    • write actions from the change logs, and the name-guess fallback;
    • events.jsonl and triggers.jsonl;
    • 120 wakes sorted in order;
    • an unknown format flagged;
    • judge_reads across Read, Grep, Bash loops and globs.
  • test_always_on_agent_verifier.py covers an end-to-end grade staged into a fake judge sandbox, the fallback without a timeline, a run that turns the agent judge off being refused, and the defaults.
  • tst/unit/agents/claude_code/test_always_on_judge_config.py checks that the judge's settings are ones the Claude Code agent's card takes.
  • pytest tst/unit/task_step tst/unit/agents/claude_code passes (3,372 passed); ruff check is clean.
  • Full tst/unit: 3 failures, in test_disk_recipe and test_docker_image_github_clone. They also fail on main.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no blocking issue was found.

Summary

Adds searchable files for always-on runs when use_trajectory is enabled.

  • Always-on judges can search every wake's tool calls.
  • Always-on judge calls get longer limits and fewer tools.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Timeline[Stored timeline] --> Stage[Match trajectories to wakes]
    Trajectories[Wake trajectories] --> Stage
    Stage --> Files[Wake files, index, events and triggers]
    Files --> Sandbox[Judge sandbox]
    Sandbox --> Judge[Agent judge]
    Judge --> Results[Grades and reading records]
Loading

Reviews (15) · Last reviewed commit: "Merge main into #680" · Reviewed by Greptile

…ored trajectory; record every app change whole

turn_events parses a stored turn trajectory (OTel spans, OpenInference's included;
Claude Code and Gemini stream-json) into the agent's text and each tool call with its
input and result, dropping what every turn or span repeats (tool lists, the
conversation so far, signatures, token counts). readable_result unwraps MCP
envelopes, decodes base64 email bodies and leaves images out. The OTel tool-span
reading is shared with compact_otel_trajectory, which behaves as before.

always_on's stored timeline now keeps every change of a changelog read with its
whole summary (no 200-change cap per read, no 200-character cut); the run's
metadata record is unchanged. render_for_judge cuts a summary for display only and
says when an older run didn't record some changes.

Fixtures are redacted recordings of a real always_on wake and a Claude Code turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 2 commits November 13, 2026 17:07
…p, and override points for staging and judge config

- judge_timeout_retries (unset: a timeout fails at once, as before): an agent judge
  call that runs out of time is tried again; the timed-out attempt's trajectory is
  fetched and logged under judge_output_discrepancies (kind judge_timeout), or
  carried on JudgeTimeoutError. Running out of time is the poll's own deadline, or a
  failed task whose error code says so (`<harness>.timeout`); only a code that
  doesn't tell falls back to the message's "timed out after Ns".
- The Claude Code agent fails a turn that ran out of its timeout with error code
  claude_code.timeout (was cli_error), as other harnesses name theirs.
- The Claude Code agent's config takes peer_tools (default true): false leaves the
  SDK's peer_agents MCP server out of a turn, for an agent that must not message
  the task's other agents.
- judge_max_turns (unset: the judge's own): sent as the judge's max_turns.
- stage_trajectory: a subclass may stage the run in the judge's sandbox itself
  (StagedTrajectory); None, the default, loads the trajectory as before.
- judge_agent_config: extra judge settings, {} by default.
- Run.firings_read: the firings of every log that could be read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…searches every wake staged as files

always_on_staging builds the run's files from its stored timeline and each wake's
trajectory, matched by trajectory_url: wakes/wake_NNN.jsonl (what woke the
assistant with the user's whole message, every tool call with input and result,
its text, every app change, messages, nudges, reads), events.jsonl, triggers.jsonl
and index.md (actions and messages with file:line). Any line too long to read whole
keeps a preview and moves its long fields to wakes/results/. index.md's write actions
are calls of a tool that made one of the wake's app changes (the changes' actors),
or, when the apps keep no change log, a guess from the tool name; the index says
which. Missing or unreadable trajectories and turns compaction didn't shrink are
recorded and logged.

The verifier accepts use_trajectory with the agent judge only, writes the files
concurrently, adds a prompt note on reading them, records the staging and what each
judge call read (judge_reads), and gives the judge only Bash and Read of its own
tools (builtin_tools only), with no peer-agents tools (peer_tools false): no helper
agents, web, writes or messaging other agents. Its defaults: 2400 s and
400 turns per judge call, one timeout retry, at most 3 batches at once. Without a
stored timeline it falls back to rubrics_verifier's per-turn loading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 force-pushed the sarahchitty/fd-5840-verifier-hooks branch from a87e0b4 to 24c5385 Compare November 14, 2026 01:09
@sarah-chitty-1
sarah-chitty-1 force-pushed the sarahchitty/fd-5840-agent-judge branch from 1e2db49 to 1ccdf35 Compare November 14, 2026 01:09
sarah-chitty-1 and others added 2 commits November 13, 2026 18:12
The verifier's timeout retry already recognises a Claude Code timeout from its
message ("timed out after Ns"), so the new claude_code.timeout code isn't
needed. Dropping it leaves this PR with no behaviour change for anyone who
doesn't opt in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 5 commits November 13, 2026 18:28
…ol name; a failed Gemini turn keeps its error message

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en loading the trajectory fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ites in the index, compare trigger times parsed, find wakes in text results; log a count, not every untrimmed trajectory

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 10 commits November 13, 2026 18:49
…rded it (Claude Code blocks, a GenAI chat span's thinking blocks)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g line in its wake when the trajectory recorded it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ks on an LLM span

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssage.contents.N.message_content.text), as GPT models are called

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 5 commits November 16, 2026 10:07
…rgin past the poll, so the poll's deadline is the only timeout signal

Per review: the judge's own limit raced the poll's, so a timeout was also read from a failed task's code or
message, which could mistake a tool's "timed out after N" for the judge running out. The judge is now allowed
judge_timeout_seconds + 60 when retries are on; the poll always fires first and cancels the task. The code and
message matching is gone. Without retries the judge is sent the same timeout as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ard, so an override can fit its settings to that judge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… when its card names its tools that way

Per Greptile: a Codex or OpenCode judge names its tools differently and would refuse Claude Code's names, failing
the config before grading. The judge's card now decides: its agent-config schema's builtin_tools names. Another
harness keeps its own tools (no peer tools either way), and the log says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 merged commit 8fea030 into main Nov 16, 2026
35 checks passed
@sarah-chitty-1
sarah-chitty-1 deleted the sarahchitty/fd-5840-agent-judge branch November 16, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants