Repository navigation
feat(always_on_agent_verifier): with use_trajectory, the agent judge searches every wake staged as files - #680
Merged
Conversation
…ored trajectory; record every app change whole turn_events parses a stored turn trajectory (OTel spans, OpenInference's included; Claude Code and Gemini stream-json) into the agent's text and each tool call with its input and result, dropping what every turn or span repeats (tool lists, the conversation so far, signatures, token counts). readable_result unwraps MCP envelopes, decodes base64 email bodies and leaves images out. The OTel tool-span reading is shared with compact_otel_trajectory, which behaves as before. always_on's stored timeline now keeps every change of a changelog read with its whole summary (no 200-change cap per read, no 200-character cut); the run's metadata record is unchanged. render_for_judge cuts a summary for display only and says when an older run didn't record some changes. Fixtures are redacted recordings of a real always_on wake and a Claude Code turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p, and override points for staging and judge config
- judge_timeout_retries (unset: a timeout fails at once, as before): an agent judge
call that runs out of time is tried again; the timed-out attempt's trajectory is
fetched and logged under judge_output_discrepancies (kind judge_timeout), or
carried on JudgeTimeoutError. Running out of time is the poll's own deadline, or a
failed task whose error code says so (`<harness>.timeout`); only a code that
doesn't tell falls back to the message's "timed out after Ns".
- The Claude Code agent fails a turn that ran out of its timeout with error code
claude_code.timeout (was cli_error), as other harnesses name theirs.
- The Claude Code agent's config takes peer_tools (default true): false leaves the
SDK's peer_agents MCP server out of a turn, for an agent that must not message
the task's other agents.
- judge_max_turns (unset: the judge's own): sent as the judge's max_turns.
- stage_trajectory: a subclass may stage the run in the judge's sandbox itself
(StagedTrajectory); None, the default, loads the trajectory as before.
- judge_agent_config: extra judge settings, {} by default.
- Run.firings_read: the firings of every log that could be read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…searches every wake staged as files always_on_staging builds the run's files from its stored timeline and each wake's trajectory, matched by trajectory_url: wakes/wake_NNN.jsonl (what woke the assistant with the user's whole message, every tool call with input and result, its text, every app change, messages, nudges, reads), events.jsonl, triggers.jsonl and index.md (actions and messages with file:line). Any line too long to read whole keeps a preview and moves its long fields to wakes/results/. index.md's write actions are calls of a tool that made one of the wake's app changes (the changes' actors), or, when the apps keep no change log, a guess from the tool name; the index says which. Missing or unreadable trajectories and turns compaction didn't shrink are recorded and logged. The verifier accepts use_trajectory with the agent judge only, writes the files concurrently, adds a prompt note on reading them, records the staging and what each judge call read (judge_reads), and gives the judge only Bash and Read of its own tools (builtin_tools only), with no peer-agents tools (peer_tools false): no helper agents, web, writes or messaging other agents. Its defaults: 2400 s and 400 turns per judge call, one timeout retry, at most 3 batches at once. Without a stored timeline it falls back to rubrics_verifier's per-turn loading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
force-pushed
the
sarahchitty/fd-5840-verifier-hooks
branch
from
November 14, 2026 01:09
a87e0b4 to
24c5385
Compare
sarah-chitty-1
force-pushed
the
sarahchitty/fd-5840-agent-judge
branch
from
November 14, 2026 01:09
1e2db49 to
1ccdf35
Compare
The verifier's timeout retry already recognises a Claude Code timeout from its
message ("timed out after Ns"), so the new claude_code.timeout code isn't
needed. Dropping it leaves this PR with no behaviour change for anyone who
doesn't opt in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ol name; a failed Gemini turn keeps its error message Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en loading the trajectory fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ites in the index, compare trigger times parsed, find wakes in text results; log a count, not every untrimmed trajectory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rded it (Claude Code blocks, a GenAI chat span's thinking blocks) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g line in its wake when the trajectory recorded it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ks on an LLM span Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssage.contents.N.message_content.text), as GPT models are called Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale
approved these changes
Nov 16, 2026
…rgin past the poll, so the poll's deadline is the only timeout signal Per review: the judge's own limit raced the poll's, so a timeout was also read from a failed task's code or message, which could mistake a tool's "timed out after N" for the judge running out. The judge is now allowed judge_timeout_seconds + 60 when retries are on; the poll always fires first and cancels the task. The code and message matching is gone. Without retries the judge is sent the same timeout as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ard, so an override can fit its settings to that judge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… when its card names its tools that way Per Greptile: a Codex or OpenCode judge names its tools differently and would refuse Claude Code's names, failing the config before grading. The judge's card now decides: its agent-config schema's builtin_tools names. Another harness keeps its own tools (no peer tools either way), and the log says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Third of three stacked PRs (Linear FD-5840); it builds on #679. The always-on process judge can now read every tool call of every wake.
The problem.
always_on_agent_verifiergrades from the user's chat and a summary of each wake, and refusesuse_trajectory.What this PR adds.
always_on_staging.pybuilds the run's files from its stored timeline and each wake's trajectory. Each trajectory is matched to its wake by the timeline'strajectory_url, never by list position. The files:wakes/wake_NNN.jsonl, one event per line. First what woke the assistant, with the user's whole message. Then every tool call with its input and result, and the assistant's own text. Then the rest of the wake in time order: every app change, its closing reply, messages to the user, nudges, and the user's reads.events.jsonl: every wake in one file.triggers.jsonl: every trigger firing.index.md: a block per wake with its time, what woke it, the user's message, its write actions and messages (each withfile:line), and errors.wakes/results/. Lines longer than 1,800 characters there are split across lines, nothing dropped (a JSON file that needed it becomes.txt).assistant_thinkingline in its wake.index.mdsays which method it used. In a run with change logs, a write-sounding call of a tool no change names (an app without a log) is listed as a "possible action".triggers.jsonlplaces each firing after the right wake by comparing parsed times; the staging log gives a count of untrimmed trajectories, not the list.always_on_agent_verifier:use_trajectorywith the agent judge only.index.md, search, then read with an offset; later actions replace earlier ones; where the user's messages are; cite wake and call numbers; what counts as the assistant's own act.judge_reads, counting files read through Read, Grep or Bash).builtin_tools: {"only": ["Bash", "Read"]}) and no peer-agents tools (peer_tools: false): no helper agents, no web, no writes, no messaging other agents. Its CLI command is--tools Bash,Read --strict-mcp-configwith no--mcp-config. The Bash/Read selection is sent only to a judge whose card names its built-in tools that way (Claude Code); a judge that names them otherwise (Codex, OpenCode) keeps its own tools, still with no peer tools, and the log says so.Re-grade of a real run
Silent Week (
psi-ts-rw-silent-week-always-on-ab4d9621782ac6f6: 109 wakes, 3,586 app changes) was re-graded locally. The data was read-only from the hub. The judge was the Claude Code CLI on Opus 5.5 with 1M context, run with the agent's own command builder. The criteria were ML's 8 original process criteria.index.md, made one Python pass over every wake's tool calls, then opened wakes 9, 11, 18, 22, 59, 60, 71, 72, 75 and 80.events.jsonlortriggers.jsonl.Then the full path: the judge deployed the normal way (the Claude Code agent in local Docker, files staged into its sandbox, settings sent through agent-config and read back), ML's original criteria:
psi-ts-craftweb-holiday-rush-always-on-0dc41c6159628433), 4 batches of 4, 3 at oncepsi-ts-rw-fender-bender-always-on-ef40b3a3fb2082d1, cut off at wake 80)JudgeTimeoutErrorThe read-back config showed
max_turns: 400andtimeout_seconds: 2400. Those runs exposed peer-agents, web and write tools on the judge, which this PR now removes (above).Backward compatibility
use_trajectorygrades from the same prompt as before, with these default changes:judge_timeout_seconds: 1000anduse_agent_judge: false, so they grade as today until their task files turn this on.How it was tested
test_always_on_staging.py, on a redacted recorded timeline and recorded trajectories. It covers:index.md'sfile:linepointing at the right line;events.jsonlandtriggers.jsonl;judge_readsacross Read, Grep, Bash loops and globs.test_always_on_agent_verifier.pycovers an end-to-end grade staged into a fake judge sandbox, the fallback without a timeline, a run that turns the agent judge off being refused, and the defaults.tst/unit/agents/claude_code/test_always_on_judge_config.pychecks that the judge's settings are ones the Claude Code agent's card takes.pytest tst/unit/task_step tst/unit/agents/claude_codepasses (3,372 passed);ruff checkis clean.tst/unit: 3 failures, intest_disk_recipeandtest_docker_image_github_clone. They also fail on main.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no blocking issue was found.Summary
Adds searchable files for always-on runs when
use_trajectoryis enabled.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart LR Timeline[Stored timeline] --> Stage[Match trajectories to wakes] Trajectories[Wake trajectories] --> Stage Stage --> Files[Wake files, index, events and triggers] Files --> Sandbox[Judge sandbox] Sandbox --> Judge[Agent judge] Judge --> Results[Grades and reading records]Reviews (15) · Last reviewed commit: "Merge main into #680" · Reviewed by Greptile