Repository navigation
feat(rubrics_verifier): opt-in judge timeout retries, a judge turn cap, and override points for staging and judge config - #679
Merged
Conversation
…ored trajectory; record every app change whole turn_events parses a stored turn trajectory (OTel spans, OpenInference's included; Claude Code and Gemini stream-json) into the agent's text and each tool call with its input and result, dropping what every turn or span repeats (tool lists, the conversation so far, signatures, token counts). readable_result unwraps MCP envelopes, decodes base64 email bodies and leaves images out. The OTel tool-span reading is shared with compact_otel_trajectory, which behaves as before. always_on's stored timeline now keeps every change of a changelog read with its whole summary (no 200-change cap per read, no 200-character cut); the run's metadata record is unchanged. render_for_judge cuts a summary for display only and says when an older run didn't record some changes. Fixtures are redacted recordings of a real always_on wake and a Claude Code turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p, and override points for staging and judge config
- judge_timeout_retries (unset: a timeout fails at once, as before): an agent judge
call that runs out of time is tried again; the timed-out attempt's trajectory is
fetched and logged under judge_output_discrepancies (kind judge_timeout), or
carried on JudgeTimeoutError. Running out of time is the poll's own deadline, or a
failed task whose error code says so (`<harness>.timeout`); only a code that
doesn't tell falls back to the message's "timed out after Ns".
- The Claude Code agent fails a turn that ran out of its timeout with error code
claude_code.timeout (was cli_error), as other harnesses name theirs.
- The Claude Code agent's config takes peer_tools (default true): false leaves the
SDK's peer_agents MCP server out of a turn, for an agent that must not message
the task's other agents.
- judge_max_turns (unset: the judge's own): sent as the judge's max_turns.
- stage_trajectory: a subclass may stage the run in the judge's sandbox itself
(StagedTrajectory); None, the default, loads the trajectory as before.
- judge_agent_config: extra judge settings, {} by default.
- Run.firings_read: the firings of every log that could be read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
force-pushed
the
sarahchitty/fd-5840-verifier-hooks
branch
from
November 14, 2026 01:09
a87e0b4 to
24c5385
Compare
The verifier's timeout retry already recognises a Claude Code timeout from its
message ("timed out after Ns"), so the new claude_code.timeout code isn't
needed. Dropping it leaves this PR with no behaviour change for anyone who
doesn't opt in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ol name; a failed Gemini turn keeps its error message Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en loading the trajectory fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rded it (Claude Code blocks, a GenAI chat span's thinking blocks) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 14, 2026
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 14, 2026
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 14, 2026
earakely-scale
approved these changes
Nov 16, 2026
…rgin past the poll, so the poll's deadline is the only timeout signal Per review: the judge's own limit raced the poll's, so a timeout was also read from a failed task's code or message, which could mistake a tool's "timed out after N" for the judge running out. The judge is now allowed judge_timeout_seconds + 60 when retries are on; the poll always fires first and cancels the task. The code and message matching is gone. Without retries the judge is sent the same timeout as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 16, 2026
…ard, so an override can fit its settings to that judge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
added a commit
that referenced
this pull request
Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Second of three stacked PRs for always-on grading (Linear FD-5840); it builds on #678. It makes small, opt-in additions to
rubrics_verifierthat the always-on verifier in #680 uses. Every one is off unless a task or subclass turns it on.judge_timeout_retries(unset by default). When an agent judge call runs out of time, it is tried again. The timed-out attempt's trajectory is fetched and logged injudge_output_discrepancies(kind: judge_timeout), so there's a record of what the judge did. If the last attempt also runs out, the error (JudgeTimeoutError) carries that trajectory's link. A batch retries on its own, so the other batches keep going.judge_timeout_seconds+ 60 s, so the poll always fires first: it cancels the judge's task and raisesTimeoutError. The judge's own limit stays behind it as a backstop.failedbefore the deadline is never treated as a timeout, whatever its error code or message says (e.g. a tool's "timed out after 30000ms"). This works the same for every harness.judge_timeout_secondsas before.judge_max_turns(unset by default): sent to the judge asmax_turns.peer_toolssetting (default true, so every other use is unchanged). False leaves the SDK'speer_agentsMCP server out of the turn, so the agent can't message or start other agents. The card lists it.stage_trajectory, an override point. A subclass can stage the run in the judge's sandbox itself and say where (StagedTrajectory). Returning None, the default, loads the trajectory exactly as before.judge_agent_config(judge_agent_card), an override point for extra judge settings, given the judge's card so they can fit that judge;{}by default. It logs any setting the judge's card doesn't take.Run.firings_read(): the trigger firings of every log that could be read.Backward compatibility
rubrics_verifierbehaves as before: a timeout still fails at once, nothing extra is fetched, and the judge config, prompt and trajectory loading are unchanged.peer_tools: false. Its error codes and messages are as before._terminate_auto_deployed_judge, used by both paths.How it was tested
tst/unit/task_step/verifiers/test_rubrics_verifier_judge_timeout_retry.py. It covers:peer_tools: falseleaves thepeer_agentsserver out of the turn.test_rubrics_verifier_staging_cleanup.pychecks that a failed staging terminates the judge the verifier deployed (it fails without the fix).pytest tst/unit/agents/claude_code tst/unit/task_step/verifierspasses at this commit, andtst/unit/task_step tst/unit/agents/claude_codepasses with feat(always_on_agent_verifier): with use_trajectory, the agent judge searches every wake staged as files #680 on top (3,373 passed);ruff checkis clean.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new actionable issues were found.Summary
Adds opt-in judge timeout retries, a turn cap, and hooks for staging trajectories and configuring judges.
Reviews (12) · Last reviewed commit: "refactor(rubrics_verifier): judge_agent_..." · Reviewed by Greptile