Skip to content

feat(rubrics_verifier): opt-in judge timeout retries, a judge turn cap, and override points for staging and judge config - #679

Merged
sarah-chitty-1 merged 14 commits into
mainfrom
sarahchitty/fd-5840-verifier-hooks
Nov 16, 2026
Merged

sarah-chitty-1 merged 14 commits into
mainfrom
sarahchitty/fd-5840-verifier-hooks

Conversation

@sarah-chitty-1

@sarah-chitty-1 sarah-chitty-1 commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

Second of three stacked PRs for always-on grading (Linear FD-5840); it builds on #678. It makes small, opt-in additions to rubrics_verifier that the always-on verifier in #680 uses. Every one is off unless a task or subclass turns it on.

  • judge_timeout_retries (unset by default). When an agent judge call runs out of time, it is tried again. The timed-out attempt's trajectory is fetched and logged in judge_output_discrepancies (kind: judge_timeout), so there's a record of what the judge did. If the last attempt also runs out, the error (JudgeTimeoutError) carries that trajectory's link. A batch retries on its own, so the other batches keep going.
    • "Ran out of time" means only the poll's own deadline. With retries on, the judge is configured with judge_timeout_seconds + 60 s, so the poll always fires first: it cancels the judge's task and raises TimeoutError. The judge's own limit stays behind it as a backstop.
    • A judge task that ends failed before the deadline is never treated as a timeout, whatever its error code or message says (e.g. a tool's "timed out after 30000ms"). This works the same for every harness.
    • Without retries, the judge is sent judge_timeout_seconds as before.
  • judge_max_turns (unset by default): sent to the judge as max_turns.
  • The Claude Code agent's peer_tools setting (default true, so every other use is unchanged). False leaves the SDK's peer_agents MCP server out of the turn, so the agent can't message or start other agents. The card lists it.
  • stage_trajectory, an override point. A subclass can stage the run in the judge's sandbox itself and say where (StagedTrajectory). Returning None, the default, loads the trajectory exactly as before.
  • judge_agent_config(judge_agent_card), an override point for extra judge settings, given the judge's card so they can fit that judge; {} by default. It logs any setting the judge's card doesn't take.
  • Run.firings_read(): the trigger firings of every log that could be read.

Backward compatibility

  • With none of the new settings, rubrics_verifier behaves as before: a timeout still fails at once, nothing extra is fetched, and the judge config, prompt and trajectory loading are unchanged.
  • The Claude Code agent is unchanged unless a caller sets peer_tools: false. Its error codes and messages are as before.
  • One fix that applies to every verifier: if loading the trajectory into the judge's sandbox fails, a judge the verifier deployed itself is now terminated (before, it was only terminated after grading). The existing termination code moved into one helper, _terminate_auto_deployed_judge, used by both paths.

How it was tested

  • New: tst/unit/task_step/verifiers/test_rubrics_verifier_judge_timeout_retry.py. It covers:
    • the default failing at once;
    • a retry that keeps the timed-out trajectory;
    • with retries, the judge is sent the timeout plus the 60 s margin, and without them the timeout as before;
    • a judge task that fails before the deadline is not retried, whatever its message or code says;
    • round-trips and validation;
    • extra config sent only when the judge's card takes it.
  • A Claude Code agent test checks that peer_tools: false leaves the peer_agents server out of the turn.
  • New: test_rubrics_verifier_staging_cleanup.py checks that a failed staging terminates the judge the verifier deployed (it fails without the fix).
  • pytest tst/unit/agents/claude_code tst/unit/task_step/verifiers passes at this commit, and tst/unit/task_step tst/unit/agents/claude_code passes with feat(always_on_agent_verifier): with use_trajectory, the agent judge searches every wake staged as files #680 on top (3,373 passed); ruff check is clean.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new actionable issues were found.

Summary

Adds opt-in judge timeout retries, a turn cap, and hooks for staging trajectories and configuring judges.

  • Agent judges can retry poll timeouts and stop at a chosen turn limit.
  • Verifier subclasses can stage judge input and add judge-specific settings.
  • Always-on run records keep full app-change details and expose readable log firings.
  • A turn reader turns OTel, Claude Code, and Gemini records into readable events.
  • Claude Code turns can leave out peer-agent tools.

Reviews (12) · Last reviewed commit: "refactor(rubrics_verifier): judge_agent_..." · Reviewed by Greptile

…ored trajectory; record every app change whole

turn_events parses a stored turn trajectory (OTel spans, OpenInference's included;
Claude Code and Gemini stream-json) into the agent's text and each tool call with its
input and result, dropping what every turn or span repeats (tool lists, the
conversation so far, signatures, token counts). readable_result unwraps MCP
envelopes, decodes base64 email bodies and leaves images out. The OTel tool-span
reading is shared with compact_otel_trajectory, which behaves as before.

always_on's stored timeline now keeps every change of a changelog read with its
whole summary (no 200-change cap per read, no 200-character cut); the run's
metadata record is unchanged. render_for_judge cuts a summary for display only and
says when an older run didn't record some changes.

Fixtures are redacted recordings of a real always_on wake and a Claude Code turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p, and override points for staging and judge config

- judge_timeout_retries (unset: a timeout fails at once, as before): an agent judge
  call that runs out of time is tried again; the timed-out attempt's trajectory is
  fetched and logged under judge_output_discrepancies (kind judge_timeout), or
  carried on JudgeTimeoutError. Running out of time is the poll's own deadline, or a
  failed task whose error code says so (`<harness>.timeout`); only a code that
  doesn't tell falls back to the message's "timed out after Ns".
- The Claude Code agent fails a turn that ran out of its timeout with error code
  claude_code.timeout (was cli_error), as other harnesses name theirs.
- The Claude Code agent's config takes peer_tools (default true): false leaves the
  SDK's peer_agents MCP server out of a turn, for an agent that must not message
  the task's other agents.
- judge_max_turns (unset: the judge's own): sent as the judge's max_turns.
- stage_trajectory: a subclass may stage the run in the judge's sandbox itself
  (StagedTrajectory); None, the default, loads the trajectory as before.
- judge_agent_config: extra judge settings, {} by default.
- Run.firings_read: the firings of every log that could be read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 force-pushed the sarahchitty/fd-5840-verifier-hooks branch from a87e0b4 to 24c5385 Compare November 14, 2026 01:09
The verifier's timeout retry already recognises a Claude Code timeout from its
message ("timed out after Ns"), so the new claude_code.timeout code isn't
needed. Dropping it leaves this PR with no behaviour change for anyone who
doesn't opt in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 5 commits November 13, 2026 18:28
…ol name; a failed Gemini turn keeps its error message

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en loading the trajectory fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rded it (Claude Code blocks, a GenAI chat span's thinking blocks)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 2 commits November 13, 2026 18:54
…ks on an LLM span

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 2 commits November 13, 2026 19:15
…ssage.contents.N.message_content.text), as GPT models are called

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from sarahchitty/fd-5840-turn-reader to main November 16, 2026 18:03
…rgin past the poll, so the poll's deadline is the only timeout signal

Per review: the judge's own limit raced the poll's, so a timeout was also read from a failed task's code or
message, which could mistake a tool's "timed out after N" for the judge running out. The judge is now allowed
judge_timeout_seconds + 60 when retries are on; the poll always fires first and cancels the task. The code and
message matching is gone. Without retries the judge is sent the same timeout as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 16, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ard, so an override can fit its settings to that judge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 16, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 merged commit 18b8104 into main Nov 16, 2026
35 of 36 checks passed
@sarah-chitty-1
sarah-chitty-1 deleted the sarahchitty/fd-5840-verifier-hooks branch November 16, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants