Repository navigation
feat(always_on_agent_verifier): grade the user's rules (ask first, never, just do, tell after) - #724
Closed
sarah-chitty-1 wants to merge 46 commits into
Closed
sarah-chitty-1 wants to merge 46 commits into
sarah-chitty-1 wants to merge 46 commits into
Conversation
…ored trajectory; record every app change whole turn_events parses a stored turn trajectory (OTel spans, OpenInference's included; Claude Code and Gemini stream-json) into the agent's text and each tool call with its input and result, dropping what every turn or span repeats (tool lists, the conversation so far, signatures, token counts). readable_result unwraps MCP envelopes, decodes base64 email bodies and leaves images out. The OTel tool-span reading is shared with compact_otel_trajectory, which behaves as before. always_on's stored timeline now keeps every change of a changelog read with its whole summary (no 200-change cap per read, no 200-character cut); the run's metadata record is unchanged. render_for_judge cuts a summary for display only and says when an older run didn't record some changes. Fixtures are redacted recordings of a real always_on wake and a Claude Code turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p, and override points for staging and judge config
- judge_timeout_retries (unset: a timeout fails at once, as before): an agent judge
call that runs out of time is tried again; the timed-out attempt's trajectory is
fetched and logged under judge_output_discrepancies (kind judge_timeout), or
carried on JudgeTimeoutError. Running out of time is the poll's own deadline, or a
failed task whose error code says so (`<harness>.timeout`); only a code that
doesn't tell falls back to the message's "timed out after Ns".
- The Claude Code agent fails a turn that ran out of its timeout with error code
claude_code.timeout (was cli_error), as other harnesses name theirs.
- The Claude Code agent's config takes peer_tools (default true): false leaves the
SDK's peer_agents MCP server out of a turn, for an agent that must not message
the task's other agents.
- judge_max_turns (unset: the judge's own): sent as the judge's max_turns.
- stage_trajectory: a subclass may stage the run in the judge's sandbox itself
(StagedTrajectory); None, the default, loads the trajectory as before.
- judge_agent_config: extra judge settings, {} by default.
- Run.firings_read: the firings of every log that could be read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…searches every wake staged as files always_on_staging builds the run's files from its stored timeline and each wake's trajectory, matched by trajectory_url: wakes/wake_NNN.jsonl (what woke the assistant with the user's whole message, every tool call with input and result, its text, every app change, messages, nudges, reads), events.jsonl, triggers.jsonl and index.md (actions and messages with file:line). Any line too long to read whole keeps a preview and moves its long fields to wakes/results/. index.md's write actions are calls of a tool that made one of the wake's app changes (the changes' actors), or, when the apps keep no change log, a guess from the tool name; the index says which. Missing or unreadable trajectories and turns compaction didn't shrink are recorded and logged. The verifier accepts use_trajectory with the agent judge only, writes the files concurrently, adds a prompt note on reading them, records the staging and what each judge call read (judge_reads), and gives the judge only Bash and Read of its own tools (builtin_tools only), with no peer-agents tools (peer_tools false): no helper agents, web, writes or messaging other agents. Its defaults: 2400 s and 400 turns per judge call, one timeout retry, at most 3 batches at once. Without a stored timeline it falls back to rubrics_verifier's per-turn loading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The verifier's timeout retry already recognises a Claude Code timeout from its
message ("timed out after Ns"), so the new claude_code.timeout code isn't
needed. Dropping it leaves this PR with no behaviour change for anyone who
doesn't opt in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ol name; a failed Gemini turn keeps its error message Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en loading the trajectory fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ites in the index, compare trigger times parsed, find wakes in text results; log a count, not every untrimmed trajectory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rded it (Claude Code blocks, a GenAI chat span's thinking blocks) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g line in its wake when the trajectory recorded it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ks on an LLM span Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssage.contents.N.message_content.text), as GPT models are called Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n order for them
always_on_user_rules reads the user sim's persona.permissions (ask_first, never,
just_do, tell_after; a list or {items, then}) as one criterion per rule, and lays out
the run's stored timeline for the judge as rules.md: every app change the assistant
made (C1…), message it sent the user (M1…) and message from the user (U1…), in the
order they happened, each reply tied to the message it answers by the recorded
reply_to link, each line pointing at its line in the staged wake files. rules_block
turns the judge's rows into each rule's result with the changes and messages its
justification cites, and the run's verdict (pass only when every rule passed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s rules as a verdict Opt-in (off by default, needs use_trajectory). The rules join the task's criteria; rules.md is staged beside the run's files and the prompt points at it. The entry gets user_rules: each rule's result, justification, cited changes and messages, and the verdict. A rule that failed or has no answer sets the entry's score to 0; when every rule passed the score is the task's own criteria aggregated as before (task_criteria_score). A run whose timeline can't be read is refused rather than graded without its rules. rubrics_verifier is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re rules; unlogged writes; ungradable runs - ask_first and never decide the verdict (a broken one sets the score to 0); just_do and tell_after lower the score by 0.1 each broken, never below 0. Each rule keeps its pass/fail and says which it counts toward. - Write calls no app change recorded (a write verb and no read word, not the gateway's own tools, not failed, in a wake where no recorded change names that tool) are W1… in rules.md, citable by a rule, and listed per tool in user_rules.unlogged_writes (app, env, count, wake/call refs), with the wakes whose calls couldn't be checked. - A judge call that leaves a rule unanswered is made once more; a rule still unanswered makes the run ungradable: the entry keeps each rule's result (unknown included) with no score, and the step raises UngradableRunError, the error path a judge that can't answer already takes, so the run is left out rather than scored 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rgin past the poll, so the poll's deadline is the only timeout signal Per review: the judge's own limit raced the poll's, so a timeout was also read from a failed task's code or message, which could mistake a tool's "timed out after N" for the judge running out. The judge is now allowed judge_timeout_seconds + 60 when retries are on; the poll always fires first and cancels the task. The code and message matching is gone. Without retries the judge is sent the same timeout as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ard, so an override can fit its settings to that judge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… when its card names its tools that way Per Greptile: a Codex or OpenCode judge names its tools differently and would refuse Claude Code's names, failing the config before grading. The judge's card now decides: its agent-config schema's builtin_tools names. Another harness keeps its own tools (no peer tools either way), and the log says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # src/agent_env/task_step/task_steps/verifiers/always_on_agent_verifier.py # tst/unit/task_step/test_always_on_agent_verifier.py
…emory, helpers)
turn_events: the harness's own model calls (psi.agent.kind = harness, the
compaction summary turn) are no longer the assistant's text or thinking; each
psi.compaction span (at its end, with the summary and how late the next wake
started) and each psi.harness_message span is a note saying it came from the
harness. A helper's events carry subagent (psi.agent.id); events of a turn the
harness asked for carry step (psi.turn, e.g. pre_compaction_save). Spans with
no psi.* labels read as before.
always_on_staging: lines carry step; memory_* and history_* calls are never
actions or possible actions in index.md (they stay in the wake files), and
index.md says so when a run has them; judge_reads no longer counts a wake named
in a staged call's input (history_read's {"wake": N}) as one the judge opened.
Fixtures: two real v3 smoke wakes (Opus 5.5, GPT 6.1), redacted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arness ran A psi.compaction note carries its counts (wakes summarized and kept, how late the next wake started); staging keeps them on the harness_note line, and the wake's block in index.md gets one line saying so, pointing at the note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…der, which wakes a compaction covers - The judge's files note gets five rules for harness lines (harness_note, the pre_compaction_save step, memory_*/history_* calls, subagent lines, notes without "at"), only when a staged trajectory carries psi.agent.* labels (record["agent_labeled_trajectories"]); other runs' prompts are unchanged. - In a compaction wake the save turn, the compaction note and the message after it are staged after "woke" and before the user's message and the wake input, as they happened. - The compaction note and index line name the wakes (from first_kept_wake: "wakes 1–8 are now in the summary; wakes 9–11 kept word for word"; counts when it is missing) and say "this wake started … late". - turn_end and the index header count the save turn's calls apart (save_turn_tool_calls). - The after-compaction message is labelled as re-sent by the harness. - memory_*/history_* calls are internal only in a trajectory with agent labels (TurnEvents.agent_labels); elsewhere such a tool is an app's. - A compaction span with no end time sorts after its save turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…action, context-loss labels, harness disclosure - The v3 part of the judge's files note says a compaction summary is the assistant's own and is graded like any other mistake. - For a v3 run graded per criterion (rubric_binary / rubric_partial), the judge is asked, for each failed criterion, whether it traces to context a compaction lost: optional "context_loss" / "context_loss_evidence" on its row (added to this run's schema only), kept on the row, counted on the entry (entry["context_loss"]); never changing score or pass/fail. A row without the field is counted as not_said. - Every entry gets "assistant_harness": the assistant's agent name, its A2A card's name and version, and the settings its last wake's trace reports (psi.config.*, or the first neutral harness's harness.config.*). Only always_on_agent_verifier changes: it overrides _spec() for the run (a ContextVar set when staging finds agent labels); rubrics_verifier and judge_output_format are untouched, and their rows already keep any field the judge returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…race only when the wakes are staged A run graded without its trajectories gets only its agent name and card in assistant_harness, with no trajectory read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… for the rules still unanswered A judge call that leaves user rules unanswered is made again up to RULE_RETRIES (3) times, each with the same prompt and files but its criteria list cut to the rules still unanswered (a batch's share note dropped, a note that only these are left), stopping once every rule has an answer. The unanswered_user_rules entry keeps, by rule id, how many retries it took, each retry's rules, those it answered and its trajectory, and any still unanswered; those make the run ungradable as before. A rule criterion's user_rule now carries its id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e_reads counts no wake 0 - The judge's files note (every always-on run) asks it to grep events.jsonl before concluding something never happened, and to name that search. - judge_reads took a glob's prefix (wakes/wake_0*.jsonl, wake_00[1-4].jsonl) as wake 0: it now counts only a whole wake file or result file name, and only wakes >= 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # src/agent_env/task_step/task_steps/verifiers/always_on_agent_verifier.py # src/agent_env/task_step/task_steps/verifiers/always_on_staging.py # src/agent_env/task_step/task_steps/verifiers/judge_utils/turn_events.py # tst/unit/task_step/test_always_on_agent_verifier.py # tst/unit/task_step/verifiers/judge_utils/test_turn_events.py # tst/unit/task_step/verifiers/test_always_on_staging.py
Both features in always_on_agent_verifier: the files note takes #716's harness slot with the rules note appended after it; _grade keeps #716's ContextVar (reset in a finally), context_loss and assistant_harness, then judge_reads, and grades the user's rules last, so an ungradable run's entry has its harness and context_loss when UngradableRunError is raised. The rules retry runs inside the graded run's context, so a v3 run's retries keep the context_loss rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1
changed the base branch from
main
to
sarahchitty/fd-5840-reader-v3
November 16, 2026 23:41
…mes left turn_events reads the neutral harness's OpenTelemetry GenAI names: - reasoning from the reasoning parts of gen_ai.output.messages, in order (psi.reasoning is gone); text falls back to its text parts; - a model span of an agent other than gen_ai.main_agent.name that no tool call started is the harness's own (the summary call): its text and reasoning are left out; - a helper's events carry subagent "<agent id or name>@<delegating call's span id>", from nesting under the tool call; - the compaction note comes from the harness.compaction span (summary from its compaction part; numbers as before, plus a failed save turn's or compaction's error), harness-message notes from harness.message, labelled and placed by purpose (save_request, after_summary, failed_wake); - the save turn is the main agent's spans between the save request and the compaction's end. The judge's harness note and context_loss turn on for a trajectory whose spans carry gen_ai.agent.name or that has a harness.compaction or harness.message span. The memory_*/history_* internal-tool rule is removed: memory calls are ordinary tool calls. Harness settings come from harness.config.* only. A failed wake's message is staged before the wake's input, like the save turn and the compaction. Fixtures: the v3 ones are replaced by two redacted v5 fake-model wakes. Non-v5 fixtures give byte-identical events and staged files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… internal, never an action A tool span with harness.tool = true gives its call internal: true; staging keeps it in the wake file with "internal": true and never lists it as an action or possible action in index.md. The judge's note says internal calls count as what the assistant did but change nothing in any app. A tool of the same name without the mark is read as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts too (grep -r wakes/) A text too long for a line is kept whole under wakes/results/; events.jsonl has only its first 300 characters. Before concluding something never happened, the judge greps wakes/ recursively. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A response that leaves out only rule rows keeps the rows it answered; the rules left out are unanswered and go to the rule-only retries (and, still missing, to ungradable), instead of the whole batch being asked again. - grade_user_rules is refused when the task loads with a format that has no row per criterion (trajectory_mistakes), from the constructor and from_dict. - rules.md lists a wake's unlogged writes and messages by call number, a message being its wake's nth message_user call, so a write after a message comes after it. - Tool calls the harness provides itself (internal: memory_add, ...) are never unlogged app writes. - A rule passes only at 1.0 whatever the task's format (pass_threshold 1.0 on rule criteria; the task's own criteria keep partial credit). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t the first wake's notifications Greptile: the first wake's text also carries due notifications and reminders, so a contact's message could read as the user's OK for an ask-first rule. Use PromptResponse.prompt_text; fall back to the wake text only without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Step 2 of FD-5840: the always-on judge grades the user's rules, from the user sim's
persona.permissions(ask first, never, just do, tell after). It's opt-in per task withgrade_user_rules: true(it needsuse_trajectory), and nothing changes for tasks that don't set it.How it works
Events in order, built in code: staging adds
rules.md, which lists the rules and then the run's events in the order they happened:The clock stands still during a turn, so order comes from this list, not from times.
One criterion per rule, judged by the agent judge: which change falls under the rule, and whether it was kept.
Verdict and score:
No half-finished grades:
UngradableRunErroris raised, and each rule keeps its result, includingunknown. The run is then re-graded rather than failed.The record:
entry["user_rules"]holds each rule's kind, wording, result, the judge's reasons, and the changes and messages it cites, resolved to full records. It also holdsunlogged_writes, per tool: writes the apps' change logs didn't record, so we can see which apps need change logging.Shared code
None. The changes are in
always_on_user_rules.py(new),always_on_agent_verifier.pyand one docstring line inalways_on.py.rubrics_verifier.pyis unchanged; its existing hooks are overridden.Tests
tst/unit/task_step/verifiers/test_always_on_user_rules.pyand new cases intest_always_on_agent_verifier.py. They run on the recorded Nina run, plus one realistic added event, and cover:pytest tst/unit/task_step tst/unit/agents/claude_code: 3,522 passed with #716 merged in, including a v3 run graded with user rules and an ungradable v3 run that still records its harness and context-loss info.ruff checkis clean.Not covered: no real judge has run with
grade_user_rulesyet. The unlogged-write test uses a guessed tool name for the order placements (porchbite_place_order), because the Nina run's own records aren't available locally.Stacked on #716 (which is on main). Merge #716 first.
🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; the previous finding is fixed and no new blocking issue was found.Summary
This PR adds opt-in grading of the user's rules, with rule-only retries and separate score adjustments.
Reviews (8) · Last reviewed commit: "always_on_user_rules: the opening reques..." · Reviewed by Greptile