Repository navigation
feat(always_on_agent_verifier): read harness traces with compaction, memory and helpers (OpenTelemetry GenAI names) - #716
Conversation
…ored trajectory; record every app change whole turn_events parses a stored turn trajectory (OTel spans, OpenInference's included; Claude Code and Gemini stream-json) into the agent's text and each tool call with its input and result, dropping what every turn or span repeats (tool lists, the conversation so far, signatures, token counts). readable_result unwraps MCP envelopes, decodes base64 email bodies and leaves images out. The OTel tool-span reading is shared with compact_otel_trajectory, which behaves as before. always_on's stored timeline now keeps every change of a changelog read with its whole summary (no 200-change cap per read, no 200-character cut); the run's metadata record is unchanged. render_for_judge cuts a summary for display only and says when an older run didn't record some changes. Fixtures are redacted recordings of a real always_on wake and a Claude Code turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p, and override points for staging and judge config
- judge_timeout_retries (unset: a timeout fails at once, as before): an agent judge
call that runs out of time is tried again; the timed-out attempt's trajectory is
fetched and logged under judge_output_discrepancies (kind judge_timeout), or
carried on JudgeTimeoutError. Running out of time is the poll's own deadline, or a
failed task whose error code says so (`<harness>.timeout`); only a code that
doesn't tell falls back to the message's "timed out after Ns".
- The Claude Code agent fails a turn that ran out of its timeout with error code
claude_code.timeout (was cli_error), as other harnesses name theirs.
- The Claude Code agent's config takes peer_tools (default true): false leaves the
SDK's peer_agents MCP server out of a turn, for an agent that must not message
the task's other agents.
- judge_max_turns (unset: the judge's own): sent as the judge's max_turns.
- stage_trajectory: a subclass may stage the run in the judge's sandbox itself
(StagedTrajectory); None, the default, loads the trajectory as before.
- judge_agent_config: extra judge settings, {} by default.
- Run.firings_read: the firings of every log that could be read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…searches every wake staged as files always_on_staging builds the run's files from its stored timeline and each wake's trajectory, matched by trajectory_url: wakes/wake_NNN.jsonl (what woke the assistant with the user's whole message, every tool call with input and result, its text, every app change, messages, nudges, reads), events.jsonl, triggers.jsonl and index.md (actions and messages with file:line). Any line too long to read whole keeps a preview and moves its long fields to wakes/results/. index.md's write actions are calls of a tool that made one of the wake's app changes (the changes' actors), or, when the apps keep no change log, a guess from the tool name; the index says which. Missing or unreadable trajectories and turns compaction didn't shrink are recorded and logged. The verifier accepts use_trajectory with the agent judge only, writes the files concurrently, adds a prompt note on reading them, records the staging and what each judge call read (judge_reads), and gives the judge only Bash and Read of its own tools (builtin_tools only), with no peer-agents tools (peer_tools false): no helper agents, web, writes or messaging other agents. Its defaults: 2400 s and 400 turns per judge call, one timeout retry, at most 3 batches at once. Without a stored timeline it falls back to rubrics_verifier's per-turn loading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The verifier's timeout retry already recognises a Claude Code timeout from its
message ("timed out after Ns"), so the new claude_code.timeout code isn't
needed. Dropping it leaves this PR with no behaviour change for anyone who
doesn't opt in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ol name; a failed Gemini turn keeps its error message Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en loading the trajectory fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ites in the index, compare trigger times parsed, find wakes in text results; log a count, not every untrimmed trajectory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rded it (Claude Code blocks, a GenAI chat span's thinking blocks) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g line in its wake when the trajectory recorded it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ks on an LLM span Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssage.contents.N.message_content.text), as GPT models are called Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rgin past the poll, so the poll's deadline is the only timeout signal Per review: the judge's own limit raced the poll's, so a timeout was also read from a failed task's code or message, which could mistake a tool's "timed out after N" for the judge running out. The judge is now allowed judge_timeout_seconds + 60 when retries are on; the poll always fires first and cancels the task. The code and message matching is gone. Without retries the judge is sent the same timeout as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ard, so an override can fit its settings to that judge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… when its card names its tools that way Per Greptile: a Codex or OpenCode judge names its tools differently and would refuse Claude Code's names, failing the config before grading. The judge's card now decides: its agent-config schema's builtin_tools names. Another harness keeps its own tools (no peer tools either way), and the log says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emory, helpers)
turn_events: the harness's own model calls (psi.agent.kind = harness, the
compaction summary turn) are no longer the assistant's text or thinking; each
psi.compaction span (at its end, with the summary and how late the next wake
started) and each psi.harness_message span is a note saying it came from the
harness. A helper's events carry subagent (psi.agent.id); events of a turn the
harness asked for carry step (psi.turn, e.g. pre_compaction_save). Spans with
no psi.* labels read as before.
always_on_staging: lines carry step; memory_* and history_* calls are never
actions or possible actions in index.md (they stay in the wake files), and
index.md says so when a run has them; judge_reads no longer counts a wake named
in a staged call's input (history_read's {"wake": N}) as one the judge opened.
Fixtures: two real v3 smoke wakes (Opus 5.5, GPT 6.1), redacted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arness ran A psi.compaction note carries its counts (wakes summarized and kept, how late the next wake started); staging keeps them on the harness_note line, and the wake's block in index.md gets one line saying so, pointing at the note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…der, which wakes a compaction covers - The judge's files note gets five rules for harness lines (harness_note, the pre_compaction_save step, memory_*/history_* calls, subagent lines, notes without "at"), only when a staged trajectory carries psi.agent.* labels (record["agent_labeled_trajectories"]); other runs' prompts are unchanged. - In a compaction wake the save turn, the compaction note and the message after it are staged after "woke" and before the user's message and the wake input, as they happened. - The compaction note and index line name the wakes (from first_kept_wake: "wakes 1–8 are now in the summary; wakes 9–11 kept word for word"; counts when it is missing) and say "this wake started … late". - turn_end and the index header count the save turn's calls apart (save_turn_tool_calls). - The after-compaction message is labelled as re-sent by the harness. - memory_*/history_* calls are internal only in a trajectory with agent labels (TurnEvents.agent_labels); elsewhere such a tool is an app's. - A compaction span with no end time sorts after its save turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…action, context-loss labels, harness disclosure - The v3 part of the judge's files note says a compaction summary is the assistant's own and is graded like any other mistake. - For a v3 run graded per criterion (rubric_binary / rubric_partial), the judge is asked, for each failed criterion, whether it traces to context a compaction lost: optional "context_loss" / "context_loss_evidence" on its row (added to this run's schema only), kept on the row, counted on the entry (entry["context_loss"]); never changing score or pass/fail. A row without the field is counted as not_said. - Every entry gets "assistant_harness": the assistant's agent name, its A2A card's name and version, and the settings its last wake's trace reports (psi.config.*, or the first neutral harness's harness.config.*). Only always_on_agent_verifier changes: it overrides _spec() for the run (a ContextVar set when staging finds agent labels); rubrics_verifier and judge_output_format are untouched, and their rows already keep any field the judge returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…race only when the wakes are staged A run graded without its trajectories gets only its agent name and card in assistant_harness, with no trajectory read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e_reads counts no wake 0 - The judge's files note (every always-on run) asks it to grep events.jsonl before concluding something never happened, and to name that search. - judge_reads took a glob's prefix (wakes/wake_0*.jsonl, wake_00[1-4].jsonl) as wake 0: it now counts only a whole wake file or result file name, and only wakes >= 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # src/agent_env/task_step/task_steps/verifiers/always_on_agent_verifier.py # src/agent_env/task_step/task_steps/verifiers/always_on_staging.py # src/agent_env/task_step/task_steps/verifiers/judge_utils/turn_events.py # tst/unit/task_step/test_always_on_agent_verifier.py # tst/unit/task_step/verifiers/judge_utils/test_turn_events.py # tst/unit/task_step/verifiers/test_always_on_staging.py
Both features in always_on_agent_verifier: the files note takes #716's harness slot with the rules note appended after it; _grade keeps #716's ContextVar (reset in a finally), context_loss and assistant_harness, then judge_reads, and grades the user's rules last, so an ungradable run's entry has its harness and context_loss when UngradableRunError is raised. The rules retry runs inside the graded run's context, so a v3 run's retries keep the context_loss rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mes left turn_events reads the neutral harness's OpenTelemetry GenAI names: - reasoning from the reasoning parts of gen_ai.output.messages, in order (psi.reasoning is gone); text falls back to its text parts; - a model span of an agent other than gen_ai.main_agent.name that no tool call started is the harness's own (the summary call): its text and reasoning are left out; - a helper's events carry subagent "<agent id or name>@<delegating call's span id>", from nesting under the tool call; - the compaction note comes from the harness.compaction span (summary from its compaction part; numbers as before, plus a failed save turn's or compaction's error), harness-message notes from harness.message, labelled and placed by purpose (save_request, after_summary, failed_wake); - the save turn is the main agent's spans between the save request and the compaction's end. The judge's harness note and context_loss turn on for a trajectory whose spans carry gen_ai.agent.name or that has a harness.compaction or harness.message span. The memory_*/history_* internal-tool rule is removed: memory calls are ordinary tool calls. Harness settings come from harness.config.* only. A failed wake's message is staged before the wake's input, like the save turn and the compaction. Fixtures: the v3 ones are replaced by two redacted v5 fake-model wakes. Non-v5 fixtures give byte-identical events and staged files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… internal, never an action A tool span with harness.tool = true gives its call internal: true; staging keeps it in the wake file with "internal": true and never lists it as an action or possible action in index.md. The judge's note says internal calls count as what the assistant did but change nothing in any app. A tool of the same name without the mark is read as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts too (grep -r wakes/) A text too long for a line is kept whole under wakes/results/; events.jsonl has only its first 300 characters. Before concluding something never happened, the judge greps wakes/ recursively. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andrewpark-scaleai
left a comment
There was a problem hiding this comment.
lgtm. the context-loss fields only turn on for staged runs of a harness that labels its spans, and the trace reads fail soft. small nits: _parts checks isinstance(messages, list) after iterating it, so a null or numeric gen_ai.output.messages throws and that wake's tool calls drop out of what the judge sees. and use_trajectory runs that don't opt in still get the new absence rule in the prompt, which seems fine but worth knowing.
…tead of dropping the span Review (andrewpark-scaleai): a null or numeric gen_ai.output.messages threw while iterating, so that wake's tool calls dropped out of what the judge sees. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…harness.* labels, never on gen_ai.agent.name alone Review (Edgar): another OpenTelemetry harness that sets the standard agent name got the harness reading, and a hand-off between two agents lost the second agent's words. Now only a trace carrying a harness's own labels (a harness.compaction or harness.message span, or a harness.* attribute) has its summary call left out, its save turn marked, and the notes and fields that go with them. Plain OpenTelemetry traces read as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the harness version, timeline-less runs staged
- A harness.compaction span's written_by goes into the compaction facts and its note. The judge is told whose a
summary's losses are by who wrote it: "assistant" (its own model) is graded as its own; "harness" is not held
against the assistant, and the judge names the loss. A compaction span without written_by (or with another value)
raises TrajectoryFormatError, and the verifier leaves the run ungraded (UngradableRunError naming the wake).
- assistant_harness carries harness.version from the harness.usage span (harness_usage replaces harness_settings).
- A run with no stored timeline (not kept, or unreadable) has its turns read through turn_events and staged by
stage_run by position (one wake file a turn, index.md and the files note say so), so the judge never reads a
raw trajectory.
- No reading for older harness versions: a compaction span is placed by its end time and the save turn ends where the
compaction that follows it ends. The old-harness fixture (turn_otel_wake4.json) is gone; the recorded fake-model
wakes are the v7 shape (turn_otel_v7_compaction_{claude,gpt}.json: written_by and harness.version added).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments Outside DiffThese findings could not be posted inline.
|
… a failed one included Greptile: with no stored timeline nothing recorded each turn's state or closing reply, and the staging dropped the trajectory's final result, so a turn that ended in an error (e.g. a quota error) read as complete. The turn_end line now takes both from the trajectory's last final result: failed with its text on an error, completed otherwise; the index lists a failed turn as an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What and why
The always-on judge has to read runs from a harness that compacts its history and has memory tools. That's the PSI neutral harness from v5, synthetic-pipelines #2048. Read the old way, such a run is misread: every compaction summary shows up as the assistant's own words, and every memory call can be listed as an action on the user's apps.
The harness labels its trace with the OpenTelemetry GenAI conventions plus a few neutral
harness.*names, so this reader has no PSI-only names:git grep 'psi\.'over the verifiers returns nothing. Runs without these labels read exactly as before.What the reader understands
reasoningparts in a model call'sgen_ai.output.messagesgen_ai.agent.nameisn't the main agent (gen_ai.main_agent.*) and isn't nested under a tool call: the summary callharness.compactionspan (itscompactionpart andharness.compaction.*)harness.messagespans, byharness.message.purposeharness.tool = trueinternal, and never an actionsubagent)harness.config.*on theharness.usagespanassistant_harnesssettings in the recordGrading additions, for runs with these labels
context_losswhen the failure traces to something a compaction lost. The entry gets a count, and the score is unchanged.assistant_harness.judge_reads: no longer counts a "wake 0" from glob patterns.How it was checked
turn_otel_v5_compaction_claude.jsonandturn_otel_v5_compaction_gpt.json, taken from the harness's v5 traces with a fake model.slow_searchcall is an app action, and everymemory_addandhistory_searchcall is internal.pytest tst/unit/task_step tst/unit/agents/claude_code: 3,502 passed.ruff checkis clean.Not covered by a real trace yet: helper agents, failed wakes and late-start timings. They're tested from the harness's description only. A real v5 trace follows once v5 is registered.
Harness v7 and the label gate
harness.compactionorharness.messagespan, or aharness.*attribute). The standardgen_ai.agent.namealone doesn't turn it on, so other OpenTelemetry harnesses, and hand-offs between agents, read as plain OpenTelemetry.harness.compaction.written_byon every summary andharness.versionon every wake. A summary withoutwritten_bymakes the run ungradable. Earlier harness versions aren't read; their runs are re-run on v7.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new actionable issue was found in the latest changes.Summary
Adds harness-aware trace reading and stages turns consistently when no timeline is stored.
All three supplied, unnumbered previous findings are addressed: searches include long-field files, the no-label test excludes the harness trace, and staged turns retain their final error.
Reviews (17) · Last reviewed commit: "fix(always_on_staging): a turn staged by..." · Reviewed by Greptile