Skip to content

feat(always_on_agent_verifier): read harness traces with compaction, memory and helpers (OpenTelemetry GenAI names) - #716

Merged
sarah-chitty-1 merged 39 commits into
mainfrom
sarahchitty/fd-5840-reader-v3
Nov 17, 2026
Merged

sarah-chitty-1 merged 39 commits into
mainfrom
sarahchitty/fd-5840-reader-v3

Conversation

@sarah-chitty-1

@sarah-chitty-1 sarah-chitty-1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

The always-on judge has to read runs from a harness that compacts its history and has memory tools. That's the PSI neutral harness from v5, synthetic-pipelines #2048. Read the old way, such a run is misread: every compaction summary shows up as the assistant's own words, and every memory call can be listed as an action on the user's apps.

The harness labels its trace with the OpenTelemetry GenAI conventions plus a few neutral harness.* names, so this reader has no PSI-only names: git grep 'psi\.' over the verifiers returns nothing. Runs without these labels read exactly as before.

What the reader understands

In the trace What the judge sees
reasoning parts in a model call's gen_ai.output.messages the assistant's reasoning, in order, before that call's text and tool calls
a model span whose gen_ai.agent.name isn't the main agent (gen_ai.main_agent.*) and isn't nested under a tool call: the summary call never the assistant's words
the harness.compaction span (its compaction part and harness.compaction.*) a harness note: which wakes are now in the summary, which were kept word for word, and how late this wake started
harness.message spans, by harness.message.purpose harness notes: the save request, the after-summary message, and failed-wake notes
main-agent spans between a save request and the compaction's end the "save your notes" turn, staged before the wake's user message, which is when it ran
tool spans with harness.tool = true the harness's own tools: kept in the wake files as internal, and never an action
spans nested under a tool call, with another agent's name helper-agent lines (subagent)
harness.config.* on the harness.usage span assistant_harness settings in the record

Grading additions, for runs with these labels

  • Judge instructions: the judge is told what harness notes, the save turn, internal tools and helper lines are. It's also told that the compaction summary is the assistant's own, and is graded normally.
  • Context loss: the judge marks a failed criterion context_loss when the failure traces to something a compaction lost. The entry gets a count, and the score is unchanged.
  • Harness disclosure: every always-on entry gets assistant_harness.
  • For all always-on runs: the judge must search every wake before it claims that something never happened.
  • judge_reads: no longer counts a "wake 0" from glob patterns.

How it was checked

  • Fixtures: turn_otel_v5_compaction_claude.json and turn_otel_v5_compaction_gpt.json, taken from the harness's v5 traces with a fake model.
  • The refreshed traces: every slow_search call is an app action, and every memory_add and history_search call is internal.
  • Old fixtures: they give byte-identical events and staged files.
  • The earlier version against real traces: an independent review compared 36 wakes from three models against the raw traces, with nothing missing or doubled. That was on the v3 traces, before the rename.
  • pytest tst/unit/task_step tst/unit/agents/claude_code: 3,502 passed. ruff check is clean.

Not covered by a real trace yet: helper agents, failed wakes and late-start timings. They're tested from the harness's description only. A real v5 trace follows once v5 is registered.

Harness v7 and the label gate

  • The harness reading turns on only for a trace that carries a harness's own labels (a harness.compaction or harness.message span, or a harness.* attribute). The standard gen_ai.agent.name alone doesn't turn it on, so other OpenTelemetry harnesses, and hand-offs between agents, read as plain OpenTelemetry.
  • It reads the PSI neutral harness v7 (synthetic-pipelines #2135): harness.compaction.written_by on every summary and harness.version on every wake. A summary without written_by makes the run ungradable. Earlier harness versions aren't read; their runs are re-run on v7.
  • Runs with no stored always-on timeline are read the same way, turn by turn, so the judge never gets raw spans.
  • Checked: v7 traces from the real image, for Claude, GPT and Gemini, through this reader and staging; and both Thanksgiving v6 runs re-graded with the same verdicts on all 7 criteria.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new actionable issue was found in the latest changes.

Summary

Adds harness-aware trace reading and stages turns consistently when no timeline is stored.

  • The judge reads harness traces as assistant events and harness notes.
  • The judge stages each assistant turn even when a run has no timeline.
  • The judge records compaction context loss and harness details with each result.

All three supplied, unnumbered previous findings are addressed: searches include long-field files, the no-label test excludes the harness trace, and staged turns retain their final error.

Reviews (17) · Last reviewed commit: "fix(always_on_staging): a turn staged by..." · Reviewed by Greptile

sarah-chitty-1 and others added 28 commits November 13, 2026 15:36
…ored trajectory; record every app change whole

turn_events parses a stored turn trajectory (OTel spans, OpenInference's included;
Claude Code and Gemini stream-json) into the agent's text and each tool call with its
input and result, dropping what every turn or span repeats (tool lists, the
conversation so far, signatures, token counts). readable_result unwraps MCP
envelopes, decodes base64 email bodies and leaves images out. The OTel tool-span
reading is shared with compact_otel_trajectory, which behaves as before.

always_on's stored timeline now keeps every change of a changelog read with its
whole summary (no 200-change cap per read, no 200-character cut); the run's
metadata record is unchanged. render_for_judge cuts a summary for display only and
says when an older run didn't record some changes.

Fixtures are redacted recordings of a real always_on wake and a Claude Code turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p, and override points for staging and judge config

- judge_timeout_retries (unset: a timeout fails at once, as before): an agent judge
  call that runs out of time is tried again; the timed-out attempt's trajectory is
  fetched and logged under judge_output_discrepancies (kind judge_timeout), or
  carried on JudgeTimeoutError. Running out of time is the poll's own deadline, or a
  failed task whose error code says so (`<harness>.timeout`); only a code that
  doesn't tell falls back to the message's "timed out after Ns".
- The Claude Code agent fails a turn that ran out of its timeout with error code
  claude_code.timeout (was cli_error), as other harnesses name theirs.
- The Claude Code agent's config takes peer_tools (default true): false leaves the
  SDK's peer_agents MCP server out of a turn, for an agent that must not message
  the task's other agents.
- judge_max_turns (unset: the judge's own): sent as the judge's max_turns.
- stage_trajectory: a subclass may stage the run in the judge's sandbox itself
  (StagedTrajectory); None, the default, loads the trajectory as before.
- judge_agent_config: extra judge settings, {} by default.
- Run.firings_read: the firings of every log that could be read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…searches every wake staged as files

always_on_staging builds the run's files from its stored timeline and each wake's
trajectory, matched by trajectory_url: wakes/wake_NNN.jsonl (what woke the
assistant with the user's whole message, every tool call with input and result,
its text, every app change, messages, nudges, reads), events.jsonl, triggers.jsonl
and index.md (actions and messages with file:line). Any line too long to read whole
keeps a preview and moves its long fields to wakes/results/. index.md's write actions
are calls of a tool that made one of the wake's app changes (the changes' actors),
or, when the apps keep no change log, a guess from the tool name; the index says
which. Missing or unreadable trajectories and turns compaction didn't shrink are
recorded and logged.

The verifier accepts use_trajectory with the agent judge only, writes the files
concurrently, adds a prompt note on reading them, records the staging and what each
judge call read (judge_reads), and gives the judge only Bash and Read of its own
tools (builtin_tools only), with no peer-agents tools (peer_tools false): no helper
agents, web, writes or messaging other agents. Its defaults: 2400 s and
400 turns per judge call, one timeout retry, at most 3 batches at once. Without a
stored timeline it falls back to rubrics_verifier's per-turn loading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The verifier's timeout retry already recognises a Claude Code timeout from its
message ("timed out after Ns"), so the new claude_code.timeout code isn't
needed. Dropping it leaves this PR with no behaviour change for anyone who
doesn't opt in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ol name; a failed Gemini turn keeps its error message

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en loading the trajectory fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ites in the index, compare trigger times parsed, find wakes in text results; log a count, not every untrimmed trajectory

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rded it (Claude Code blocks, a GenAI chat span's thinking blocks)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g line in its wake when the trajectory recorded it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ks on an LLM span

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssage.contents.N.message_content.text), as GPT models are called

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rgin past the poll, so the poll's deadline is the only timeout signal

Per review: the judge's own limit raced the poll's, so a timeout was also read from a failed task's code or
message, which could mistake a tool's "timed out after N" for the judge running out. The judge is now allowed
judge_timeout_seconds + 60 when retries are on; the poll always fires first and cancels the task. The code and
message matching is gone. Without retries the judge is sent the same timeout as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ard, so an override can fit its settings to that judge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… when its card names its tools that way

Per Greptile: a Codex or OpenCode judge names its tools differently and would refuse Claude Code's names, failing
the config before grading. The judge's card now decides: its agent-config schema's builtin_tools names. Another
harness keeps its own tools (no peer tools either way), and the log says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emory, helpers)

turn_events: the harness's own model calls (psi.agent.kind = harness, the
compaction summary turn) are no longer the assistant's text or thinking; each
psi.compaction span (at its end, with the summary and how late the next wake
started) and each psi.harness_message span is a note saying it came from the
harness. A helper's events carry subagent (psi.agent.id); events of a turn the
harness asked for carry step (psi.turn, e.g. pre_compaction_save). Spans with
no psi.* labels read as before.

always_on_staging: lines carry step; memory_* and history_* calls are never
actions or possible actions in index.md (they stay in the wake files), and
index.md says so when a run has them; judge_reads no longer counts a wake named
in a staged call's input (history_read's {"wake": N}) as one the judge opened.

Fixtures: two real v3 smoke wakes (Opus 5.5, GPT 6.1), redacted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arness ran

A psi.compaction note carries its counts (wakes summarized and kept, how late
the next wake started); staging keeps them on the harness_note line, and the
wake's block in index.md gets one line saying so, pointing at the note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…der, which wakes a compaction covers

- The judge's files note gets five rules for harness lines (harness_note, the
  pre_compaction_save step, memory_*/history_* calls, subagent lines, notes
  without "at"), only when a staged trajectory carries psi.agent.* labels
  (record["agent_labeled_trajectories"]); other runs' prompts are unchanged.
- In a compaction wake the save turn, the compaction note and the message
  after it are staged after "woke" and before the user's message and the wake
  input, as they happened.
- The compaction note and index line name the wakes (from first_kept_wake:
  "wakes 1–8 are now in the summary; wakes 9–11 kept word for word"; counts
  when it is missing) and say "this wake started … late".
- turn_end and the index header count the save turn's calls apart
  (save_turn_tool_calls).
- The after-compaction message is labelled as re-sent by the harness.
- memory_*/history_* calls are internal only in a trajectory with agent
  labels (TurnEvents.agent_labels); elsewhere such a tool is an app's.
- A compaction span with no end time sorts after its save turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 requested a review from a team as a code owner November 16, 2026 19:42
…action, context-loss labels, harness disclosure

- The v3 part of the judge's files note says a compaction summary is the
  assistant's own and is graded like any other mistake.
- For a v3 run graded per criterion (rubric_binary / rubric_partial), the
  judge is asked, for each failed criterion, whether it traces to context a
  compaction lost: optional "context_loss" / "context_loss_evidence" on its
  row (added to this run's schema only), kept on the row, counted on the
  entry (entry["context_loss"]); never changing score or pass/fail. A row
  without the field is counted as not_said.
- Every entry gets "assistant_harness": the assistant's agent name, its A2A
  card's name and version, and the settings its last wake's trace reports
  (psi.config.*, or the first neutral harness's harness.config.*).

Only always_on_agent_verifier changes: it overrides _spec() for the run
(a ContextVar set when staging finds agent labels); rubrics_verifier and
judge_output_format are untouched, and their rows already keep any field the
judge returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 2 commits November 16, 2026 11:58
…race only when the wakes are staged

A run graded without its trajectories gets only its agent name and card in
assistant_harness, with no trajectory read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e_reads counts no wake 0

- The judge's files note (every always-on run) asks it to grep events.jsonl
  before concluding something never happened, and to name that search.
- judge_reads took a glob's prefix (wakes/wake_0*.jsonl, wake_00[1-4].jsonl)
  as wake 0: it now counts only a whole wake file or result file name, and
  only wakes >= 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from sarahchitty/fd-5840-agent-judge to main November 16, 2026 23:29
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	src/agent_env/task_step/task_steps/verifiers/always_on_agent_verifier.py
#	src/agent_env/task_step/task_steps/verifiers/always_on_staging.py
#	src/agent_env/task_step/task_steps/verifiers/judge_utils/turn_events.py
#	tst/unit/task_step/test_always_on_agent_verifier.py
#	tst/unit/task_step/verifiers/judge_utils/test_turn_events.py
#	tst/unit/task_step/verifiers/test_always_on_staging.py
sarah-chitty-1 added a commit that referenced this pull request Nov 16, 2026
Both features in always_on_agent_verifier: the files note takes #716's harness
slot with the rules note appended after it; _grade keeps #716's ContextVar
(reset in a finally), context_loss and assistant_harness, then judge_reads, and
grades the user's rules last, so an ungradable run's entry has its harness and
context_loss when UngradableRunError is raised. The rules retry runs inside the
graded run's context, so a v3 run's retries keep the context_loss rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 2 commits November 16, 2026 15:48
…mes left

turn_events reads the neutral harness's OpenTelemetry GenAI names:
- reasoning from the reasoning parts of gen_ai.output.messages, in order
  (psi.reasoning is gone); text falls back to its text parts;
- a model span of an agent other than gen_ai.main_agent.name that no tool
  call started is the harness's own (the summary call): its text and
  reasoning are left out;
- a helper's events carry subagent "<agent id or name>@<delegating call's
  span id>", from nesting under the tool call;
- the compaction note comes from the harness.compaction span (summary from
  its compaction part; numbers as before, plus a failed save turn's or
  compaction's error), harness-message notes from harness.message, labelled
  and placed by purpose (save_request, after_summary, failed_wake);
- the save turn is the main agent's spans between the save request and the
  compaction's end.

The judge's harness note and context_loss turn on for a trajectory whose
spans carry gen_ai.agent.name or that has a harness.compaction or
harness.message span. The memory_*/history_* internal-tool rule is removed:
memory calls are ordinary tool calls. Harness settings come from
harness.config.* only. A failed wake's message is staged before the wake's
input, like the save turn and the compaction.

Fixtures: the v3 ones are replaced by two redacted v5 fake-model wakes.
Non-v5 fixtures give byte-identical events and staged files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… internal, never an action

A tool span with harness.tool = true gives its call internal: true; staging
keeps it in the wake file with "internal": true and never lists it as an
action or possible action in index.md. The judge's note says internal calls
count as what the assistant did but change nothing in any app. A tool of the
same name without the mark is read as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 16, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	src/agent_env/task_step/task_steps/verifiers/always_on_agent_verifier.py
@sarah-chitty-1 sarah-chitty-1 changed the title feat(always_on_agent_verifier): read the PSI neutral harness v3 trace (compaction, memory, helpers) feat(always_on_agent_verifier): read harness traces with compaction, memory and helpers (OpenTelemetry GenAI names) Nov 16, 2026
…ts too (grep -r wakes/)

A text too long for a line is kept whole under wakes/results/; events.jsonl
has only its first 300 characters. Before concluding something never
happened, the judge greps wakes/ recursively.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	tst/unit/task_step/test_always_on_agent_verifier.py

@andrewpark-scaleai andrewpark-scaleai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. the context-loss fields only turn on for staged runs of a harness that labels its spans, and the trace reads fail soft. small nits: _parts checks isinstance(messages, list) after iterating it, so a null or numeric gen_ai.output.messages throws and that wake's tool calls drop out of what the judge sees. and use_trajectory runs that don't opt in still get the new absence rule in the prompt, which seems fine but worth knowing.

…tead of dropping the span

Review (andrewpark-scaleai): a null or numeric gen_ai.output.messages threw while
iterating, so that wake's tool calls dropped out of what the judge sees.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…harness.* labels, never on gen_ai.agent.name alone

Review (Edgar): another OpenTelemetry harness that sets the standard agent name
got the harness reading, and a hand-off between two agents lost the second
agent's words. Now only a trace carrying a harness's own labels (a
harness.compaction or harness.message span, or a harness.* attribute) has its
summary call left out, its save turn marked, and the notes and fields that go
with them. Plain OpenTelemetry traces read as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the harness version, timeline-less runs staged

- A harness.compaction span's written_by goes into the compaction facts and its note. The judge is told whose a
  summary's losses are by who wrote it: "assistant" (its own model) is graded as its own; "harness" is not held
  against the assistant, and the judge names the loss. A compaction span without written_by (or with another value)
  raises TrajectoryFormatError, and the verifier leaves the run ungraded (UngradableRunError naming the wake).
- assistant_harness carries harness.version from the harness.usage span (harness_usage replaces harness_settings).
- A run with no stored timeline (not kept, or unreadable) has its turns read through turn_events and staged by
  stage_run by position (one wake file a turn, index.md and the files note say so), so the judge never reads a
  raw trajectory.
- No reading for older harness versions: a compaction span is placed by its end time and the save turn ends where the
  compaction that follows it ends. The old-harness fixture (turn_otel_wake4.json) is gone; the recorded fake-model
  wakes are the v7 shape (turn_otel_v7_compaction_{claude,gpt}.json: written_by and harness.version added).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Nov 17, 2026

Copy link
Copy Markdown

Comments Outside Diff

These findings could not be posted inline.

  • P1 Dependent tests lose their traces tst/unit/task_step/verifiers/recorded/turn_otel_wake4.json:3 ▶

    Deleting this fixture breaks PR feat(always_on_agent_verifier): grade the user's rules (ask first, never, just do, tell after) #724's new tests in tst/unit/task_step/verifiers/test_always_on_user_rules.py. Its nina_turns() still reads turn_otel_wake4.json, so tests using that helper raise FileNotFoundError. Its memory-tool test also reads the renamed turn_otel_v5_compaction_gpt.json.

    Keep the dinner-run fixture and update the dependent PR's compaction reference before merging the stack.

… a failed one included

Greptile: with no stored timeline nothing recorded each turn's state or closing
reply, and the staging dropped the trajectory's final result, so a turn that ended
in an error (e.g. a quota error) read as complete. The turn_end line now takes
both from the trajectory's last final result: failed with its text on an error,
completed otherwise; the index lists a failed turn as an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 merged commit 38b1fd1 into main Nov 17, 2026
35 checks passed
@sarah-chitty-1
sarah-chitty-1 deleted the sarahchitty/fd-5840-reader-v3 branch November 17, 2026 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants