Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
98a7493
feat(verifiers): read an agent turn's tool calls and text from any st…
sarah-chitty-1 Nov 13, 2026
24c5385
feat(rubrics_verifier): opt-in judge timeout retries, a judge turn ca…
sarah-chitty-1 Nov 14, 2026
1ccdf35
feat(always_on_agent_verifier): with use_trajectory, the agent judge …
sarah-chitty-1 Nov 14, 2026
a732ea6
fix(claude-code): keep a timed-out turn's error code cli_error
sarah-chitty-1 Nov 14, 2026
b3f15e4
Merge #679's timeout error-code revert into #680
sarah-chitty-1 Nov 14, 2026
7dc7dd8
fix(turn_events): a generically named tool span takes the prompt's to…
sarah-chitty-1 Nov 14, 2026
0f3e2e7
fix(rubrics_verifier): a judge the verifier deployed is terminated wh…
sarah-chitty-1 Nov 14, 2026
6fcaa72
Merge #678's Greptile fixes into #679
sarah-chitty-1 Nov 14, 2026
222407d
Merge #679's Greptile fixes into #680
sarah-chitty-1 Nov 14, 2026
6b5828d
fix(always_on_staging): split long side-file lines, label unlogged wr…
sarah-chitty-1 Nov 14, 2026
c0ab41f
feat(turn_events): keep the agent's thinking when the trajectory reco…
sarah-chitty-1 Nov 14, 2026
a8abe2d
Merge #679 into #680
sarah-chitty-1 Nov 14, 2026
1aa7336
Merge #678's thinking reader into #679
sarah-chitty-1 Nov 14, 2026
86408d7
feat(always_on_staging): the agent's thinking is an assistant_thinkin…
sarah-chitty-1 Nov 14, 2026
34e3f11
feat(turn_events): read the PSI harness's psi.reasoning thinking bloc…
sarah-chitty-1 Nov 14, 2026
ae4aa3d
Merge #678 into #679
sarah-chitty-1 Nov 14, 2026
075041b
Merge #679 into #680
sarah-chitty-1 Nov 14, 2026
36a8648
fix(turn_events): read an OpenInference Responses-API span's text (me…
sarah-chitty-1 Nov 14, 2026
50d9256
Merge #678 into #679
sarah-chitty-1 Nov 14, 2026
a2bd86a
Merge #679 into #680
sarah-chitty-1 Nov 14, 2026
9e342a7
feat(always_on): the user's rules as criteria, and the run's events i…
sarah-chitty-1 Nov 14, 2026
4539639
feat(always_on_agent_verifier): grade_user_rules grades the user sim'…
sarah-chitty-1 Nov 14, 2026
913b212
feat(always_on_agent_verifier): user rules split into verdict and sco…
sarah-chitty-1 Nov 14, 2026
a7f5e57
fix(rubrics_verifier): with timeout retries, the judge gets a 60 s ma…
sarah-chitty-1 Nov 16, 2026
d9b567d
Merge #679 into #680
sarah-chitty-1 Nov 16, 2026
522493b
Merge #680 into the user-rules branch
sarah-chitty-1 Nov 16, 2026
869c175
refactor(rubrics_verifier): judge_agent_config is given the judge's c…
sarah-chitty-1 Nov 16, 2026
caf8e2d
Merge #679 into #680
sarah-chitty-1 Nov 16, 2026
8b0778d
fix(always_on_agent_verifier): narrow the judge to Bash and Read only…
sarah-chitty-1 Nov 16, 2026
28af44b
Merge #680 into the user-rules branch
sarah-chitty-1 Nov 16, 2026
fade3df
feat(verifiers): read the PSI neutral harness v3 trace (compaction, m…
sarah-chitty-1 Nov 16, 2026
8f85f62
feat(always_on_staging): a line in index.md for each compaction the h…
sarah-chitty-1 Nov 16, 2026
88e3b5e
fix(always_on staging): v3 review fixes — judge instructions, wake or…
sarah-chitty-1 Nov 16, 2026
4bc67a8
feat(always_on_agent_verifier): v3 runs — the assistant owns its comp…
sarah-chitty-1 Nov 16, 2026
33a2aa0
fix(always_on_agent_verifier): read the harness's settings from the t…
sarah-chitty-1 Nov 16, 2026
a63eebe
fix(always_on_agent_verifier): ask the judge up to 3 more times, only…
sarah-chitty-1 Nov 16, 2026
621eca5
fix(always_on judge): search every wake before claiming absence; judg…
sarah-chitty-1 Nov 16, 2026
c668603
Merge main into #716
sarah-chitty-1 Nov 16, 2026
495c078
Merge #716 (reader v3) into the user-rules branch
sarah-chitty-1 Nov 16, 2026
5f2fee9
refactor(verifiers): read the PSI harness v5 trace names; no psi.* na…
sarah-chitty-1 Nov 16, 2026
e5ae1eb
feat(verifiers): a tool the harness provides itself (harness.tool) is…
sarah-chitty-1 Nov 16, 2026
def64e7
Merge #716 into #724
sarah-chitty-1 Nov 16, 2026
3de93f9
fix(always_on_agent_verifier): the absence rule searches the long tex…
sarah-chitty-1 Nov 17, 2026
575bb86
fix(always_on_agent_verifier): Greptile's five on the user rules
sarah-chitty-1 Nov 17, 2026
c70a0bd
Merge #716 into #724
sarah-chitty-1 Nov 17, 2026
bbee851
always_on_user_rules: the opening request is the user's own words, no…
sarah-chitty-1 Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/agent_env/task_step/task_steps/always_on.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@
sim's session (``context_id``), which rate_user_satisfaction asks in once the run is over.
- An always_on_agent_verifier (``verifiers/always_on_agent_verifier.py``) grades the step's prompt from the user's chat
and the run's context, the run wake by wake among it (``render_for_judge``), and, with use_trajectory, every wake's
tool calls staged as files; a plain rubrics_verifier on it is refused when the task loads.
tool calls staged as files, and with grade_user_rules the user sim's rules, checked against the order of the
changes, messages and replies recorded here; a plain rubrics_verifier on it is refused when the task loads.
- The agent: the message_user rules go in its system prompt when the step sets one and the agent's card takes it,
else on its first wake; its session is the step's ``context_id`` when set, as prompt_agent's is, else the run's own.
- When the step ends it puts back the gateway settings it changed for the run (the user chat's roles and stay_quiet,
Expand Down

Large diffs are not rendered by default.

97 changes: 74 additions & 23 deletions src/agent_env/task_step/task_steps/verifiers/always_on_staging.py

Large diffs are not rendered by default.

524 changes: 524 additions & 0 deletions src/agent_env/task_step/task_steps/verifiers/always_on_user_rules.py

Large diffs are not rendered by default.

Large diffs are not rendered by default.

531 changes: 530 additions & 1 deletion tst/unit/task_step/test_always_on_agent_verifier.py

Large diffs are not rendered by default.

114 changes: 109 additions & 5 deletions tst/unit/task_step/verifiers/judge_utils/test_turn_events.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,17 +116,18 @@ def test_the_agent_s_thinking_is_kept_when_it_was_recorded_and_an_empty_block_is
("thinking", "Ask before paying $855."), ("text", "Asking the user.")]


def test_the_psi_harness_s_reasoning_on_an_llm_span_comes_before_that_call_s_text():
def test_an_llm_span_s_reasoning_parts_come_in_order_before_that_call_s_text_and_an_empty_one_is_dropped():
spans = [{"name": "ChatCompletion", "start_time": "1", "attributes": {
"openinference.span.kind": "LLM",
"llm.output_messages.0.message.role": "assistant",
"llm.output_messages.0.message.content": "I'll ask Nina first.",
"psi.reasoning": json.dumps([{"type": "thinking", "thinking": ""},
{"type": "thinking", "thinking": "Paying $855 needs her OK."}])}}]
"gen_ai.output.messages": json.dumps([{"role": "assistant", "finish_reason": "stop", "parts": [
{"type": "reasoning", "content": ""}, {"type": "reasoning", "content": "Paying $855 needs her OK."},
{"type": "reasoning", "content": "So ask."}, {"type": "text", "content": "I'll ask Nina first."}]}])}}]

events = [(e["type"], e["text"]) for e in turn_events(json.dumps(spans)).events if e["type"] != "final"]

assert events == [("thinking", "Paying $855 needs her OK."), ("text", "I'll ask Nina first.")]
assert events == [("thinking", "Paying $855 needs her OK.\nSo ask."), ("text", "I'll ask Nina first.")]


def test_a_responses_api_llm_span_s_text_and_reasoning_are_read():
Expand All @@ -135,7 +136,9 @@ def test_a_responses_api_llm_span_s_text_and_reasoning_are_read():
"llm.output_messages.0.message.role": "assistant",
"llm.output_messages.0.message.contents.0.message_content.type": "text",
"llm.output_messages.0.message.contents.0.message_content.text": "Booked for Friday.",
"psi.reasoning": json.dumps([{"type": "thinking", "thinking": "She said Friday works."}])}}]
"gen_ai.output.messages": json.dumps([{"role": "assistant", "finish_reason": "stop", "parts": [
{"type": "reasoning", "content": "She said Friday works."},
{"type": "text", "content": "Booked for Friday."}]}])}}]

events = [(e["type"], e["text"]) for e in turn_events(json.dumps(spans)).events if e["type"] != "final"]

Expand Down Expand Up @@ -171,3 +174,104 @@ def test_the_compactor_rubrics_verifier_uses_still_reads_tool_spans_the_same_way
assert events == [{"type": "tool_call", "tool": "email_send", "input": {"to": "a@b"}, "timestamp": "1"},
{"type": "tool_result", "tool": "email_send", "output": {"ok": True}, "timestamp": "2"}]
assert files == [] and trajectory_filter.tool_span_io(span) == ("email_send", {"to": "a@b"}, {"ok": True})


def _v5(name: str) -> list[dict]:
"""A recorded wake of the PSI neutral harness v5 (redacted from a fake-model run): it starts with a compaction."""
return json.loads((RECORDED / f"turn_otel_v5_compaction_{name}.json").read_text())


def _summary(spans: list[dict]) -> str:
span = next(s for s in spans if s["name"] == "harness.compaction")
return json.loads(span["attributes"]["gen_ai.output.messages"])[0]["parts"][0]["content"]


def test_a_v5_compaction_wake_keeps_the_assistant_s_words_and_makes_the_harness_s_parts_notes():
spans = _v5("claude")
summary = _summary(spans)

traj = turn_events(json.dumps(spans))
events = traj.events

assert traj.agent_labels is True
assert [(e["type"], e.get("step")) for e in events] == [
("note", "pre_compaction_save"), ("thinking", "pre_compaction_save"), ("text", "pre_compaction_save"),
("tool_call", "pre_compaction_save"), ("thinking", "pre_compaction_save"), ("text", "pre_compaction_save"),
("note", None), ("note", "after_compaction"),
("thinking", None), ("text", None), ("tool_call", None), ("thinking", None), ("text", None)]
save, compaction, after = (e["text"] for e in events if e["type"] == "note")
assert save.startswith("Message the harness sent the assistant (step pre_compaction_save; not from the user):\n"
"[Harness: older history is about to be summarized")
assert compaction == ("Compaction summary (written by the harness from the model's own summary turn; it replaces "
"the older history the model sees; wakes 1–3 are now in the summary; wakes 4–5 kept word for "
f"word):\n{summary}")
assert next(e for e in events if "compaction" in e)["compaction"] == {
"wakes_summarized": 3, "wakes_kept": 2, "first_kept_wake": 4}
assert after.startswith("Message the harness sent the assistant (step after_compaction; re-sent by the harness, "
"it repeats the user's earlier words):\n[Harness: older history was summarized")
# The summary call is the harness's (agent "harness", under no tool call): neither its text nor its reasoning is
# the assistant's, though its OpenInference output still holds the summary as text.
own = [e["text"] for e in events if e["type"] in ("text", "thinking")]
assert not any(summary in t or "step 12" in t for t in own)
assert [(c["tool"], c.get("internal")) for c in _calls(traj)] == [("memory_add", True), ("slow_search", None)]


def test_a_compaction_with_no_end_time_comes_after_its_save_turn_and_its_numbers_and_errors_are_said():
spans = _v5("claude")
span = next(s for s in spans if s["name"] == "harness.compaction")
del span["end_time"], span["attributes"]["harness.compaction.first_kept_wake"]
span["attributes"].update({"harness.compaction.next_wake_waited_s": 8.694,
"harness.compaction.save_error": "APITimeoutError: timed out"})

events = turn_events(json.dumps(spans)).events

kinds = [(e["type"], e.get("step")) for e in events]
assert kinds.index(("note", None)) == len(kinds) - 1 - kinds[::-1].index(("text", "pre_compaction_save")) + 1
compaction = next(e["text"] for e in events if "compaction" in e)
assert ("; 3 wakes summarized; the last 2 kept word for word; the save turn before it failed: APITimeoutError: "
"timed out; this wake started 8.694 s late (real time) waiting for it):") in compaction


def test_a_v5_gpt_save_turn_keeps_its_reasoning_and_calls_marked_as_the_harness_s_step():
events = turn_events(json.dumps(_v5("gpt"))).events

save = [e for e in events if e.get("step") == "pre_compaction_save" and e["type"] != "note"]
assert [(e["type"], e.get("tool"), e.get("text")) for e in save] == [
("thinking", None, "GPT reasoning 10."), ("tool_call", "memory_add", None),
("thinking", None, "GPT reasoning 11."), ("text", None, "Done (11).")]
assert [e["text"] for e in events if e["type"] == "text"] == ["Done (11).", "Done (14)."]


def test_a_helper_agent_s_spans_are_named_by_its_agent_and_the_call_that_started_it():
spans = _v5("claude")
delegate = next(s for s in spans if s["name"] == "slow_search")
helper = [s for s in spans if s["name"] == "generation" and s["start_time"] > delegate["start_time"]]
for s in helper:
s["parent_id"] = delegate["context"]["span_id"]
s["attributes"]["gen_ai.agent.name"] = "researcher"

events = turn_events(json.dumps(spans)).events

assert [(e["type"], e.get("subagent")) for e in events if e.get("subagent")] == [
("thinking", f"researcher@{delegate['context']['span_id']}"), ("text", f"researcher@{delegate['context']['span_id']}")]


def test_a_failed_wake_s_message_is_a_note_in_its_step():
span = {"name": "harness.message", "start_time": "1", "attributes": {
"gen_ai.agent.name": "harness", "gen_ai.main_agent.name": "Orchestrator",
"harness.message.purpose": "failed_wake",
"harness.message.text": "[Harness: your last wake ended with an error before it finished (TimeoutError).]"}}

[note] = turn_events(json.dumps([span])).events

assert note == {"type": "note", "step": "failed_wake", "text": (
"Message the harness sent the assistant (step failed_wake; not from the user):\n"
"[Harness: your last wake ended with an error before it finished (TimeoutError).]")}


def test_spans_without_agent_labels_read_as_before():
events = turn_events((RECORDED / "turn_otel_wake4.json").read_bytes()).events

assert not any("step" in e or "subagent" in e for e in events)
assert "note" not in {e["type"] for e in events}
assert turn_events((RECORDED / "turn_otel_wake4.json").read_bytes()).agent_labels is False
Loading
Loading