Skip to content

feat(always_on): the judge can see what the assistant was told before the run (opt-in assistant_instructions) - #756

Closed
sarah-chitty-1 wants to merge 1 commit into
mainfrom
sarahchitty/fd-6216-judge-sees-instructions
Closed

sarah-chitty-1 wants to merge 1 commit into
mainfrom
sarahchitty/fd-6216-judge-sees-instructions

Conversation

@sarah-chitty-1

@sarah-chitty-1 sarah-chitty-1 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

Always-on tasks give the assistant standing instructions in the always_on step's system_prompt, such as the user's rules it was told up front. The always-on judge never saw them, so it couldn't tell a rule the assistant was told from one it had to learn by asking. Linear FD-6216.

Opt-in; tasks that don't ask for it behave exactly as before.

What it adds

  • always_on records what it gave the assistant. When the step configures the agent, the run's metadata gets agent_system_prompt: the agent, whether a prompt was sent, and its length, or why_not when none was sent. The full text goes to the stored record in the object store; if that write fails, the metadata keeps the text. stored_system_prompt(record) reads it back.
  • The judge can see it. There's a new run_context_for_judge part, assistant_instructions; the default list is unchanged.
    • With use_trajectory: the judge gets assistant_instructions.md next to index.md, plus one line in its files note.
    • Direct judges: a prompt section, capped at 60k characters.
    • Nothing recorded (older runs, no prompt sent, or an unreadable record): the judge gets a line saying why, and grading goes on as normal.

Example

{"type": "always_on_agent_verifier", "use_agent_judge": true, "use_trajectory": true,
 "run_context_for_judge": ["conversation", "trigger_log", "assistant_instructions"]}

Tests

13 new tests, using a system prompt with a block of the user's standing rules. tst/unit/task_step passes, apart from the 2 known test_mcp_cli_builder failures (missing click).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 4/5

No blocking behavior defect was found, but the repository's named-limit requirement must be satisfied before merging.

Fix All in CursorFindings

  1. P2 Error preview limit is unnamed ▶
Fix with agent prompt
### Issue 1
src/agent_env/task_step/task_steps/verifiers/always_on_agent_verifier.py:374
The new error preview cuts exception text with an inline `200`. The repository requires numeric limits to be named class or instance variables. Give this limit a descriptive name so readers can see what it controls without hunting through string slices. This repository requirement must be satisfied before merging.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Records the system prompt sent by always_on and adds opt-in assistant_instructions for judges.

  • Always-on runs keep the system prompt sent to the assistant.
  • Judges can see the assistant's starting instructions when tasks opt in.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Send assistant system prompt] --> B[Keep prompt summary in run metadata]
  B --> C{Object store write}
  C -->|Succeeds| D[Store full prompt with timeline]
  C -->|Fails| E[Keep prompt text in metadata]
  D --> F{Judge opts in}
  E --> F
  F -->|No| G[Existing judge context]
  F -->|Yes, staged| H[assistant_instructions.md and files note]
  F -->|Yes, not staged| I[Capped prompt section]
Loading

Reviews (1) · Last reviewed commit: "always_on: the judge can see what the as..." · Reviewed by Greptile

The always_on step now records the system prompt it configured the agent with:
the run's metadata keeps agent_system_prompt (agent, sent or why not, length)
and the stored record keeps the text (stored_system_prompt reads it).

always_on_agent_verifier takes a new run_context_for_judge part,
assistant_instructions (off by default). A staged run gets it as
assistant_instructions.md beside the index, named in the files note;
otherwise it is a section of the judge's prompt. A run whose record doesn't
keep it, whose step sent none, or whose record can't be read gets a line
saying so, and is graded as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 requested a review from a team as a code owner November 18, 2026 01:00
except Exception as e:
logger.warning("Verifier '%s': the assistant's system prompt could not be read (%s: %s)", self.verifier_id,
type(e).__name__, e)
return None, f"The run's record of it couldn't be read ({type(e).__name__}: {str(e)[:200]})."

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Error preview limit is unnamed

The new error preview cuts exception text with an inline 200. The repository requires numeric limits to be named class or instance variables. Give this limit a descriptive name so readers can see what it controls without hunting through string slices. This repository requirement must be satisfied before merging.

Rule Used: Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. (source)

Learned From
scaleapi/scaleapi#126388

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/verifiers/always_on_agent_verifier.py
Line: 374

Comment:
**Error preview limit is unnamed**

The new error preview cuts exception text with an inline `200`. The repository requires numeric limits to be named class or instance variables. Give this limit a descriptive name so readers can see what it controls without hunting through string slices. This repository requirement must be satisfied before merging.

**Rule Used:** Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. ([source](https://app.greptile.com/-/switch-tenant?tenant=scale-ai&next=%2Fcustom-context%3Fmemory%3D002e0051-41ad-46c1-9098-47433c580150))

**Learned From**
[scaleapi/scaleapi#126388](https://github.com/scaleapi/scaleapi/pull/126388)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Claude Code Fix in Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant