Repository navigation
feat(always_on): the judge can see what the assistant was told before the run (opt-in assistant_instructions) - #756
Closed
sarah-chitty-1 wants to merge 1 commit into
Closed
sarah-chitty-1 wants to merge 1 commit into
sarah-chitty-1 wants to merge 1 commit into
Conversation
The always_on step now records the system prompt it configured the agent with: the run's metadata keeps agent_system_prompt (agent, sent or why not, length) and the stored record keeps the text (stored_system_prompt reads it). always_on_agent_verifier takes a new run_context_for_judge part, assistant_instructions (off by default). A staged run gets it as assistant_instructions.md beside the index, named in the files note; otherwise it is a section of the judge's prompt. A run whose record doesn't keep it, whose step sent none, or whose record can't be read gets a line saying so, and is graded as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| except Exception as e: | ||
| logger.warning("Verifier '%s': the assistant's system prompt could not be read (%s: %s)", self.verifier_id, | ||
| type(e).__name__, e) | ||
| return None, f"The run's record of it couldn't be read ({type(e).__name__}: {str(e)[:200]})." |
There was a problem hiding this comment.
Error preview limit is unnamed
The new error preview cuts exception text with an inline 200. The repository requires numeric limits to be named class or instance variables. Give this limit a descriptive name so readers can see what it controls without hunting through string slices. This repository requirement must be satisfied before merging.
Rule Used: Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. (source)
Learned From
scaleapi/scaleapi#126388
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/verifiers/always_on_agent_verifier.py
Line: 374
Comment:
**Error preview limit is unnamed**
The new error preview cuts exception text with an inline `200`. The repository requires numeric limits to be named class or instance variables. Give this limit a descriptive name so readers can see what it controls without hunting through string slices. This repository requirement must be satisfied before merging.
**Rule Used:** Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. ([source](https://app.greptile.com/-/switch-tenant?tenant=scale-ai&next=%2Fcustom-context%3Fmemory%3D002e0051-41ad-46c1-9098-47433c580150))
**Learned From**
[scaleapi/scaleapi#126388](https://github.com/scaleapi/scaleapi/pull/126388)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Always-on tasks give the assistant standing instructions in the
always_onstep'ssystem_prompt, such as the user's rules it was told up front. The always-on judge never saw them, so it couldn't tell a rule the assistant was told from one it had to learn by asking. Linear FD-6216.Opt-in; tasks that don't ask for it behave exactly as before.
What it adds
always_onrecords what it gave the assistant. When the step configures the agent, the run's metadata getsagent_system_prompt: the agent, whether a prompt was sent, and its length, orwhy_notwhen none was sent. The full text goes to the stored record in the object store; if that write fails, the metadata keeps the text.stored_system_prompt(record)reads it back.run_context_for_judgepart,assistant_instructions; the default list is unchanged.use_trajectory: the judge getsassistant_instructions.mdnext toindex.md, plus one line in its files note.Example
{"type": "always_on_agent_verifier", "use_agent_judge": true, "use_trajectory": true, "run_context_for_judge": ["conversation", "trigger_log", "assistant_instructions"]}Tests
13 new tests, using a system prompt with a block of the user's standing rules.
tst/unit/task_steppasses, apart from the 2 knowntest_mcp_cli_builderfailures (missingclick).🤖 Generated with Claude Code
No blocking behavior defect was found, but the repository's named-limit requirement must be satisfied before merging.
Fix with agent prompt
Summary
Records the system prompt sent by
always_onand adds opt-inassistant_instructionsfor judges.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Send assistant system prompt] --> B[Keep prompt summary in run metadata] B --> C{Object store write} C -->|Succeeds| D[Store full prompt with timeline] C -->|Fails| E[Keep prompt text in metadata] D --> F{Judge opts in} E --> F F -->|No| G[Existing judge context] F -->|Yes, staged| H[assistant_instructions.md and files note] F -->|Yes, not staged| I[Capped prompt section]Reviews (1) · Last reviewed commit: "always_on: the judge can see what the as..." · Reviewed by Greptile