Repository navigation
fix(agents): no refusal as judge context; the language comes from the pasted data - #12171
Merged
Merged
Conversation
…he language
Issue 1, false refusal right after a refusal (live test 2026-10-14): the
scope judge got the halt event's fixed refusal as `<last_assistant_turn>`,
so after "Wie wird das Wetter morgen in Wien?" was refused `needs_data`,
"Welcher Tag im April war der wärmste, und kannst du ihn im Plot
markieren?" was refused `needs_data` too. `judge_context` now skips every
fixed reply (`policy.fixed_replies`, all refusals in all languages, plus
the outage line) and passes the last reply a model wrote, or none; the
withheld placeholder among the user turns still marks the refused turn.
The rubric says a question about the user's own data is in scope and
`needs_data` is only for data the service would have to fetch. Eight new
scope eval cases (`data_question`, three `after_refusal`).
Issue 2, language from the pasted data (owner request): a paste before any
typed message or choice sets the session language (new source `pasted`)
from the notes judge's `lang`, else the dataset judge's `lang` for the
part holding the column names; no extra model call. `en` is no signal.
Detection still replaces `pasted`; a paste never replaces `detected` or
`chosen` (the route re-reads the session after the judges). The dataset
route returns `language: {language, source}`; the page syncs its chip
(`lang: de · from data`) and shows one notice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Several API, architecture, and module-level descriptions still expose the obsolete three-source language contract.
3 open findings
What changed in this PR
Fixes false refusals after prior refusals and derives session language from pasted data.
Changes:
- Excludes fixed service replies from scope-judge context and expands scope rules.
- Adds the
pastedlanguage source across backend and frontend. - Adds regression tests, eval cases, documentation, and changelog coverage.
| File | Description |
|---|---|
tests/unit/agents/runtime/test_service_flow.py |
Tests pasted-language service flow. |
tests/unit/agents/runtime/test_plugins.py |
Tests refusal filtering and language precedence. |
tests/unit/agents/runtime/test_language.py |
Tests pasted-language selection. |
docs/reference/api.md |
Documents the dataset language response. |
docs/concepts/agent-network.md |
Describes the updated language and scope design. |
changelog.d/judge-context-data-language.md |
Records both fixes. |
app/src/sections/agent-chat/messages.ts |
Adds the pasted-language notice. |
app/src/sections/agent-chat/LanguageChip.tsx |
Displays the new source label. |
app/src/sections/agent-chat/LanguageChip.test.tsx |
Tests the source label. |
app/src/lib/agent.ts |
Extends client language types and parsing. |
app/src/lib/agent.test.ts |
Tests parsing pasted. |
app/src/hooks/useAgentSession.ts |
Synchronizes language after a paste. |
app/src/hooks/useAgentSession.test.ts |
Tests synchronization and notices. |
api/routers/agent.py |
Updates the BFF route documentation. |
agents/stream.py |
Documents the fourth language source. |
agents/README.md |
Updates agent behavior and eval guidance. |
agents/main.py |
Derives and stores language during ingestion. |
agents/evals/scope.py |
Updates scope-eval documentation. |
agents/evals/scope.evalset.json |
Adds refusal and data-question cases. |
agents/evals/import_case.py |
Preserves pasted languages in imported cases. |
agents/anyplot/session_state.py |
Extends session source semantics. |
agents/anyplot/prompts/scope_judge.md |
Clarifies data-question classification. |
agents/anyplot/policy.py |
Collects fixed service replies. |
agents/anyplot/plugins/scope_guard.py |
Filters service replies from judge context. |
agents/anyplot/language.py |
Implements pasted-language precedence. |
agents/anyplot/agent.py |
Describes pasted language to the root agent. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Copilot review of #12171: the hook's state-machine overview, the design doc's SSE and frontend bullets, and the API reference's refusal and done rows name the pasted source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Refusal reasons are lost for follow-ups, language precedence is not atomic, and the local browser mock lacks the new response behavior.
6 open findings
3 resolved since last review
🧠 Review effort: Balanced
…view round 2 Copilot's second round on #12171: - A judged refusal now stores `[message withheld by scope policy: <code>]` (`out_of_scope` or `needs_data`; `attack` reads as `out_of_scope`), so "do it anyway" after a `needs_data` refusal gets `needs_data` again while no refusal text enters the judge's context. Refusals before the judge keep the bare placeholder. The rubric names both forms and says the placeholder's verdict belongs to the refused message only (in-053, a plot change after a refusal, regressed without that line). The eval cases use the coded placeholders; adv-067/adv-068 are the German and English continuations after a `needs_data` refusal. - `upload_dataset`: documented why the re-read and the conditional language write are atomic (no suspending await in between on the in-memory session service). - `import_case`: matches the placeholder by prefix; the session_language test covers `pasted` (version language first, session fallback). - BFF mock: a German paste sets `de`/`pasted` and the dataset answer carries `language`; detection yields to it as in the service. - Changelog and API doc: an English or unclear paste leaves the current language unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
MarkusNeusinger
added a commit
that referenced
this pull request
Oct 14, 2026
Brings in #12171 (the dataset judge returns the headers' language, the notes screen returns the notes' language, the session language from a paste). Conflicts: - agents/main.py: main changed _judge_dataset to return the headers' language; this branch moved it into dataset_guard.py. The route keeps calling check_dataset, which now returns the headers' language from judge_dataset (the first part's verdict); run_data_code ignores it. - docs/reference/api.md: main's text taken, this branch's four additions (422 code, the data-code paragraph, done.dataset.code, the BFF code) applied on top. - docs/concepts/agent-network.md and agents/README.md: both sides kept (the language field and source next to 422 code and run_data_code; the scope case count is 219, the prep case count 44 with 11 data-code cases). scope_judge.md, scope.evalset.json (ids unique), policy, scope_guard and import_case merged without conflicts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two fixes from the owner's live test of 2026-10-14.
needs_data), the next message, "Welcher Tag im April war der wärmste, und kannst du ihn im Plot markieren?", was also refused.judge_contextpassed the fixed refusal (the before_run halt event, authormodel) to the judge as<last_assistant_turn>.policy.fixed_replies()andscope_guard.is_service_text()skip every service-written refusal, in every language. The judge sees the last reply a model wrote, or none.in_scope. A weather word, date or place alone never makes a messageneeds_data.needs_data3/3, and both together gavein_scope3/3 for all three cases.lang.encounts as no signal.defaultor an earlierpastedlanguage, neverdetectedorchosen.language: {language, source}. The page syncs its chip (lang: de · from data) and shows one notice.Measurements
eu) passed: 203 cases, adversarial recall 100 %, false refusals 0 %, exact 99.0 %.data_question, three of themafter_refusal, German and English) and adv-066 are all right.Plan
N/A
Test plan
pytest tests/unit tests/integration: 10,094 passed. ruff, ruff format, mypy and the changelog check pass.