Repository navigation
fix(agents): reviewer confirms requested texts when there is no probe - #12454
Merged
Merged
Conversation
…as no probe R writes no probe, so every ggplot2 request for a specific text (a title, an axis label) shipped as needs_attention with UNVERIFIED_LINE, although the image showed the text (live test 2026-10-29, line-basic, "Temperatur (°C)"). Owner decision of 2026-10-29: without a probe, the reviewer that already reviews every render confirms the texts in the image. Design (decided and documented here): - Gate: a new RuntimeAdapter.review_texts flag. RRuntime sets True; PythonRuntime keeps False, so every Python run is unchanged (see the finding below). The flag is read only when the render has no drawn texts (drawn is None) and AGENT_EXPECT_CHECK is not off. - Transport: ReviewRequest.expected_texts, filled only for such a render. Both request renderers add a fixed heading (REVIEW_EXPECTED_HEADING) and the texts fenced as <user_message>; an empty list adds nothing, so the reviewer's static prompts (reviewer.md, reviewer_catalogue.md, scorer.md), its output schemas and every review request of a render with the probe are byte-identical. No new model call: the texts ride in the review the pipeline already makes. - Answer shape: the existing defect grammar. The reviewer reports a text it cannot find as one finding under SC-04 (SC-02 for a number format in the catalogue rubric; the checklist has no SC-02, so SC-04 there) whose observed part quotes it. grounding.reviewed_misses reads these back from every line of the answer, whatever the points: a line names a requested text when a quoted passage normalises to it (a quoted "Temperatur (°C)" never names a requested "Temperatur"); a line without quotes names each requested text it contains whole. - Status and repair mirror the probe's miss: under AGENT_EXPECT_CHECK= repair a reported miss sends the round to the repair whatever the points (as a VISIBLE_DEFECT_IDS line does), the render is not clean for AGENT_SHIP_BEST, and a shipped miss keeps the reviewer's line as a residual line plus the text in PlotShows.requested_not_found. Under note it is the residual line only. - UNVERIFIED_LINE is set at render time as before and cleared only by a readable review of that render, so an unreviewed render (budget, deadline, unreadable answer, not_rereviewed, padded canvas) keeps it: never a silent pass. A readable review that names no miss counts as confirming the texts. Finding for the owner: altair, pygal and lets-plot (Python) write no probe either and still get UNVERIFIED_LINE for every requested text. Moving them to the reviewer is setting PythonRuntime.review_texts (or a per-library value) to True, which changes their review requests, so it is left to the owner. Tests: reviewed_misses (quotes, typographic quotes, shorter-in-longer, SC-02, other ids), finish() with a confirmed and a missed text, the request rendering in both modes and its absence without texts, the runtime flags, and catalogue flow runs without drawn texts: confirmed -> ok; missed -> repair round despite passing points; unfixed miss in repair and note mode -> needs_attention with the reviewer's line; unreadable review -> UNVERIFIED_LINE stays. The probe flow tests now assert that the review request carries no texts. Docs: docs/concepts/agent-network.md, the AGENT_EXPECT_CHECK setting, the R runtime docstring, and a changelog fragment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
…t too altair, pygal and lets-plot render without matplotlib and write no probe, so every requested text there got the same "could not be checked" note as ggplot2. The owner's decision (reviewer confirms the texts when there is no probe) covers them: `PythonRuntime.review_texts` is now a property by library (`TEXTLESS_LIBRARIES`), and `RuntimeAdapter.review_texts` a read-only protocol member. matplotlib, seaborn and plotnine keep the probe's check; the Python prompts are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
…form
The Gemini Flash pre-PR check found two ways `reviewed_misses` misread a
review line: a requested text with quotes of its own (`Zone "A"`) was
split by the quote pattern and never named (a silent pass), and a line
that also quoted what the image shows instead ("(the y axis reads
"Score")") named that text as missing when the user had asked for it too.
A line that quotes only the number it saw dropped a missing number format.
A line in the heading's own form, `the requested text "<text>"`, now names
exactly the texts it cites so, compared with every quotation mark kept
(`_norm(quotes=True)`). Other lines keep the quoted-passage rule, and fall
back to whole-text containment for a text with quotes of its own or a
number format.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The checklist branch remains insufficiently covered because its distinct Verdict schema and defect parsing are only exercised through shared helpers, so end-to-end tests for confirmed text, reported misses, and unreadable reviews are needed.
1 open issue
What changed in this PR
This pull request enables reviewer-based verification of requested text for renders without text probes, including ggplot2, Altair, Pygal, and Lets-Plot. Reported missing text is propagated through repair and result grounding, while existing probe-based behavior remains unchanged for other runtimes.
Changes:
- Adds runtime opt-in support for reviewer-backed text verification.
- Carries expected texts into eligible review requests and parses reported misses.
- Integrates missing-text findings into repair and residual-result handling.
- Adds tests, documentation, and a changelog entry.
| File | Summary | Review findings |
|---|---|---|
tests/unit/agents/runtime/test_runtimes.py |
Verifies runtime review-text configuration. | No findings. |
tests/unit/agents/runtime/test_grounding.py |
Tests reviewer-line parsing and prompt construction. | No findings. |
tests/unit/agents/runtime/test_catalogue_flow.py |
Tests confirmation, repair, residual, and unreadable-review flows. | No findings. |
docs/concepts/agent-network.md |
Documents reviewer-backed requested-text verification. | No findings. |
changelog.d/review-requested-texts.md |
Records the user-visible fix. | No findings. |
agents/anyplot/settings.py |
Documents expanded expected-text checking behavior. | No findings. |
agents/anyplot/schemas.py |
Adds expected texts to review requests. | No findings. |
agents/anyplot/render/runtimes/r.py |
Enables reviewer checks for R renders. | Nit (1 vote): Runtime prompt instructions remain stale and should be updated with prompt tests. |
agents/anyplot/render/runtimes/python.py |
Enables reviewer checks for supported probe-less Python libraries. | No findings. |
agents/anyplot/render/contract.py |
Extends the runtime adapter contract. | No findings. |
agents/anyplot/pipeline.py |
Integrates reviewer text checks into rendering, repair, and result handling. | Moderate (1 vote): Checklist defect limits can drop requested-text findings. Nit (4 votes): The checklist branch lacks end-to-end flow coverage. |
agents/anyplot/grounding.py |
Parses reviewer-reported requested-text misses. | Moderate (1 vote): Fallback parsing can misclassify unrelated quoted passages as missing requested text. |
Review effort: Lite
…to end Copilot's review of #12454 (and the Flash pre-PR check): the checklist reviewer's Verdict schema was exercised only through the shared helpers. Three flow cases now run it: a confirmed text ships ok, a reported miss goes to the repair, and an answer that is no verdict keeps the note. The R adapter's requested_check slot no longer says the result marks the texts as not checked: the reviewer looks for them now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
needs attentionwith "the requested texts could not be checked", although the image was right (ggplot2 live test on anyplot.ai, 2026-10-29). Owner decision of 2026-10-29: the reviewer now gets the requested texts and confirms them in the image, in the review call it already makes (no new model call).RuntimeAdapter.review_textsgates it (R: on; Python: on foraltair,pygal,letsplot, off for matplotlib, seaborn and plotnine, which keep the probe's check unchanged).ReviewRequest.expected_textscarries the texts only for such a render, under a fixed heading, fenced as user text. The reviewer reports a text it cannot find as anSC-04(orSC-02) line in the prescribed formthe requested text "<text>";grounding.reviewed_missesreads those lines back, and the pipeline treats them like the probe's misses (a repair underAGENT_EXPECT_CHECK=repair, a residual line if no round fixes it). The "could not be checked" note stays only when no readable review answered.reviewed_misses(a requested text with quotes of its own was never named; a passage quoted for what the image shows was named as missing) and a dropped number format; all three are fixed in f29100d with tests. Its third finding (no checklist-mode flow test) is left: the checklist path calls the samereviewed_misses, and the catalogue mode is the default.Plan
N/A (follow-up of the ggplot2 live test;
agentic/specs/261024-agent-all-libraries.mdcovers the R runtime)Test plan
uv run ruff check agents tests && uv run ruff format --check agents testsuv run --extra typecheck --extra agents mypy agents(115 files, clean)uv run --extra agents --extra test pytest tests/unit/agents -q -n auto(13,090 passed, 27 skipped for plotting libraries not installed locally)okwithout a note; a reported miss → one repair round although the score passed; an unfixed miss inrepairandnotemode → the reviewer'sSC-04line; an unreadable review → the note stays; probe renders → the review request carries no textsuv run python -m tools.changelog check --base origin/mainokwithout the "could not be checked" note