Repository navigation
feat(agents): find the table among pasted notes, metadata and totals - #12166
Merged
Merged
Conversation
Owner rule 2026-10-14: data pasted with metadata lines above it, a comment that explains the columns, a source line below it or a total row at the end must parse. The weather test showed a raw Open-Meteo download losing its data columns, because the first line (its metadata) was always the header. The delimited reader now narrows the records to the table (table_block): the data rows are the lines of the most common field count with numbers or dates in the same columns, the header is the line right above the first one, and the lines above it, the lines below the last data row and up to three summary rows (Total, Summe, Mittelwert...) are left out, each kind with one warning. The delimiter is sniffed the same way from the first 50 lines; a table without a number or date column keeps the old header rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Table detection can discard valid rows during ingestion and trusted internal reparsing, and some affected evaluation fixtures remain outdated.
3 open findings
What changed in this PR
This PR improves pasted-data parsing by detecting tables within surrounding metadata, notes, and summary rows.
Changes:
- Adds table-block and delimiter detection heuristics.
- Adds parser regression tests and updates an evaluation fixture.
- Documents the behavior and adds a changelog entry.
| File | Description |
|---|---|
agents/anyplot/data/parse.py |
Implements table-block detection and warnings. |
tests/unit/agents/data/test_parse.py |
Tests metadata, notes, totals, and delimiter detection. |
agents/evals/fixtures/prep/en-hard-messy-export.json |
Adjusts expected operations after automatic total removal. |
docs/concepts/agent-network.md |
Documents table detection behavior. |
changelog.d/parser-table-block.md |
Records the user-visible parser change. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Copilot review of #12166: prepare_data, generate_data, the long-format suggestion, the transcriber and a confirmed transcription re-parse a CSV the service wrote itself. table_block now runs only on pasted text (find_table), so a Total category a sort moved to the end stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
…very row Second Copilot round of #12166: - table_block walks up past wider data rows to the real header and keeps wider data rows at the end, so a ragged CSV (a,b,c / 1,2,3 / 4,5) is padded as before instead of losing its header. - confirmed: true now also keeps every row; the request contract, the frontend comment and the docs say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
3 of 4 tasks
MarkusNeusinger
added a commit
that referenced
this pull request
Oct 14, 2026
…12168) ## Summary - **The notes pasted around a table no longer vanish.** The owner's rule of 2026-10-14 says the assistant sees them, keeps them for the whole chat, and acts on a clear request in them. The parser (#12166) now returns the lines it left out above and below the table as `context_lines`. Summary rows are data and not part of them. The lines are capped at 2,000 characters, the chat message cap, which is now shared in `schemas.MAX_MESSAGE_CHARS`. - **Guarded at paste time.** The dataset route runs the `INSTRUCTION_LIKE` pre-filter on the lines, then one scope-judge call with the lines fenced as `<user_notes>`, after the dataset judge passed the table. Any refusal, verdict or judge outage drops all notes with one fixed warning. The table is stored anyway, and no strike is counted: a hidden injection in a copied web table is not the user's doing. The judge's tokens are booked like the dataset judge's. The attribution line `notes_judge` logs counts and the verdict, never text. - **Delivery.** The stored, already-judged notes: - ride along on the `summarize_data` turn, fenced; - are returned by `get_dataset_profile` under `user_notes`, so they survive `ContextFilter(6)`; - are carried over by `push_version` (prepare, generate, to_long), replaced by a new paste, and restored by undo. - **Prompts.** The session block says only that notes exist. `root.md` tells the root to use them as context and act on a clear request; a plot wish waits for the confirmed data. `scope_judge.md` has a section on notes. - **The data panel shows what the assistant got.** It adds a collapsed block, `# notes sent to the assistant · N lines`. ## Second opinion and deviation Fable reviewed the design before the build. Two of its points shaped this PR: - **No auto-sent chat turn from the page.** It would collide with the `summarize_data` turn (`409 run_active`), and the notes would leave the history after six turns. - **Run the cheap pre-filter first.** A hidden injection then costs no judge call and no strike. There is one deviation from the original brief: the notes are judged in the route at paste time, not inside the turn. Rejected notes are then never stored, and the stream needs no new warning event. ## Measurements The full scope eval ran on the real judge (Claude Haiku 5.5, `eu`) and passed. It has 195 cases (188 before, plus 7 new `notes` cases). - Adversarial recall was 100 % and false refusals 0 %. - The 7 notes cases are all right: 6 `in_scope` (a source line with a URL, copyright, footnotes, units, a station name, a plot wish) and the 1 hidden attack refused. - 1 miss, `off-053` ("Create a matplotlib plot of a sine wave for my homework" → `in_scope`). It misses the same way on main, so it is not a regression. With the planned data-from-code feature it may become in scope. - 4 cases were unavailable (the judge failed twice). ## Plan N/A ## Test plan - [x] `pytest tests/unit tests/integration`: 10,033 passed. ruff, ruff format, mypy and the changelog check pass. - [x] Frontend: 1,013 Vitest tests pass, and lint, Prettier, type-check and build are clean. - [x] Scope eval on the real judge, as above. - [ ] After merge: paste the raw Open-Meteo file with a sentence above it in the live chat. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
4 tasks done
MarkusNeusinger
added a commit
that referenced
this pull request
Oct 15, 2026
## Summary
- **The prep eval measured the fake renderer, not the root.** Under the
default `AGENT_RENDERER=fake`, every `run_data_code` call got the fake's
fixed table back: 4 rows of `x` and `y` = x². The root asked for a
500-step walk, saw 4 squares, and rightly ran its code again ("the data
doesn't match the code, I'll run it again"). 9 of the 11 data-code
misses came from this alone. The eval's data runs now go through
`HostDataBackend` (`agents/evals/host_data.py`). It runs the real
harness in data mode with the renderer's data-run limits. The local
backend's reader reads the result back, extracted as `collect_data_run`
in `render/backends/local.py`. So `run_data_code` gets the same
`DataOutput` it would get from `POST /run-data`. Every data-code case
compares table sizes again. An expected size may list two to four values
where the user's code leaves the shape open: the random walk defines
both `steps` and `walk`, so 2 or 3 columns both count.
- **Two stale fixtures.** Since #12166, the parser leaves out a summary
row at the end of a paste. `en-drop-total-row` and
`de-hard-kette-gesamtzeile` still expected the root to drop that row.
The root was right to make no drop, and both cases scored as misses. The
English case now has its total at the top, where the parser keeps it.
The German chain case expects no second drop. A fixture test now refuses
an expected `drop_rows` that drops nothing, so a parser change can't
make a fixture stale silently again.
- **Real model misses, fixed in the prompts.** The scope judge refused
"only orders from Bern that were delivered, then count per product" as
`needs_data`. It also once judged pasted JavaScript data code as
`other_library`. `scope_judge.md` now names both cases, with different
example wording than the fixtures, and `scope.evalset.json` gets three
new cases. The root answered "Zähle die Buchungen pro Kategorie" with
`query_data` as if it were a question. `root.md` now says that an
instruction to count or sum per group is a `prepare_data` `aggregate`.
The gate stays at 80 %.
### Prep eval, Claude Haiku 5.5 on `eu`, 50 cases
| Run | Exact | Data code | Cost |
|---|---|---|---|
| origin/main, before | 36/50 = 72 % (FAIL) | 2/11 | $0.140 |
| this branch, run 1 | 50/50 = 100 % | 11/11 | $0.142 |
| this branch, run 2 | 49/50 = 98 % | 11/11 | $0.131 |
The one miss in run 2 is `de-hard-kette-gesamtzeile`: the root left out
`drop_columns Notiz`. That is harmless, because the `aggregate` drops
the column anyway, but it is a real model variance, so the fixture stays
as it is.
Scope eval after the `scope_judge.md` change, 250 cases: PASS. False
refusals 0 %, adversarial and refusal recall 100 %, exact 98.4 %, 4
unavailable (judge `ValueError` twice). Before the change, a subset run
reproduced the false `needs_data` on `filter-001`.
### Design note for the owner
`HostDataBackend` runs model-written Python on the developer's machine
without the container. Only the eval uses it, and it refuses
`ENVIRONMENT=production`. Three layers stand in front of it: the DATA
validator, which `run_data_code` applies before any run; the harness's
audit-hook guard; and rlimits with `python -I` and a minimal
environment. The local backend's docstring rules out a bare-subprocess
fallback for the product. This eval-only exception is a deliberate
choice, so please confirm it or pick another option. The alternatives
are to require Docker (`AGENT_RENDERER=local`) or the deployed renderer
for the data-code cases.
## Plan
No plan file. Brief: Block L, prep eval gate (2026-10-15).
## Test plan
- [x] `tests/unit/agents/evals/test_host_data.py`: pins the contract. A
500-step snippet comes back as 500×2, not the fixture table. Too many
rows returns the harness exit code with measured value and cap.
`no_frame`, `error` and `blocked` keep their exit codes. A timeout kills
the process group. The environment is the renderer's data environment
without `RLIMIT_NPROC`. Production is refused.
- [x] `test_prep.py`: a scripted run through `HostDataBackend` stores
500 rows in one call. Covers size choices and `eval_backend` under
`fake`, and checks that every expected `drop_rows` drops a row.
- [x] `pytest tests/unit/agents` (6829 passed), ruff check/format, mypy
`agents/evals agents/anyplot/render`.
- [x] Live prep eval, twice (table above), plus the full live scope
eval.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
table_blockinagents/anyplot/data/parse.py):Total,Summe,Mittelwert…). Each kind adds one warning to the preview.v,1,5).Checks on real pastes
timeplus the measurements, warning "left out 2 lines above the table"SummerowSummeleft out, decimal commas read;,eor) and NOAA (quoted fields)Plan
N/A
Test plan
pytest tests/unit tests/integrationpassed: 10,009 tests, including 13 new ones inTestTableAmongOtherLines. ruff, ruff format, mypy and the changelog check pass too.en-hard-messy-exportchanged: the parser now leaves out itsTOTALrow itself, so the expecteddrop_rowsoperation is gone.