[MarkusNeusinger]

Summary

• The prep eval measured the fake renderer, not the root. Under the default AGENT_RENDERER=fake, every run_data_code call got the fake's fixed table back: 4 rows of x and y = x². The root asked for a 500-step walk, saw 4 squares, and rightly ran its code again ("the data doesn't match the code, I'll run it again"). 9 of the 11 data-code misses came from this alone. The eval's data runs now go through HostDataBackend (agents/evals/host_data.py). It runs the real harness in data mode with the renderer's data-run limits. The local backend's reader reads the result back, extracted as collect_data_run in render/backends/local.py. So run_data_code gets the same DataOutput it would get from POST /run-data. Every data-code case compares table sizes again. An expected size may list two to four values where the user's code leaves the shape open: the random walk defines both steps and walk, so 2 or 3 columns both count.
• Two stale fixtures. Since feat(agents): find the table among pasted notes, metadata and totals #12166, the parser leaves out a summary row at the end of a paste. en-drop-total-row and de-hard-kette-gesamtzeile still expected the root to drop that row. The root was right to make no drop, and both cases scored as misses. The English case now has its total at the top, where the parser keeps it. The German chain case expects no second drop. A fixture test now refuses an expected drop_rows that drops nothing, so a parser change can't make a fixture stale silently again.
• Real model misses, fixed in the prompts. The scope judge refused "only orders from Bern that were delivered, then count per product" as needs_data. It also once judged pasted JavaScript data code as other_library. scope_judge.md now names both cases, with different example wording than the fixtures, and scope.evalset.json gets three new cases. The root answered "Zähle die Buchungen pro Kategorie" with query_data as if it were a question. root.md now says that an instruction to count or sum per group is a prepare_data aggregate.

The gate stays at 80 %.

Prep eval, Claude Haiku 5.5 on eu, 50 cases

Run	Exact	Data code	Cost
origin/main, before	36/50 = 72 % (FAIL)	2/11	$0.140
this branch, run 1	50/50 = 100 %	11/11	$0.142
this branch, run 2	49/50 = 98 %	11/11	$0.131

The one miss in run 2 is de-hard-kette-gesamtzeile: the root left out drop_columns Notiz. That is harmless, because the aggregate drops the column anyway, but it is a real model variance, so the fixture stays as it is.

Scope eval after the scope_judge.md change, 250 cases: PASS. False refusals 0 %, adversarial and refusal recall 100 %, exact 98.4 %, 4 unavailable (judge ValueError twice). Before the change, a subset run reproduced the false needs_data on filter-001.

Design note for the owner

HostDataBackend runs model-written Python on the developer's machine without the container. Only the eval uses it, and it refuses ENVIRONMENT=production. Three layers stand in front of it: the DATA validator, which run_data_code applies before any run; the harness's audit-hook guard; and rlimits with python -I and a minimal environment. The local backend's docstring rules out a bare-subprocess fallback for the product. This eval-only exception is a deliberate choice, so please confirm it or pick another option. The alternatives are to require Docker (AGENT_RENDERER=local) or the deployed renderer for the data-code cases.

Plan

No plan file. Brief: Block L, prep eval gate (2026-10-15).

Test plan

• tests/unit/agents/evals/test_host_data.py: pins the contract. A 500-step snippet comes back as 500×2, not the fixture table. Too many rows returns the harness exit code with measured value and cap. no_frame, error and blocked keep their exit codes. A timeout kills the process group. The environment is the renderer's data environment without RLIMIT_NPROC. Production is refused.
• test_prep.py: a scripted run through HostDataBackend stores 500 rows in one call. Covers size choices and eval_backend under fake, and checks that every expected drop_rows drops a row.
• pytest tests/unit/agents (6829 passed), ruff check/format, mypy agents/evals agents/anyplot/render.
• Live prep eval, twice (table above), plus the full live scope eval.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

[Copilot]

🔵 Needs a closer look

Executing model-written Python without container isolation requires explicit owner security review.

1 open finding

• Drain stderr during timeout cleanup to prevent process wait hangs · New

What changed in this PR

Corrects the prep eval so it measures the root agent’s data preparation rather than the fake renderer’s fixed output.

Changes:

• Executes eval data snippets through the real harness and compares table sizes.
• Updates stale fixtures and adds regression coverage.
• Clarifies prompt handling of aggregation requests, filters, and pasted data code.

File	Description
tests/​unit/​agents/​evals/​test_prep.py	Tests real data output, size alternatives, and fixture consistency.
tests/​unit/​agents/​evals/​test_host_data.py	Tests host execution and failure handling.
changelog.d/​agent-prep-eval-gate.md	Records eval and prompt fixes.
agents/​README.md	Documents revised eval behavior.
agents/​evals/​scope.evalset.json	Adds scope-classification cases.
agents/​evals/​prep.py	Selects the host backend and compares table sizes.
agents/​evals/​host_data.py	Adds eval-only host data execution.
agents/​evals/​fixtures/​prep/​en-drop-total-row.json	Moves the total row before data rows.
agents/​evals/​fixtures/​prep/​en-code-python-random-walk.json	Allows two or three output columns.
agents/​evals/​fixtures/​prep/​de-hard-kette-gesamtzeile.json	Removes the obsolete row-drop expectation.
agents/​anyplot/​render/​backends/​local.py	Extracts shared data-output collection.
agents/​anyplot/​prompts/​scope_judge.md	Clarifies filters and pasted data code.
agents/​anyplot/​prompts/​root.md	Distinguishes aggregation instructions from questions.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 92.44186% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
agents/anyplot/render/backends/local.py	80.00%	4 Missing ⚠️
agents/evals/host_data.py	94.80%	4 Missing ⚠️
agents/evals/prep.py	93.54%	4 Missing ⚠️
agents/anyplot/render/png.py	83.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

🔵 Needs a closer look

Host execution of model-written code requires owner security review, and malformed-operation matching still needs correction.

1 open finding

• Prevent rejected tool calls from aborting evaluation · New

1 resolved since last review

• Drain stderr during timeout cleanup to prevent process wait hangs

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

Running model-generated Python directly on the host introduces an intentional isolation exception that requires owner security review.

1 open finding

• Prevent rejected tool calls from aborting evaluation

🧠 Review effort: Balanced

[MarkusNeusinger]

Owner decision (2026-10-15): keep option (a), the eval runs model-written data snippets on the host under HostDataBackend (validator inside run_data, DataGuard audit hook, rlimits with the production CPU clamp, -I, a fresh minimal env, development/test only). Docker or the deployed renderer are not required for the prep eval.