diff --application: one finding per changed agent, the first unresolved hop, grouped uncertainty (#914)

[pengfei-threemoonslab]

Closes #914

What a reviewer gets

diff --application opens with a Findings: line and one finding per changed agent, then the rows. The JSON gains an additive top-level summary with the same content, which the text is rendered from, so #915 can print the same finding.

• Capability. What the changed tools newly reach (Application diff rows should name what a bound tool reaches: endpoint, action, credential, model-supplied arguments #872, Tool reach: name effects beyond HTTP — databases, subprocesses, files, cloud SDKs and messaging #913, Application diff: read tools bound as objects — MCP toolsets, agent-as-tool and hosted tools #910): the outbound call or library effect, host, credential names, model-controlled arguments; for a changed tool, still reaches or changed reach (the same call newly taking a model-controlled argument or sending a credential).
• First unresolved hop. A finding with an unread binding, an unread reach or an incomplete agent graph is partial and says not read beyond <place>; each tool line keeps its first unread hop and a count of the rest. comparison_status still speaks of bindings only, so a compared answer can carry a partial finding.
• Grouped uncertainty. not_established rows with the same reason (names and places aside) are one line with a count and one example; effect evidence is counted once; limits no row states are one line per agent.
• A question specific to the change. A new reach (with the credentials it sends and the arguments the model controls), the same call newly taking an argument, a removed tool, a changed one and what changed.
• Detail. Per-row text follows under Detail: for up to 10 rows. Above that the text says how many rows it did not print; --json always carries every row, gap and reach limit. No new flag.

Presentation only

• No row, status, direction, gap, scope or exit code moves. summary is derived from the sanitized rows and names the rows it summarizes by index; it states nothing a row does not.
• application_comparison_schema_version stays 0.4: it is unreleased (v1.2.0 shipped 0.2), as Tool reach: name effects beyond HTTP — databases, subprocesses, files, cloud SDKs and messaging #913 did for reach.effects. summary extends it. comparison_id digests summary with the rows, so the documented recomputation still holds.
• benchmark/application-q2/summarize.py leaves summary out of answer_id, so a hand score keeps judging the answer it read.

Evidence

Development corpus (49 PRs), origin/main at e703b41b against this branch's src/ from a snapshot worktree (benchmark/application-q2/run.py --engine <snapshot>/shipgate): every member ok; summarize.py output is identical (statuses, row counts by kind, sides with calls or unresolved hops, with and without the 2026-10-05 scores); 49 of 49 answers have the same answer_id, status and rows.

PR	text before	text after	before the rows
VidulaWickramasinghe/O.R.I.O.N#126 (237 rows)	11,926 lines	24 lines	21
Tiendat2703/MIS_TALENT#7 (2 rows)	59 lines	67 lines	10
jpka/attest#3 (2 rows)	23 lines	29 lines	7

Small answers grow by the summary and keep their rows under Detail:. benchmark/application-q2/goldens/ holds the text of six real answers (those three, xju2/hepagent#72, buriro-ezekia/cinescout-ai#9, cianfhoghlaim/tuatha#1) and, beside each, the reduced answer tests/test_application_summary.py renders in CI; goldens.py regenerates and checks them from a run.

Tests

• tests/test_application_summary.py: findings from git fixtures (reach with credential and model-controlled argument, an unread hop, a removal, a changed argument, 24 rows sharing a cause), every summary statement traceable to the rows it names, the JSON differing only by summary, no dependence on key order, control characters in names, the goldens and the docs quoting lines the goldens print.
• tests/test_application_q2_benchmark.py: answer_id ignores summary.
• tests/test_distribution_surface_parity.py and docs/distribution-surfaces.md: the new module and its no-claim note.
• Full CI mirror (-n auto -m "not perf" --ignore=tests/test_adapter_static_only.py) exits 1 only on test_check_unmodelled_host_config_keys.py[local_settings_enabled_plugins-*], which fails on this machine only; tests/test_adapter_static_only.py, ruff check . and scripts/generate_schemas.py --check exit 0.

Notes for the reviewer

• On origin/main contract form hot fix Tiendat2703/MIS_TALENT#7 yields 2 rows; the 2026-10-05 ledger records 3 (Finance_Agent's load_and_validate changed). Runs of main just before Application diff: read tools bound as objects — MCP toolsets, agent-as-tool and hosted tools #910 already lack the row, so it went missing between that ledger's build and Application diff: read tools bound as objects — MCP toolsets, agent-as-tool and hosted tools #910. This PR neither causes nor changes it; the new "Also not read" line names Finance_Agent as having unread limits.
• O.R.I.O.N#126 reports four findings named orion: the agent is built in api_main.py, main.py, voice_main.py and wake_main.py, each its own agent in the comparison.

🤖 Generated with Claude Code