From d93d78e5c9a4761dabb1329a9cf9ae677d691d3d Mon Sep 17 00:00:00 2001 From: Samran Asif Date: Wed, 9 Sep 2026 17:15:31 +0500 Subject: [PATCH] docs: assess whether this project should expose an MCP server Phase 16 of the plan asks for this evaluation and assumes the answer: "all four are strong candidates". Working through them one at a time, that is not what the code says. This project's verdict: mostly no - it would compromise the benchmark. The plan's assumption holds for api-verity-lab and devrepro-doctor, which answer bounded questions from files on disk and already emit structured artifacts. It does not hold for tooltrace-bench, where letting an agent invoke the scorer is a contamination problem rather than an engineering one -- an agent that can call `run` can run itself, retry until the score improves, and report the best result with the same provenance as an honest one. And it half holds for aihwbench, where the constraint is not danger but meaning: a benchmark invoked concurrently by two agents produces two wrong numbers and says nothing about it. Each document states what would be exposed, what would deliberately not be, and what has to be true first. Nothing is built. The point is to record the reasoning while it is fresh rather than to ship a surface nobody has thought about, and to be clear that this is an assessment rather than a capability the project has. The recurring "what has to be true first" across all four is path confinement: every candidate tool takes a file path, and an MCP server that hands an agent unrestricted filesystem reach through a path argument is a real hazard whatever the tool does with it. --- docs/mcp-exposure.md | 65 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 docs/mcp-exposure.md diff --git a/docs/mcp-exposure.md b/docs/mcp-exposure.md new file mode 100644 index 0000000..0b78840 --- /dev/null +++ b/docs/mcp-exposure.md @@ -0,0 +1,65 @@ +# Should tooltrace-bench expose an MCP server? + +**Verdict: mostly no — and this is the one project in the family where the +answer is not "yes".** The reading commands are fine; exposing the *running* +ones to an agent would compromise the benchmark. + +This is an assessment, not a feature. Nothing here ships today. + +## The direction that already exists + +This project is already an MCP **client**. `tooltrace/agents/mcp.py` speaks +JSON-RPC 2.0 over stdio to any MCP server, captures a tool inventory and +records every call into the trace, with conformance fixtures exercising +`initialize` → `tools/list` → `tools/call` against a fake server. + +That is the useful direction here: MCP servers are a thing this benchmark +*measures*, not a thing it needs to become. + +## Why exposing the runner is a bad idea + +The other three siblings answer questions. This one scores agents. Letting an +agent invoke the scorer is a different proposition, for three reasons: + +1. **Contamination.** A benchmark whose tasks an agent can enumerate at will, + through a tool the agent controls, stops measuring generalisation. The task + corpus is deliberately fingerprinted and contamination-tracked + (`TaskDefinitionV2.contamination`); handing out a `tools/list` of every task + works against that on purpose. +2. **Self-evaluation.** An agent that can call `run` can run itself, retry + until a score improves, and report the best one. Nothing in the protocol + distinguishes that from a legitimate call, and the resulting number would be + presented with the same provenance as an honest one. +3. **Cost and duration.** A benchmark run takes minutes and spawns sandboxes. + Tool calls are interactive; this is not interactive work. + +None of these are hypothetical failure modes of MCP — they are what happens +when the thing being measured controls the measurement. + +## What would be reasonable to expose + +The read side, which has none of those problems: + +| Tool | Answers | +|---|---| +| `tasks` | What tasks exist, with domain and difficulty | +| `scorers` | What scorers exist and what each asserts | +| `report` | Summarise an existing result bundle | +| `compare` | Compare two committed bundles | +| `stats` | Confidence intervals and flakiness for an existing run | + +Every one of these reads artifacts that already exist. None of them can be used +to obtain a score. + +## What has to be true first + +1. **A decision about task disclosure.** Even `tasks` hands over the corpus + inventory. That may be fine — the tasks are public in the repository — but + it should be a decision rather than a side effect. +2. **Bundle path confinement**, as everywhere else. +3. **Nothing that writes.** `run`, `benchmark`, `sweep` and `perturb` stay out, + permanently, for the reasons above rather than for want of effort. + +The honest summary: this project's MCP story is the client it already has. The +server would be a small convenience with a real integrity cost attached, and +the cost is larger than the convenience.