From 38afa4771384e6267363368a3cdce3240cf4cbcc Mon Sep 17 00:00:00 2001 From: Samran Asif Date: Wed, 9 Sep 2026 17:15:25 +0500 Subject: [PATCH] docs: assess whether this project should expose an MCP server Phase 16 of the plan asks for this evaluation and assumes the answer: "all four are strong candidates". Working through them one at a time, that is not what the code says. This project's verdict: yes, for a read-only subset. The plan's assumption holds for api-verity-lab and devrepro-doctor, which answer bounded questions from files on disk and already emit structured artifacts. It does not hold for tooltrace-bench, where letting an agent invoke the scorer is a contamination problem rather than an engineering one -- an agent that can call `run` can run itself, retry until the score improves, and report the best result with the same provenance as an honest one. And it half holds for aihwbench, where the constraint is not danger but meaning: a benchmark invoked concurrently by two agents produces two wrong numbers and says nothing about it. Each document states what would be exposed, what would deliberately not be, and what has to be true first. Nothing is built. The point is to record the reasoning while it is fresh rather than to ship a surface nobody has thought about, and to be clear that this is an assessment rather than a capability the project has. The recurring "what has to be true first" across all four is path confinement: every candidate tool takes a file path, and an MCP server that hands an agent unrestricted filesystem reach through a path argument is a real hazard whatever the tool does with it. --- docs/mcp-exposure.md | 66 ++++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 67 insertions(+) create mode 100644 docs/mcp-exposure.md diff --git a/docs/mcp-exposure.md b/docs/mcp-exposure.md new file mode 100644 index 0000000..71e8dc9 --- /dev/null +++ b/docs/mcp-exposure.md @@ -0,0 +1,66 @@ +# Should api-verity-lab expose an MCP server? + +**Verdict: yes, for a deliberately small read-only subset. Not yet built.** + +This is an assessment, not a feature. Nothing here ships today; the point is to +record what the surface would be and what has to be true before it exists. + +## Why this project fits + +An agent editing an API spec has a question this tool already answers exactly: +*did that change break anything, and for whom?* The answer is deterministic, +fast, needs no network, and is already emitted as a structured `result-v1` +artifact rather than prose. That is close to an ideal MCP tool: a typed +question with a typed answer and no side effects. + +It also fits the failure mode MCP tools tend to have. An agent asking a model +"is this breaking?" gets a plausible answer. Asking this tool gets one with a +rule id behind it that the agent can quote and a human can check. + +## What would be exposed + +Nine of the nineteen commands are pure functions of files on disk: + +| Tool | Answers | +|---|---| +| `validate` | Is this document a valid contract, and what is wrong with it? | +| `diff` | What changed between these two versions? | +| `breaking` | Which changes break consumers, and in which direction? | +| `changelog` | Render those changes as a human changelog | +| `coverage` | Which operations does this test suite actually touch? | +| `rules` | What rules exist, with ids and severities? | +| `plugins` | Which spec formats are installed? | + +Each already supports `--json` and each emits a `result-v1` artifact, so the +MCP schema is the schema the project already publishes. That is most of the +work already done. + +## What would deliberately not be exposed + +- `drift`, `replay`, `baseline`, `regression` — these contact a target. An + agent should not be able to send traffic to an arbitrary base URL because a + prompt told it to. If they are ever exposed it must be behind an explicit + allowlist supplied by the human, not a parameter the model chooses. +- `mock`, `serve`, `server-db` — these start listeners and hold state. A tool + call that leaves a process running is not a tool call. +- `export` — writes bundles to disk. + +The split is not arbitrary: it is the same boundary +[SAFETY_MODEL.md](safety-model.md) already draws between commands that read and +commands that reach out. + +## What has to be true first + +1. **A stable tool schema.** `result-v1` is versioned, but the MCP tool + *inputs* would be a new public contract with the same + never-change-a-meaning rule as [exit codes](exit-codes.md). Worth designing + once rather than growing. +2. **Path confinement.** Every exposed tool takes file paths. An MCP server + handing an agent unrestricted read access to the filesystem via a spec-path + argument is a real hazard; the server would need a configured root. +3. **A decision about who ships it.** An `apiverity-mcp` extra keeps the core + dependency-free, which matters more here than convenience. + +None of these is hard. They are simply not free, and the honest position is +that this is a good idea that has not been built rather than a capability the +project has. diff --git a/mkdocs.yml b/mkdocs.yml index 28427b0..486d571 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -37,6 +37,7 @@ nav: - Spec support: spec-support.md - Protocol support: protocol-support.md - Exit codes: exit-codes.md + - MCP exposure: mcp-exposure.md - Guides: - CI contract gate: ci.md - SDK: sdk.md