From b74297f3bb50371446f2a59d5b15458905a5edaf Mon Sep 17 00:00:00 2001 From: Samran Asif Date: Wed, 9 Sep 2026 17:15:28 +0500 Subject: [PATCH] docs: assess whether this project should expose an MCP server Phase 16 of the plan asks for this evaluation and assumes the answer: "all four are strong candidates". Working through them one at a time, that is not what the code says. This project's verdict: yes - the strongest case of the four. The plan's assumption holds for api-verity-lab and devrepro-doctor, which answer bounded questions from files on disk and already emit structured artifacts. It does not hold for tooltrace-bench, where letting an agent invoke the scorer is a contamination problem rather than an engineering one -- an agent that can call `run` can run itself, retry until the score improves, and report the best result with the same provenance as an honest one. And it half holds for aihwbench, where the constraint is not danger but meaning: a benchmark invoked concurrently by two agents produces two wrong numbers and says nothing about it. Each document states what would be exposed, what would deliberately not be, and what has to be true first. Nothing is built. The point is to record the reasoning while it is fresh rather than to ship a surface nobody has thought about, and to be clear that this is an assessment rather than a capability the project has. The recurring "what has to be true first" across all four is path confinement: every candidate tool takes a file path, and an MCP server that hands an agent unrestricted filesystem reach through a path argument is a real hazard whatever the tool does with it. --- docs/MCP-EXPOSURE.md | 68 ++++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + pyproject.toml | 1 + 3 files changed, 70 insertions(+) create mode 100644 docs/MCP-EXPOSURE.md diff --git a/docs/MCP-EXPOSURE.md b/docs/MCP-EXPOSURE.md new file mode 100644 index 0000000..a03b4f2 --- /dev/null +++ b/docs/MCP-EXPOSURE.md @@ -0,0 +1,68 @@ +# Should devrepro-doctor expose an MCP server? + +**Verdict: yes — the strongest case of the four sibling projects — for the +read-only commands only, and never for `fix`.** + +This is an assessment, not a feature. Nothing here ships today. + +## Why this project fits better than the others + +*"Why won't this build on my machine?"* is a question agents get asked +constantly and answer badly, because the honest answer requires looking at the +machine. This tool looks at the machine and reports evidence. + +Two properties it already has make the fit unusually good: + +- **Read-only is the design, not a mode.** The scanning path does not modify + anything; that is the project's stated promise, not a flag. An MCP tool built + on it inherits that. +- **Output is already sanitized.** `devrepro/privacy/` redacts before anything + is serialized, and `tests/test_privacy.py` holds it there. An MCP server + hands its output to a model, which may forward it anywhere; a diagnostics + tool without that gate would be a genuinely bad thing to expose. + +The second point is the one worth dwelling on. Most machine-inspection tools +are unsafe to put behind an agent precisely because their output is full of +paths, usernames, tokens and environment variables. This one already treats +that as a correctness property. + +## What would be exposed + +| Tool | Answers | +|---|---| +| `doctor` | What is wrong with this machine, with evidence | +| `check` | Does this machine meet this project's declared requirements? | +| `info` | What is installed, and which versions? | +| `which` / `path` | Which binary actually resolves, and why that one? | +| `preflight` | Is this machine ready before a long build? | +| `diff` | Why does it work there and not here? | +| `snapshot` | Produce a sanitized manifest to hand to someone else | +| `rules` | What is checked, and what does each finding mean? | + +Every one already supports `--json`, so the MCP result schema is the existing +`ScanReport`. + +## What must never be exposed + +**`fix`.** It executes remediations, gated on an explicit `--yes`. That gate +exists because the project's rule is that nothing above LOW risk is applied +without a human agreeing. An MCP tool call has no human in it: the model +decides to call it. Exposing `fix` would move the confirmation from a person to +a model, which is precisely the thing the gate was built to prevent. + +`serve`, `server-backup` and `server-restore` are out for the same reason as +elsewhere: they start processes or move data. + +## What has to be true first + +1. **Exit codes do not survive the translation.** MCP returns structured + content, not a process status, and this project's most important signal is + `BLOCKED = 2` — which, as [EXIT-CODES.md](EXIT-CODES.md) records, means + something entirely different in the sibling projects. The tool result would + need to carry the verdict explicitly rather than relying on a convention + that does not cross the boundary. +2. **Scan cost.** A full `doctor` run takes seconds and shells out repeatedly. + That is fine for a CLI and slow for an interactive tool call; the server + would want a cached report with an explicit refresh. +3. **Confinement.** `check --project` takes a path. Same concern as anywhere + else: the server needs a configured root rather than trusting an argument. diff --git a/mkdocs.yml b/mkdocs.yml index 3826d11..8cbb61d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -41,6 +41,7 @@ nav: - Plugins: PLUGINS.md - Privacy: PRIVACY.md - Exit codes: EXIT-CODES.md + - MCP exposure: MCP-EXPOSURE.md - Releasing: release.md - Project: - Contributing: contributing.md diff --git a/pyproject.toml b/pyproject.toml index 29212d2..a3a987e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -102,6 +102,7 @@ ignore = [ ] [tool.ruff.lint.per-file-ignores] +"scripts/check_docs_site.py" = ["S603", "RUF100"] # Same shared-script situation: the gh endpoint is built from the hardcoded # REPOS list above it, not from user input. "scripts/fetch_competitor_meta.py" = ["S603", "S607", "RUF100"]