Skip to content

MCP documentation checker and README counts are outdated #1987

Closed

Description

@rongzhichen

What the harness reports, and why it is wrong

The MCP documentation checker reports that the server's tools were not
read because tools/list returns context, impact, link, path, and tests,
while its assertion expects only impact, path, and tests.

It also reports that source=True is an unsupported argument to tests.
The argument actually belongs to context. The tool-name matcher omits
context and link, so it attributes context's arguments to a preceding
recognized tool.

The server correctly exposes five tools, and context's schema accepts
source as a boolean. These failures come from outdated assumptions in
the checker.

How you isolated it

Compared the checker from these commits:

A — before:
dbcb49c

B — after:
40e4ed5

Loaded the checker source from each commit and executed it in memory
against the same server and documentation files. Those server and
documentation files are unchanged between the two commits. No parser
extraction or engine solve was involved.

Inspected:

  • The server's tools/list response and parameter schemas.
  • The checker's VERB matcher.
  • The expected tool-set assertion.
  • The (tool, argument) associations returned by documented().

Both arms received the same five server tools.

A: the documentation check reported five failures and returned 1.
B: the documentation check reported ok and returned 0.

Also compared three synthetic examples:

  1. tests() selects tests; context(task="example", source=True) explains flow.

    A associated task and source with tests.
    B associated both arguments with context and accepted them.

  2. MCP link with site=app.py:12 and target=run.

    A associated neither argument with a tool and rejected them.
    B associated both arguments with link and accepted them.

  3. MCP tests with source=True.

    Both arms rejected this invalid argument. This negative control
    confirms the fix preserves argument validation.

The checker changes are limited to recognizing the current tool names
and expecting the current five-tool set.

Activity

  1. github-actions commented on Oct 27, 2026

    @github-actions
    Contributor

    Closed by #1988, which merged into merge/dev-into-0.2.1 and reached dev in 30b456b.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testTest, fixture or gate coverage only

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions