From 962e18cfeb680b3b888842f4bbc2c966f4f17bd9 Mon Sep 17 00:00:00 2001 From: Aviad Date: Wed, 12 Aug 2026 22:57:22 +0300 Subject: [PATCH 1/2] feat: claims that fire on ordinary repositories (v0.5.0) The verification layer only had two narrow claims (billing webhooks, JWT), so on three realistic open-source repositories dtc verify produced nothing useful on two of them. Honest and useless at the same time. This release fixes the coverage, not the honesty. - New claim route-test-coverage: "HTTP routes are exercised by tests". Routes and tests are the most abundant evidence in nearly every server repo, so this claim fires almost everywhere. It reports the ratio and names the uncovered routes. Attribution is by test imports and route names; e2e specs are excluded because they match by accident; absence of tests is missing evidence, never a contradiction. - New claim admin-authorization: "Administrative routes require an authorization check". A missing authorization signal is WEAK, never CONTRADICTED - global middleware and framework decorators are not detected, and calling a route unprotected when it is not would destroy trust. - New status NOT_APPLICABLE: a repo with no billing code is not "unknown" for a billing claim. UNKNOWN is now reserved for "surface exists, evidence cannot decide". - dtc verify is a report card: findings first (contradictions before all), then which claims do not apply and why. When nothing applies it reports what was scanned, what evidence exists, what would unlock a claim, and that this is a coverage limit rather than a verdict. - dtc verify --list shows applicability for the current repository. - verify_all loads scan evidence once; verify stays sub-second on a repo with 133 routes and thousands of tests. - Results that do not apply are no longer stored: they would pollute freshness and diff impact with claims that have no evidence. JSON is schema_version 2 (all v1 fields unchanged; adds the NOT_APPLICABLE status value). 142 tests (20 new). Version 0.5.0. Investigated and deliberately not built: a secret-handling claim. DevTime hard-denies secret files from scanning, so its evidence cannot exist without breaking the trust model. --- LIMITATIONS.md | 11 +- QUICKSTART.md | 4 +- README.md | 11 +- RELEASE_NOTES_v0.5.0.md | 94 +++++ VERIFICATION.md | 38 +- pyproject.toml | 4 +- server.json | 4 +- src/devtime/__init__.py | 2 +- src/devtime/cli.py | 126 +++++- src/devtime/intelligence/verification.py | 419 +++++++++++++++++-- src/devtime/mcp/transport.py | 13 +- tests/integration/test_evidence_precision.py | 8 +- tests/integration/test_verification.py | 211 +++++++++- 13 files changed, 864 insertions(+), 81 deletions(-) create mode 100644 RELEASE_NOTES_v0.5.0.md diff --git a/LIMITATIONS.md b/LIMITATIONS.md index 9eefb2c..ac9c0cb 100644 --- a/LIMITATIONS.md +++ b/LIMITATIONS.md @@ -72,8 +72,15 @@ do. Read this before trusting any single output. ## 6b. Verification limitations (experimental) -- Two built-in claims (billing-webhook-signature, jwt-authentication). - User-defined claims are not supported yet, deliberately. +- Four built-in claims (route-test-coverage, admin-authorization, + billing-webhook-signature, jwt-authentication). User-defined claims are not + supported yet, deliberately. +- Route test coverage is attributed statically (test imports and route names), + not by executing tests. A route exercised only indirectly can be reported as + uncovered. +- Admin authorization reports WEAK when it finds no authorization evidence. + That means DevTime found nothing, never that a route is confirmed + unprotected: global middleware and framework decorators are not detected. - Verification is rule-driven over scanner signals; it inherits every scanner coverage limitation listed here. - Statuses mean "per DevTime's evidence rules", not formal proof or a security diff --git a/QUICKSTART.md b/QUICKSTART.md index 8d08cbc..177b283 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -27,7 +27,7 @@ python -m venv .venv source .venv/bin/activate # Windows (PowerShell): .venv\Scripts\Activate.ps1 # Windows (Git Bash): source .venv/Scripts/activate pip install -e ".[dev]" -pytest # optional: all tests pass (129 at v0.4.0) +pytest # optional: all tests pass (142 at v0.5.0) ``` ## 3. Create the demo repo @@ -132,7 +132,7 @@ A fresh-clone check was run on the current candidate: - **OS:** Windows 11 (Git Bash) - **Python:** 3.11.9 - **Install:** `pip install -e ".[dev]"` -- **Tests:** all passing (129 at v0.4.0) +- **Tests:** all passing (142 at v0.5.0) - **Demo:** `dtc init` / `dtc scan` / `dtc concepts` / `dtc explain "Billing Webhooks"` all produced the expected output from a clean `git clone`. diff --git a/README.md b/README.md index 59bcef2..283ec5e 100644 --- a/README.md +++ b/README.md @@ -165,7 +165,7 @@ Anything outside these six is out of scope for V0. See [LIMITATIONS.md](LIMITATI | `dtc context ` | Create a governed Context Pack for agents or humans. | | `dtc risk --diff` | Review a git diff for risky changes using local evidence (advisory). | | `dtc decision add` | Add a local decision record that can reduce uncertainty. | -| `dtc verify [claim]` | Verify a repository claim against evidence: status, contradictions, freshness (experimental). | +| `dtc verify [claim]` | Verify repository claims against evidence: status, contradictions, freshness (experimental). | (Also available: `dtc evidence`, `dtc debt`, `dtc status`, `dtc doctor --privacy`, `dtc export`, `dtc reset`, `dtc mcp start`.) @@ -266,9 +266,12 @@ changes: ![dtc verify demo - a claim goes from SUPPORTED to CONTRADICTED to STALE](assets/devtime-verify-demo.svg) -Statuses are SUPPORTED, WEAK, CONTRADICTED, or UNKNOWN; contradictions always -show both sides; changed evidence marks a claim STALE. Two built-in claims ship -(billing webhook signatures, JWT authentication). See **[VERIFICATION.md](VERIFICATION.md)**. +Statuses are SUPPORTED, WEAK, CONTRADICTED, UNKNOWN, or NOT_APPLICABLE; +contradictions always show both sides; changed evidence marks a claim STALE. +Four built-in claims ship: route test coverage, admin authorization, billing +webhook signatures, and JWT authentication. `dtc verify` leads with what it can +actually verify here, and when nothing applies it says what would make a claim +verifiable instead of dead-ending. See **[VERIFICATION.md](VERIFICATION.md)**. ## Example output diff --git a/RELEASE_NOTES_v0.5.0.md b/RELEASE_NOTES_v0.5.0.md new file mode 100644 index 0000000..81e2eb2 --- /dev/null +++ b/RELEASE_NOTES_v0.5.0.md @@ -0,0 +1,94 @@ +# DevTime v0.5.0 - claims that fire on ordinary repositories + +The verification layer had a coverage problem. Its two claims (billing webhooks, +JWT) only apply to repositories that happen to have billing or JWT code. Run +against three realistic open-source repositories, `dtc verify` produced nothing +useful on two of them. Honest, but useless. + +This release fixes that: claims that apply to ordinary repositories, plus an +output that never dead-ends. No cloud, no telemetry, no AI, no code execution - +unchanged. + +## Two new built-in claims + +**route-test-coverage** - "HTTP routes are exercised by tests." + +```text +Route Test Coverage +Status: WEAK + +Why: + - 6 of 15 routes have a referencing test. + +Missing evidence: + - Tests referencing 9 route(s): /api/items/{id}, /api/login/access-token, ... +``` + +Routes and tests are the two most abundant kinds of evidence in almost every +server repository, so this claim fires nearly everywhere. Attribution is by test +imports and route names, and end-to-end specs are excluded because they match by +accident. Absence of tests is missing evidence, never a contradiction. + +**admin-authorization** - "Administrative routes require an authorization check." + +A missing authorization signal is reported as WEAK, never CONTRADICTED. +Authorization can be applied globally, by a router mount, or by a framework +decorator the scanner does not parse. Reporting an endpoint as unprotected when +it is not would destroy the trust this tool is built on, so DevTime says what it +found and what it cannot see. + +## New status: NOT_APPLICABLE + +A repository with no billing code is not "unknown" for a billing claim. The +claim simply does not apply. NOT_APPLICABLE says that plainly, and UNKNOWN is +now reserved for the harder case: the surface exists but the evidence cannot +decide. + +## dtc verify is a report card + +Findings first (contradictions before everything else), then a compact list of +claims that do not apply and why. + +When nothing applies, DevTime no longer dead-ends. It reports what it scanned, +what evidence it collected, what would make a claim verifiable, and states +plainly that this is a coverage limit rather than a verdict on your code: + +```text +No built-in claim applies to this repository yet. + +DevTime scanned 129 files and found 199 signals. +Evidence collected: doc=199 + +Built-in claims become verifiable when a repository has: + - HTTP routes and tests (route-test-coverage) + ... + +This is a coverage limit, not a verdict on your repository. +``` + +`dtc verify --list` now shows which claims apply to the current repository. + +## Compatibility + +- JSON output is `schema_version: 2`. Every version 1 field is unchanged; the + only addition is the NOT_APPLICABLE status value. +- Results that do not apply are no longer stored. Storing them would pollute + freshness and diff impact with claims that have no evidence. +- No command, concept, or MCP tool was renamed or removed. + +## Notes + +- 142 passing tests (20 new). +- Verified against three real repositories: an Express-based project went from + no findings to "59 of 133 routes have a referencing test"; a FastAPI template + went to "6 of 15 routes" plus SUPPORTED JWT authentication; a Go project + correctly reports that no claim applies and explains why. +- `dtc verify` remains sub-second on a repository with 133 routes and thousands + of tests. +- A secret-handling claim was investigated and deliberately not built: DevTime + hard-denies secret files from scanning, so evidence for that claim cannot + exist without breaking the trust model. + +## Names + +- PyPI distribution: `devtime-ei`. Python import: `devtime`. CLI: `dtc`. diff --git a/VERIFICATION.md b/VERIFICATION.md index 3cc8e18..af1ed73 100644 --- a/VERIFICATION.md +++ b/VERIFICATION.md @@ -39,7 +39,8 @@ guarantee. | SUPPORTED | Required behavior evidence exists in the current scan. | | WEAK | The claim's surface exists, but the proving evidence is missing. | | CONTRADICTED | Credible evidence conflicts with the claim. Both sides are always shown. | -| UNKNOWN | No relevant surface was found, or coverage cannot responsibly decide. | +| UNKNOWN | The surface exists, but coverage cannot responsibly decide. | +| NOT_APPLICABLE | The repository has no surface this claim is about. | ## Freshness @@ -55,8 +56,22 @@ same time: the last verification supported it, but its evidence changed since. Freshness only tracks files that were evidence for the claim. Unrelated changes never mark a claim stale. +NOT_APPLICABLE matters as much as the others. A repository with no billing code +is not "unknown" for a billing claim; the claim simply does not apply, and saying +so plainly is more useful than an ominous UNKNOWN. + ## Built-in claims +- **route-test-coverage** (v0.5) - "HTTP routes are exercised by tests." Reports + how many routes have a referencing test and names the ones that do not. + Attribution is by test imports and route names; end-to-end specs are excluded + because they match by accident. Absence of tests is missing evidence, never a + contradiction. +- **admin-authorization** (v0.5) - "Administrative routes require an + authorization check." A missing authorization signal is WEAK, never + CONTRADICTED: authorization can be applied globally or by a wrapper the + scanner cannot see, and reporting an endpoint as unprotected when it is not + would destroy the trust this tool is built on. - **billing-webhook-signature** - "Incoming billing webhooks verify the payment provider's signature." - **jwt-authentication** (v0.3) - "Authentication uses JWT access tokens." @@ -85,6 +100,16 @@ Claim impact: Only a claim's recorded evidence files count. A diff touching unrelated files never flags a claim, and nothing is printed when no verified claim is affected. +## The report card (v0.5) + +`dtc verify` leads with what it can actually say about your repository: +findings first (contradictions before everything else), then a compact list of +claims that do not apply and why. + +When no claim applies, DevTime does not dead-end. It reports what it scanned, +what evidence it collected, what would make a claim verifiable, and states +plainly that this is a coverage limit rather than a verdict on your code. + ## Trust model - Deterministic and rule-driven. No AI, no network, no code execution. @@ -100,14 +125,15 @@ never flags a claim, and nothing is printed when no verified claim is affected. - Heuristic scanner: evidence comes from static patterns, not execution. - Signature verification is recognized for known provider patterns (e.g. Stripe `constructEvent`); custom schemes may not be detected. -- Two built-in claims. User-defined claims are deliberately not +- Four built-in claims. User-defined claims are deliberately not supported yet: the claim model must earn trust before it grows a configuration language. - Coverage follows scanner language support; see [LIMITATIONS.md](LIMITATIONS.md). ## Where this is going -Next candidates, in order: more built-in claims over well-covered domains -(webhook idempotency), more contradiction detectors, and machine-readable -claim impact in risk output. User-defined -claims come after built-in claims prove trustworthy on real repositories. +Next candidates, in order: language coverage beyond the current TypeScript and +Python focus (Go repositories currently produce no routes at all), more built-in +claims over well-covered domains, and machine-readable claim impact in risk +output. User-defined claims come after built-in claims prove trustworthy on real +repositories. diff --git a/pyproject.toml b/pyproject.toml index 3ee41ef..c52eb05 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "devtime-ei" -version = "0.4.0" +version = "0.5.0" description = "Local-first Engineering Intelligence for software repositories" readme = "README.md" requires-python = ">=3.11" @@ -50,7 +50,7 @@ dev = [ Homepage = "https://github.com/Shakargy/devtime" Repository = "https://github.com/Shakargy/devtime" Issues = "https://github.com/Shakargy/devtime/issues" -"Release Notes" = "https://github.com/Shakargy/devtime/releases/tag/v0.4.0" +"Release Notes" = "https://github.com/Shakargy/devtime/releases/tag/v0.5.0" Demo = "https://youtu.be/1Hiu3Y9J_SI" [project.scripts] diff --git a/server.json b/server.json index 2fdce24..46decdb 100644 --- a/server.json +++ b/server.json @@ -8,12 +8,12 @@ "source": "github" }, "websiteUrl": "https://github.com/Shakargy/devtime", - "version": "0.4.0", + "version": "0.5.0", "packages": [ { "registryType": "pypi", "identifier": "devtime-ei", - "version": "0.4.0", + "version": "0.5.0", "transport": { "type": "stdio" } diff --git a/src/devtime/__init__.py b/src/devtime/__init__.py index bc80f49..8d3a49d 100644 --- a/src/devtime/__init__.py +++ b/src/devtime/__init__.py @@ -1,6 +1,6 @@ """DevTime - local-first Engineering Intelligence for repository memory.""" -__version__ = "0.4.0" +__version__ = "0.5.0" # Version metadata (Builder Edition, Chapter 20). EVIDENCE_MODEL = "2026.06.1" diff --git a/src/devtime/cli.py b/src/devtime/cli.py index eaff4c9..2cf28be 100644 --- a/src/devtime/cli.py +++ b/src/devtime/cli.py @@ -97,51 +97,71 @@ def verify( conn = connection.connect() try: if list_claims: + # Relevance is computed live so the list answers the useful question: + # which of these claims apply to THIS repository? + current = {r.claim_slug: r for r in ver.verify_all(conn)} rows = [] for slug, definition in ver.BUILTIN_CLAIMS.items(): latest = ver.load_latest_verification(conn, slug) freshness, changed = ver.freshness_for(conn, slug) + live = current.get(slug) rows.append( { "claim_id": slug, "name": definition.name, "statement": definition.statement, + "applies_here": bool( + live and live.status in ver.APPLICABLE_STATUSES + ), + "current_status": live.status if live else None, "last_status": latest[0]["status"] if latest else None, "freshness": freshness, "changed_evidence": changed, } ) + rows.sort(key=lambda r: (not r["applies_here"], r["claim_id"])) if as_json: - console.print_json(_json.dumps({"schema_version": "1", "claims": rows})) + console.print_json(_json.dumps({"schema_version": "2", "claims": rows})) else: console.print("[bold]Built-in claims[/bold]\n") for r in rows: + if not r["applies_here"]: + console.print(f" [dim]{r['claim_id']} (not applicable here)[/dim]") + continue status_txt = r["last_status"] or "never verified" console.print(f" {r['claim_id']}") console.print(f" {r['statement']}") - console.print(f" last status: {status_txt} freshness: {r['freshness']}") + console.print( + f" current: {r['current_status']} " + f"last verified: {status_txt} freshness: {r['freshness']}" + ) for p in r["changed_evidence"]: console.print(f" changed since verification: {p}", markup=False) console.print("") return - slugs = [claim] if claim else list(ver.BUILTIN_CLAIMS.keys()) - results = [] - for slug in slugs: + if claim: try: - result = ver.verify_claim(conn, slug) + results = [ver.verify_claim(conn, claim)] except KeyError: - console.print(f"[red]Unknown claim:[/red] {slug}") + console.print(f"[red]Unknown claim:[/red] {claim}") console.print("Run [bold]dtc verify --list[/bold] to see built-in claims.") raise typer.Exit(code=1) - ver.save_verification(conn, result) - results.append(result) + else: + results = ver.verify_all(conn) + + # Only real verifications are recorded. A claim that does not apply to + # this repository was not verified, so storing it would pollute + # freshness and diff impact with claims that have no evidence. + for result in results: + if result.status in ver.APPLICABLE_STATUSES: + ver.save_verification(conn, result) if as_json: console.print_json( _json.dumps( { - "schema_version": "1", + "schema_version": "2", "command": "verify", "results": [r.to_dict() for r in results], } @@ -149,11 +169,94 @@ def verify( ) return - for result in results: + _print_report(results, single=bool(claim)) + finally: + conn.close() + + +def _print_report(results: list, single: bool) -> None: + """Report card: what DevTime can and cannot verify about this repository.""" + from devtime.intelligence import verification as ver + + applicable = [r for r in results if r.status in ver.APPLICABLE_STATUSES] + not_applicable = [r for r in results if r.status == ver.NOT_APPLICABLE] + + for result in applicable: + _print_verification(result) + + if not_applicable and not single: + console.print("[dim]Not applicable to this repository:[/dim]") + for r in not_applicable: + reason = r.why[0] if r.why else "No relevant surface was found." + console.print(f" - {r.claim_slug}: {reason}", markup=False) + console.print("") + + if single and not applicable: + # An explicitly requested claim that does not apply still explains itself. + for result in not_applicable: _print_verification(result) + return + + if not applicable: + _print_nothing_verifiable() + + +def _print_nothing_verifiable() -> None: + """Never a dead end: say what was scanned and what would unlock a claim.""" + from devtime.db import connection + + conn = connection.connect() + try: + scan = conn.execute( + "SELECT id, file_count, signal_count FROM scans WHERE status = 'completed' " + "ORDER BY started_at DESC LIMIT 1" + ).fetchone() + kinds = [] + if scan: + kinds = conn.execute( + "SELECT kind, COUNT(*) c FROM signals WHERE scan_id = ? " + "GROUP BY kind ORDER BY c DESC LIMIT 6", + (scan["id"],), + ).fetchall() finally: conn.close() + console.print("[bold]No built-in claim applies to this repository yet.[/bold]") + console.print("") + if scan: + console.print( + f"DevTime scanned {scan['file_count']} files and found " + f"{scan['signal_count']} signals.", + markup=False, + ) + if kinds: + summary = ", ".join(f"{k['kind']}={k['c']}" for k in kinds) + console.print(f"Evidence collected: {summary}", markup=False) + else: + console.print( + "No evidence was extracted, which usually means this repository's " + "language or framework is outside current scanner coverage.", + markup=False, + ) + console.print("") + console.print("Built-in claims become verifiable when a repository has:") + console.print(" - HTTP routes and tests (route-test-coverage)") + console.print(" - admin, staff, or back-office routes (admin-authorization)") + console.print(" - JWT usage or a JWT dependency (jwt-authentication)") + console.print(" - billing webhooks or a payment provider (billing-webhook-signature)") + console.print("") + console.print( + "This is a coverage limit, not a verdict on your repository. " + "Scanner support is strongest on TypeScript, Next.js, Express, and " + "FastAPI-style code; see LIMITATIONS.md.", + markup=False, + ) + console.print( + "If DevTime missed something your repository clearly has, that is worth " + "an issue: https://github.com/Shakargy/devtime/issues", + markup=False, + ) + def _print_verification(result) -> None: color = { @@ -161,6 +264,7 @@ def _print_verification(result) -> None: "WEAK": "yellow", "CONTRADICTED": "red", "UNKNOWN": "cyan", + "NOT_APPLICABLE": "dim", }.get(result.status, "white") console.print(f"[bold]{result.claim_name}[/bold]") console.print(f"Claim: {result.statement}") diff --git a/src/devtime/intelligence/verification.py b/src/devtime/intelligence/verification.py index ffca201..89e7a56 100644 --- a/src/devtime/intelligence/verification.py +++ b/src/devtime/intelligence/verification.py @@ -6,19 +6,22 @@ missing evidence, coverage limitations, and freshness - never with confidence the evidence cannot back. -V0.2 scope, deliberately narrow: +Scope, deliberately narrow: - Built-in claims only (no user-defined claim files yet). - - One claim domain: billing webhook signature verification. - - Four statuses: SUPPORTED, WEAK, CONTRADICTED, UNKNOWN. - - Freshness from file fingerprints: FRESH, STALE, NEEDS_VERIFICATION. - Deterministic and rule-driven. No AI, no network, no code execution. + - Freshness from file fingerprints: FRESH, STALE, NEEDS_VERIFICATION. Statuses (documented meaning, per repository evidence policy - not formal proof): - SUPPORTED required behavior evidence exists in the current scan. - WEAK the claim's surface exists, but the proving evidence is missing. - CONTRADICTED credible evidence conflicts with the claim; both sides are shown. - UNKNOWN the repository shows no relevant surface, or coverage cannot - responsibly decide. + SUPPORTED required behavior evidence exists in the current scan. + WEAK the claim's surface exists, but the proving evidence is missing. + CONTRADICTED credible evidence conflicts with the claim; both sides are shown. + UNKNOWN the surface exists but coverage cannot responsibly decide. + NOT_APPLICABLE the repository has no surface this claim is about (v0.5.0). + +NOT_APPLICABLE matters as much as the others. A repository with no billing code +is not "unknown" for a billing claim - the claim simply does not apply, and +saying so plainly is more honest than an ominous UNKNOWN. UNKNOWN is reserved +for the harder case: the surface exists, but the evidence cannot decide. Freshness is separate from truth: FRESH supporting evidence files are unchanged since verification. @@ -41,6 +44,10 @@ WEAK = "WEAK" CONTRADICTED = "CONTRADICTED" UNKNOWN = "UNKNOWN" +NOT_APPLICABLE = "NOT_APPLICABLE" + +# Statuses that mean "this claim has something to say about this repository". +APPLICABLE_STATUSES = (SUPPORTED, WEAK, CONTRADICTED, UNKNOWN) # Freshness FRESH = "FRESH" @@ -117,8 +124,11 @@ class VerificationResult: engine_version: str def to_dict(self) -> dict: + # schema_version 2 (v0.5.0): adds the NOT_APPLICABLE status value. All + # version 1 fields are unchanged, so consumers that ignore unknown + # status values keep working. return { - "schema_version": "1", + "schema_version": "2", "claim_id": self.claim_slug, "claim_name": self.claim_name, "statement": self.statement, @@ -155,6 +165,18 @@ class ClaimDefinition: statement="Authentication uses JWT access tokens.", category="authentication", ), + "route-test-coverage": ClaimDefinition( + slug="route-test-coverage", + name="Route Test Coverage", + statement="HTTP routes are exercised by tests.", + category="testing", + ), + "admin-authorization": ClaimDefinition( + slug="admin-authorization", + name="Admin Authorization", + statement="Administrative routes require an authorization check.", + category="security", + ), } # JWT library names for dependency evidence. @@ -210,6 +232,23 @@ def _is_billingish(hay: str) -> bool: # The engine # --------------------------------------------------------------------------- # +def _no_scan_result(definition: ClaimDefinition) -> VerificationResult: + return VerificationResult( + claim_slug=definition.slug, + claim_name=definition.name, + statement=definition.statement, + status=UNKNOWN, + why=["No completed scan exists. Run dtc scan first."], + supporting=[], + contradictions=[], + missing=["A completed repository scan."], + limitations=_LIMITATIONS, + scan_id=None, + verified_at=_now(), + engine_version=__version__, + ) + + def verify_claim(conn: sqlite3.Connection, slug: str) -> VerificationResult: """Verify one built-in claim against the latest completed scan.""" definition = BUILTIN_CLAIMS.get(slug) @@ -218,24 +257,31 @@ def verify_claim(conn: sqlite3.Connection, slug: str) -> VerificationResult: scan_id = _latest_scan_id(conn) if scan_id is None: - return VerificationResult( - claim_slug=definition.slug, - claim_name=definition.name, - statement=definition.statement, - status=UNKNOWN, - why=["No completed scan exists. Run dtc scan first."], - supporting=[], - contradictions=[], - missing=["A completed repository scan."], - limitations=_LIMITATIONS, - scan_id=None, - verified_at=_now(), - engine_version=__version__, - ) + return _no_scan_result(definition) rows = _load_signals(conn, scan_id) - evaluator = _EVALUATORS[slug] - return evaluator(definition, rows, scan_id) + return _EVALUATORS[slug](definition, rows, scan_id) + + +def verify_all(conn: sqlite3.Connection) -> list[VerificationResult]: + """Verify every built-in claim, loading scan evidence exactly once. + + Applicable results (something to say about this repository) sort first, so + the first thing a user reads is what DevTime actually found. + """ + scan_id = _latest_scan_id(conn) + if scan_id is None: + return [_no_scan_result(d) for d in BUILTIN_CLAIMS.values()] + + rows = _load_signals(conn, scan_id) + results = [ + _EVALUATORS[slug](definition, rows, scan_id) + for slug, definition in BUILTIN_CLAIMS.items() + ] + # Contradictions first: they are the findings a user most needs to see. + order = {CONTRADICTED: 0, SUPPORTED: 1, WEAK: 2, UNKNOWN: 3, NOT_APPLICABLE: 4} + results.sort(key=lambda r: (order.get(r.status, 9), r.claim_slug)) + return results def _verify_billing_webhook_signature( @@ -335,12 +381,12 @@ def _verify_billing_webhook_signature( if not signature_tests: missing.append("A test that exercises webhook signature verification.") else: - status = UNKNOWN - why.append( - "No billing webhook surface was found in the scanned files. " - "The claim does not apply, or the surface is outside scanner coverage." + return _not_applicable( + definition, + scan_id, + "No billing webhook surface was found in the scanned files.", + "A billing webhook route, handler, or payment provider dependency.", ) - missing.append("Any billing webhook route, handler, or provider dependency.") return VerificationResult( claim_slug=definition.slug, @@ -458,12 +504,12 @@ def _verify_jwt_authentication( ) missing.append("JWT access-token usage (login/bearer/authorization context).") else: - status = UNKNOWN - why.append( - "No JWT evidence was found in the scanned files. The claim does not " - "apply, or the usage is outside scanner coverage." + return _not_applicable( + definition, + scan_id, + "No JWT evidence was found in the scanned files.", + "JWT usage, a JWT library dependency, or documentation referencing JWT.", ) - missing.append("Any JWT usage, dependency, or documentation.") return VerificationResult( claim_slug=definition.slug, @@ -481,9 +527,310 @@ def _verify_jwt_authentication( ) +# --------------------------------------------------------------------------- # +# Route test coverage (v0.5.0) +# --------------------------------------------------------------------------- # + +# Route path segments that carry no identity and must not be used for matching. +_GENERIC_SEGMENTS = {"api", "v1", "v2", "v3", "app", "index", "route", "routes", "src"} + + +def _module_token(path: str) -> str: + """The distinctive file stem of an implementation file, lowercased.""" + stem = path.rsplit("/", 1)[-1] + for suffix in (".ts", ".tsx", ".js", ".jsx", ".py", ".mjs", ".cjs"): + if stem.endswith(suffix): + stem = stem[: -len(suffix)] + break + return stem.lower() + + +def _route_tokens(route_path: str) -> list[str]: + """Distinctive, non-generic segments of a route path.""" + out = [] + for seg in route_path.lower().replace("\\", "/").split("/"): + seg = seg.strip() + if not seg or seg.startswith(("[", ":", "{", "<")) or seg in _GENERIC_SEGMENTS: + continue + if len(seg) < 3: + continue + out.append(seg) + return out + + +def _verify_route_test_coverage( + definition: ClaimDefinition, rows: list[sqlite3.Row], scan_id: str +) -> VerificationResult: + """Verify that HTTP routes are exercised by tests. + + Matching is deliberately conservative and explainable. A route counts as + covered when a test file either imports the route's implementation module, + or names a distinctive segment of the route path. Test files are aggregated + first so the comparison stays linear in test FILES, not test cases (large + repos have thousands of test cases across a few dozen files). + + Absence of tests is missing evidence, never a contradiction. + """ + # Aggregate tests per file: imports + a single blob of test names. + test_imports: dict[str, set[str]] = {} + test_blobs: dict[str, list[str]] = {} + for row in rows: + if row["kind"] != "test": + continue + try: + meta = json.loads(row["metadata_json"] or "{}") + except json.JSONDecodeError: + meta = {} + if meta.get("e2e"): + # E2E specs match by accident (Reality Validation finding); they are + # weak evidence for concepts and unreliable for route attribution. + continue + path = row["path"] + imports = test_imports.setdefault(path, set()) + for imp in meta.get("imports") or []: + imports.add(str(imp).lower()) + test_blobs.setdefault(path, []).append(str(row["name"] or "").lower()) + + # One joined blob per test file keeps matching linear in test FILES and turns + # each check into a single substring scan. + test_name_blob = {p: " ".join(names) for p, names in test_blobs.items()} + test_import_blob = {p: " ".join(sorted(i)) for p, i in test_imports.items()} + + # Deduplicate routes: several methods on one path are one surface to cover. + routes: dict[tuple[str, str], sqlite3.Row] = {} + for row in rows: + if row["kind"] != "route": + continue + try: + meta = json.loads(row["metadata_json"] or "{}") + except json.JSONDecodeError: + meta = {} + route_path = str(meta.get("path") or row["name"] or "").strip() + routes.setdefault((row["path"], route_path.lower()), row) + + if not routes: + return _not_applicable( + definition, + scan_id, + "No HTTP routes were found in the scanned files.", + "Any HTTP route (Express, Next.js, or FastAPI style).", + ) + + covered: list[tuple[str, str, str]] = [] # (impl path, route path, reason) + uncovered: list[tuple[str, str]] = [] + for (impl_path, route_path), row in sorted(routes.items()): + token = _module_token(impl_path) + reason = "" + # 1. A test that imports the implementation module. + if token and len(token) >= 3: + for test_path, blob in test_import_blob.items(): + if token in blob: + reason = f"{test_path} imports {token}" + break + # 2. A test whose names mention a distinctive segment of the route path. + if not reason: + segments = _route_tokens(route_path) + for test_path, blob in test_name_blob.items(): + if segments and any(seg in blob for seg in segments): + reason = f"{test_path} names {segments[0]}" + break + if reason: + covered.append((impl_path, route_path, reason)) + else: + uncovered.append((impl_path, route_path)) + + total = len(routes) + n_covered = len(covered) + # Evidence is bounded (responses and stored fingerprints must stay bounded), + # and truncation is disclosed below rather than hidden. + _EVIDENCE_CAP = 25 + sha_by_path = {row["path"]: row["sha256"] for row in rows} + supporting = [ + EvidenceRef( + path=impl, + observation=f"Route {route or impl} is referenced by a test ({reason}).", + kind="route", + strength="moderate", + sha256=sha_by_path.get(impl), + ) + for impl, route, reason in covered[:_EVIDENCE_CAP] + ] + + why = [f"{n_covered} of {total} routes have a referencing test."] + missing: list[str] = [] + if n_covered == total: + status = SUPPORTED + why.append("Every detected route has at least one test referencing it.") + else: + status = WEAK + why.append( + "Routes without a referencing test are not proven to be exercised." + ) + shown = [r or p for p, r in uncovered[:8]] + missing.append( + f"Tests referencing {total - n_covered} route(s): " + ", ".join(shown) + + (" ..." if len(uncovered) > 8 else "") + ) + + limitations = _LIMITATIONS + [ + "Coverage is attributed by test imports and route names, not by executing " + "tests; a route exercised only indirectly may be reported as uncovered.", + "End-to-end specs are excluded from attribution because they match by " + "accident.", + ] + if len(covered) > _EVIDENCE_CAP: + limitations.append( + f"Evidence is capped at {_EVIDENCE_CAP} routes; freshness tracks only " + f"those recorded files, not all {len(covered)} covered routes." + ) + return VerificationResult( + claim_slug=definition.slug, + claim_name=definition.name, + statement=definition.statement, + status=status, + why=why, + supporting=supporting, + contradictions=[], + missing=missing, + limitations=limitations, + scan_id=scan_id, + verified_at=_now(), + engine_version=__version__, + ) + + +# --------------------------------------------------------------------------- # +# Admin authorization (v0.5.0) +# --------------------------------------------------------------------------- # + +_ADMIN_TOKENS = ("admin", "superuser", "staff", "backoffice", "back-office") +_AUTHZ_TOKENS = ( + "requireadmin", "require_admin", "isadmin", "is_admin", "adminonly", + "admin_only", "hasrole", "has_role", "authorize", "authorization", + "permission", "rbac", "requireauth", "require_auth", "isauthenticated", + "current_user", "get_current_user", "authmiddleware", "auth_middleware", +) + + +def _verify_admin_authorization( + definition: ClaimDefinition, rows: list[sqlite3.Row], scan_id: str +) -> VerificationResult: + """Verify that administrative routes require an authorization check. + + Honesty rule for this claim: a missing authorization signal is WEAK, never + CONTRADICTED. Authorization can be applied globally, by a decorator, or by a + wrapper the scanner cannot see. Telling someone their admin endpoint is + unprotected when it is not would destroy the trust this tool is built on. + """ + admin_routes: list[sqlite3.Row] = [] + authz_files: set[str] = set() + authz_rows: list[sqlite3.Row] = [] + + for row in rows: + hay = _hay(row) + kind = row["kind"] + if kind == "route" and any(t in hay for t in _ADMIN_TOKENS): + admin_routes.append(row) + if kind in ("middleware", "auth_dependency") or any( + t in hay for t in _AUTHZ_TOKENS + ): + if kind in ("middleware", "auth_dependency", "route", "test"): + authz_files.add(row["path"]) + if kind in ("middleware", "auth_dependency"): + authz_rows.append(row) + + if not admin_routes: + return _not_applicable( + definition, + scan_id, + "No administrative routes were found in the scanned files.", + "An admin, staff, or back-office route.", + ) + + protected: list[sqlite3.Row] = [] + unprotected: list[sqlite3.Row] = [] + for row in admin_routes: + hay = _hay(row) + # Authorization evidence in the route's own file, or in the route itself. + if row["path"] in authz_files or any(t in hay for t in _AUTHZ_TOKENS): + protected.append(row) + else: + unprotected.append(row) + + supporting = [ + _ref(r, "Admin route shows an authorization check in its file.", "moderate") + for r in protected[:5] + ] + [ + _ref(r, "Authorization middleware or dependency.", "moderate") + for r in authz_rows[:2] + ] + + total = len(admin_routes) + why = [f"{len(protected)} of {total} administrative route(s) show an " + "authorization check."] + missing: list[str] = [] + + if not unprotected: + status = SUPPORTED + why.append("Every detected admin route has authorization evidence.") + else: + status = WEAK + why.append( + "No authorization evidence was found for the remaining admin route(s). " + "This is missing evidence, not proof that they are unprotected." + ) + missing.append( + "Authorization evidence for: " + + ", ".join(sorted({r["path"] for r in unprotected})[:6]) + ) + + limitations = _LIMITATIONS + [ + "Authorization applied globally (a server-wide middleware, a router " + "mount, or a framework decorator the scanner does not parse) is not " + "detected. A WEAK result means DevTime found no evidence, never that a " + "route is confirmed unprotected.", + ] + return VerificationResult( + claim_slug=definition.slug, + claim_name=definition.name, + statement=definition.statement, + status=status, + why=why, + supporting=supporting, + contradictions=[], + missing=missing, + limitations=limitations, + scan_id=scan_id, + verified_at=_now(), + engine_version=__version__, + ) + + +def _not_applicable( + definition: ClaimDefinition, scan_id: str, reason: str, would_need: str +) -> VerificationResult: + """This claim has no surface in this repository. Say so plainly.""" + return VerificationResult( + claim_slug=definition.slug, + claim_name=definition.name, + statement=definition.statement, + status=NOT_APPLICABLE, + why=[reason, "This claim does not apply to this repository."], + supporting=[], + contradictions=[], + missing=[f"Would become verifiable with: {would_need}"], + limitations=_LIMITATIONS, + scan_id=scan_id, + verified_at=_now(), + engine_version=__version__, + ) + + _EVALUATORS = { "billing-webhook-signature": _verify_billing_webhook_signature, "jwt-authentication": _verify_jwt_authentication, + "route-test-coverage": _verify_route_test_coverage, + "admin-authorization": _verify_admin_authorization, } diff --git a/src/devtime/mcp/transport.py b/src/devtime/mcp/transport.py index a06ff71..1427d5c 100644 --- a/src/devtime/mcp/transport.py +++ b/src/devtime/mcp/transport.py @@ -100,11 +100,14 @@ def get_context_pack(concept: str, mode: str = "risk") -> dict: def verify_claim(claim_id: str = "") -> dict: """Verify a repository claim against scanned evidence (read-only compute). - Returns status (SUPPORTED / WEAK / CONTRADICTED / UNKNOWN), why, - supporting evidence with file paths, both-sided contradictions, missing - evidence, and coverage limitations. Call with no claim_id to list the - built-in claims. Results are computed fresh and NOT persisted (this - server stays read-only); use `dtc verify` in a terminal to record one. + Returns status (SUPPORTED / WEAK / CONTRADICTED / UNKNOWN / + NOT_APPLICABLE), why, supporting evidence with file paths, both-sided + contradictions, missing evidence, and coverage limitations. + NOT_APPLICABLE means the repository has no surface this claim is about, + which is different from UNKNOWN (surface exists, evidence cannot + decide). Call with no claim_id to list the built-in claims. Results are + computed fresh and NOT persisted (this server stays read-only); use + `dtc verify` in a terminal to record one. """ if not paths.is_initialized(): return _NOT_INITIALIZED diff --git a/tests/integration/test_evidence_precision.py b/tests/integration/test_evidence_precision.py index 11f4cb0..90968ca 100644 --- a/tests/integration/test_evidence_precision.py +++ b/tests/integration/test_evidence_precision.py @@ -14,14 +14,14 @@ # --- version ------------------------------------------------------------------ -def test_version_is_release_0_4_0(): - # v0.4.0 release: package metadata and __version__ agree on the release version. +def test_version_is_release_0_5_0(): + # v0.5.0 release: package metadata and __version__ agree on the release version. import importlib.metadata as m - assert devtime.__version__ == "0.4.0" + assert devtime.__version__ == "0.5.0" # Distribution is published as "devtime-ei" (the name "devtime" is reserved on # PyPI); the import package and the dtc command stay "devtime"/"dtc". - assert m.version("devtime-ei") == "0.4.0" + assert m.version("devtime-ei") == "0.5.0" # --- P0 Authentication headline precision ------------------------------------ diff --git a/tests/integration/test_verification.py b/tests/integration/test_verification.py index 797d5ef..89361ff 100644 --- a/tests/integration/test_verification.py +++ b/tests/integration/test_verification.py @@ -100,11 +100,14 @@ def test_contradicted_on_disabled_stub(tmp_path, monkeypatch): assert "stub" in c.summary.lower() or "stub" in c.observed_side.lower() -def test_unknown_when_no_billing_surface(tmp_path, monkeypatch): +def test_not_applicable_when_no_billing_surface(tmp_path, monkeypatch): + # v0.5.0: a repo with no billing code is NOT_APPLICABLE, not an ominous + # UNKNOWN, and it says what would make the claim verifiable. _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) _init_scan(tmp_path, monkeypatch) result = _verify() - assert result.status == ver.UNKNOWN + assert result.status == ver.NOT_APPLICABLE + assert any("would become verifiable" in m.lower() for m in result.missing) def test_supported_with_stub_elsewhere_reports_contradiction_but_stays_supported( @@ -200,7 +203,7 @@ def test_cli_verify_json_schema(tmp_path, monkeypatch): result = runner.invoke(app, ["verify", "billing-webhook-signature", "--json"]) assert result.exit_code == 0 payload = json.loads(result.stdout) - assert payload["schema_version"] == "1" + assert payload["schema_version"] == "2" r = payload["results"][0] for key in ("claim_id", "status", "why", "supporting_evidence", "contradictions", "missing_evidence", "limitations"): @@ -298,11 +301,11 @@ def test_jwt_docs_vs_invitation_only_is_contradicted(tmp_path, monkeypatch): assert "invitation" in c.observed_side.lower() -def test_jwt_unknown_without_any_jwt_surface(tmp_path, monkeypatch): +def test_jwt_not_applicable_without_any_jwt_surface(tmp_path, monkeypatch): _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) _init_scan(tmp_path, monkeypatch) result = _verify("jwt-authentication") - assert result.status == ver.UNKNOWN + assert result.status == ver.NOT_APPLICABLE def test_signal_metadata_survives_persistence(tmp_path, monkeypatch): @@ -328,7 +331,7 @@ def test_verify_all_returns_both_claims(tmp_path, monkeypatch): assert result.exit_code == 0 payload = json.loads(result.stdout) ids = {r["claim_id"] for r in payload["results"]} - assert ids == {"billing-webhook-signature", "jwt-authentication"} + assert ids == set(ver.BUILTIN_CLAIMS) # --- diff-aware claim impact (v0.4.0) --------------------------------------------- @@ -371,3 +374,199 @@ def test_no_verifications_means_no_impact(tmp_path, monkeypatch): assert impact == [] finally: conn.close() + + +# --- v0.5.0: claims that fire on ordinary repositories --------------------------- + +EXPRESS_ROUTES = """ +import express from "express"; +import { listUsers } from "../services/users"; +const router = express.Router(); +router.get("/api/users", listUsers); +router.post("/api/reports", createReport); +export default router; +""" + +USERS_TEST = """ +import { listUsers } from "../src/routes/users-router"; +import { describe, it } from "vitest"; +describe("users", () => { it("lists users", () => {}); }); +""" + +ADMIN_ROUTES_PROTECTED = """ +import express from "express"; +import { requireAdmin } from "./require-admin"; +const router = express.Router(); +router.get("/admin/users", requireAdmin, listAdminUsers); +export default router; +""" + +ADMIN_ROUTES_BARE = """ +import express from "express"; +const router = express.Router(); +router.get("/admin/users", listAdminUsers); +export default router; +""" + + +def test_route_test_coverage_supported_when_all_routes_covered(tmp_path, monkeypatch): + _repo(tmp_path, { + "src/routes/users-router.ts": EXPRESS_ROUTES, + "tests/users.test.ts": USERS_TEST, + }) + _init_scan(tmp_path, monkeypatch) + result = _verify("route-test-coverage") + assert result.status == ver.SUPPORTED + assert any(" of " in w for w in result.why) # reports the ratio + + +def test_route_test_coverage_weak_and_names_uncovered_routes(tmp_path, monkeypatch): + _repo(tmp_path, { + "src/routes/users-router.ts": EXPRESS_ROUTES, + "src/routes/billing-router.ts": + 'import express from "express";\n' + 'const router = express.Router();\n' + 'router.get("/api/invoices", listInvoices);\n', + "tests/users.test.ts": USERS_TEST, + }) + _init_scan(tmp_path, monkeypatch) + result = _verify("route-test-coverage") + assert result.status == ver.WEAK + # Absence of tests is missing evidence, never a contradiction. + assert result.contradictions == [] + assert any("invoices" in m for m in result.missing) + + +def test_route_test_coverage_not_applicable_without_routes(tmp_path, monkeypatch): + _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) + _init_scan(tmp_path, monkeypatch) + result = _verify("route-test-coverage") + assert result.status == ver.NOT_APPLICABLE + + +def test_route_test_coverage_ignores_e2e_specs(tmp_path, monkeypatch): + # E2E specs match by accident, so they must not count as route attribution. + _repo(tmp_path, { + "src/routes/users-router.ts": EXPRESS_ROUTES, + "tests-e2e/users.e2e.spec.ts": USERS_TEST, + }) + _init_scan(tmp_path, monkeypatch) + result = _verify("route-test-coverage") + assert result.status == ver.WEAK + + +def test_admin_authorization_supported_with_authz_evidence(tmp_path, monkeypatch): + _repo(tmp_path, { + "src/routes/admin-router.ts": ADMIN_ROUTES_PROTECTED, + "src/routes/require-admin.ts": + "export function requireAdmin(req, res, next) { return next(); }\n", + }) + _init_scan(tmp_path, monkeypatch) + result = _verify("admin-authorization") + assert result.status == ver.SUPPORTED + + +def test_admin_authorization_missing_authz_is_weak_never_contradicted( + tmp_path, monkeypatch +): + # The honesty rule for this claim: authorization can be applied globally or + # by a wrapper the scanner cannot see, so a missing signal is WEAK. + _repo(tmp_path, {"src/routes/admin-router.ts": ADMIN_ROUTES_BARE}) + _init_scan(tmp_path, monkeypatch) + result = _verify("admin-authorization") + assert result.status == ver.WEAK + assert result.contradictions == [] + assert any("not proof" in w.lower() for w in result.why) + assert any("globally" in lim for lim in result.limitations) + + +def test_admin_authorization_not_applicable_without_admin_routes(tmp_path, monkeypatch): + _repo(tmp_path, {"src/routes/users-router.ts": EXPRESS_ROUTES}) + _init_scan(tmp_path, monkeypatch) + result = _verify("admin-authorization") + assert result.status == ver.NOT_APPLICABLE + + +# --- verify_all and the report card ---------------------------------------------- + +def test_verify_all_returns_every_claim_contradictions_first(tmp_path, monkeypatch): + _repo(tmp_path, { + "apps/web/pages/api/stripe/webhook.ts": STUB_HANDLER, + "package.json": PACKAGE_WITH_STRIPE, + "src/routes/users-router.ts": EXPRESS_ROUTES, + }) + _init_scan(tmp_path, monkeypatch) + conn = connection.connect() + try: + results = ver.verify_all(conn) + finally: + conn.close() + assert {r.claim_slug for r in results} == set(ver.BUILTIN_CLAIMS) + assert results[0].status == ver.CONTRADICTED # most important finding first + # NOT_APPLICABLE results sort last. + assert results[-1].status == ver.NOT_APPLICABLE + + +def test_not_applicable_results_are_not_saved(tmp_path, monkeypatch): + # Storing a claim that does not apply would pollute freshness and diff impact. + _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) + _init_scan(tmp_path, monkeypatch) + assert runner.invoke(app, ["verify"]).exit_code == 0 + conn = connection.connect() + try: + ver.ensure_verifications_table(conn) + rows = conn.execute( + "SELECT status FROM verifications WHERE status = ?", (ver.NOT_APPLICABLE,) + ).fetchall() + assert rows == [] + finally: + conn.close() + + +def test_cli_report_never_dead_ends(tmp_path, monkeypatch): + # A repository where nothing applies must still explain itself. + _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) + _init_scan(tmp_path, monkeypatch) + result = runner.invoke(app, ["verify"]) + assert result.exit_code == 0 + out = result.stdout + assert "No built-in claim applies" in out + assert "become verifiable" in out + assert "coverage limit" in out + assert "github.com/Shakargy/devtime/issues" in out + + +def test_cli_report_lists_not_applicable_reasons(tmp_path, monkeypatch): + _repo(tmp_path, { + "src/routes/users-router.ts": EXPRESS_ROUTES, + "tests/users.test.ts": USERS_TEST, + }) + _init_scan(tmp_path, monkeypatch) + result = runner.invoke(app, ["verify"]) + assert result.exit_code == 0 + assert "Not applicable to this repository" in result.stdout + assert "billing-webhook-signature" in result.stdout + + +def test_cli_list_marks_applicability(tmp_path, monkeypatch): + _repo(tmp_path, { + "src/routes/users-router.ts": EXPRESS_ROUTES, + "tests/users.test.ts": USERS_TEST, + }) + _init_scan(tmp_path, monkeypatch) + result = runner.invoke(app, ["verify", "--list", "--json"]) + assert result.exit_code == 0 + payload = json.loads(result.stdout) + assert payload["schema_version"] == "2" + by_id = {c["claim_id"]: c for c in payload["claims"]} + assert by_id["route-test-coverage"]["applies_here"] is True + assert by_id["billing-webhook-signature"]["applies_here"] is False + + +def test_single_claim_that_does_not_apply_still_explains_itself(tmp_path, monkeypatch): + _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) + _init_scan(tmp_path, monkeypatch) + result = runner.invoke(app, ["verify", "billing-webhook-signature"]) + assert result.exit_code == 0 + assert "NOT_APPLICABLE" in result.stdout + assert "does not apply" in result.stdout From 76515dd40cec11c37b407180237a2e4d13fd72d2 Mon Sep 17 00:00:00 2001 From: Aviad Date: Wed, 12 Aug 2026 23:17:31 +0300 Subject: [PATCH 2/2] fix: support MCP SDK 2.0, which removed mcp.server.fastmcp The MCP Python SDK released 2.0.0 and renamed its high-level server: mcp.server.fastmcp.FastMCP became mcp.server.MCPServer. Because the extra is declared as mcp>=1.2, every fresh `pipx install "devtime-ei[mcp]"` resolved to 2.0 and failed with ModuleNotFoundError before the server could start. CI caught it on this branch; it was already broken for users on PyPI. Resolve the server class across both SDK generations rather than pinning users to one, and add a regression test asserting the resolved class still exposes tool/list_tools/call_tool/run so a future rename cannot pass silently. Verified on both: MCP 1.28.1 (local) and MCP 2.0.0 (clean venv, wheel install) register all four tools, and verify_claim returns evidence-backed results on both. --- RELEASE_NOTES_v0.5.0.md | 11 +++++++++++ src/devtime/mcp/transport.py | 26 +++++++++++++++++++++---- tests/integration/test_mcp_transport.py | 15 ++++++++++++++ 3 files changed, 48 insertions(+), 4 deletions(-) diff --git a/RELEASE_NOTES_v0.5.0.md b/RELEASE_NOTES_v0.5.0.md index 81e2eb2..cfc3574 100644 --- a/RELEASE_NOTES_v0.5.0.md +++ b/RELEASE_NOTES_v0.5.0.md @@ -68,6 +68,17 @@ This is a coverage limit, not a verdict on your repository. `dtc verify --list` now shows which claims apply to the current repository. +## Fixed: MCP SDK 2.0 broke every fresh install + +The MCP Python SDK released 2.0.0, which removed `mcp.server.fastmcp`. Any new +`pipx install "devtime-ei[mcp]"` resolved to the new SDK and could not start the +server at all. DevTime now supports both SDK generations (`MCPServer` in 2.x, +`FastMCP` in 1.x), verified against both, with a regression test so a future +rename cannot pass silently. + +If you installed the MCP extra recently and `dtc mcp start` failed, this +release fixes it. + ## Compatibility - JSON output is `schema_version: 2`. Every version 1 field is unchanged; the diff --git a/src/devtime/mcp/transport.py b/src/devtime/mcp/transport.py index 1427d5c..e9ecc06 100644 --- a/src/devtime/mcp/transport.py +++ b/src/devtime/mcp/transport.py @@ -52,14 +52,32 @@ class McpDependencyMissing(RuntimeError): INSTALL_HINT = 'MCP support needs the optional dependency: pip install "devtime-ei[mcp]"' -def build_server(): - """Build the FastMCP server with the read-only tool surface registered.""" - try: +def _server_class(): + """Return the SDK's server class across MCP SDK generations. + + The SDK renamed its high-level server in 2.0: `mcp.server.fastmcp.FastMCP` + became `mcp.server.MCPServer`. Both expose the surface DevTime uses (a + `tool()` decorator, async `list_tools`/`call_tool`, and a stdio `run`), so + both are supported rather than pinning users to one generation. + """ + try: # MCP SDK 2.x + from mcp.server import MCPServer + + return MCPServer + except ImportError: + pass + try: # MCP SDK 1.x from mcp.server.fastmcp import FastMCP + + return FastMCP except ImportError as exc: # pragma: no cover - exercised via CLI test raise McpDependencyMissing(McpDependencyMissing.INSTALL_HINT) from exc - server = FastMCP(SERVER_NAME, instructions=SERVER_INSTRUCTIONS) + +def build_server(): + """Build the MCP server with the read-only tool surface registered.""" + server_class = _server_class() + server = server_class(name=SERVER_NAME, instructions=SERVER_INSTRUCTIONS) @server.tool() def list_concepts(limit: int = 50) -> list[dict] | dict: diff --git a/tests/integration/test_mcp_transport.py b/tests/integration/test_mcp_transport.py index 82e3b59..1c37163 100644 --- a/tests/integration/test_mcp_transport.py +++ b/tests/integration/test_mcp_transport.py @@ -89,3 +89,18 @@ def test_cli_mcp_preview_shows_implemented_tools(): for tool in IMPLEMENTED_TOOLS: assert tool in result.stdout assert "read-only" in result.stdout + + +def test_server_class_resolves_across_sdk_generations(): + """The SDK renamed FastMCP to MCPServer in 2.0. + + mcp 2.0.0 removed `mcp.server.fastmcp`, which broke every fresh install of + devtime-ei[mcp]. Both generations must resolve, and the resolved class must + expose the surface DevTime relies on. + """ + from devtime.mcp.transport import _server_class + + cls = _server_class() + assert cls.__name__ in ("MCPServer", "FastMCP") + for attr in ("tool", "list_tools", "call_tool", "run"): + assert hasattr(cls, attr), f"server class is missing {attr}"