diff --git a/LIMITATIONS.md b/LIMITATIONS.md index 9eefb2c..ac9c0cb 100644 --- a/LIMITATIONS.md +++ b/LIMITATIONS.md @@ -72,8 +72,15 @@ do. Read this before trusting any single output. ## 6b. Verification limitations (experimental) -- Two built-in claims (billing-webhook-signature, jwt-authentication). - User-defined claims are not supported yet, deliberately. +- Four built-in claims (route-test-coverage, admin-authorization, + billing-webhook-signature, jwt-authentication). User-defined claims are not + supported yet, deliberately. +- Route test coverage is attributed statically (test imports and route names), + not by executing tests. A route exercised only indirectly can be reported as + uncovered. +- Admin authorization reports WEAK when it finds no authorization evidence. + That means DevTime found nothing, never that a route is confirmed + unprotected: global middleware and framework decorators are not detected. - Verification is rule-driven over scanner signals; it inherits every scanner coverage limitation listed here. - Statuses mean "per DevTime's evidence rules", not formal proof or a security diff --git a/QUICKSTART.md b/QUICKSTART.md index 8d08cbc..177b283 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -27,7 +27,7 @@ python -m venv .venv source .venv/bin/activate # Windows (PowerShell): .venv\Scripts\Activate.ps1 # Windows (Git Bash): source .venv/Scripts/activate pip install -e ".[dev]" -pytest # optional: all tests pass (129 at v0.4.0) +pytest # optional: all tests pass (142 at v0.5.0) ``` ## 3. Create the demo repo @@ -132,7 +132,7 @@ A fresh-clone check was run on the current candidate: - **OS:** Windows 11 (Git Bash) - **Python:** 3.11.9 - **Install:** `pip install -e ".[dev]"` -- **Tests:** all passing (129 at v0.4.0) +- **Tests:** all passing (142 at v0.5.0) - **Demo:** `dtc init` / `dtc scan` / `dtc concepts` / `dtc explain "Billing Webhooks"` all produced the expected output from a clean `git clone`. diff --git a/README.md b/README.md index 59bcef2..283ec5e 100644 --- a/README.md +++ b/README.md @@ -165,7 +165,7 @@ Anything outside these six is out of scope for V0. See [LIMITATIONS.md](LIMITATI | `dtc context ` | Create a governed Context Pack for agents or humans. | | `dtc risk --diff` | Review a git diff for risky changes using local evidence (advisory). | | `dtc decision add` | Add a local decision record that can reduce uncertainty. | -| `dtc verify [claim]` | Verify a repository claim against evidence: status, contradictions, freshness (experimental). | +| `dtc verify [claim]` | Verify repository claims against evidence: status, contradictions, freshness (experimental). | (Also available: `dtc evidence`, `dtc debt`, `dtc status`, `dtc doctor --privacy`, `dtc export`, `dtc reset`, `dtc mcp start`.) @@ -266,9 +266,12 @@ changes: ![dtc verify demo - a claim goes from SUPPORTED to CONTRADICTED to STALE](assets/devtime-verify-demo.svg) -Statuses are SUPPORTED, WEAK, CONTRADICTED, or UNKNOWN; contradictions always -show both sides; changed evidence marks a claim STALE. Two built-in claims ship -(billing webhook signatures, JWT authentication). See **[VERIFICATION.md](VERIFICATION.md)**. +Statuses are SUPPORTED, WEAK, CONTRADICTED, UNKNOWN, or NOT_APPLICABLE; +contradictions always show both sides; changed evidence marks a claim STALE. +Four built-in claims ship: route test coverage, admin authorization, billing +webhook signatures, and JWT authentication. `dtc verify` leads with what it can +actually verify here, and when nothing applies it says what would make a claim +verifiable instead of dead-ending. See **[VERIFICATION.md](VERIFICATION.md)**. ## Example output diff --git a/RELEASE_NOTES_v0.5.0.md b/RELEASE_NOTES_v0.5.0.md new file mode 100644 index 0000000..cfc3574 --- /dev/null +++ b/RELEASE_NOTES_v0.5.0.md @@ -0,0 +1,105 @@ +# DevTime v0.5.0 - claims that fire on ordinary repositories + +The verification layer had a coverage problem. Its two claims (billing webhooks, +JWT) only apply to repositories that happen to have billing or JWT code. Run +against three realistic open-source repositories, `dtc verify` produced nothing +useful on two of them. Honest, but useless. + +This release fixes that: claims that apply to ordinary repositories, plus an +output that never dead-ends. No cloud, no telemetry, no AI, no code execution - +unchanged. + +## Two new built-in claims + +**route-test-coverage** - "HTTP routes are exercised by tests." + +```text +Route Test Coverage +Status: WEAK + +Why: + - 6 of 15 routes have a referencing test. + +Missing evidence: + - Tests referencing 9 route(s): /api/items/{id}, /api/login/access-token, ... +``` + +Routes and tests are the two most abundant kinds of evidence in almost every +server repository, so this claim fires nearly everywhere. Attribution is by test +imports and route names, and end-to-end specs are excluded because they match by +accident. Absence of tests is missing evidence, never a contradiction. + +**admin-authorization** - "Administrative routes require an authorization check." + +A missing authorization signal is reported as WEAK, never CONTRADICTED. +Authorization can be applied globally, by a router mount, or by a framework +decorator the scanner does not parse. Reporting an endpoint as unprotected when +it is not would destroy the trust this tool is built on, so DevTime says what it +found and what it cannot see. + +## New status: NOT_APPLICABLE + +A repository with no billing code is not "unknown" for a billing claim. The +claim simply does not apply. NOT_APPLICABLE says that plainly, and UNKNOWN is +now reserved for the harder case: the surface exists but the evidence cannot +decide. + +## dtc verify is a report card + +Findings first (contradictions before everything else), then a compact list of +claims that do not apply and why. + +When nothing applies, DevTime no longer dead-ends. It reports what it scanned, +what evidence it collected, what would make a claim verifiable, and states +plainly that this is a coverage limit rather than a verdict on your code: + +```text +No built-in claim applies to this repository yet. + +DevTime scanned 129 files and found 199 signals. +Evidence collected: doc=199 + +Built-in claims become verifiable when a repository has: + - HTTP routes and tests (route-test-coverage) + ... + +This is a coverage limit, not a verdict on your repository. +``` + +`dtc verify --list` now shows which claims apply to the current repository. + +## Fixed: MCP SDK 2.0 broke every fresh install + +The MCP Python SDK released 2.0.0, which removed `mcp.server.fastmcp`. Any new +`pipx install "devtime-ei[mcp]"` resolved to the new SDK and could not start the +server at all. DevTime now supports both SDK generations (`MCPServer` in 2.x, +`FastMCP` in 1.x), verified against both, with a regression test so a future +rename cannot pass silently. + +If you installed the MCP extra recently and `dtc mcp start` failed, this +release fixes it. + +## Compatibility + +- JSON output is `schema_version: 2`. Every version 1 field is unchanged; the + only addition is the NOT_APPLICABLE status value. +- Results that do not apply are no longer stored. Storing them would pollute + freshness and diff impact with claims that have no evidence. +- No command, concept, or MCP tool was renamed or removed. + +## Notes + +- 142 passing tests (20 new). +- Verified against three real repositories: an Express-based project went from + no findings to "59 of 133 routes have a referencing test"; a FastAPI template + went to "6 of 15 routes" plus SUPPORTED JWT authentication; a Go project + correctly reports that no claim applies and explains why. +- `dtc verify` remains sub-second on a repository with 133 routes and thousands + of tests. +- A secret-handling claim was investigated and deliberately not built: DevTime + hard-denies secret files from scanning, so evidence for that claim cannot + exist without breaking the trust model. + +## Names + +- PyPI distribution: `devtime-ei`. Python import: `devtime`. CLI: `dtc`. diff --git a/VERIFICATION.md b/VERIFICATION.md index 3cc8e18..af1ed73 100644 --- a/VERIFICATION.md +++ b/VERIFICATION.md @@ -39,7 +39,8 @@ guarantee. | SUPPORTED | Required behavior evidence exists in the current scan. | | WEAK | The claim's surface exists, but the proving evidence is missing. | | CONTRADICTED | Credible evidence conflicts with the claim. Both sides are always shown. | -| UNKNOWN | No relevant surface was found, or coverage cannot responsibly decide. | +| UNKNOWN | The surface exists, but coverage cannot responsibly decide. | +| NOT_APPLICABLE | The repository has no surface this claim is about. | ## Freshness @@ -55,8 +56,22 @@ same time: the last verification supported it, but its evidence changed since. Freshness only tracks files that were evidence for the claim. Unrelated changes never mark a claim stale. +NOT_APPLICABLE matters as much as the others. A repository with no billing code +is not "unknown" for a billing claim; the claim simply does not apply, and saying +so plainly is more useful than an ominous UNKNOWN. + ## Built-in claims +- **route-test-coverage** (v0.5) - "HTTP routes are exercised by tests." Reports + how many routes have a referencing test and names the ones that do not. + Attribution is by test imports and route names; end-to-end specs are excluded + because they match by accident. Absence of tests is missing evidence, never a + contradiction. +- **admin-authorization** (v0.5) - "Administrative routes require an + authorization check." A missing authorization signal is WEAK, never + CONTRADICTED: authorization can be applied globally or by a wrapper the + scanner cannot see, and reporting an endpoint as unprotected when it is not + would destroy the trust this tool is built on. - **billing-webhook-signature** - "Incoming billing webhooks verify the payment provider's signature." - **jwt-authentication** (v0.3) - "Authentication uses JWT access tokens." @@ -85,6 +100,16 @@ Claim impact: Only a claim's recorded evidence files count. A diff touching unrelated files never flags a claim, and nothing is printed when no verified claim is affected. +## The report card (v0.5) + +`dtc verify` leads with what it can actually say about your repository: +findings first (contradictions before everything else), then a compact list of +claims that do not apply and why. + +When no claim applies, DevTime does not dead-end. It reports what it scanned, +what evidence it collected, what would make a claim verifiable, and states +plainly that this is a coverage limit rather than a verdict on your code. + ## Trust model - Deterministic and rule-driven. No AI, no network, no code execution. @@ -100,14 +125,15 @@ never flags a claim, and nothing is printed when no verified claim is affected. - Heuristic scanner: evidence comes from static patterns, not execution. - Signature verification is recognized for known provider patterns (e.g. Stripe `constructEvent`); custom schemes may not be detected. -- Two built-in claims. User-defined claims are deliberately not +- Four built-in claims. User-defined claims are deliberately not supported yet: the claim model must earn trust before it grows a configuration language. - Coverage follows scanner language support; see [LIMITATIONS.md](LIMITATIONS.md). ## Where this is going -Next candidates, in order: more built-in claims over well-covered domains -(webhook idempotency), more contradiction detectors, and machine-readable -claim impact in risk output. User-defined -claims come after built-in claims prove trustworthy on real repositories. +Next candidates, in order: language coverage beyond the current TypeScript and +Python focus (Go repositories currently produce no routes at all), more built-in +claims over well-covered domains, and machine-readable claim impact in risk +output. User-defined claims come after built-in claims prove trustworthy on real +repositories. diff --git a/pyproject.toml b/pyproject.toml index 3ee41ef..c52eb05 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "devtime-ei" -version = "0.4.0" +version = "0.5.0" description = "Local-first Engineering Intelligence for software repositories" readme = "README.md" requires-python = ">=3.11" @@ -50,7 +50,7 @@ dev = [ Homepage = "https://github.com/Shakargy/devtime" Repository = "https://github.com/Shakargy/devtime" Issues = "https://github.com/Shakargy/devtime/issues" -"Release Notes" = "https://github.com/Shakargy/devtime/releases/tag/v0.4.0" +"Release Notes" = "https://github.com/Shakargy/devtime/releases/tag/v0.5.0" Demo = "https://youtu.be/1Hiu3Y9J_SI" [project.scripts] diff --git a/server.json b/server.json index 2fdce24..46decdb 100644 --- a/server.json +++ b/server.json @@ -8,12 +8,12 @@ "source": "github" }, "websiteUrl": "https://github.com/Shakargy/devtime", - "version": "0.4.0", + "version": "0.5.0", "packages": [ { "registryType": "pypi", "identifier": "devtime-ei", - "version": "0.4.0", + "version": "0.5.0", "transport": { "type": "stdio" } diff --git a/src/devtime/__init__.py b/src/devtime/__init__.py index bc80f49..8d3a49d 100644 --- a/src/devtime/__init__.py +++ b/src/devtime/__init__.py @@ -1,6 +1,6 @@ """DevTime - local-first Engineering Intelligence for repository memory.""" -__version__ = "0.4.0" +__version__ = "0.5.0" # Version metadata (Builder Edition, Chapter 20). EVIDENCE_MODEL = "2026.06.1" diff --git a/src/devtime/cli.py b/src/devtime/cli.py index eaff4c9..2cf28be 100644 --- a/src/devtime/cli.py +++ b/src/devtime/cli.py @@ -97,51 +97,71 @@ def verify( conn = connection.connect() try: if list_claims: + # Relevance is computed live so the list answers the useful question: + # which of these claims apply to THIS repository? + current = {r.claim_slug: r for r in ver.verify_all(conn)} rows = [] for slug, definition in ver.BUILTIN_CLAIMS.items(): latest = ver.load_latest_verification(conn, slug) freshness, changed = ver.freshness_for(conn, slug) + live = current.get(slug) rows.append( { "claim_id": slug, "name": definition.name, "statement": definition.statement, + "applies_here": bool( + live and live.status in ver.APPLICABLE_STATUSES + ), + "current_status": live.status if live else None, "last_status": latest[0]["status"] if latest else None, "freshness": freshness, "changed_evidence": changed, } ) + rows.sort(key=lambda r: (not r["applies_here"], r["claim_id"])) if as_json: - console.print_json(_json.dumps({"schema_version": "1", "claims": rows})) + console.print_json(_json.dumps({"schema_version": "2", "claims": rows})) else: console.print("[bold]Built-in claims[/bold]\n") for r in rows: + if not r["applies_here"]: + console.print(f" [dim]{r['claim_id']} (not applicable here)[/dim]") + continue status_txt = r["last_status"] or "never verified" console.print(f" {r['claim_id']}") console.print(f" {r['statement']}") - console.print(f" last status: {status_txt} freshness: {r['freshness']}") + console.print( + f" current: {r['current_status']} " + f"last verified: {status_txt} freshness: {r['freshness']}" + ) for p in r["changed_evidence"]: console.print(f" changed since verification: {p}", markup=False) console.print("") return - slugs = [claim] if claim else list(ver.BUILTIN_CLAIMS.keys()) - results = [] - for slug in slugs: + if claim: try: - result = ver.verify_claim(conn, slug) + results = [ver.verify_claim(conn, claim)] except KeyError: - console.print(f"[red]Unknown claim:[/red] {slug}") + console.print(f"[red]Unknown claim:[/red] {claim}") console.print("Run [bold]dtc verify --list[/bold] to see built-in claims.") raise typer.Exit(code=1) - ver.save_verification(conn, result) - results.append(result) + else: + results = ver.verify_all(conn) + + # Only real verifications are recorded. A claim that does not apply to + # this repository was not verified, so storing it would pollute + # freshness and diff impact with claims that have no evidence. + for result in results: + if result.status in ver.APPLICABLE_STATUSES: + ver.save_verification(conn, result) if as_json: console.print_json( _json.dumps( { - "schema_version": "1", + "schema_version": "2", "command": "verify", "results": [r.to_dict() for r in results], } @@ -149,11 +169,94 @@ def verify( ) return - for result in results: + _print_report(results, single=bool(claim)) + finally: + conn.close() + + +def _print_report(results: list, single: bool) -> None: + """Report card: what DevTime can and cannot verify about this repository.""" + from devtime.intelligence import verification as ver + + applicable = [r for r in results if r.status in ver.APPLICABLE_STATUSES] + not_applicable = [r for r in results if r.status == ver.NOT_APPLICABLE] + + for result in applicable: + _print_verification(result) + + if not_applicable and not single: + console.print("[dim]Not applicable to this repository:[/dim]") + for r in not_applicable: + reason = r.why[0] if r.why else "No relevant surface was found." + console.print(f" - {r.claim_slug}: {reason}", markup=False) + console.print("") + + if single and not applicable: + # An explicitly requested claim that does not apply still explains itself. + for result in not_applicable: _print_verification(result) + return + + if not applicable: + _print_nothing_verifiable() + + +def _print_nothing_verifiable() -> None: + """Never a dead end: say what was scanned and what would unlock a claim.""" + from devtime.db import connection + + conn = connection.connect() + try: + scan = conn.execute( + "SELECT id, file_count, signal_count FROM scans WHERE status = 'completed' " + "ORDER BY started_at DESC LIMIT 1" + ).fetchone() + kinds = [] + if scan: + kinds = conn.execute( + "SELECT kind, COUNT(*) c FROM signals WHERE scan_id = ? " + "GROUP BY kind ORDER BY c DESC LIMIT 6", + (scan["id"],), + ).fetchall() finally: conn.close() + console.print("[bold]No built-in claim applies to this repository yet.[/bold]") + console.print("") + if scan: + console.print( + f"DevTime scanned {scan['file_count']} files and found " + f"{scan['signal_count']} signals.", + markup=False, + ) + if kinds: + summary = ", ".join(f"{k['kind']}={k['c']}" for k in kinds) + console.print(f"Evidence collected: {summary}", markup=False) + else: + console.print( + "No evidence was extracted, which usually means this repository's " + "language or framework is outside current scanner coverage.", + markup=False, + ) + console.print("") + console.print("Built-in claims become verifiable when a repository has:") + console.print(" - HTTP routes and tests (route-test-coverage)") + console.print(" - admin, staff, or back-office routes (admin-authorization)") + console.print(" - JWT usage or a JWT dependency (jwt-authentication)") + console.print(" - billing webhooks or a payment provider (billing-webhook-signature)") + console.print("") + console.print( + "This is a coverage limit, not a verdict on your repository. " + "Scanner support is strongest on TypeScript, Next.js, Express, and " + "FastAPI-style code; see LIMITATIONS.md.", + markup=False, + ) + console.print( + "If DevTime missed something your repository clearly has, that is worth " + "an issue: https://github.com/Shakargy/devtime/issues", + markup=False, + ) + def _print_verification(result) -> None: color = { @@ -161,6 +264,7 @@ def _print_verification(result) -> None: "WEAK": "yellow", "CONTRADICTED": "red", "UNKNOWN": "cyan", + "NOT_APPLICABLE": "dim", }.get(result.status, "white") console.print(f"[bold]{result.claim_name}[/bold]") console.print(f"Claim: {result.statement}") diff --git a/src/devtime/intelligence/verification.py b/src/devtime/intelligence/verification.py index ffca201..89e7a56 100644 --- a/src/devtime/intelligence/verification.py +++ b/src/devtime/intelligence/verification.py @@ -6,19 +6,22 @@ missing evidence, coverage limitations, and freshness - never with confidence the evidence cannot back. -V0.2 scope, deliberately narrow: +Scope, deliberately narrow: - Built-in claims only (no user-defined claim files yet). - - One claim domain: billing webhook signature verification. - - Four statuses: SUPPORTED, WEAK, CONTRADICTED, UNKNOWN. - - Freshness from file fingerprints: FRESH, STALE, NEEDS_VERIFICATION. - Deterministic and rule-driven. No AI, no network, no code execution. + - Freshness from file fingerprints: FRESH, STALE, NEEDS_VERIFICATION. Statuses (documented meaning, per repository evidence policy - not formal proof): - SUPPORTED required behavior evidence exists in the current scan. - WEAK the claim's surface exists, but the proving evidence is missing. - CONTRADICTED credible evidence conflicts with the claim; both sides are shown. - UNKNOWN the repository shows no relevant surface, or coverage cannot - responsibly decide. + SUPPORTED required behavior evidence exists in the current scan. + WEAK the claim's surface exists, but the proving evidence is missing. + CONTRADICTED credible evidence conflicts with the claim; both sides are shown. + UNKNOWN the surface exists but coverage cannot responsibly decide. + NOT_APPLICABLE the repository has no surface this claim is about (v0.5.0). + +NOT_APPLICABLE matters as much as the others. A repository with no billing code +is not "unknown" for a billing claim - the claim simply does not apply, and +saying so plainly is more honest than an ominous UNKNOWN. UNKNOWN is reserved +for the harder case: the surface exists, but the evidence cannot decide. Freshness is separate from truth: FRESH supporting evidence files are unchanged since verification. @@ -41,6 +44,10 @@ WEAK = "WEAK" CONTRADICTED = "CONTRADICTED" UNKNOWN = "UNKNOWN" +NOT_APPLICABLE = "NOT_APPLICABLE" + +# Statuses that mean "this claim has something to say about this repository". +APPLICABLE_STATUSES = (SUPPORTED, WEAK, CONTRADICTED, UNKNOWN) # Freshness FRESH = "FRESH" @@ -117,8 +124,11 @@ class VerificationResult: engine_version: str def to_dict(self) -> dict: + # schema_version 2 (v0.5.0): adds the NOT_APPLICABLE status value. All + # version 1 fields are unchanged, so consumers that ignore unknown + # status values keep working. return { - "schema_version": "1", + "schema_version": "2", "claim_id": self.claim_slug, "claim_name": self.claim_name, "statement": self.statement, @@ -155,6 +165,18 @@ class ClaimDefinition: statement="Authentication uses JWT access tokens.", category="authentication", ), + "route-test-coverage": ClaimDefinition( + slug="route-test-coverage", + name="Route Test Coverage", + statement="HTTP routes are exercised by tests.", + category="testing", + ), + "admin-authorization": ClaimDefinition( + slug="admin-authorization", + name="Admin Authorization", + statement="Administrative routes require an authorization check.", + category="security", + ), } # JWT library names for dependency evidence. @@ -210,6 +232,23 @@ def _is_billingish(hay: str) -> bool: # The engine # --------------------------------------------------------------------------- # +def _no_scan_result(definition: ClaimDefinition) -> VerificationResult: + return VerificationResult( + claim_slug=definition.slug, + claim_name=definition.name, + statement=definition.statement, + status=UNKNOWN, + why=["No completed scan exists. Run dtc scan first."], + supporting=[], + contradictions=[], + missing=["A completed repository scan."], + limitations=_LIMITATIONS, + scan_id=None, + verified_at=_now(), + engine_version=__version__, + ) + + def verify_claim(conn: sqlite3.Connection, slug: str) -> VerificationResult: """Verify one built-in claim against the latest completed scan.""" definition = BUILTIN_CLAIMS.get(slug) @@ -218,24 +257,31 @@ def verify_claim(conn: sqlite3.Connection, slug: str) -> VerificationResult: scan_id = _latest_scan_id(conn) if scan_id is None: - return VerificationResult( - claim_slug=definition.slug, - claim_name=definition.name, - statement=definition.statement, - status=UNKNOWN, - why=["No completed scan exists. Run dtc scan first."], - supporting=[], - contradictions=[], - missing=["A completed repository scan."], - limitations=_LIMITATIONS, - scan_id=None, - verified_at=_now(), - engine_version=__version__, - ) + return _no_scan_result(definition) rows = _load_signals(conn, scan_id) - evaluator = _EVALUATORS[slug] - return evaluator(definition, rows, scan_id) + return _EVALUATORS[slug](definition, rows, scan_id) + + +def verify_all(conn: sqlite3.Connection) -> list[VerificationResult]: + """Verify every built-in claim, loading scan evidence exactly once. + + Applicable results (something to say about this repository) sort first, so + the first thing a user reads is what DevTime actually found. + """ + scan_id = _latest_scan_id(conn) + if scan_id is None: + return [_no_scan_result(d) for d in BUILTIN_CLAIMS.values()] + + rows = _load_signals(conn, scan_id) + results = [ + _EVALUATORS[slug](definition, rows, scan_id) + for slug, definition in BUILTIN_CLAIMS.items() + ] + # Contradictions first: they are the findings a user most needs to see. + order = {CONTRADICTED: 0, SUPPORTED: 1, WEAK: 2, UNKNOWN: 3, NOT_APPLICABLE: 4} + results.sort(key=lambda r: (order.get(r.status, 9), r.claim_slug)) + return results def _verify_billing_webhook_signature( @@ -335,12 +381,12 @@ def _verify_billing_webhook_signature( if not signature_tests: missing.append("A test that exercises webhook signature verification.") else: - status = UNKNOWN - why.append( - "No billing webhook surface was found in the scanned files. " - "The claim does not apply, or the surface is outside scanner coverage." + return _not_applicable( + definition, + scan_id, + "No billing webhook surface was found in the scanned files.", + "A billing webhook route, handler, or payment provider dependency.", ) - missing.append("Any billing webhook route, handler, or provider dependency.") return VerificationResult( claim_slug=definition.slug, @@ -458,12 +504,12 @@ def _verify_jwt_authentication( ) missing.append("JWT access-token usage (login/bearer/authorization context).") else: - status = UNKNOWN - why.append( - "No JWT evidence was found in the scanned files. The claim does not " - "apply, or the usage is outside scanner coverage." + return _not_applicable( + definition, + scan_id, + "No JWT evidence was found in the scanned files.", + "JWT usage, a JWT library dependency, or documentation referencing JWT.", ) - missing.append("Any JWT usage, dependency, or documentation.") return VerificationResult( claim_slug=definition.slug, @@ -481,9 +527,310 @@ def _verify_jwt_authentication( ) +# --------------------------------------------------------------------------- # +# Route test coverage (v0.5.0) +# --------------------------------------------------------------------------- # + +# Route path segments that carry no identity and must not be used for matching. +_GENERIC_SEGMENTS = {"api", "v1", "v2", "v3", "app", "index", "route", "routes", "src"} + + +def _module_token(path: str) -> str: + """The distinctive file stem of an implementation file, lowercased.""" + stem = path.rsplit("/", 1)[-1] + for suffix in (".ts", ".tsx", ".js", ".jsx", ".py", ".mjs", ".cjs"): + if stem.endswith(suffix): + stem = stem[: -len(suffix)] + break + return stem.lower() + + +def _route_tokens(route_path: str) -> list[str]: + """Distinctive, non-generic segments of a route path.""" + out = [] + for seg in route_path.lower().replace("\\", "/").split("/"): + seg = seg.strip() + if not seg or seg.startswith(("[", ":", "{", "<")) or seg in _GENERIC_SEGMENTS: + continue + if len(seg) < 3: + continue + out.append(seg) + return out + + +def _verify_route_test_coverage( + definition: ClaimDefinition, rows: list[sqlite3.Row], scan_id: str +) -> VerificationResult: + """Verify that HTTP routes are exercised by tests. + + Matching is deliberately conservative and explainable. A route counts as + covered when a test file either imports the route's implementation module, + or names a distinctive segment of the route path. Test files are aggregated + first so the comparison stays linear in test FILES, not test cases (large + repos have thousands of test cases across a few dozen files). + + Absence of tests is missing evidence, never a contradiction. + """ + # Aggregate tests per file: imports + a single blob of test names. + test_imports: dict[str, set[str]] = {} + test_blobs: dict[str, list[str]] = {} + for row in rows: + if row["kind"] != "test": + continue + try: + meta = json.loads(row["metadata_json"] or "{}") + except json.JSONDecodeError: + meta = {} + if meta.get("e2e"): + # E2E specs match by accident (Reality Validation finding); they are + # weak evidence for concepts and unreliable for route attribution. + continue + path = row["path"] + imports = test_imports.setdefault(path, set()) + for imp in meta.get("imports") or []: + imports.add(str(imp).lower()) + test_blobs.setdefault(path, []).append(str(row["name"] or "").lower()) + + # One joined blob per test file keeps matching linear in test FILES and turns + # each check into a single substring scan. + test_name_blob = {p: " ".join(names) for p, names in test_blobs.items()} + test_import_blob = {p: " ".join(sorted(i)) for p, i in test_imports.items()} + + # Deduplicate routes: several methods on one path are one surface to cover. + routes: dict[tuple[str, str], sqlite3.Row] = {} + for row in rows: + if row["kind"] != "route": + continue + try: + meta = json.loads(row["metadata_json"] or "{}") + except json.JSONDecodeError: + meta = {} + route_path = str(meta.get("path") or row["name"] or "").strip() + routes.setdefault((row["path"], route_path.lower()), row) + + if not routes: + return _not_applicable( + definition, + scan_id, + "No HTTP routes were found in the scanned files.", + "Any HTTP route (Express, Next.js, or FastAPI style).", + ) + + covered: list[tuple[str, str, str]] = [] # (impl path, route path, reason) + uncovered: list[tuple[str, str]] = [] + for (impl_path, route_path), row in sorted(routes.items()): + token = _module_token(impl_path) + reason = "" + # 1. A test that imports the implementation module. + if token and len(token) >= 3: + for test_path, blob in test_import_blob.items(): + if token in blob: + reason = f"{test_path} imports {token}" + break + # 2. A test whose names mention a distinctive segment of the route path. + if not reason: + segments = _route_tokens(route_path) + for test_path, blob in test_name_blob.items(): + if segments and any(seg in blob for seg in segments): + reason = f"{test_path} names {segments[0]}" + break + if reason: + covered.append((impl_path, route_path, reason)) + else: + uncovered.append((impl_path, route_path)) + + total = len(routes) + n_covered = len(covered) + # Evidence is bounded (responses and stored fingerprints must stay bounded), + # and truncation is disclosed below rather than hidden. + _EVIDENCE_CAP = 25 + sha_by_path = {row["path"]: row["sha256"] for row in rows} + supporting = [ + EvidenceRef( + path=impl, + observation=f"Route {route or impl} is referenced by a test ({reason}).", + kind="route", + strength="moderate", + sha256=sha_by_path.get(impl), + ) + for impl, route, reason in covered[:_EVIDENCE_CAP] + ] + + why = [f"{n_covered} of {total} routes have a referencing test."] + missing: list[str] = [] + if n_covered == total: + status = SUPPORTED + why.append("Every detected route has at least one test referencing it.") + else: + status = WEAK + why.append( + "Routes without a referencing test are not proven to be exercised." + ) + shown = [r or p for p, r in uncovered[:8]] + missing.append( + f"Tests referencing {total - n_covered} route(s): " + ", ".join(shown) + + (" ..." if len(uncovered) > 8 else "") + ) + + limitations = _LIMITATIONS + [ + "Coverage is attributed by test imports and route names, not by executing " + "tests; a route exercised only indirectly may be reported as uncovered.", + "End-to-end specs are excluded from attribution because they match by " + "accident.", + ] + if len(covered) > _EVIDENCE_CAP: + limitations.append( + f"Evidence is capped at {_EVIDENCE_CAP} routes; freshness tracks only " + f"those recorded files, not all {len(covered)} covered routes." + ) + return VerificationResult( + claim_slug=definition.slug, + claim_name=definition.name, + statement=definition.statement, + status=status, + why=why, + supporting=supporting, + contradictions=[], + missing=missing, + limitations=limitations, + scan_id=scan_id, + verified_at=_now(), + engine_version=__version__, + ) + + +# --------------------------------------------------------------------------- # +# Admin authorization (v0.5.0) +# --------------------------------------------------------------------------- # + +_ADMIN_TOKENS = ("admin", "superuser", "staff", "backoffice", "back-office") +_AUTHZ_TOKENS = ( + "requireadmin", "require_admin", "isadmin", "is_admin", "adminonly", + "admin_only", "hasrole", "has_role", "authorize", "authorization", + "permission", "rbac", "requireauth", "require_auth", "isauthenticated", + "current_user", "get_current_user", "authmiddleware", "auth_middleware", +) + + +def _verify_admin_authorization( + definition: ClaimDefinition, rows: list[sqlite3.Row], scan_id: str +) -> VerificationResult: + """Verify that administrative routes require an authorization check. + + Honesty rule for this claim: a missing authorization signal is WEAK, never + CONTRADICTED. Authorization can be applied globally, by a decorator, or by a + wrapper the scanner cannot see. Telling someone their admin endpoint is + unprotected when it is not would destroy the trust this tool is built on. + """ + admin_routes: list[sqlite3.Row] = [] + authz_files: set[str] = set() + authz_rows: list[sqlite3.Row] = [] + + for row in rows: + hay = _hay(row) + kind = row["kind"] + if kind == "route" and any(t in hay for t in _ADMIN_TOKENS): + admin_routes.append(row) + if kind in ("middleware", "auth_dependency") or any( + t in hay for t in _AUTHZ_TOKENS + ): + if kind in ("middleware", "auth_dependency", "route", "test"): + authz_files.add(row["path"]) + if kind in ("middleware", "auth_dependency"): + authz_rows.append(row) + + if not admin_routes: + return _not_applicable( + definition, + scan_id, + "No administrative routes were found in the scanned files.", + "An admin, staff, or back-office route.", + ) + + protected: list[sqlite3.Row] = [] + unprotected: list[sqlite3.Row] = [] + for row in admin_routes: + hay = _hay(row) + # Authorization evidence in the route's own file, or in the route itself. + if row["path"] in authz_files or any(t in hay for t in _AUTHZ_TOKENS): + protected.append(row) + else: + unprotected.append(row) + + supporting = [ + _ref(r, "Admin route shows an authorization check in its file.", "moderate") + for r in protected[:5] + ] + [ + _ref(r, "Authorization middleware or dependency.", "moderate") + for r in authz_rows[:2] + ] + + total = len(admin_routes) + why = [f"{len(protected)} of {total} administrative route(s) show an " + "authorization check."] + missing: list[str] = [] + + if not unprotected: + status = SUPPORTED + why.append("Every detected admin route has authorization evidence.") + else: + status = WEAK + why.append( + "No authorization evidence was found for the remaining admin route(s). " + "This is missing evidence, not proof that they are unprotected." + ) + missing.append( + "Authorization evidence for: " + + ", ".join(sorted({r["path"] for r in unprotected})[:6]) + ) + + limitations = _LIMITATIONS + [ + "Authorization applied globally (a server-wide middleware, a router " + "mount, or a framework decorator the scanner does not parse) is not " + "detected. A WEAK result means DevTime found no evidence, never that a " + "route is confirmed unprotected.", + ] + return VerificationResult( + claim_slug=definition.slug, + claim_name=definition.name, + statement=definition.statement, + status=status, + why=why, + supporting=supporting, + contradictions=[], + missing=missing, + limitations=limitations, + scan_id=scan_id, + verified_at=_now(), + engine_version=__version__, + ) + + +def _not_applicable( + definition: ClaimDefinition, scan_id: str, reason: str, would_need: str +) -> VerificationResult: + """This claim has no surface in this repository. Say so plainly.""" + return VerificationResult( + claim_slug=definition.slug, + claim_name=definition.name, + statement=definition.statement, + status=NOT_APPLICABLE, + why=[reason, "This claim does not apply to this repository."], + supporting=[], + contradictions=[], + missing=[f"Would become verifiable with: {would_need}"], + limitations=_LIMITATIONS, + scan_id=scan_id, + verified_at=_now(), + engine_version=__version__, + ) + + _EVALUATORS = { "billing-webhook-signature": _verify_billing_webhook_signature, "jwt-authentication": _verify_jwt_authentication, + "route-test-coverage": _verify_route_test_coverage, + "admin-authorization": _verify_admin_authorization, } diff --git a/src/devtime/mcp/transport.py b/src/devtime/mcp/transport.py index a06ff71..e9ecc06 100644 --- a/src/devtime/mcp/transport.py +++ b/src/devtime/mcp/transport.py @@ -52,14 +52,32 @@ class McpDependencyMissing(RuntimeError): INSTALL_HINT = 'MCP support needs the optional dependency: pip install "devtime-ei[mcp]"' -def build_server(): - """Build the FastMCP server with the read-only tool surface registered.""" - try: +def _server_class(): + """Return the SDK's server class across MCP SDK generations. + + The SDK renamed its high-level server in 2.0: `mcp.server.fastmcp.FastMCP` + became `mcp.server.MCPServer`. Both expose the surface DevTime uses (a + `tool()` decorator, async `list_tools`/`call_tool`, and a stdio `run`), so + both are supported rather than pinning users to one generation. + """ + try: # MCP SDK 2.x + from mcp.server import MCPServer + + return MCPServer + except ImportError: + pass + try: # MCP SDK 1.x from mcp.server.fastmcp import FastMCP + + return FastMCP except ImportError as exc: # pragma: no cover - exercised via CLI test raise McpDependencyMissing(McpDependencyMissing.INSTALL_HINT) from exc - server = FastMCP(SERVER_NAME, instructions=SERVER_INSTRUCTIONS) + +def build_server(): + """Build the MCP server with the read-only tool surface registered.""" + server_class = _server_class() + server = server_class(name=SERVER_NAME, instructions=SERVER_INSTRUCTIONS) @server.tool() def list_concepts(limit: int = 50) -> list[dict] | dict: @@ -100,11 +118,14 @@ def get_context_pack(concept: str, mode: str = "risk") -> dict: def verify_claim(claim_id: str = "") -> dict: """Verify a repository claim against scanned evidence (read-only compute). - Returns status (SUPPORTED / WEAK / CONTRADICTED / UNKNOWN), why, - supporting evidence with file paths, both-sided contradictions, missing - evidence, and coverage limitations. Call with no claim_id to list the - built-in claims. Results are computed fresh and NOT persisted (this - server stays read-only); use `dtc verify` in a terminal to record one. + Returns status (SUPPORTED / WEAK / CONTRADICTED / UNKNOWN / + NOT_APPLICABLE), why, supporting evidence with file paths, both-sided + contradictions, missing evidence, and coverage limitations. + NOT_APPLICABLE means the repository has no surface this claim is about, + which is different from UNKNOWN (surface exists, evidence cannot + decide). Call with no claim_id to list the built-in claims. Results are + computed fresh and NOT persisted (this server stays read-only); use + `dtc verify` in a terminal to record one. """ if not paths.is_initialized(): return _NOT_INITIALIZED diff --git a/tests/integration/test_evidence_precision.py b/tests/integration/test_evidence_precision.py index 11f4cb0..90968ca 100644 --- a/tests/integration/test_evidence_precision.py +++ b/tests/integration/test_evidence_precision.py @@ -14,14 +14,14 @@ # --- version ------------------------------------------------------------------ -def test_version_is_release_0_4_0(): - # v0.4.0 release: package metadata and __version__ agree on the release version. +def test_version_is_release_0_5_0(): + # v0.5.0 release: package metadata and __version__ agree on the release version. import importlib.metadata as m - assert devtime.__version__ == "0.4.0" + assert devtime.__version__ == "0.5.0" # Distribution is published as "devtime-ei" (the name "devtime" is reserved on # PyPI); the import package and the dtc command stay "devtime"/"dtc". - assert m.version("devtime-ei") == "0.4.0" + assert m.version("devtime-ei") == "0.5.0" # --- P0 Authentication headline precision ------------------------------------ diff --git a/tests/integration/test_mcp_transport.py b/tests/integration/test_mcp_transport.py index 82e3b59..1c37163 100644 --- a/tests/integration/test_mcp_transport.py +++ b/tests/integration/test_mcp_transport.py @@ -89,3 +89,18 @@ def test_cli_mcp_preview_shows_implemented_tools(): for tool in IMPLEMENTED_TOOLS: assert tool in result.stdout assert "read-only" in result.stdout + + +def test_server_class_resolves_across_sdk_generations(): + """The SDK renamed FastMCP to MCPServer in 2.0. + + mcp 2.0.0 removed `mcp.server.fastmcp`, which broke every fresh install of + devtime-ei[mcp]. Both generations must resolve, and the resolved class must + expose the surface DevTime relies on. + """ + from devtime.mcp.transport import _server_class + + cls = _server_class() + assert cls.__name__ in ("MCPServer", "FastMCP") + for attr in ("tool", "list_tools", "call_tool", "run"): + assert hasattr(cls, attr), f"server class is missing {attr}" diff --git a/tests/integration/test_verification.py b/tests/integration/test_verification.py index 797d5ef..89361ff 100644 --- a/tests/integration/test_verification.py +++ b/tests/integration/test_verification.py @@ -100,11 +100,14 @@ def test_contradicted_on_disabled_stub(tmp_path, monkeypatch): assert "stub" in c.summary.lower() or "stub" in c.observed_side.lower() -def test_unknown_when_no_billing_surface(tmp_path, monkeypatch): +def test_not_applicable_when_no_billing_surface(tmp_path, monkeypatch): + # v0.5.0: a repo with no billing code is NOT_APPLICABLE, not an ominous + # UNKNOWN, and it says what would make the claim verifiable. _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) _init_scan(tmp_path, monkeypatch) result = _verify() - assert result.status == ver.UNKNOWN + assert result.status == ver.NOT_APPLICABLE + assert any("would become verifiable" in m.lower() for m in result.missing) def test_supported_with_stub_elsewhere_reports_contradiction_but_stays_supported( @@ -200,7 +203,7 @@ def test_cli_verify_json_schema(tmp_path, monkeypatch): result = runner.invoke(app, ["verify", "billing-webhook-signature", "--json"]) assert result.exit_code == 0 payload = json.loads(result.stdout) - assert payload["schema_version"] == "1" + assert payload["schema_version"] == "2" r = payload["results"][0] for key in ("claim_id", "status", "why", "supporting_evidence", "contradictions", "missing_evidence", "limitations"): @@ -298,11 +301,11 @@ def test_jwt_docs_vs_invitation_only_is_contradicted(tmp_path, monkeypatch): assert "invitation" in c.observed_side.lower() -def test_jwt_unknown_without_any_jwt_surface(tmp_path, monkeypatch): +def test_jwt_not_applicable_without_any_jwt_surface(tmp_path, monkeypatch): _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) _init_scan(tmp_path, monkeypatch) result = _verify("jwt-authentication") - assert result.status == ver.UNKNOWN + assert result.status == ver.NOT_APPLICABLE def test_signal_metadata_survives_persistence(tmp_path, monkeypatch): @@ -328,7 +331,7 @@ def test_verify_all_returns_both_claims(tmp_path, monkeypatch): assert result.exit_code == 0 payload = json.loads(result.stdout) ids = {r["claim_id"] for r in payload["results"]} - assert ids == {"billing-webhook-signature", "jwt-authentication"} + assert ids == set(ver.BUILTIN_CLAIMS) # --- diff-aware claim impact (v0.4.0) --------------------------------------------- @@ -371,3 +374,199 @@ def test_no_verifications_means_no_impact(tmp_path, monkeypatch): assert impact == [] finally: conn.close() + + +# --- v0.5.0: claims that fire on ordinary repositories --------------------------- + +EXPRESS_ROUTES = """ +import express from "express"; +import { listUsers } from "../services/users"; +const router = express.Router(); +router.get("/api/users", listUsers); +router.post("/api/reports", createReport); +export default router; +""" + +USERS_TEST = """ +import { listUsers } from "../src/routes/users-router"; +import { describe, it } from "vitest"; +describe("users", () => { it("lists users", () => {}); }); +""" + +ADMIN_ROUTES_PROTECTED = """ +import express from "express"; +import { requireAdmin } from "./require-admin"; +const router = express.Router(); +router.get("/admin/users", requireAdmin, listAdminUsers); +export default router; +""" + +ADMIN_ROUTES_BARE = """ +import express from "express"; +const router = express.Router(); +router.get("/admin/users", listAdminUsers); +export default router; +""" + + +def test_route_test_coverage_supported_when_all_routes_covered(tmp_path, monkeypatch): + _repo(tmp_path, { + "src/routes/users-router.ts": EXPRESS_ROUTES, + "tests/users.test.ts": USERS_TEST, + }) + _init_scan(tmp_path, monkeypatch) + result = _verify("route-test-coverage") + assert result.status == ver.SUPPORTED + assert any(" of " in w for w in result.why) # reports the ratio + + +def test_route_test_coverage_weak_and_names_uncovered_routes(tmp_path, monkeypatch): + _repo(tmp_path, { + "src/routes/users-router.ts": EXPRESS_ROUTES, + "src/routes/billing-router.ts": + 'import express from "express";\n' + 'const router = express.Router();\n' + 'router.get("/api/invoices", listInvoices);\n', + "tests/users.test.ts": USERS_TEST, + }) + _init_scan(tmp_path, monkeypatch) + result = _verify("route-test-coverage") + assert result.status == ver.WEAK + # Absence of tests is missing evidence, never a contradiction. + assert result.contradictions == [] + assert any("invoices" in m for m in result.missing) + + +def test_route_test_coverage_not_applicable_without_routes(tmp_path, monkeypatch): + _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) + _init_scan(tmp_path, monkeypatch) + result = _verify("route-test-coverage") + assert result.status == ver.NOT_APPLICABLE + + +def test_route_test_coverage_ignores_e2e_specs(tmp_path, monkeypatch): + # E2E specs match by accident, so they must not count as route attribution. + _repo(tmp_path, { + "src/routes/users-router.ts": EXPRESS_ROUTES, + "tests-e2e/users.e2e.spec.ts": USERS_TEST, + }) + _init_scan(tmp_path, monkeypatch) + result = _verify("route-test-coverage") + assert result.status == ver.WEAK + + +def test_admin_authorization_supported_with_authz_evidence(tmp_path, monkeypatch): + _repo(tmp_path, { + "src/routes/admin-router.ts": ADMIN_ROUTES_PROTECTED, + "src/routes/require-admin.ts": + "export function requireAdmin(req, res, next) { return next(); }\n", + }) + _init_scan(tmp_path, monkeypatch) + result = _verify("admin-authorization") + assert result.status == ver.SUPPORTED + + +def test_admin_authorization_missing_authz_is_weak_never_contradicted( + tmp_path, monkeypatch +): + # The honesty rule for this claim: authorization can be applied globally or + # by a wrapper the scanner cannot see, so a missing signal is WEAK. + _repo(tmp_path, {"src/routes/admin-router.ts": ADMIN_ROUTES_BARE}) + _init_scan(tmp_path, monkeypatch) + result = _verify("admin-authorization") + assert result.status == ver.WEAK + assert result.contradictions == [] + assert any("not proof" in w.lower() for w in result.why) + assert any("globally" in lim for lim in result.limitations) + + +def test_admin_authorization_not_applicable_without_admin_routes(tmp_path, monkeypatch): + _repo(tmp_path, {"src/routes/users-router.ts": EXPRESS_ROUTES}) + _init_scan(tmp_path, monkeypatch) + result = _verify("admin-authorization") + assert result.status == ver.NOT_APPLICABLE + + +# --- verify_all and the report card ---------------------------------------------- + +def test_verify_all_returns_every_claim_contradictions_first(tmp_path, monkeypatch): + _repo(tmp_path, { + "apps/web/pages/api/stripe/webhook.ts": STUB_HANDLER, + "package.json": PACKAGE_WITH_STRIPE, + "src/routes/users-router.ts": EXPRESS_ROUTES, + }) + _init_scan(tmp_path, monkeypatch) + conn = connection.connect() + try: + results = ver.verify_all(conn) + finally: + conn.close() + assert {r.claim_slug for r in results} == set(ver.BUILTIN_CLAIMS) + assert results[0].status == ver.CONTRADICTED # most important finding first + # NOT_APPLICABLE results sort last. + assert results[-1].status == ver.NOT_APPLICABLE + + +def test_not_applicable_results_are_not_saved(tmp_path, monkeypatch): + # Storing a claim that does not apply would pollute freshness and diff impact. + _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) + _init_scan(tmp_path, monkeypatch) + assert runner.invoke(app, ["verify"]).exit_code == 0 + conn = connection.connect() + try: + ver.ensure_verifications_table(conn) + rows = conn.execute( + "SELECT status FROM verifications WHERE status = ?", (ver.NOT_APPLICABLE,) + ).fetchall() + assert rows == [] + finally: + conn.close() + + +def test_cli_report_never_dead_ends(tmp_path, monkeypatch): + # A repository where nothing applies must still explain itself. + _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) + _init_scan(tmp_path, monkeypatch) + result = runner.invoke(app, ["verify"]) + assert result.exit_code == 0 + out = result.stdout + assert "No built-in claim applies" in out + assert "become verifiable" in out + assert "coverage limit" in out + assert "github.com/Shakargy/devtime/issues" in out + + +def test_cli_report_lists_not_applicable_reasons(tmp_path, monkeypatch): + _repo(tmp_path, { + "src/routes/users-router.ts": EXPRESS_ROUTES, + "tests/users.test.ts": USERS_TEST, + }) + _init_scan(tmp_path, monkeypatch) + result = runner.invoke(app, ["verify"]) + assert result.exit_code == 0 + assert "Not applicable to this repository" in result.stdout + assert "billing-webhook-signature" in result.stdout + + +def test_cli_list_marks_applicability(tmp_path, monkeypatch): + _repo(tmp_path, { + "src/routes/users-router.ts": EXPRESS_ROUTES, + "tests/users.test.ts": USERS_TEST, + }) + _init_scan(tmp_path, monkeypatch) + result = runner.invoke(app, ["verify", "--list", "--json"]) + assert result.exit_code == 0 + payload = json.loads(result.stdout) + assert payload["schema_version"] == "2" + by_id = {c["claim_id"]: c for c in payload["claims"]} + assert by_id["route-test-coverage"]["applies_here"] is True + assert by_id["billing-webhook-signature"]["applies_here"] is False + + +def test_single_claim_that_does_not_apply_still_explains_itself(tmp_path, monkeypatch): + _repo(tmp_path, {"src/util/math.ts": "export const add = (a, b) => a + b;\n"}) + _init_scan(tmp_path, monkeypatch) + result = runner.invoke(app, ["verify", "billing-webhook-signature"]) + assert result.exit_code == 0 + assert "NOT_APPLICABLE" in result.stdout + assert "does not apply" in result.stdout