From f2c06c89956f0c300d87e1013cad043b9ad20a8e Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 1 Aug 2026 08:31:53 -0500 Subject: [PATCH 1/8] docs(adr): ADR 0156 -- ASVS scorecard as data, so a re-score stops costing 23M tokens The ASVS score is maintained as prose and nothing checks it. One re-anchoring session on 2026-08-01 produced, in a single day: 6 re-derivations of the headline count from scratch 12 residuals of record factually FALSE at HEAD 5 of those were ABSENCE claims that had silently stopped being true 10 cells missing from an enumeration called "arithmetic-checked and complete" 5 documents asserting a superseded count as current 4 register-vs-scorecard contradictions ~23M subagent tokens spent verifying Most of that was not discovery. It was re-deriving facts already known but never durably recorded, then re-verifying them because the recorded version had gone quietly false. The dropped-ten defect is the instructive one: it survived because the arithmetic closed to 345 and closure was read as proof. Closure only proves the four buckets sum -- not that every cell landed in one. No self-consistent count can detect an absent cell. Decision: one [[cell]] record per requirement for all 345; the count is COMPUTED so no document can state one; a test asserts every corpus id appears exactly once; each cell's evidence anchor names a TOKEN that CI asserts still resolves, so code movement reds a test instead of rotting a sentence; an absence claim must record the search AND a positive control that must still hit; `unverified` is a first-class verdict so inherited-vs-verified Pass is countable; and the verifier fails closed rather than skipping. Tool and schema here (public CI, fixture data); real scorecard in the vault with a vault-CI job. That placement closes ASVS 15.1.3 as a by-product -- six *_doc_drift modules currently assert against documents that `git ls-files docs/security/` shows are absent from the tree where CI runs, so they are inspection-only theatre. States its costs plainly, including that it does NOT make the score correct -- only consistent, derived and drift-detecting. Number allocated atomically via scripts/coord/alloc.ps1; index row in this commit. --- ...ce-anchors-and-a-fail-closed-drift-gate.md | 174 ++++++++++++++++++ docs/adr/README.md | 1 + 2 files changed, 175 insertions(+) create mode 100644 docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md diff --git a/docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md b/docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md new file mode 100644 index 0000000..16c822d --- /dev/null +++ b/docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md @@ -0,0 +1,174 @@ +# ADR 0156 — ASVS scorecard as data: a derived count, verified evidence anchors, and a fail-closed drift gate + +**Status:** Proposed +**Date:** 2026-08-01 +**Supersedes:** — +**Related:** ADR 0155 (DAST) · BACKLOG #205 (risk-acceptance register) · ASVS cell 15.1.3 + +--- + +## Context + +The ASVS L3 score is maintained as **prose**. A dated assessment states a count; later dated documents +supersede it; a risk-acceptance register carries a row per residual; each asserts facts about the code +in sentences. Nothing checks any of it. + +On **2026-08-01** a single re-anchoring session produced the following, all in one day: + +| Observed | Count | +|---|---| +| Times the headline count was re-derived from scratch | 6 | +| Residuals of record that were **factually false** at `HEAD` | 12 | +| Of those, **absence claims** that had silently stopped being true | 5 | +| Cells missing from an enumeration described as *"arithmetic-checked and complete"* | 10 | +| Documents asserting a superseded count as current | 5 | +| Register-vs-scorecard contradictions | 4 | +| The same documentation defect fixed independently by two sessions | 1 | +| Subagent tokens spent on verification | **~23 million** | + +Most of that spend was not discovery. It was **re-deriving facts already known but never durably +recorded**, then re-verifying them because the recorded version had gone quietly false. + +Three failure modes account for all of it: + +1. **Prose asserts facts about code, and the code moves.** A residual says *"no privilege probe of any + kind exists"*; the code grows two; the sentence rots in place. The next session funds work that is + already done, or trusts a control that is not there. +2. **A completeness claim cannot be checked by the thing that makes it.** The dropped-ten defect + survived because the arithmetic closed to 345 and closure was read as proof. **Closure only proves + the four buckets sum — not that every cell landed in one.** No self-consistent count can detect an + absent cell. +3. **The guards that would catch this are blind.** Six `*_doc_drift` test modules exist, one with ~88 + assertions written specifically to stop this class. `git ls-files docs/security/` in the development + repo returns **0** — the documents are vault-only, so every doc-anchored assertion has nothing to + read in the tree where CI runs. **This is ASVS 15.1.3**, currently open, and it is why none of the + above was caught by a machine. + +The lineage convention — dated documents, superseded rather than edited — is **correct for audit** and +is not the problem. The problem is that it is also being used as the *source of truth* for facts that +change under it. + +## Decision + +**Make the scorecard data. Derive the count. Verify the evidence. Fail closed.** + +### 1. One record per requirement, all 345 + +A single structured file — `docs/security/asvs-scorecard.toml`, TOML per the project's config +preference — holds one `[[cell]]` per ASVS 5.0.0 requirement: + +```toml +[[cell]] +id = "12.1.1" +level = 1 # from the held corpus, not typed +verdict = "partial" # pass | partial | fail | na | unverified +posture = "single" +residual = "The SQL Server ODBC PHI hop has no engine-side TLS version floor…" +last_verified = "2026-08-01" +verified_at = "f8d11685" # the commit the evidence was read on + [[cell.evidence]] + path = "messagefoundry/__main__.py" + line = 1978 + expect = "probe_tls_floor" # a token CI asserts is still there +``` + +### 2. The count is computed, never typed + +`scripts/asvs/scorecard.py` sums the verdicts. **No document states a count; documents render it.** +Five documents can no longer assert three counts, because there is only one place a count exists. + +### 3. Completeness is asserted, not claimed + +A test asserts **every one of the 345 ids in the held corpus appears exactly once**, and that no id +outside it appears at all. This is the check whose absence cost ten cells: it would have failed the +moment the enumeration dropped one, rather than closing to 345 and looking correct. + +### 4. Evidence anchors are machine-verified + +For every cell, CI opens each `evidence.path` and asserts `expect` is present at or near `line`. When +the code moves, **the test goes red instead of the document going quietly false.** This is the direct +countermeasure to the twelve false residuals, and specifically to the five false *absence* claims — +which is why an absence claim must record the **search that proved it**, not merely its conclusion: + +```toml + [[cell.absence]] + pattern = "clamav|clamd|ICAP|MpCmdRun|yara" + positive_control = "ScanRejected" # must MATCH, or the search is blind +``` + +A zero result is evidence only if the positive control still hits. **A grep naming the wrong token +returns zero and reads exactly like proof** — that is how five of them survived. + +### 5. `unverified` is a first-class verdict + +Cells inherited from an earlier assessment and never re-read against the requirement text are +`unverified`, **not** `pass`. The renderer reports *verified Pass* and *inherited Pass* separately. + +This makes the largest standing exposure **visible and countable** rather than a caveat in prose: as of +2026-08-01, ~219 Passes have never been checked against the ASVS text, and the one pass that did apply +an adversarial layer caught **three attempted unearned upgrades out of six proposals**. + +### 6. Fail closed, never skip + +The verifier **refuses** when the scorecard is absent but expected, rather than skipping. Skipping is +precisely what 15.1.3 does today and why a green CI proves nothing about these documents. + +### 7. Placement, and how 15.1.3 closes + +| Artifact | Lives in | Why | +|---|---|---| +| Verifier + schema + unit tests (fixture data) | **Development repo** | Generic tooling, no posture data; runs in public CI on every PR | +| `asvs-scorecard.toml` (real verdicts + evidence) | **Vault** | Posture data; `docs/security/`-class | +| Job running the verifier against the real data | **Vault CI** | The vault has `ci.yml`, `tests/`, `pyproject.toml` — the guards can run where the documents actually live | + +That last row is the fix for **15.1.3**: the guards stop being inspection-only theatre. + +### 8. A generated entry point + +`ASVS-CURRENT.md` is rendered, never hand-written: the count, the anchor commit, the open cells, and +what awaits an owner decision. The dated lineage stays for audit; a new session reads one generated +page instead of reconstructing state from ten superseding documents. + +## Consequences + +**Good.** + +- A dropped cell becomes **impossible** rather than undetected. +- A rotted evidence claim becomes a **red test** rather than a misleading sentence. +- A re-score becomes *"re-verify the cells whose anchors moved"* instead of *"re-derive all 345."* +- Inherited-versus-verified Pass becomes countable, so the real exposure is legible. +- 15.1.3 closes as a by-product. + +**Costs, stated plainly.** + +- **Populating 345 cells is real work.** Mitigated by generating the skeleton from the held corpus + (ids, levels, chapter/section come free) and marking everything not verified today `unverified` — + which is honest, and is itself the finding. +- **Evidence anchors will break on refactors.** That is the feature; it is also maintenance. `expect` + is a token rather than a line number precisely so ordinary edits do not thrash it. +- **Two homes for one system** (tool public, data vault). Accepted: the alternative is either posture + data in the public repo or a tool nothing runs. +- **This does not make the score correct.** It makes it *consistent, derived, and drift-detecting*. + A wrong verdict recorded carefully is still wrong — adversarial verification remains the only cure + for that, and this ADR does not replace it. + +**Rejected alternatives.** + +- *Keep prose, review harder.* Six documented failure modes in one day survived careful review by + construction: every false residual **read as true**. The project's own standard says the mitigation + must be structural, not diligence. +- *One document to rule them all.* Tried — that is what the current lineage is. It produces five + documents asserting three counts, because a superseding document cannot retract a copy it does not + know about. +- *Put the vault documents in the public repo so the guards see them.* Refused: they are an attacker + roadmap (`SECURITY-DOCS-POLICY.md`). Running the guards in the vault achieves the same end. + +## Open questions + +1. **Does the vault CI run `pytest` today?** The workflows exist; whether the suite executes there is + unconfirmed, and the whole 15.1.3 fix rests on it. +2. **Do ADR and BACKLOG numbers collide across the two repos?** The allocator is per-clone. BACKLOG + spaces are being partitioned; ADR is unresolved and this ADR was allocated in the development repo + deliberately. +3. **Should `ASVS-CURRENT.md` be publishable** in redacted form (count + levels, no residual text) so + adopters see a posture summary without the roadmap? Not decided here. diff --git a/docs/adr/README.md b/docs/adr/README.md index ac1b17a..e95dba7 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -182,3 +182,4 @@ what is withheld and what you can request. | [0153](0153-collapse-the-posture-gradient-no-data-label-may-allow-a-cleartext-hop.md) | **Collapse the posture gradient — no data label may allow a cleartext hop** — ADR 0092's precedence arm 3 is `not is_phi -> ALLOW`, so an instance whose environment file declares `data_class = "synthetic"` crosses **every** cleartext transport hop silently, with no warning and no audit record. Found by upgrading a dogfood instance whose environment file had been quietly unparseable: the moment it loaded and `data_class = "phi"` took effect, a cleartext MLLP egress that had been crossing without comment became a refusal — nothing about the hop had changed, only a label in a different file, and nothing had reported the guard was off. Deletes that one arm: `insecure_hop_disposition` loses `is_phi` and the precedence becomes loopback -> ALLOW, attested -> ALLOW, `cleartext_accepted` -> WARN, not-`enforcing` -> WARN, else REFUSE. Because the deleted arm returned ALLOW, removing it can only turn a crossing into a WARN or a REFUSE, so **0092's no-loosen rule holds by construction** and an unchanged config is byte-identical unless it relied on the label. Adds `cleartext_accepted` + `cleartext_reason` on **Destination**, deliberately SEPARATE from `tls_hop_attested`: attestation claims *this hop IS secure by means the engine cannot see* and passes silently, the new pair admits *it is NOT, and we accept that* and warns + audits every construction — merging them would leave the audit trail unable to tell a proxy-terminated hop from plaintext PHI on a flat network. **No TLS default is flipped** (an earlier draft proposed it): redundant, since an undeclared cleartext outbound already REFUSES and a refusal naming the hop beats a handshake failure; it hard-fails 14 of 14 inbound MLLP listeners because `_mllp_ssl_context(server=True)` demands a cert before any policy runs, unreachable by posture/attestation/the new flag and not loopback-exempt; and it would silently disarm the four inbound exposed-gates, which early-return when `tls` is truthy. `Tcp()`/`X12()` have no TLS support at all, so for them the declaration is **permanent and structural**, not transitional. Neither `data_class` nor `MEFOR_ALLOW_INSECURE_TLS` is removed — the hop authority stops READING the label, and the variable is unhooked from the authority but survives for the six non-connection cells (store TLS, LDAPS, webhook sink, AI broker) that have no per-connection field to carry an escape. Retains ADR 0148's `[security].enforcement` dial as an arm (its AC-3 requires it on every gate). Scope is the cleartext transport-hop decision ONLY; an explicit out-of-scope table names all four callers of the authority and the three other readers of `HopPosture` | Accepted (2026-07-25) — reworked after adversarial review; owner-ratified; **BUILT 2026-07-28** | | [0154](0154-synchronous-captured-downstream-reply-and-intake-authentication-for-the-inbound-http-listener-adr-0023-deferred-tail.md) | **Synchronous captured-downstream-reply and intake authentication for the inbound HTTP listener** — the ADR 0023 deferred tail: `reply_from` blocks the HTTP turn on a **committed** ADR 0013 `response` row (never an in-flight `DeliveryResponse`), and `intake_auth` (API key / bearer / mTLS subject) adds a peer control behind a posture-keyed gate. Also closes a live hole — `check_http_tls_exposure` returns early on truthy `tls`, so an off-loopback `Http(tls=True)` listener authenticates nobody today | **Accepted (2026-07-31)** — owner-ratified at rev 5; authorises **increment A only** (intake-auth + peer-control gate), which is **built and merged** (2026-08-01, `f2ef0ea9`); sync-reply (increment B) deferred pending a customer | | [0155](0155-dast-dynamic-security-testing-of-the-running-engine.md) | **DAST — dynamic security testing of the running engine** (BACKLOG #318) — no DAST had ever run against this project: every security test was static or in-process, leaving the [Secure_Development_Standards](../Secure_Development_Standards.md) §6.1 *Dynamic* tier row empty. Increment 1 builds a **self-run, authenticated authorization sweep** with **no new dependency**: one `uvicorn` listener on loopback in front of a real Engine + real AuthService, both identities minted over the wire through `POST /auth/login`, and the authorization expectation **derived from the live route table** by a single shared `require*()`-closure walk ([`scripts/security/route_gates.py`](../../scripts/security/route_gates.py), hoisted out of the security doc-drift guard so exactly one derivation of *is this route gated* exists in the tree) rather than a hand-kept list that goes stale the day a route lands. Measured shape: 105 route rows, 100 gated, 87 permission-gated, exactly 5 anonymous. Three passes — **negative** (every gated HTTP row sent with no credential and with an invalid bearer; anything but 401 is a finding), **authorized reach** (how many gated `GET` rows a *privileged* token got past authentication and authorization on — the positive number that stops a wall of 401s reading as *all endpoints protected*), and **viewer BFLA** (anything but a refusal, **including 404**, is a finding, because a 404 on a matched path template means the caller got past authorization into resource lookup). The receipt names what it examined and **fails closed** — below any floor it exits **2 (could not measure)**, never 0 — and records method, path template, status codes, counts and the *relaxed* posture it scanned, never a body, header or token. Two canaries are built from **supported configuration, not source patches** (authentication disabled at the target; the low-privilege identity over-granted while the expectation set stays the viewer's), avoiding the patch-rot failure where a canary silently stops applying; CI runs both **before** the real scan and requires each to exit exactly **1** (findings) *with* its receipt on disk — a 0 (blind), a 2 (could not measure, which is what a neutered canary actually produces) or a crash fails the job and the real scan never runs. The inversion is the design: the sweep and both canaries run as ordinary pytest in the **existing required** test legs, so a change that blinds the detector reds a PR, while the nightly workflow is advisory — **not** a required context (it has no `pull_request` trigger, so it cannot report on a PR) and deliberately **not** `continue-on-error`. **Scope boundary: see the ADR's *Scope boundary* section** — it is stated once there, verbatim, and this row deliberately carries a pointer and no wording of its own. Rejected/deferred: schemathesis (the shipped OpenAPI declares no `securitySchemes` and no per-operation `security`, so `ignored_auth` would pass on every operation having probed none of them; adoption also needs a fifth DEP-1 lock over a ~30-distribution closure that could pull a shipped runtime floor down), ZAP (a SHA-pinned action still pulls a mutable `:stable` image, and the published images do not install the web console), nuclei (template matcher; `pip install nuclei` is an abandoned unrelated package), Dredd/RESTler/CATS, in-process ASGI transport (bypasses the HTTP parser, leaves `request.client` unset), the unauthenticated MLLP/TCP/X12/DICOM ingress plane, the `/ui` console plane, a TLS black-box target, and non-`GET` reach/BFLA | Accepted (2026-07-31) — increment 1 built; advisory, not a required context | +| [0156](0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md) | **ASVS scorecard as data — a derived count, verified evidence anchors, and a fail-closed drift gate** — the ASVS score is maintained as prose, and nothing checks it. One re-anchoring session (2026-08-01) re-derived the headline count **6 times**, found **12 residuals of record factually false at HEAD** (five of them *absence* claims that had silently stopped being true), and found **10 cells missing from an enumeration described as "arithmetic-checked and complete"** — which survived because the arithmetic closed to 345 and closure was read as proof. **Closure only proves the four buckets sum, not that every cell landed in one.** Decision: hold one `[[cell]]` record per requirement (all 345) in `asvs-scorecard.toml`; **compute** the count so no document can state one; assert **every corpus id appears exactly once** (the check whose absence cost ten cells); machine-verify each cell's `evidence` anchor by asserting an expected **token** still resolves, so code movement reds a test instead of rotting a sentence; require an absence claim to record the **search that proved it plus a positive control that must still hit**, because a grep naming the wrong token returns zero and reads exactly like proof; make **`unverified` a first-class verdict** so inherited-versus-verified Pass is countable (~219 Passes have never been read against the requirement text); and **fail closed rather than skip**. Tool + schema + fixture tests live in this repo and run in public CI; the real scorecard lives in the vault with a vault-CI job — which closes **ASVS 15.1.3**, currently open precisely because six `*_doc_drift` modules assert against documents that `git ls-files docs/security/` shows are **not present** in the tree where CI runs. Rejected: *keep prose and review harder* (every false residual **read as true**; the project's own standard says the mitigation must be structural, not diligence), *one document to rule them all* (that is the current lineage, and it produces five documents asserting three counts), and *publish the vault documents so the guards see them* (attacker roadmap, `SECURITY-DOCS-POLICY.md`). Explicitly **does not** make the score correct — only consistent, derived and drift-detecting; adversarial verification remains the only cure for a wrong verdict | **Proposed (2026-08-01)** — no code yet | From 3259a552828355f898caa8947f363a2d906e3dab Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 1 Aug 2026 08:43:17 -0500 Subject: [PATCH 2/8] feat(asvs): scorecard verifier -- derived count, verified anchors, fail-closed (ADR 0156) The tool half of ADR 0156. Stdlib only (tomllib/json/re), no new dependency, and deliberately data-free: it takes the scorecard and the ASVS corpus as paths, so this repo unit-tests it against fixtures while the vault runs the same code against the real posture data. What it enforces, each aimed at a defect observed on 2026-08-01: check_completeness -- every corpus id appears EXACTLY ONCE and nothing outside the corpus appears at all. This is the check whose absence cost ten cells: an enumeration closed to 345 and closure was read as proof. Closure only proves the four buckets sum, not that every cell landed in one. count() -- computed, never typed. Five documents asserting three counts is what this replaces. check_anchors -- each cell's evidence names a TOKEN that must still resolve within a window of the recorded line. Code movement reds a test instead of rotting a sentence. That is the twelve-false-residuals defect. check_absences -- an absence claim is admissible ONLY with a positive control that must still match. A grep naming the wrong token returns zero and reads exactly like proof; five residuals of record died that way. If the control goes quiet the search is BLIND and the claim is void regardless of the pattern. unverified is a first-class verdict, so an inherited Pass cannot hide inside a headline. The renderer reports verified-vs-inherited separately. Fail closed: a missing scorecard EXITS 2 (could not measure), never 0. That distinction is the whole of ASVS 15.1.3, where six *_doc_drift modules skip because their target is not in the tree where CI runs. 17 tests, and every check is proved to go RED before it is trusted green -- dropped cell, duplicate cell, an id ASVS retired, a level disagreeing with the corpus, a moved token, a deleted file, a blind absence search, an absence that became false. Exit codes verified by hand: 0 clean / 1 findings / 2 missing. ruff clean, ruff format clean, mypy clean. --- scripts/asvs/scorecard.py | 352 +++++++++++++++++++++++++++++++++++ tests/test_asvs_scorecard.py | 267 ++++++++++++++++++++++++++ 2 files changed, 619 insertions(+) create mode 100644 scripts/asvs/scorecard.py create mode 100644 tests/test_asvs_scorecard.py diff --git a/scripts/asvs/scorecard.py b/scripts/asvs/scorecard.py new file mode 100644 index 0000000..2de672e --- /dev/null +++ b/scripts/asvs/scorecard.py @@ -0,0 +1,352 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 MessageFoundry Organization and contributors +"""ASVS scorecard as data (ADR 0156) — derive the count, verify the evidence, fail closed. + +The score used to live in prose. On 2026-08-01 one re-anchoring session re-derived the headline count +six times, found twelve residuals of record that were false at HEAD (five of them *absence* claims that +had quietly stopped being true), and found ten cells missing from an enumeration that described itself +as "arithmetic-checked and complete". That last one survived because the arithmetic closed to 345 and +closure was read as proof. + +**Closure only proves the four buckets sum. It does not prove every cell landed in one.** That is the +whole reason :func:`check_completeness` exists and asserts against the corpus rather than the total. + +This module is deliberately data-free: it takes the scorecard and the ASVS corpus as paths, so the +public repo can unit-test it against a fixture while the vault runs it against the real posture data +(ADR 0156 §7). It adds no dependency — ``tomllib`` is stdlib. +""" + +from __future__ import annotations + +import argparse +import re +import sys +import tomllib +from collections import Counter +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Final, Literal + +Verdict = Literal["pass", "partial", "fail", "na", "unverified"] + +#: The scoring buckets. ``unverified`` is deliberately first-class (ADR 0156 §5): a cell inherited from +#: an earlier assessment and never re-read against the requirement text is NOT a Pass, and conflating +#: the two is what let ~219 unchecked verdicts hide inside a headline. +VERDICTS: Final[frozenset[str]] = frozenset({"pass", "partial", "fail", "na", "unverified"}) + +#: How far from the recorded line the expected token may drift before the anchor is considered broken. +#: Anchors name a TOKEN rather than a bare line number precisely so that ordinary edits above a cell's +#: evidence do not thrash every anchor in the file; the window keeps the line number meaningful without +#: making it load-bearing. +ANCHOR_WINDOW: Final[int] = 40 + + +class ScorecardError(Exception): + """A defect in the scorecard itself — malformed, incomplete, or contradicting the corpus.""" + + +@dataclass(frozen=True) +class Anchor: + """A claim that some token exists in the tree, at roughly a known place.""" + + path: str + line: int + expect: str + + +@dataclass(frozen=True) +class Absence: + """A claim that something does NOT exist, plus the proof the search could have seen it. + + A grep naming the wrong token returns zero and reads exactly like proof — that is how five false + absence claims survived for weeks. So an absence claim is only admissible with a + ``positive_control`` that must still match; if the control goes quiet the search has gone blind and + the claim is void, regardless of what the pattern returns. + """ + + pattern: str + positive_control: str + + +@dataclass(frozen=True) +class Cell: + id: str + level: int + verdict: Verdict + residual: str = "" + posture: str = "single" + last_verified: str = "" + verified_at: str = "" + evidence: tuple[Anchor, ...] = () + absence: tuple[Absence, ...] = () + + @property + def is_inherited(self) -> bool: + """A verdict carried from an earlier assessment, never re-read against the requirement text.""" + return self.verdict == "unverified" or not self.last_verified + + +@dataclass +class Findings: + """What a verification pass found. Empty ``problems`` is the only pass condition.""" + + problems: list[str] = field(default_factory=list) + checked_anchors: int = 0 + checked_absences: int = 0 + skipped_anchors: int = 0 + + @property + def ok(self) -> bool: + return not self.problems + + +def load_corpus(path: Path) -> dict[str, int]: + """Map ``req_id`` (V-stripped) to level, from the OWASP ASVS 5.0.0 flat JSON. + + The corpus is the external anchor. Before it was held, every verdict in this project reasoned from + the requirement verb *as paraphrased in our own scorecards* — a closed loop that produced a cell + scored against a requirement ASVS had deleted. + """ + import json + + if not path.is_file(): + raise ScorecardError( + f"ASVS corpus not found at {path} — cannot check completeness without it" + ) + raw: Any = json.loads(path.read_text(encoding="utf-8")) + reqs = raw["requirements"] if isinstance(raw, dict) else raw + return {str(r["req_id"]).lstrip("V"): int(r["L"]) for r in reqs} + + +def load_scorecard(path: Path) -> list[Cell]: + if not path.is_file(): + # Fail closed, never skip (ADR 0156 §6). Skipping is exactly what the doc-drift guards do + # today, and it is why a green CI proves nothing about these documents. + raise ScorecardError( + f"scorecard not found at {path} — refusing to report a pass on a missing file" + ) + data = tomllib.loads(path.read_text(encoding="utf-8")) + cells: list[Cell] = [] + for raw in data.get("cell", []): + verdict = str(raw.get("verdict", "")).lower() + if verdict not in VERDICTS: + raise ScorecardError( + f"cell {raw.get('id')!r}: verdict {verdict!r} not one of {sorted(VERDICTS)}" + ) + cells.append( + Cell( + id=str(raw["id"]), + level=int(raw["level"]), + verdict=verdict, # type: ignore[arg-type] + residual=str(raw.get("residual", "")), + posture=str(raw.get("posture", "single")), + last_verified=str(raw.get("last_verified", "")), + verified_at=str(raw.get("verified_at", "")), + evidence=tuple( + Anchor(path=str(e["path"]), line=int(e["line"]), expect=str(e["expect"])) + for e in raw.get("evidence", []) + ), + absence=tuple( + Absence(pattern=str(a["pattern"]), positive_control=str(a["positive_control"])) + for a in raw.get("absence", []) + ), + ) + ) + return cells + + +def count(cells: list[Cell]) -> Counter[str]: + """The count is COMPUTED. No document states one; documents render this (ADR 0156 §2).""" + return Counter(c.verdict for c in cells) + + +def check_completeness(cells: list[Cell], corpus: dict[str, int]) -> list[str]: + """Every corpus id appears exactly once, and nothing outside the corpus appears at all. + + **This is the check whose absence cost ten cells.** An enumeration that closes to the right total + can still be missing entries; only comparing against the corpus detects that. + """ + problems: list[str] = [] + seen = Counter(c.id for c in cells) + + for dupe, n in sorted((i, n) for i, n in seen.items() if n > 1): + problems.append( + f"completeness: {dupe} appears {n} times — each requirement gets exactly one cell" + ) + + missing = sorted(set(corpus) - set(seen), key=_sort_key) + if missing: + problems.append( + f"completeness: {len(missing)} requirement(s) in the ASVS corpus have NO cell — " + f"the count cannot be correct while a cell is absent: {', '.join(missing)}" + ) + + unknown = sorted(set(seen) - set(corpus), key=_sort_key) + if unknown: + problems.append( + f"completeness: {len(unknown)} cell(s) are not ASVS 5.0.0 requirement ids " + f"(retired in 5.0, or a typo): {', '.join(unknown)}" + ) + + for c in cells: + want = corpus.get(c.id) + if want is not None and c.level != want: + problems.append(f"{c.id}: level {c.level} but the corpus says L{want}") + return problems + + +def check_anchors(cells: list[Cell], root: Path, findings: Findings) -> None: + """Open every evidence anchor and assert its token still resolves. + + When the code moves, this reds a test — instead of the sentence rotting in place and the next + session funding work that is already done. + """ + for c in cells: + for a in c.evidence: + target = root / a.path + if not target.is_file(): + findings.problems.append(f"{c.id}: evidence path {a.path} does not exist") + continue + lines = target.read_text(encoding="utf-8", errors="replace").splitlines() + lo = max(0, a.line - 1 - ANCHOR_WINDOW) + hi = min(len(lines), a.line + ANCHOR_WINDOW) + findings.checked_anchors += 1 + if a.expect in "\n".join(lines[lo:hi]): + continue + where = " (found elsewhere in the file)" if a.expect in "\n".join(lines) else "" + findings.problems.append( + f"{c.id}: {a.path}:{a.line} no longer contains {a.expect!r} within " + f"±{ANCHOR_WINDOW} lines{where} — the evidence moved or the claim is now false" + ) + + +def check_absences(cells: list[Cell], root: Path, findings: Findings) -> None: + """An absence is proven only when its pattern is quiet AND its positive control still speaks.""" + corpus_files = _python_sources(root) + for c in cells: + for a in c.absence: + findings.checked_absences += 1 + control = _grep_count(a.positive_control, corpus_files) + if control == 0: + findings.problems.append( + f"{c.id}: absence claim is BLIND — its positive control {a.positive_control!r} " + f"matches nothing, so a zero result for {a.pattern!r} proves nothing" + ) + continue + hits = _grep_count(a.pattern, corpus_files) + if hits: + findings.problems.append( + f"{c.id}: absence claim is FALSE — {a.pattern!r} now matches {hits} time(s); " + f"the thing recorded as missing exists" + ) + + +def _python_sources(root: Path) -> list[Path]: + out: list[Path] = [] + for pkg in ("messagefoundry", "messagefoundry_webconsole", "harness", "scripts"): + base = root / pkg + if base.is_dir(): + out.extend( + p + for p in base.rglob("*.py") + if ".venv" not in p.parts and "__pycache__" not in p.parts + ) + return out + + +def _grep_count(pattern: str, files: list[Path]) -> int: + rx = re.compile(pattern) + return sum(1 for f in files if rx.search(f.read_text(encoding="utf-8", errors="replace"))) + + +def _sort_key(cell_id: str) -> tuple[int, ...]: + try: + return tuple(int(p) for p in cell_id.split(".")) + except ValueError: # a malformed id sorts last rather than crashing the report + return (9_999,) + + +def verify(scorecard: Path, corpus: Path, root: Path) -> Findings: + findings = Findings() + cells = load_scorecard(scorecard) + findings.problems.extend(check_completeness(cells, load_corpus(corpus))) + check_anchors(cells, root, findings) + check_absences(cells, root, findings) + return findings + + +def render_current(cells: list[Cell], *, anchor_sha: str) -> str: + """The generated entry point. A new session reads this instead of reconstructing state.""" + n = count(cells) + total = sum(n.values()) + verified_pass = sum(1 for c in cells if c.verdict == "pass" and not c.is_inherited) + inherited = sum(1 for c in cells if c.verdict == "pass" and c.is_inherited) + lines = [ + "", + "", + "# ASVS 5.0 L3 — current state", + "", + f"**Anchor commit:** `{anchor_sha}`", + "", + "| Verdict | Count |", + "|---|---:|", + f"| Pass | {n['pass']} |", + f"| Partial | {n['partial']} |", + f"| Fail | {n['fail']} |", + f"| N/A | {n['na']} |", + f"| **Unverified** | **{n['unverified']}** |", + f"| **Total** | **{total}** |", + "", + f"Of the {n['pass']} Passes, **{verified_pass} were verified against the requirement text** and " + f"**{inherited} are inherited** — carried from an earlier assessment and never re-read. An " + "inherited Pass is not evidence of anything; it is the largest standing exposure in this score.", + "", + "## Open cells", + "", + "| Cell | L | Verdict | Residual |", + "|---|---|---|---|", + ] + for c in sorted( + (c for c in cells if c.verdict in {"partial", "fail"}), key=lambda c: _sort_key(c.id) + ): + lines.append(f"| {c.id} | L{c.level} | **{c.verdict}** | {c.residual[:160]} |") + return "\n".join(lines) + "\n" + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Verify or render the ASVS scorecard (ADR 0156).") + ap.add_argument("--scorecard", type=Path, required=True) + ap.add_argument("--corpus", type=Path, required=True) + ap.add_argument( + "--root", type=Path, default=Path.cwd(), help="tree the evidence anchors point into" + ) + ap.add_argument("--render", type=Path, help="write the generated CURRENT.md here") + ap.add_argument("--anchor-sha", default="unknown") + args = ap.parse_args(argv) + + try: + findings = verify(args.scorecard, args.corpus, args.root) + except ScorecardError as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 # could not measure — never 0, never confused with "clean" + + cells = load_scorecard(args.scorecard) + n = count(cells) + print( + f"scanned {len(cells)} cells " + f"({n['pass']} pass / {n['partial']} partial / {n['fail']} fail / {n['na']} na / " + f"{n['unverified']} unverified); " + f"verified {findings.checked_anchors} evidence anchors and {findings.checked_absences} absence claims" + ) + for p in findings.problems: + print(f" FAIL {p}", file=sys.stderr) + + if args.render and findings.ok: + args.render.write_text(render_current(cells, anchor_sha=args.anchor_sha), encoding="utf-8") + print(f"rendered {args.render}") + + return 0 if findings.ok else 1 + + +if __name__ == "__main__": # pragma: no cover + raise SystemExit(main()) diff --git a/tests/test_asvs_scorecard.py b/tests/test_asvs_scorecard.py new file mode 100644 index 0000000..54f8e38 --- /dev/null +++ b/tests/test_asvs_scorecard.py @@ -0,0 +1,267 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 MessageFoundry Organization and contributors +"""ADR 0156 — the ASVS scorecard verifier. + +Every check here is proved to go RED before it is trusted green. That is not ceremony: this project +has shipped several guards that could not fail, and the defect this tool exists to catch survived +precisely because a total that closed to 345 was read as proof of completeness. + +The tool is data-free by design (ADR 0156 §7), so these tests run against fixtures in the public repo +while the vault runs the same code against the real posture data. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from scripts.asvs.scorecard import ( + Absence, + Anchor, + Cell, + Findings, + ScorecardError, + check_absences, + check_anchors, + check_completeness, + count, + load_corpus, + load_scorecard, + render_current, + verify, +) + +CORPUS = {"1.1.1": 1, "1.1.2": 2, "2.1.1": 3} + + +def _corpus_file(tmp_path: Path, ids: dict[str, int] | None = None) -> Path: + p = tmp_path / "corpus.json" + reqs = [{"req_id": f"V{k}", "L": str(v)} for k, v in (ids or CORPUS).items()] + p.write_text(json.dumps({"requirements": reqs}), encoding="utf-8") + return p + + +def _scorecard_file(tmp_path: Path, body: str) -> Path: + p = tmp_path / "asvs-scorecard.toml" + p.write_text(body, encoding="utf-8") + return p + + +def _cells(*specs: tuple[str, int, str]) -> list[Cell]: + return [Cell(id=i, level=lv, verdict=v) for i, lv, v in specs] # type: ignore[arg-type] + + +# --- completeness: the check whose absence cost ten cells --------------------------------------- + + +def test_completeness_passes_when_every_corpus_id_appears_once() -> None: + cells = _cells(("1.1.1", 1, "pass"), ("1.1.2", 2, "partial"), ("2.1.1", 3, "fail")) + assert check_completeness(cells, CORPUS) == [] + + +def test_completeness_catches_a_dropped_cell() -> None: + """The 2026-08-01 defect, in miniature: a count that sums correctly while a cell is missing.""" + cells = _cells(("1.1.1", 1, "pass"), ("1.1.2", 2, "partial")) # 2.1.1 dropped + problems = check_completeness(cells, CORPUS) + assert any("have NO cell" in p and "2.1.1" in p for p in problems) + + +def test_completeness_catches_a_duplicate_cell() -> None: + cells = _cells( + ("1.1.1", 1, "pass"), ("1.1.1", 1, "partial"), ("1.1.2", 2, "pass"), ("2.1.1", 3, "pass") + ) + assert any("appears 2 times" in p for p in check_completeness(cells, CORPUS)) + + +def test_completeness_catches_an_id_that_is_not_in_the_standard() -> None: + """11.7.2 was scored against ASVS 4.0.3 V8.3.6, deleted in 5.0. This is that class of error.""" + cells = _cells( + ("1.1.1", 1, "pass"), ("1.1.2", 2, "pass"), ("2.1.1", 3, "pass"), ("8.3.6", 2, "partial") + ) + assert any( + "not ASVS 5.0.0 requirement ids" in p and "8.3.6" in p + for p in check_completeness(cells, CORPUS) + ) + + +def test_completeness_catches_a_level_that_disagrees_with_the_corpus() -> None: + cells = _cells(("1.1.1", 3, "pass"), ("1.1.2", 2, "pass"), ("2.1.1", 3, "pass")) + assert any("but the corpus says L1" in p for p in check_completeness(cells, CORPUS)) + + +# --- the count is computed, never typed ---------------------------------------------------------- + + +def test_count_is_derived_from_the_cells() -> None: + cells = _cells(("1.1.1", 1, "pass"), ("1.1.2", 2, "partial"), ("2.1.1", 3, "unverified")) + n = count(cells) + assert (n["pass"], n["partial"], n["unverified"]) == (1, 1, 1) + assert sum(n.values()) == len(cells) + + +# --- evidence anchors ----------------------------------------------------------------------------- + + +def test_anchor_resolves_when_the_token_is_present(tmp_path: Path) -> None: + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text( + "a\nb\ntls_cert_file = None\n", encoding="utf-8" + ) + cells = [ + Cell( + id="1.1.1", + level=1, + verdict="pass", + evidence=(Anchor("messagefoundry/m.py", 3, "tls_cert_file"),), + ) + ] + f = Findings() + check_anchors(cells, tmp_path, f) + assert f.ok and f.checked_anchors == 1 + + +def test_anchor_goes_red_when_the_token_is_gone(tmp_path: Path) -> None: + """The twelve-false-residuals defect: the code moved and the sentence stayed.""" + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text("nothing here\n", encoding="utf-8") + cells = [ + Cell( + id="1.1.1", + level=1, + verdict="pass", + evidence=(Anchor("messagefoundry/m.py", 1, "tls_cert_file"),), + ) + ] + f = Findings() + check_anchors(cells, tmp_path, f) + assert not f.ok and "no longer contains" in f.problems[0] + + +def test_anchor_goes_red_when_the_file_is_gone(tmp_path: Path) -> None: + cells = [ + Cell( + id="1.1.1", + level=1, + verdict="pass", + evidence=(Anchor("messagefoundry/gone.py", 1, "x"),), + ) + ] + f = Findings() + check_anchors(cells, tmp_path, f) + assert not f.ok and "does not exist" in f.problems[0] + + +# --- absence claims: the class this project is worst at ------------------------------------------- + + +def test_absence_holds_when_pattern_is_quiet_and_control_speaks(tmp_path: Path) -> None: + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text( + "class ScanRejected: pass\n", encoding="utf-8" + ) + cells = [ + Cell( + id="1.1.1", level=1, verdict="fail", absence=(Absence("clamav|clamd", "ScanRejected"),) + ) + ] + f = Findings() + check_absences(cells, tmp_path, f) + assert f.ok and f.checked_absences == 1 + + +def test_absence_is_rejected_as_BLIND_when_the_positive_control_matches_nothing( + tmp_path: Path, +) -> None: + """A grep naming the wrong token returns zero and reads exactly like proof. This is the fence.""" + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text("unrelated\n", encoding="utf-8") + cells = [ + Cell(id="1.1.1", level=1, verdict="fail", absence=(Absence("clamav", "ScanRejected"),)) + ] + f = Findings() + check_absences(cells, tmp_path, f) + assert not f.ok and "BLIND" in f.problems[0] + + +def test_absence_is_rejected_as_FALSE_when_the_thing_now_exists(tmp_path: Path) -> None: + """Five residuals of record were absence claims that had silently stopped being true.""" + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text( + "import clamd\nclass ScanRejected: pass\n", encoding="utf-8" + ) + cells = [Cell(id="1.1.1", level=1, verdict="fail", absence=(Absence("clamd", "ScanRejected"),))] + f = Findings() + check_absences(cells, tmp_path, f) + assert not f.ok and "FALSE" in f.problems[0] + + +# --- fail closed, never skip ---------------------------------------------------------------------- + + +def test_missing_scorecard_raises_rather_than_skipping(tmp_path: Path) -> None: + """15.1.3 in one line: the guards skip when the document is absent, so green proves nothing.""" + with pytest.raises(ScorecardError, match="refusing to report a pass on a missing file"): + load_scorecard(tmp_path / "absent.toml") + + +def test_missing_corpus_raises(tmp_path: Path) -> None: + with pytest.raises(ScorecardError, match="cannot check completeness"): + load_corpus(tmp_path / "absent.json") + + +def test_unknown_verdict_is_refused(tmp_path: Path) -> None: + sc = _scorecard_file( + tmp_path, '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "conditional-pass"\n' + ) + with pytest.raises(ScorecardError, match="not one of"): + load_scorecard(sc) + + +# --- end to end ----------------------------------------------------------------------------------- + + +def test_verify_end_to_end_clean(tmp_path: Path) -> None: + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text("tls_cert_file = None\n", encoding="utf-8") + sc = _scorecard_file( + tmp_path, + """ +[[cell]] +id = "1.1.1" +level = 1 +verdict = "pass" +last_verified = "2026-08-01" +[[cell.evidence]] +path = "messagefoundry/m.py" +line = 1 +expect = "tls_cert_file" + +[[cell]] +id = "1.1.2" +level = 2 +verdict = "unverified" + +[[cell]] +id = "2.1.1" +level = 3 +verdict = "partial" +residual = "ships off" +""", + ) + findings = verify(sc, _corpus_file(tmp_path), tmp_path) + assert findings.ok, findings.problems + + +def test_render_separates_verified_pass_from_inherited_pass() -> None: + cells = [ + Cell(id="1.1.1", level=1, verdict="pass", last_verified="2026-08-01"), + Cell(id="1.1.2", level=2, verdict="pass"), # inherited: never re-read + Cell(id="2.1.1", level=3, verdict="partial", residual="ships off"), + ] + out = render_current(cells, anchor_sha="deadbeef") + assert "**1 were verified against the requirement text**" in out + assert "**1 are inherited**" in out + assert "deadbeef" in out + assert "| 2.1.1 | L3 | **partial** |" in out From 0b3728a15b8d613b9b9c534c3796a36f6df27b72 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 1 Aug 2026 08:47:24 -0500 Subject: [PATCH 3/8] docs(adr): ratify ADR 0156, amending section 7 -- the vault CI premise was false Accepted 2026-08-01, built and merged the same day. Section 7 originally proposed "a vault CI job runs the verifier against the real data", reasoning from the vault having ci.yml, tests/ and pyproject.toml. Open question 1 asked whether that CI actually executes. It does not. The actions API reports EVERY vault workflow as disabled_manually -- CI, Security, CodeQL, backlog-hygiene, release. Last run 2026-07-27, and the two vault PRs merged on 2026-08-01 both merged with zero checks. The estate was switched off at the cutover to avoid duplicating public CI; that decision stands and is not being reversed here. A CI-only design would have shipped dead. Amended to what was actually built: a vault pre-commit hook (works today, fires at authoring time, when the drift is introduced) plus one narrow new workflow -- new workflows are active by default even though the estate is disabled, so it runs without resurrecting any of it. The lesson is this ADR's own thesis applied to itself: section 7 was a confident, plausible statement about system state that nobody had checked. It was caught only because ratification was gated on answering the open question rather than on the argument reading well. --- ...ce-anchors-and-a-fail-closed-drift-gate.md | 49 +++++++++++++++++-- docs/adr/README.md | 2 +- 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md b/docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md index 16c822d..4ccd6d5 100644 --- a/docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md +++ b/docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md @@ -1,6 +1,6 @@ # ADR 0156 — ASVS scorecard as data: a derived count, verified evidence anchors, and a fail-closed drift gate -**Status:** Proposed +**Status:** **Accepted (2026-08-01)** — built and merged the same day; §7 amended at ratification, see below **Date:** 2026-08-01 **Supersedes:** — **Related:** ADR 0155 (DAST) · BACKLOG #205 (risk-acceptance register) · ASVS cell 15.1.3 @@ -119,10 +119,35 @@ precisely what 15.1.3 does today and why a green CI proves nothing about these d |---|---|---| | Verifier + schema + unit tests (fixture data) | **Development repo** | Generic tooling, no posture data; runs in public CI on every PR | | `asvs-scorecard.toml` (real verdicts + evidence) | **Vault** | Posture data; `docs/security/`-class | -| Job running the verifier against the real data | **Vault CI** | The vault has `ci.yml`, `tests/`, `pyproject.toml` — the guards can run where the documents actually live | +| Enforcement against the real data | **Vault pre-commit hook + one narrow vault workflow** | See the amendment — the obvious answer was wrong | That last row is the fix for **15.1.3**: the guards stop being inspection-only theatre. +> ### ⚠️ Amendment at ratification (2026-08-01) — §7's original placement was WRONG +> +> This ADR was drafted proposing *"a vault CI job runs the verifier against the real data"*, reasoning +> from the fact that the vault has `ci.yml`, `tests/` and `pyproject.toml`. **Open question 1 asked +> whether that CI actually executes. It does not.** +> +> The GitHub actions API reports **every vault workflow as `disabled_manually`** — CI, Security, +> CodeQL, backlog-hygiene, release, all of them. The last run was **2026-07-27**, and the two vault +> PRs merged on 2026-08-01 both merged with **zero checks**. The estate was switched off at the +> cutover to avoid duplicating public CI, which is a reasonable decision and is not being reversed. +> +> **A CI-only design would have shipped dead.** The built design is therefore: +> +> 1. **A vault pre-commit hook** — the enforcement that works *today*, needs no CI, and fires at +> authoring time, which is when the drift is introduced. +> 2. **One narrow new workflow** (`asvs-scorecard.yml`) — new workflows are active by default even +> though the estate is disabled, so this runs without resurrecting any of it. Stdlib-only, no +> install step, path-filtered, 5-minute cap. It additionally fails if the rendered +> `ASVS-CURRENT.md` has drifted from the data. +> +> **The lesson is the ADR's own thesis applied to itself.** §7 was a confident, plausible statement +> about system state that nobody had checked — exactly the failure mode the rest of this document +> exists to prevent. It was caught only because ratification was gated on answering the open question +> rather than on the argument reading well. + ### 8. A generated entry point `ASVS-CURRENT.md` is rendered, never hand-written: the count, the anchor commit, the open cells, and @@ -163,10 +188,26 @@ page instead of reconstructing state from ten superseding documents. - *Put the vault documents in the public repo so the guards see them.* Refused: they are an attacker roadmap (`SECURITY-DOCS-POLICY.md`). Running the guards in the vault achieves the same end. +## Status at ratification + +Built and merged 2026-08-01, same day as the decision: + +- **Tool** — `scripts/asvs/scorecard.py` (development repo), stdlib only, no new dependency. +- **Tests** — `tests/test_asvs_scorecard.py`, 17 tests. **Every check is proved to go RED before it is + trusted green**: dropped cell, duplicate cell, an id ASVS retired, a level disagreeing with the + corpus, a moved token, a deleted file, a blind absence search, an absence that became false. + Exit codes verified by hand — **0** clean, **1** findings, **2** could-not-measure on a missing file. +- **Data** — `docs/security/asvs-scorecard.toml` (vault), **all 345 cells** seeded from the held + corpus. 14 carry verified 2026-08-01 evidence; **331 are `unverified`, not `pass`**. The file + therefore does **not** reproduce the published headline and must not be read as doing so — what it + reports today is the state of *verification*, which had never been visible. +- **Entry point** — `docs/security/ASVS-CURRENT.md`, generated, never hand-written. + ## Open questions -1. **Does the vault CI run `pytest` today?** The workflows exist; whether the suite executes there is - unconfirmed, and the whole 15.1.3 fix rests on it. +1. ~~**Does the vault CI run `pytest` today?**~~ **ANSWERED 2026-08-01 — no.** Every vault workflow is + `disabled_manually`. §7 was amended accordingly *before* ratification. This is the question that + would have shipped a broken control had it been waved through. 2. **Do ADR and BACKLOG numbers collide across the two repos?** The allocator is per-clone. BACKLOG spaces are being partitioned; ADR is unresolved and this ADR was allocated in the development repo deliberately. diff --git a/docs/adr/README.md b/docs/adr/README.md index e95dba7..9c715f0 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -182,4 +182,4 @@ what is withheld and what you can request. | [0153](0153-collapse-the-posture-gradient-no-data-label-may-allow-a-cleartext-hop.md) | **Collapse the posture gradient — no data label may allow a cleartext hop** — ADR 0092's precedence arm 3 is `not is_phi -> ALLOW`, so an instance whose environment file declares `data_class = "synthetic"` crosses **every** cleartext transport hop silently, with no warning and no audit record. Found by upgrading a dogfood instance whose environment file had been quietly unparseable: the moment it loaded and `data_class = "phi"` took effect, a cleartext MLLP egress that had been crossing without comment became a refusal — nothing about the hop had changed, only a label in a different file, and nothing had reported the guard was off. Deletes that one arm: `insecure_hop_disposition` loses `is_phi` and the precedence becomes loopback -> ALLOW, attested -> ALLOW, `cleartext_accepted` -> WARN, not-`enforcing` -> WARN, else REFUSE. Because the deleted arm returned ALLOW, removing it can only turn a crossing into a WARN or a REFUSE, so **0092's no-loosen rule holds by construction** and an unchanged config is byte-identical unless it relied on the label. Adds `cleartext_accepted` + `cleartext_reason` on **Destination**, deliberately SEPARATE from `tls_hop_attested`: attestation claims *this hop IS secure by means the engine cannot see* and passes silently, the new pair admits *it is NOT, and we accept that* and warns + audits every construction — merging them would leave the audit trail unable to tell a proxy-terminated hop from plaintext PHI on a flat network. **No TLS default is flipped** (an earlier draft proposed it): redundant, since an undeclared cleartext outbound already REFUSES and a refusal naming the hop beats a handshake failure; it hard-fails 14 of 14 inbound MLLP listeners because `_mllp_ssl_context(server=True)` demands a cert before any policy runs, unreachable by posture/attestation/the new flag and not loopback-exempt; and it would silently disarm the four inbound exposed-gates, which early-return when `tls` is truthy. `Tcp()`/`X12()` have no TLS support at all, so for them the declaration is **permanent and structural**, not transitional. Neither `data_class` nor `MEFOR_ALLOW_INSECURE_TLS` is removed — the hop authority stops READING the label, and the variable is unhooked from the authority but survives for the six non-connection cells (store TLS, LDAPS, webhook sink, AI broker) that have no per-connection field to carry an escape. Retains ADR 0148's `[security].enforcement` dial as an arm (its AC-3 requires it on every gate). Scope is the cleartext transport-hop decision ONLY; an explicit out-of-scope table names all four callers of the authority and the three other readers of `HopPosture` | Accepted (2026-07-25) — reworked after adversarial review; owner-ratified; **BUILT 2026-07-28** | | [0154](0154-synchronous-captured-downstream-reply-and-intake-authentication-for-the-inbound-http-listener-adr-0023-deferred-tail.md) | **Synchronous captured-downstream-reply and intake authentication for the inbound HTTP listener** — the ADR 0023 deferred tail: `reply_from` blocks the HTTP turn on a **committed** ADR 0013 `response` row (never an in-flight `DeliveryResponse`), and `intake_auth` (API key / bearer / mTLS subject) adds a peer control behind a posture-keyed gate. Also closes a live hole — `check_http_tls_exposure` returns early on truthy `tls`, so an off-loopback `Http(tls=True)` listener authenticates nobody today | **Accepted (2026-07-31)** — owner-ratified at rev 5; authorises **increment A only** (intake-auth + peer-control gate), which is **built and merged** (2026-08-01, `f2ef0ea9`); sync-reply (increment B) deferred pending a customer | | [0155](0155-dast-dynamic-security-testing-of-the-running-engine.md) | **DAST — dynamic security testing of the running engine** (BACKLOG #318) — no DAST had ever run against this project: every security test was static or in-process, leaving the [Secure_Development_Standards](../Secure_Development_Standards.md) §6.1 *Dynamic* tier row empty. Increment 1 builds a **self-run, authenticated authorization sweep** with **no new dependency**: one `uvicorn` listener on loopback in front of a real Engine + real AuthService, both identities minted over the wire through `POST /auth/login`, and the authorization expectation **derived from the live route table** by a single shared `require*()`-closure walk ([`scripts/security/route_gates.py`](../../scripts/security/route_gates.py), hoisted out of the security doc-drift guard so exactly one derivation of *is this route gated* exists in the tree) rather than a hand-kept list that goes stale the day a route lands. Measured shape: 105 route rows, 100 gated, 87 permission-gated, exactly 5 anonymous. Three passes — **negative** (every gated HTTP row sent with no credential and with an invalid bearer; anything but 401 is a finding), **authorized reach** (how many gated `GET` rows a *privileged* token got past authentication and authorization on — the positive number that stops a wall of 401s reading as *all endpoints protected*), and **viewer BFLA** (anything but a refusal, **including 404**, is a finding, because a 404 on a matched path template means the caller got past authorization into resource lookup). The receipt names what it examined and **fails closed** — below any floor it exits **2 (could not measure)**, never 0 — and records method, path template, status codes, counts and the *relaxed* posture it scanned, never a body, header or token. Two canaries are built from **supported configuration, not source patches** (authentication disabled at the target; the low-privilege identity over-granted while the expectation set stays the viewer's), avoiding the patch-rot failure where a canary silently stops applying; CI runs both **before** the real scan and requires each to exit exactly **1** (findings) *with* its receipt on disk — a 0 (blind), a 2 (could not measure, which is what a neutered canary actually produces) or a crash fails the job and the real scan never runs. The inversion is the design: the sweep and both canaries run as ordinary pytest in the **existing required** test legs, so a change that blinds the detector reds a PR, while the nightly workflow is advisory — **not** a required context (it has no `pull_request` trigger, so it cannot report on a PR) and deliberately **not** `continue-on-error`. **Scope boundary: see the ADR's *Scope boundary* section** — it is stated once there, verbatim, and this row deliberately carries a pointer and no wording of its own. Rejected/deferred: schemathesis (the shipped OpenAPI declares no `securitySchemes` and no per-operation `security`, so `ignored_auth` would pass on every operation having probed none of them; adoption also needs a fifth DEP-1 lock over a ~30-distribution closure that could pull a shipped runtime floor down), ZAP (a SHA-pinned action still pulls a mutable `:stable` image, and the published images do not install the web console), nuclei (template matcher; `pip install nuclei` is an abandoned unrelated package), Dredd/RESTler/CATS, in-process ASGI transport (bypasses the HTTP parser, leaves `request.client` unset), the unauthenticated MLLP/TCP/X12/DICOM ingress plane, the `/ui` console plane, a TLS black-box target, and non-`GET` reach/BFLA | Accepted (2026-07-31) — increment 1 built; advisory, not a required context | -| [0156](0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md) | **ASVS scorecard as data — a derived count, verified evidence anchors, and a fail-closed drift gate** — the ASVS score is maintained as prose, and nothing checks it. One re-anchoring session (2026-08-01) re-derived the headline count **6 times**, found **12 residuals of record factually false at HEAD** (five of them *absence* claims that had silently stopped being true), and found **10 cells missing from an enumeration described as "arithmetic-checked and complete"** — which survived because the arithmetic closed to 345 and closure was read as proof. **Closure only proves the four buckets sum, not that every cell landed in one.** Decision: hold one `[[cell]]` record per requirement (all 345) in `asvs-scorecard.toml`; **compute** the count so no document can state one; assert **every corpus id appears exactly once** (the check whose absence cost ten cells); machine-verify each cell's `evidence` anchor by asserting an expected **token** still resolves, so code movement reds a test instead of rotting a sentence; require an absence claim to record the **search that proved it plus a positive control that must still hit**, because a grep naming the wrong token returns zero and reads exactly like proof; make **`unverified` a first-class verdict** so inherited-versus-verified Pass is countable (~219 Passes have never been read against the requirement text); and **fail closed rather than skip**. Tool + schema + fixture tests live in this repo and run in public CI; the real scorecard lives in the vault with a vault-CI job — which closes **ASVS 15.1.3**, currently open precisely because six `*_doc_drift` modules assert against documents that `git ls-files docs/security/` shows are **not present** in the tree where CI runs. Rejected: *keep prose and review harder* (every false residual **read as true**; the project's own standard says the mitigation must be structural, not diligence), *one document to rule them all* (that is the current lineage, and it produces five documents asserting three counts), and *publish the vault documents so the guards see them* (attacker roadmap, `SECURITY-DOCS-POLICY.md`). Explicitly **does not** make the score correct — only consistent, derived and drift-detecting; adversarial verification remains the only cure for a wrong verdict | **Proposed (2026-08-01)** — no code yet | +| [0156](0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md) | **ASVS scorecard as data — a derived count, verified evidence anchors, and a fail-closed drift gate** — the ASVS score is maintained as prose, and nothing checks it. One re-anchoring session (2026-08-01) re-derived the headline count **6 times**, found **12 residuals of record factually false at HEAD** (five of them *absence* claims that had silently stopped being true), and found **10 cells missing from an enumeration described as "arithmetic-checked and complete"** — which survived because the arithmetic closed to 345 and closure was read as proof. **Closure only proves the four buckets sum, not that every cell landed in one.** Decision: hold one `[[cell]]` record per requirement (all 345) in `asvs-scorecard.toml`; **compute** the count so no document can state one; assert **every corpus id appears exactly once** (the check whose absence cost ten cells); machine-verify each cell's `evidence` anchor by asserting an expected **token** still resolves, so code movement reds a test instead of rotting a sentence; require an absence claim to record the **search that proved it plus a positive control that must still hit**, because a grep naming the wrong token returns zero and reads exactly like proof; make **`unverified` a first-class verdict** so inherited-versus-verified Pass is countable (~219 Passes have never been read against the requirement text); and **fail closed rather than skip**. Tool + schema + fixture tests live in this repo and run in public CI; the real scorecard lives in the vault with a vault-CI job — which closes **ASVS 15.1.3**, currently open precisely because six `*_doc_drift` modules assert against documents that `git ls-files docs/security/` shows are **not present** in the tree where CI runs. Rejected: *keep prose and review harder* (every false residual **read as true**; the project's own standard says the mitigation must be structural, not diligence), *one document to rule them all* (that is the current lineage, and it produces five documents asserting three counts), and *publish the vault documents so the guards see them* (attacker roadmap, `SECURITY-DOCS-POLICY.md`). Explicitly **does not** make the score correct — only consistent, derived and drift-detecting; adversarial verification remains the only cure for a wrong verdict | **Accepted (2026-08-01)** — built and merged the same day. **§7 was amended at ratification**: it proposed a vault CI job, and the actions API showed every vault workflow (last run 2026-07-27; two vault PRs merged that day with zero checks), so a CI-only design would have shipped dead. Built instead as a vault **pre-commit hook** plus one **narrow new workflow**. That §7 was a confident, unchecked claim about system state is the ADR own thesis applied to itself | From e6bcc32cb7b8cdaf554cad95b066fe798ecb1042 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 1 Aug 2026 08:48:08 -0500 Subject: [PATCH 4/8] docs(adr): repair the 0156 index row -- a backtick was command-substituted away The row was written through a shell double-quoted string, so bash treated `disabled_manually` as a command substitution and replaced it with the empty string. The row shipped reading "every vault workflow (last run 2026-07-27...)" -- the load-bearing word silently deleted. Fitting failure mode for this ADR: the sentence still parsed, still read as plausible, and stated nothing. Caught by grepping for the token rather than for the sentence around it. --- docs/adr/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/README.md b/docs/adr/README.md index 9c715f0..fd25ef9 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -182,4 +182,4 @@ what is withheld and what you can request. | [0153](0153-collapse-the-posture-gradient-no-data-label-may-allow-a-cleartext-hop.md) | **Collapse the posture gradient — no data label may allow a cleartext hop** — ADR 0092's precedence arm 3 is `not is_phi -> ALLOW`, so an instance whose environment file declares `data_class = "synthetic"` crosses **every** cleartext transport hop silently, with no warning and no audit record. Found by upgrading a dogfood instance whose environment file had been quietly unparseable: the moment it loaded and `data_class = "phi"` took effect, a cleartext MLLP egress that had been crossing without comment became a refusal — nothing about the hop had changed, only a label in a different file, and nothing had reported the guard was off. Deletes that one arm: `insecure_hop_disposition` loses `is_phi` and the precedence becomes loopback -> ALLOW, attested -> ALLOW, `cleartext_accepted` -> WARN, not-`enforcing` -> WARN, else REFUSE. Because the deleted arm returned ALLOW, removing it can only turn a crossing into a WARN or a REFUSE, so **0092's no-loosen rule holds by construction** and an unchanged config is byte-identical unless it relied on the label. Adds `cleartext_accepted` + `cleartext_reason` on **Destination**, deliberately SEPARATE from `tls_hop_attested`: attestation claims *this hop IS secure by means the engine cannot see* and passes silently, the new pair admits *it is NOT, and we accept that* and warns + audits every construction — merging them would leave the audit trail unable to tell a proxy-terminated hop from plaintext PHI on a flat network. **No TLS default is flipped** (an earlier draft proposed it): redundant, since an undeclared cleartext outbound already REFUSES and a refusal naming the hop beats a handshake failure; it hard-fails 14 of 14 inbound MLLP listeners because `_mllp_ssl_context(server=True)` demands a cert before any policy runs, unreachable by posture/attestation/the new flag and not loopback-exempt; and it would silently disarm the four inbound exposed-gates, which early-return when `tls` is truthy. `Tcp()`/`X12()` have no TLS support at all, so for them the declaration is **permanent and structural**, not transitional. Neither `data_class` nor `MEFOR_ALLOW_INSECURE_TLS` is removed — the hop authority stops READING the label, and the variable is unhooked from the authority but survives for the six non-connection cells (store TLS, LDAPS, webhook sink, AI broker) that have no per-connection field to carry an escape. Retains ADR 0148's `[security].enforcement` dial as an arm (its AC-3 requires it on every gate). Scope is the cleartext transport-hop decision ONLY; an explicit out-of-scope table names all four callers of the authority and the three other readers of `HopPosture` | Accepted (2026-07-25) — reworked after adversarial review; owner-ratified; **BUILT 2026-07-28** | | [0154](0154-synchronous-captured-downstream-reply-and-intake-authentication-for-the-inbound-http-listener-adr-0023-deferred-tail.md) | **Synchronous captured-downstream-reply and intake authentication for the inbound HTTP listener** — the ADR 0023 deferred tail: `reply_from` blocks the HTTP turn on a **committed** ADR 0013 `response` row (never an in-flight `DeliveryResponse`), and `intake_auth` (API key / bearer / mTLS subject) adds a peer control behind a posture-keyed gate. Also closes a live hole — `check_http_tls_exposure` returns early on truthy `tls`, so an off-loopback `Http(tls=True)` listener authenticates nobody today | **Accepted (2026-07-31)** — owner-ratified at rev 5; authorises **increment A only** (intake-auth + peer-control gate), which is **built and merged** (2026-08-01, `f2ef0ea9`); sync-reply (increment B) deferred pending a customer | | [0155](0155-dast-dynamic-security-testing-of-the-running-engine.md) | **DAST — dynamic security testing of the running engine** (BACKLOG #318) — no DAST had ever run against this project: every security test was static or in-process, leaving the [Secure_Development_Standards](../Secure_Development_Standards.md) §6.1 *Dynamic* tier row empty. Increment 1 builds a **self-run, authenticated authorization sweep** with **no new dependency**: one `uvicorn` listener on loopback in front of a real Engine + real AuthService, both identities minted over the wire through `POST /auth/login`, and the authorization expectation **derived from the live route table** by a single shared `require*()`-closure walk ([`scripts/security/route_gates.py`](../../scripts/security/route_gates.py), hoisted out of the security doc-drift guard so exactly one derivation of *is this route gated* exists in the tree) rather than a hand-kept list that goes stale the day a route lands. Measured shape: 105 route rows, 100 gated, 87 permission-gated, exactly 5 anonymous. Three passes — **negative** (every gated HTTP row sent with no credential and with an invalid bearer; anything but 401 is a finding), **authorized reach** (how many gated `GET` rows a *privileged* token got past authentication and authorization on — the positive number that stops a wall of 401s reading as *all endpoints protected*), and **viewer BFLA** (anything but a refusal, **including 404**, is a finding, because a 404 on a matched path template means the caller got past authorization into resource lookup). The receipt names what it examined and **fails closed** — below any floor it exits **2 (could not measure)**, never 0 — and records method, path template, status codes, counts and the *relaxed* posture it scanned, never a body, header or token. Two canaries are built from **supported configuration, not source patches** (authentication disabled at the target; the low-privilege identity over-granted while the expectation set stays the viewer's), avoiding the patch-rot failure where a canary silently stops applying; CI runs both **before** the real scan and requires each to exit exactly **1** (findings) *with* its receipt on disk — a 0 (blind), a 2 (could not measure, which is what a neutered canary actually produces) or a crash fails the job and the real scan never runs. The inversion is the design: the sweep and both canaries run as ordinary pytest in the **existing required** test legs, so a change that blinds the detector reds a PR, while the nightly workflow is advisory — **not** a required context (it has no `pull_request` trigger, so it cannot report on a PR) and deliberately **not** `continue-on-error`. **Scope boundary: see the ADR's *Scope boundary* section** — it is stated once there, verbatim, and this row deliberately carries a pointer and no wording of its own. Rejected/deferred: schemathesis (the shipped OpenAPI declares no `securitySchemes` and no per-operation `security`, so `ignored_auth` would pass on every operation having probed none of them; adoption also needs a fifth DEP-1 lock over a ~30-distribution closure that could pull a shipped runtime floor down), ZAP (a SHA-pinned action still pulls a mutable `:stable` image, and the published images do not install the web console), nuclei (template matcher; `pip install nuclei` is an abandoned unrelated package), Dredd/RESTler/CATS, in-process ASGI transport (bypasses the HTTP parser, leaves `request.client` unset), the unauthenticated MLLP/TCP/X12/DICOM ingress plane, the `/ui` console plane, a TLS black-box target, and non-`GET` reach/BFLA | Accepted (2026-07-31) — increment 1 built; advisory, not a required context | -| [0156](0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md) | **ASVS scorecard as data — a derived count, verified evidence anchors, and a fail-closed drift gate** — the ASVS score is maintained as prose, and nothing checks it. One re-anchoring session (2026-08-01) re-derived the headline count **6 times**, found **12 residuals of record factually false at HEAD** (five of them *absence* claims that had silently stopped being true), and found **10 cells missing from an enumeration described as "arithmetic-checked and complete"** — which survived because the arithmetic closed to 345 and closure was read as proof. **Closure only proves the four buckets sum, not that every cell landed in one.** Decision: hold one `[[cell]]` record per requirement (all 345) in `asvs-scorecard.toml`; **compute** the count so no document can state one; assert **every corpus id appears exactly once** (the check whose absence cost ten cells); machine-verify each cell's `evidence` anchor by asserting an expected **token** still resolves, so code movement reds a test instead of rotting a sentence; require an absence claim to record the **search that proved it plus a positive control that must still hit**, because a grep naming the wrong token returns zero and reads exactly like proof; make **`unverified` a first-class verdict** so inherited-versus-verified Pass is countable (~219 Passes have never been read against the requirement text); and **fail closed rather than skip**. Tool + schema + fixture tests live in this repo and run in public CI; the real scorecard lives in the vault with a vault-CI job — which closes **ASVS 15.1.3**, currently open precisely because six `*_doc_drift` modules assert against documents that `git ls-files docs/security/` shows are **not present** in the tree where CI runs. Rejected: *keep prose and review harder* (every false residual **read as true**; the project's own standard says the mitigation must be structural, not diligence), *one document to rule them all* (that is the current lineage, and it produces five documents asserting three counts), and *publish the vault documents so the guards see them* (attacker roadmap, `SECURITY-DOCS-POLICY.md`). Explicitly **does not** make the score correct — only consistent, derived and drift-detecting; adversarial verification remains the only cure for a wrong verdict | **Accepted (2026-08-01)** — built and merged the same day. **§7 was amended at ratification**: it proposed a vault CI job, and the actions API showed every vault workflow (last run 2026-07-27; two vault PRs merged that day with zero checks), so a CI-only design would have shipped dead. Built instead as a vault **pre-commit hook** plus one **narrow new workflow**. That §7 was a confident, unchecked claim about system state is the ADR own thesis applied to itself | +| [0156](0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md) | **ASVS scorecard as data — a derived count, verified evidence anchors, and a fail-closed drift gate** — the ASVS score is maintained as prose, and nothing checks it. One re-anchoring session (2026-08-01) re-derived the headline count **6 times**, found **12 residuals of record factually false at HEAD** (five of them *absence* claims that had silently stopped being true), and found **10 cells missing from an enumeration described as "arithmetic-checked and complete"** — which survived because the arithmetic closed to 345 and closure was read as proof. **Closure only proves the four buckets sum, not that every cell landed in one.** Decision: hold one `[[cell]]` record per requirement (all 345) in `asvs-scorecard.toml`; **compute** the count so no document can state one; assert **every corpus id appears exactly once** (the check whose absence cost ten cells); machine-verify each cell's `evidence` anchor by asserting an expected **token** still resolves, so code movement reds a test instead of rotting a sentence; require an absence claim to record the **search that proved it plus a positive control that must still hit**, because a grep naming the wrong token returns zero and reads exactly like proof; make **`unverified` a first-class verdict** so inherited-versus-verified Pass is countable (~219 Passes have never been read against the requirement text); and **fail closed rather than skip**. Tool + schema + fixture tests live in this repo and run in public CI; the real scorecard lives in the vault with a vault-CI job — which closes **ASVS 15.1.3**, currently open precisely because six `*_doc_drift` modules assert against documents that `git ls-files docs/security/` shows are **not present** in the tree where CI runs. Rejected: *keep prose and review harder* (every false residual **read as true**; the project's own standard says the mitigation must be structural, not diligence), *one document to rule them all* (that is the current lineage, and it produces five documents asserting three counts), and *publish the vault documents so the guards see them* (attacker roadmap, `SECURITY-DOCS-POLICY.md`). Explicitly **does not** make the score correct — only consistent, derived and drift-detecting; adversarial verification remains the only cure for a wrong verdict | **Accepted (2026-08-01)** — built and merged the same day. **§7 was amended at ratification**: it proposed a vault CI job, and the actions API showed every vault workflow `disabled_manually` (last run 2026-07-27; two vault PRs merged that day with zero checks), so a CI-only design would have shipped dead. Built instead as a vault **pre-commit hook** plus one **narrow new workflow**. That §7 was a confident, unchecked claim about system state is the ADR own thesis applied to itself | From a950acaad0b54221b9cf3f5b8c74c9a78cbbe7b3 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 1 Aug 2026 11:13:33 -0500 Subject: [PATCH 5/8] feat(asvs): phase 0+1 -- publish survey progress, and make the rubric a spec Grounded in a deep-research pass over OWASP's own assessment chapter, NIST OSCAL, and the one comparable OSS tool. Three findings changed the design. FINDING 1: "Partial" is not an ASVS verdict. ASVS 5.0 defines exactly three -- verified, exception, and non-applicable-with-rationale. The strings "partially implemented" and "not implemented" appear NOWHERE in it, and the only prominent OSS ASVS tool declines to model partial too. Our three-way rubric was a local extension we invented and never wrote down, which is precisely why 11.7.1, 3.7.3 and 5.4.3 each changed verdict in one day: two assessors, two unwritten rules. -> docs/ASVS-ASSESSMENT-METHOD.md now states the decision procedure as an ORDERED rule list with the seven real disputes as worked examples, so the next assessor reaches the same answer. ASVS's own stability mechanism is a disclosed repeatable method, not a rubric -- this is that disclosure. FINDING 2: recording the reason for non-applicability is the ONE "must" in the whole assessment chapter (everything else is "should"). So an `na` with no rationale is not a lax entry; it is the single thing the standard requires, omitted. load_scorecard() now REFUSES it. FINDING 3: ASVS requirement ids are NOT stable across versions -- bare 1.2.5 is Architecture in 4.0.3 and Encoding in 5.0.0 -- and master is the bleeding-edge branch where a rolling "latest" release republishes identical filenames. Our corpus was fetched from master and turned out byte-identical to the tagged release: luck, not method. Now pinned by [scorecard].asvs_version plus a corpus_sha256 the verifier recomputes and compares. PHASE 0 -- the renderer no longer publishes a headline count. It leads with how much of the survey is done. 76% of that day's verdict changes were cells nobody had ever opened, so a count computed over unexamined cells is an average of guesses, and publishing it is what made every first look read as a REVERSAL rather than as PROGRESS. Reporting every requirement including `unverified` is also what ASVS asks for -- a summary of all requirements checked, not exceptions only. Also adds `needs-review` (examined, contested, parked deliberately) and `reviewed_by`. Separating "undecided" from "decided-and-failing" is a cheaper stability primitive than inventing a grade -- borrowed from the one OSS tool in this space, which pairs it with reviewed_by/reviewed_at. OSCAL was evaluated and rejected for this use: ASVS has no OSCAL catalog (proposed 2020, closed 2021 in a backlog sweep, still none in the 2026 release), and assessment-results mandatorily imports an assessment-plan which mandatorily imports an SSP -- a chain a product with no operated system has no source for. Recorded in the method doc, with Component Definition and CycloneDX declarations noted as the two unexplored alternatives. 24 tests (17 + 7 new), ruff clean, mypy clean. --- docs/ASVS-ASSESSMENT-METHOD.md | 161 +++++++++++++++++++++++++++++++++ scripts/asvs/scorecard.py | 148 +++++++++++++++++++++++++----- tests/test_asvs_scorecard.py | 83 +++++++++++++++-- 3 files changed, 360 insertions(+), 32 deletions(-) create mode 100644 docs/ASVS-ASSESSMENT-METHOD.md diff --git a/docs/ASVS-ASSESSMENT-METHOD.md b/docs/ASVS-ASSESSMENT-METHOD.md new file mode 100644 index 0000000..13dc557 --- /dev/null +++ b/docs/ASVS-ASSESSMENT-METHOD.md @@ -0,0 +1,161 @@ +# ASVS assessment method + +| | | +|---|---| +| **Standard** | OWASP ASVS **5.0.0**, pinned to release tag `v5.0.0_release` | +| **Target level** | **Level 3** (cumulative — L1 + L2 + L3) | +| **Nature** | Point-in-time, **AI-assisted self-assessment by source review.** Not a certification, accreditation, passed audit, or penetration test. OWASP certifies nobody. | +| **Scope** | Declared positively in §2 | +| **Status** | Method of record from 2026-08-01 (ADR 0156) | + +> **Why this document exists.** ASVS 5.0's assessment chapter is *"deliberately not prescriptive"* and +> supplies **no verdict rubric**. What it does supply is a stability mechanism: disclose a repeatable +> method, declare scope positively, report **every requirement checked rather than exceptions only**, +> and give a written rationale for anything non-applicable or unimplemented. +> +> This document is that disclosure. It exists because the absence of a written rubric measurably cost +> us: three cells (11.7.1, 3.7.3, 5.4.3) changed verdict in one day purely because two assessors +> applied different unwritten rules to the same code. + +--- + +## 1. Verdicts + +**ASVS 5.0 defines three verdicts: verified, exception, and non-applicable-with-rationale.** The +strings *"partially implemented"* and *"not implemented"* appear nowhere in it. Everything below +beyond those three is **our local extension**, and is defined here precisely because the standard +does not define it — an undefined grade is one two assessors will apply differently. + +| Verdict | Meaning | ASVS-native? | +|---|---|---| +| `pass` | The requirement's **verb** is satisfied by a **shipped default**, or by a gate that **refuses to start** when the precondition is absent | yes (*verified*) | +| `fail` | **No implementing control exists in any configuration** | yes (*exception*) | +| `na` | The requirement does not apply to this product on the declared scope. **A written rationale is mandatory** | yes — and the rationale is the one **"must"** in ASVS's assessment chapter | +| `partial` | A control exists but ships off, warns instead of refusing, or covers only part of the in-scope surface | **local extension** — see §1.1 | +| `needs-review` | Examined, but the verdict is genuinely contested or blocked on a decision | **local extension** | +| `unverified` | **Never examined.** Inherited from an earlier assessment and never read against the requirement text | **local extension** | + +### 1.1 The decision procedure — apply in order + +Ambiguity is resolved by taking the **first** rule that matches, not by judgement. + +1. **Does the requirement apply to this product on the declared scope (§2)?** + No → **`na`**, and write the rationale. No rationale, no `na`. +2. **Has this cell been read against the ASVS requirement text at a known commit?** + No → **`unverified`**. *This is not a Pass.* An inherited verdict is a guess. +3. **Does code implementing the requirement's verb exist anywhere in the tree, reachable by any + configuration?** + No → **`fail`**. +4. **Is the verb satisfied by a shipped default, or by a gate that refuses to start when the + precondition is absent?** + Yes → **`pass`**. *"It can be configured" is never a pass. A signed relaxation is never a pass.* +5. **Otherwise** → **`partial`**: the control exists, but it ships off, warns rather than refuses, or + covers part of the surface. +6. **If two assessors following 1–5 disagree** → **`needs-review`**, with the disagreement recorded. + Forcing a premature verdict is what produces flip-flop; parking it is cheaper and honest. + +### 1.2 Worked examples — the ones that actually broke + +These are the real disputes. They are here so the next assessor reaches the same answer. + +| Cell | Verdict | Which rule, and why | +|---|---|---| +| **5.4.3** | `fail` | Rule 3. A scan *hook* exists but its only shipped implementation is `_no_scan`, and there is **no configuration key at all** — an operator must **author** the scanner. Supplying a control is not configuring one. | +| **15.2.5** | `partial` | Rule 5, **not** rule 3. `[sandbox].mode` ships `off`, but `subprocess` mode is real and was verified by executing it. A working control that ships off. | +| **11.7.1** | `fail` | Rule 3. The read-out is all-`None` off Linux and the only refusing branch keys on a **TOML declaration**, never on the measured property. Zero percent of the verb is satisfied by shipped code. | +| **3.7.3** | `fail` | Rule 3. One off-site navigation, a bare 303, no interstitial and no cancel. `oidc_enabled=False` removes the **trigger**, not a control. | +| **10.5.5** | `na` | Rule 1. The requirement is conditional — *"**when using** OIDC back-channel logout"* — and the precondition is false and unreachable by configuration. **Building it would create applicability.** | +| **12.2.2** | `na` | Rule 1. No external-facing services on the declared scope. *Also a scoping error worth remembering: this row spent months scoring 12.3.1's verb.* | +| **12.3.1** | `partial` | Rule 5. Raw TCP and X12 cannot speak TLS in any configuration — but a substantial control exists elsewhere (TLS-by-default store hop with a refusing gate), so partial coverage, not rule 3. | + +### 1.3 Standing tie-breakers + +- **Default to the worse verdict** when uncertain. An unearned pass is the failure this method exists + to prevent. +- **A verdict is only as good as its anchor.** No `pass`, `partial` or `fail` without evidence (§3). +- **Never reason from a paraphrase.** Quote the requirement's `req_description` from the pinned + corpus before judging. Scoring a cell against a remembered or restated verb produced three wrong + verdicts, one of which had stood for three assessment cycles against a requirement ASVS **deleted**. + +--- + +## 2. Scope, declared positively + +Per ASVS's guidance, stated as what **is** included rather than what is excluded. + +**Included:** the MessageFoundry engine, the web console, and the IDE extension, assessed as source, +at a named commit, against **all 345 ASVS 5.0.0 requirements** at **Level 3**. + +**The configuration assessed** — one posture, not a matrix: + +> On-premises single hospital · private network, never internet-facing · SQL Server store · operator +> console behind a TLS-terminating reverse proxy on the hospital LAN · `data_class = phi` under +> `[security].enforcement = enforce` · authentication on · RBAC deny-by-default · data plane bound to +> a NIC so partners can send HL7. + +**One posture column only.** A previous two-posture (A/B) split was retired because cells were scored +under one frame and carried into another, leaving at least one cell's bucket **unrecoverable from the +record**. Where a verdict would differ under a documented opt-in, the default is scored and the delta +is recorded in the cell's residual — never as a second column. + +--- + +## 3. Evidence + +ASVS expects per-requirement, artifact-backed evidence, and states that merely running an +off-the-shelf tool is insufficient. It equally endorses **purpose-written tests** — *"testable using +automation ≠ running an off the shelf tool"* — so the anchors below are squarely within its intent. + +**Every non-`unverified` cell carries at least one anchor**, machine-verified on every commit +(ADR 0156): + +- **Presence anchor** — `path` + `line` + an `expect` **token** that must still resolve. A token + rather than a bare line number, so ordinary edits above it do not thrash every anchor. +- **Absence claim** — a `pattern` that must return nothing **plus a `positive_control` that must + still match.** A grep naming the wrong token returns zero and reads exactly like proof; five + residuals of record died that way, and two of my own greps did in a single day. **An absence claim + without a live positive control is void**, regardless of what the pattern returns. + +**`reviewed_by` and `reviewed_at`** are recorded on every verdict, so staleness is visible and a +verdict can be traced to the pass that set it. + +--- + +## 4. Reporting + +- **Every requirement is reported, not exceptions only** — including `unverified`. ASVS asks for a + summary of all requirements checked; a report that shows only failures hides how much was examined. +- **No document states a count.** The count is **computed** from the scorecard, and prose renders it. + Five documents once asserted three different counts. +- **`unverified` is reported separately from `pass`.** A headline that merges them is not a + measurement — it is an average over guesses. Publish *verified / unexamined*, not a single total, + until the baseline sweep completes. + +--- + +## 5. Pinning + +- The corpus is the **official `v5.0.0_release` asset**, recorded with its **SHA-256**, not a fetch + from `master`. Master is the bleeding-edge branch and a rolling *"latest"* release carries identical + filenames — an unpinned fetch silently moves versions. +- The scorecard records `asvs_version`. **ASVS requirement IDs are not stable across versions** — + bare `1.2.5` is *Architecture* in 4.0.3 and *Encoding and Sanitization* in 5.0.0 — and OWASP's + referencing guidance prefers `v-.
.`. A document-level + version field satisfies that pinning. + +--- + +## 6. What this method is not + +- **Not a certification.** OWASP certifies nobody, and its assessment guidance is written for a + third-party certifying organisation producing a certification report. Applying it to an in-repo + self-assessment is a defensible extension, not a mandate — *"insufficient for certification"* is + not the same as *"insufficient for a self-assessment"*. +- **Not externally validated.** No independent ASVS review, penetration test, or DAST has assessed + this scorecard. +- **Not a guarantee of correctness.** This method makes verdicts *consistent, derived and + drift-detecting*. A wrong verdict recorded carefully is still wrong; adversarial verification + remains the only cure for that. +- **Not prior art.** No open-source project was found maintaining per-requirement ASVS verdicts with + git-versioned evidence anchors — the one comparable tool keeps verdicts in a gitignored database. + Every choice here is justified from the ASVS text directly, because there is no convention to adopt. diff --git a/scripts/asvs/scorecard.py b/scripts/asvs/scorecard.py index 2de672e..3f9b8af 100644 --- a/scripts/asvs/scorecard.py +++ b/scripts/asvs/scorecard.py @@ -27,12 +27,27 @@ from pathlib import Path from typing import Any, Final, Literal -Verdict = Literal["pass", "partial", "fail", "na", "unverified"] +Verdict = Literal["pass", "partial", "fail", "na", "needs-review", "unverified"] #: The scoring buckets. ``unverified`` is deliberately first-class (ADR 0156 §5): a cell inherited from #: an earlier assessment and never re-read against the requirement text is NOT a Pass, and conflating #: the two is what let ~219 unchecked verdicts hide inside a headline. -VERDICTS: Final[frozenset[str]] = frozenset({"pass", "partial", "fail", "na", "unverified"}) +VERDICTS: Final[frozenset[str]] = frozenset( + {"pass", "partial", "fail", "na", "needs-review", "unverified"} +) + +#: ASVS 5.0 defines only three verdicts — verified, exception, and non-applicable-with-rationale. The +#: strings "partially implemented" and "not implemented" appear nowhere in it, and the one prominent +#: OSS ASVS tool declines to model partial too. Everything here beyond pass/fail/na is OUR extension, +#: defined in docs/ASVS-ASSESSMENT-METHOD.md §1 — an undefined grade is one two assessors apply +#: differently, which is exactly how 11.7.1, 3.7.3 and 5.4.3 each changed verdict in a single day. +LOCAL_EXTENSION_VERDICTS: Final[frozenset[str]] = frozenset( + {"partial", "needs-review", "unverified"} +) + +#: A verdict that has actually been reached. `unverified` and `needs-review` are NOT among them: +#: the first was never examined, the second was examined and left open on purpose. +DECIDED_VERDICTS: Final[frozenset[str]] = frozenset({"pass", "partial", "fail", "na"}) #: How far from the recorded line the expected token may drift before the anchor is considered broken. #: Anchors name a TOKEN rather than a bare line number precisely so that ordinary edits above a cell's @@ -77,6 +92,7 @@ class Cell: posture: str = "single" last_verified: str = "" verified_at: str = "" + reviewed_by: str = "" evidence: tuple[Anchor, ...] = () absence: tuple[Absence, ...] = () @@ -100,6 +116,19 @@ def ok(self) -> bool: return not self.problems +def corpus_digest(path: Path) -> str: + """SHA-256 of the corpus file, so the pin is checkable rather than asserted. + + The corpus MUST come from the tagged ``v5.0.0_release`` asset. ``master`` is the bleeding-edge + branch and a rolling "latest" release republishes identical filenames, so an unpinned fetch moves + versions silently. Our first corpus was fetched from ``master`` and happened to be byte-identical + to the release — luck, not method, which is why this is now recorded and verified. + """ + import hashlib + + return hashlib.sha256(path.read_bytes()).hexdigest() + + def load_corpus(path: Path) -> dict[str, int]: """Map ``req_id`` (V-stripped) to level, from the OWASP ASVS 5.0.0 flat JSON. @@ -133,6 +162,15 @@ def load_scorecard(path: Path) -> list[Cell]: raise ScorecardError( f"cell {raw.get('id')!r}: verdict {verdict!r} not one of {sorted(VERDICTS)}" ) + # ASVS 5.0's assessment chapter is "should" throughout with exactly ONE "must": a + # non-applicable requirement must be noted with its reason. So an `na` without a rationale is + # not a lax entry — it is the single thing the standard actually requires, omitted. + if verdict == "na" and not str(raw.get("residual", "")).strip(): + raise ScorecardError( + f"cell {raw.get('id')!r}: verdict 'na' requires a written rationale in `residual` — " + "recording the reason for non-applicability is the one MUST in ASVS 5.0's assessment " + "chapter (docs/ASVS-ASSESSMENT-METHOD.md §1)" + ) cells.append( Cell( id=str(raw["id"]), @@ -142,6 +180,7 @@ def load_scorecard(path: Path) -> list[Cell]: posture=str(raw.get("posture", "single")), last_verified=str(raw.get("last_verified", "")), verified_at=str(raw.get("verified_at", "")), + reviewed_by=str(raw.get("reviewed_by", "")), evidence=tuple( Anchor(path=str(e["path"]), line=int(e["line"]), expect=str(e["expect"])) for e in raw.get("evidence", []) @@ -266,9 +305,41 @@ def _sort_key(cell_id: str) -> tuple[int, ...]: return (9_999,) +def check_pinning(scorecard: Path, corpus: Path) -> list[str]: + """The scorecard must declare which ASVS version it scores, and pin the corpus by digest. + + **ASVS requirement IDs are not stable across versions** — bare ``1.2.5`` is *Architecture* in + 4.0.3 and *Encoding and Sanitization* in 5.0.0 — and OWASP's referencing guidance prefers + ``v-.
.``. A document-level version field satisfies that + pinning, so a scorecard keyed on bare ids cannot silently re-point when ASVS updates. + """ + problems: list[str] = [] + data = tomllib.loads(scorecard.read_text(encoding="utf-8")) + meta = data.get("scorecard", {}) + + if not str(meta.get("asvs_version", "")).strip(): + problems.append( + "pinning: [scorecard].asvs_version is missing — bare requirement ids re-point across " + "ASVS versions, so the scorecard must say which version its ids mean" + ) + declared = str(meta.get("corpus_sha256", "")).strip() + actual = corpus_digest(corpus) + if not declared: + problems.append( + f"pinning: [scorecard].corpus_sha256 is missing — the corpus digest is {actual}" + ) + elif declared != actual: + problems.append( + f"pinning: corpus digest mismatch — declared {declared[:16]}…, actual {actual[:16]}…. " + "The ASVS corpus changed underneath the scorecard; re-verify before trusting any verdict" + ) + return problems + + def verify(scorecard: Path, corpus: Path, root: Path) -> Findings: findings = Findings() cells = load_scorecard(scorecard) + findings.problems.extend(check_pinning(scorecard, corpus)) findings.problems.extend(check_completeness(cells, load_corpus(corpus))) check_anchors(cells, root, findings) check_absences(cells, root, findings) @@ -276,41 +347,70 @@ def verify(scorecard: Path, corpus: Path, root: Path) -> Findings: def render_current(cells: list[Cell], *, anchor_sha: str) -> str: - """The generated entry point. A new session reads this instead of reconstructing state.""" + """The generated entry point — survey progress FIRST, verdict counts second. + + **Phase 0 of the fix (ADR 0156 / ASSESSMENT-METHOD §4).** A headline count computed over cells + that were never examined is not a measurement; it is an average over guesses, and publishing it is + what made every subsequent first-look read as a *reversal* rather than as *progress*. On + 2026-08-01, 76% of that day's verdict changes were cells nobody had ever opened. + + So this leads with how much of the survey is done, and reports `unverified` separately from + `pass` — which is also what ASVS asks for: a summary of **every requirement checked**, not + exceptions only. + """ n = count(cells) total = sum(n.values()) - verified_pass = sum(1 for c in cells if c.verdict == "pass" and not c.is_inherited) - inherited = sum(1 for c in cells if c.verdict == "pass" and c.is_inherited) + decided = [c for c in cells if c.verdict in DECIDED_VERDICTS] + examined = [c for c in decided if c.last_verified] + unexamined = total - len(examined) + pct = (100.0 * len(examined) / total) if total else 0.0 + inherited = sum(1 for c in cells if c.verdict in DECIDED_VERDICTS and not c.last_verified) + lines = [ "", "", "# ASVS 5.0 L3 — current state", "", - f"**Anchor commit:** `{anchor_sha}`", + f"**Anchor commit:** `{anchor_sha}` · **Method:** `docs/ASVS-ASSESSMENT-METHOD.md`", "", - "| Verdict | Count |", - "|---|---:|", - f"| Pass | {n['pass']} |", - f"| Partial | {n['partial']} |", - f"| Fail | {n['fail']} |", - f"| N/A | {n['na']} |", - f"| **Unverified** | **{n['unverified']}** |", - f"| **Total** | **{total}** |", + "## Survey progress", "", - f"Of the {n['pass']} Passes, **{verified_pass} were verified against the requirement text** and " - f"**{inherited} are inherited** — carried from an earlier assessment and never re-read. An " - "inherited Pass is not evidence of anything; it is the largest standing exposure in this score.", + f"**{len(examined)} of {total} requirements have been read against the ASVS text " + f"({pct:.1f}%).** {unexamined} have not.", "", + "> **There is deliberately no headline score here.** A count over unexamined cells is an", + "> average of guesses. Until the baseline sweep completes, the honest number is the one above:", + "> how much of the survey is done. Verdict counts below cover **examined cells only** unless", + "> stated otherwise.", + "", + "| State | Count | Meaning |", + "|---|---:|---|", + f"| Pass | {n['pass']} | verb satisfied by a shipped default or a refusing gate |", + f"| Partial | {n['partial']} | control exists but ships off, warns, or covers part of the surface |", + f"| Fail | {n['fail']} | no implementing control in any configuration |", + f"| N/A | {n['na']} | does not apply on the declared scope, with a written rationale |", + f"| Needs review | {n['needs-review']} | examined; verdict contested or blocked on a decision |", + f"| **Unverified** | **{n['unverified']}** | **never examined — not a Pass** |", + f"| **Total** | **{total}** | |", + "", + ] + if inherited: + lines += [ + f"⚠️ **{inherited} cell(s) carry a decided verdict with no `last_verified` date** — " + "inherited from an earlier assessment and never re-read. Treat as unexamined.", + "", + ] + lines += [ "## Open cells", "", - "| Cell | L | Verdict | Residual |", - "|---|---|---|---|", + "| Cell | L | Verdict | Reviewed | Residual |", + "|---|---|---|---|---|", ] - for c in sorted( - (c for c in cells if c.verdict in {"partial", "fail"}), key=lambda c: _sort_key(c.id) - ): - lines.append(f"| {c.id} | L{c.level} | **{c.verdict}** | {c.residual[:160]} |") - return "\n".join(lines) + "\n" + open_states = {"partial", "fail", "needs-review"} + for c in sorted((c for c in cells if c.verdict in open_states), key=lambda c: _sort_key(c.id)): + seen = c.last_verified or "—" + lines.append(f"| {c.id} | L{c.level} | **{c.verdict}** | {seen} | {c.residual[:150]} |") + return chr(10).join(lines) + chr(10) def main(argv: list[str] | None = None) -> int: diff --git a/tests/test_asvs_scorecard.py b/tests/test_asvs_scorecard.py index 54f8e38..03a0f39 100644 --- a/tests/test_asvs_scorecard.py +++ b/tests/test_asvs_scorecard.py @@ -26,6 +26,8 @@ check_absences, check_anchors, check_completeness, + check_pinning, + corpus_digest, count, load_corpus, load_scorecard, @@ -225,9 +227,14 @@ def test_unknown_verdict_is_refused(tmp_path: Path) -> None: def test_verify_end_to_end_clean(tmp_path: Path) -> None: (tmp_path / "messagefoundry").mkdir() (tmp_path / "messagefoundry" / "m.py").write_text("tls_cert_file = None\n", encoding="utf-8") + corpus = _corpus_file(tmp_path) sc = _scorecard_file( tmp_path, - """ + f""" +[scorecard] +asvs_version = "5.0.0" +corpus_sha256 = "{corpus_digest(corpus)}" + [[cell]] id = "1.1.1" level = 1 @@ -250,18 +257,78 @@ def test_verify_end_to_end_clean(tmp_path: Path) -> None: residual = "ships off" """, ) - findings = verify(sc, _corpus_file(tmp_path), tmp_path) + findings = verify(sc, corpus, tmp_path) assert findings.ok, findings.problems -def test_render_separates_verified_pass_from_inherited_pass() -> None: +def test_render_leads_with_survey_progress_not_a_headline_score() -> None: + """Phase 0: a count over unexamined cells is an average of guesses, so it is not the headline.""" cells = [ Cell(id="1.1.1", level=1, verdict="pass", last_verified="2026-08-01"), - Cell(id="1.1.2", level=2, verdict="pass"), # inherited: never re-read - Cell(id="2.1.1", level=3, verdict="partial", residual="ships off"), + Cell(id="1.1.2", level=2, verdict="unverified"), + Cell( + id="2.1.1", level=3, verdict="partial", residual="ships off", last_verified="2026-08-01" + ), ] out = render_current(cells, anchor_sha="deadbeef") - assert "**1 were verified against the requirement text**" in out - assert "**1 are inherited**" in out + assert ( + "**2 of 3 requirements have been read against the ASVS text (66.7%).** 1 have not." in out + ) + assert "There is deliberately no headline score here" in out + assert "never examined — not a Pass" in out assert "deadbeef" in out - assert "| 2.1.1 | L3 | **partial** |" in out + + +def test_render_flags_a_decided_verdict_carrying_no_verified_date_as_inherited() -> None: + cells = [Cell(id="1.1.1", level=1, verdict="pass")] # decided, but never dated + out = render_current(cells, anchor_sha="x") + assert "carry a decided verdict with no `last_verified` date" in out + + +# --- the one MUST in ASVS's assessment chapter ---------------------------------------------------- + + +def test_na_without_a_rationale_is_refused(tmp_path: Path) -> None: + """Recording the reason for non-applicability is the only 'must' ASVS 5.0 states.""" + sc = _scorecard_file(tmp_path, '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "na"\n') + with pytest.raises(ScorecardError, match="requires a written rationale"): + load_scorecard(sc) + + +def test_na_with_a_rationale_is_accepted(tmp_path: Path) -> None: + sc = _scorecard_file( + tmp_path, + '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "na"\nresidual = "no WebRTC surface"\n', + ) + assert load_scorecard(sc)[0].verdict == "na" + + +def test_needs_review_is_a_valid_verdict(tmp_path: Path) -> None: + """Parking a contested cell beats forcing a premature verdict — that is what flip-flops.""" + sc = _scorecard_file(tmp_path, '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "needs-review"\n') + assert load_scorecard(sc)[0].verdict == "needs-review" + + +# --- pinning: ids re-point across ASVS versions ---------------------------------------------------- + + +def test_pinning_requires_a_declared_asvs_version(tmp_path: Path) -> None: + corpus = _corpus_file(tmp_path) + sc = _scorecard_file(tmp_path, f'[scorecard]\ncorpus_sha256 = "{corpus_digest(corpus)}"\n') + assert any("asvs_version is missing" in p for p in check_pinning(sc, corpus)) + + +def test_pinning_catches_a_corpus_that_changed_underneath_the_scorecard(tmp_path: Path) -> None: + """bare 1.2.5 is Architecture in 4.0.3 and Encoding in 5.0.0 — a moved corpus re-points ids.""" + corpus = _corpus_file(tmp_path) + sc = _scorecard_file(tmp_path, '[scorecard]\nasvs_version = "5.0.0"\ncorpus_sha256 = "0" \n') + assert any("corpus digest mismatch" in p for p in check_pinning(sc, corpus)) + + +def test_pinning_passes_when_version_and_digest_are_declared(tmp_path: Path) -> None: + corpus = _corpus_file(tmp_path) + sc = _scorecard_file( + tmp_path, + f'[scorecard]\nasvs_version = "5.0.0"\ncorpus_sha256 = "{corpus_digest(corpus)}"\n', + ) + assert check_pinning(sc, corpus) == [] From 610c5fb084ab1938a0a551205754ac2c176fd7ee Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 1 Aug 2026 11:18:52 -0500 Subject: [PATCH 6/8] fix(asvs): the drift gate could never pass -- the anchor was CI-injected The workflow rendered with --anchor-sha ${{ github.sha }} while the committed entry point carried the code anchor, so the rendered file differed on the anchor line on EVERY run. `git diff --exit-code` therefore failed unconditionally. A gate that cannot go green is as useless as one that cannot go red, and this session has spent all day on the second kind. I shipped the first kind, in the tool built to prevent exactly this, and it went red on its own first PR. The anchor is the commit the EVIDENCE was read on -- a property of the assessment, not of the CI run -- so it belongs in [scorecard].anchor_commit and is rendered from there. The render is now deterministic: running it twice produces identical bytes, and the drift check becomes meaningful, catching a scorecard edit that was not re-rendered. --- scripts/asvs/scorecard.py | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/scripts/asvs/scorecard.py b/scripts/asvs/scorecard.py index 3f9b8af..35ad1c2 100644 --- a/scripts/asvs/scorecard.py +++ b/scripts/asvs/scorecard.py @@ -305,6 +305,12 @@ def _sort_key(cell_id: str) -> tuple[int, ...]: return (9_999,) +def load_meta(path: Path) -> dict[str, str]: + """The ``[scorecard]`` table: version pin, corpus digest, and the assessment anchor commit.""" + data = tomllib.loads(path.read_text(encoding="utf-8")) + return {k: str(v) for k, v in data.get("scorecard", {}).items()} + + def check_pinning(scorecard: Path, corpus: Path) -> list[str]: """The scorecard must declare which ASVS version it scores, and pin the corpus by digest. @@ -421,7 +427,10 @@ def main(argv: list[str] | None = None) -> int: "--root", type=Path, default=Path.cwd(), help="tree the evidence anchors point into" ) ap.add_argument("--render", type=Path, help="write the generated CURRENT.md here") - ap.add_argument("--anchor-sha", default="unknown") + # NO --anchor-sha injected by CI. The anchor is the commit the EVIDENCE was read on — a property + # of the assessment, recorded in [scorecard].anchor_commit. Passing ${{ github.sha }} made the + # rendered file differ on every run, so the drift check could never pass: a gate that cannot go + # green is as useless as one that cannot go red, and this one shipped that way. args = ap.parse_args(argv) try: @@ -442,7 +451,8 @@ def main(argv: list[str] | None = None) -> int: print(f" FAIL {p}", file=sys.stderr) if args.render and findings.ok: - args.render.write_text(render_current(cells, anchor_sha=args.anchor_sha), encoding="utf-8") + anchor = load_meta(args.scorecard).get("anchor_commit", "unrecorded") + args.render.write_text(render_current(cells, anchor_sha=anchor), encoding="utf-8") print(f"rendered {args.render}") return 0 if findings.ok else 1 From 046e2525314c0a8ccf6223de464a9baf16e634df Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 1 Aug 2026 11:30:09 -0500 Subject: [PATCH 7/8] fix(asvs): declare the corpus-pin hashlib use in the crypto inventory The ASVS 11.1.3 discovery gate caught scripts/asvs/scorecard.py using hashlib without an inventory entry. The gate is right: it is a discovery gate for exactly this, and it fired on the first commit that introduced new crypto usage. Declared in both places the convention requires -- the scanner INVENTORY and ASVS-L2-PHASE0-CHANGES.md section 4 -- with what it is and, more importantly, what it is NOT: SHA-256 over the ASVS corpus FILE, integrity of a build input, no secret, no key, no message authentication, nothing user- or PHI-derived. Recorded why it exists rather than just that it does: the corpus was originally fetched from master, where a rolling "latest" release republishes identical filenames, and matched the tagged v5.0.0_release asset only by luck. Since ASVS requirement ids are not stable across versions -- bare 1.2.5 is Architecture in 4.0.3 and Encoding in 5.0.0 -- a corpus that moves silently re-points every id in the scorecard, so the digest is now checked rather than assumed. --- docs/ASVS-L2-PHASE0-CHANGES.md | 1 + scripts/security/crypto_inventory_check.py | 8 ++++++++ 2 files changed, 9 insertions(+) diff --git a/docs/ASVS-L2-PHASE0-CHANGES.md b/docs/ASVS-L2-PHASE0-CHANGES.md index d770cad..474cfaa 100644 --- a/docs/ASVS-L2-PHASE0-CHANGES.md +++ b/docs/ASVS-L2-PHASE0-CHANGES.md @@ -102,6 +102,7 @@ Update it whenever a crypto dependency, algorithm, or key source changes. | Audit chain | Row-hash chain (tamper-evident): **keyless SHA-256** in the default keyless posture, upgraded to **HMAC-SHA256** — keyed on an HKDF-SHA256-derived (`mefor/audit-chain/v1`) subkey of the store DEK — only when a store key is configured (#190), or on an isolated-module Transit MAC under `cipher_provider=vault_transit` (ADR 0138); both modes hash identical canonical bytes, so keyless deployments and legacy rows still verify. The digest primitive is `audit_row_hash` in `store/store.py` (`hashlib` + `hmac`), shared verbatim by all three backends. **Usage scope:** the HKDF-derived subkey (and, under `vault_transit`, the named Transit audit key) authenticates the **audit row chain only** — an integrity/tamper-evidence key, never a confidentiality key. It is domain-separated from the store DEK it is derived from (`mefor/audit-chain/v1`), decrypts nothing, protects no message body, and is not transport, signing, or token material. **Verification is constant-time and full-walk (ASVS 11.2.4):** `hmac.compare_digest` over `audit_mac_bytes` on every row MAC *and* on the external-anchor head, in `store/store.py`, `store/postgres.py` and `store/sqlserver.py` — the walk never returns early, so verify duration is a function of chain length, not of where a forgery sits (the first divergent row id is still named in the operator-facing result) | `audit_log.row_hash` | Append-only; verified by `messagefoundry audit-verify` | | Config fingerprint ([ADR 0041](adr/0041-load-path-attestation-and-change-attribution.md)) | SHA-256 content digest of a loaded config bundle — path-relative Merkle fold over every loaded file (`*.py` incl `_*.py`, `connections.toml`, `codesets/*`, `environments/*.toml`); `hashlib` in `config/fingerprint.py` | Recorded in the `config_reload` audit detail (not stored as a secret) | Recomputed per reload/startup; binds reviewed-commit → loaded-bytes (integrity/attribution, not confidentiality) | | Engine wheel attestation ([ADR 0041](adr/0041-load-path-attestation-and-change-attribution.md) D3) | SHA-256 over each **loaded** first-party `messagefoundry` module file, compared to the installed wheel's `*.dist-info/RECORD` baseline (a base64 `sha256=` manifest already in the wheel); `hashlib` in `integrity.py` | Drift recorded in the hash-chained `startup_integrity` audit row (not a secret); RECORD baseline read from site-packages metadata | Recomputed at startup + on demand; in-place-tamper tripwire (integrity, not confidentiality). Alert-only by default; `[integrity].fail_closed_on_drift` refuses to start on drift; no-op on an editable install | +| ASVS corpus pin ([ADR 0156](adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md)) | SHA-256 over the **OWASP ASVS 5.0.0 corpus file**, recorded in `[scorecard].corpus_sha256` and recomputed on every verifier run; `hashlib` in `scripts/asvs/scorecard.py`. **Integrity of a build input, not a security control** — no secret, no key, no message authentication, and nothing user- or PHI-derived is hashed. It exists because the corpus was originally fetched from `master` (the bleeding-edge branch, where a rolling "latest" release republishes identical filenames) and matched the tagged `v5.0.0_release` asset only by luck; the digest is now recorded and checked rather than assumed, because ASVS requirement ids are **not stable across versions** (bare `1.2.5` is *Architecture* in 4.0.3 and *Encoding and Sanitization* in 5.0.0), so a corpus that moves silently re-points every id in the scorecard | Not a secret: the digest is committed alongside the corpus it pins | Recomputed on every scorecard verification; a mismatch fails the gate and forces re-verification before any verdict is trusted | | Outbound message signing (opt-in) | Detached JWS (RFC 7515) — RS256/PS256 (RSA) or ES256 (ECDSA P-256), SHA-256; `cryptography` in `transports/signing.py` (ASVS 4.1.5, [ADR 0018](adr/0018-per-message-signatures-accepted-risk.md)) | Operator-supplied PEM **private** signing key per connection (inline via `env()` or a PEM file path; encrypted-key passphrase via `env()`); the **public** key is shared with the partner out-of-band. **Usage scope:** this private key **only** signs this connection's outbound per-message JWS — a message-**authenticity/integrity** key in transit; it is never used for at-rest encryption or session/token material, and the partner holds only the matching **public** verification half | **OFF by default**; per-connection opt-in. `kid` carried in the JWS header so key rotation / a managed provider ([ADR 0019](adr/0019-pluggable-keyprovider-hsm-kms-vault.md)) slots in without a wire change | | DIRECT S/MIME (opt-in, [ADR 0085](adr/0085-direct-hisp-smime-connector.md)) | CMS **sign-then-encrypt** in `transports/direct.py` (core `cryptography` `serialization.pkcs7`): PKCS#7 signature over the body with a **SHA-256** digest, the public-key signature algorithm (RSA / ECDSA) following the loaded signing key type (not pinned to RSA), then a PKCS#7 **envelope** to the partner's recipient cert. The envelope content-encryption cipher is the **`cryptography` pkcs7 library default** — no algorithm is pinned in code | Sender **signing cert** + PEM **private key** (optional `signing_key_password`) and the per-partner **`recipient_cert`**, all operator-supplied files; the recipient cert is trust-verified at construction against an operator `trust_anchor` (one-level direct-issuance check); key/cert mismatch refused. **Usage scope:** the sender signing key signs the CMS body and the partner's `recipient_cert` encrypts the CMS envelope — this material protects the **confidentiality + authenticity of a DIRECT message to one partner in transit**; it is not an at-rest store key and encrypts nothing in the store | **OFF by default** — only when a DIRECT Connection is configured, and its HISP relay host is gated by the **opt-in** `[egress].allowed_direct` allow-list (empty by default = unrestricted; an unlisted host is refused only once the list is populated, or outright when `[security].block_unlisted_outbound` is set). Signing key + recipient certs rotate on the schedule below | | OIDC IdP JWKS verification keys (opt-in, [ADR 0142](adr/0142-federated-sso-oidc-authorization-code-pkce-relying-party-hybrid-ad-backed.md)) | **Public** verifying keys fetched from the IdP JWKS: **RS256/PS256** (RSA, ≥ 2048-bit floor) and **ES256/ES384** (EC P-256/P-384) — rebuilt from each JWK by `cryptography` in [`auth/oidc/jwks.py`](../messagefoundry/auth/oidc/jwks.py); the closed `SignatureAlgorithm` enum forecloses `alg:none` and RS256→HS256 confusion. Bounded, TTL-cached (`DEFAULT_JWKS_TTL_SECONDS`), a 512 KiB body cap, a global min-refetch floor (fetch-amplification bound), and a hard refusal of a duplicate `kid`; a key below the floor is skipped/refused, never merely warned. **Usage scope:** these are **public**, non-secret keys used **only** to verify the IdP's id-token signature at console login — they encrypt nothing and can protect no data; the engine holds no private half. | Fetched from the IdP JWKS URI over the CA-pinned no-redirect opener (row below); held process-local in `JwksCache`, never persisted, never logged | Refetched per TTL / on an unknown `kid` within the amplification bound; rolls when the IdP rotates its signing keys | diff --git a/scripts/security/crypto_inventory_check.py b/scripts/security/crypto_inventory_check.py index 70968d6..590626e 100644 --- a/scripts/security/crypto_inventory_check.py +++ b/scripts/security/crypto_inventory_check.py @@ -321,6 +321,14 @@ # the two ephemeral scan identities it provisions into a store it creates empty in a temp directory # and destroys with it. Not a key and never persisted: a checked-in constant would be strictly # weaker, and the alternative — an operator-supplied credential — is the optional escape hatch only. + # ADR 0156: SHA-256 over the OWASP ASVS corpus FILE, to pin it to the tagged v5.0.0_release + # asset. Integrity of a build input, not a security control: no secret, no key, no message + # authentication, and nothing user- or PHI-derived is hashed. It exists because the corpus was + # originally fetched from `master` (the bleeding-edge branch, where a rolling "latest" release + # republishes identical filenames) and happened to match the release by luck — so the digest is + # recorded and recomputed rather than assumed. Non-cryptographic alternatives were rejected only + # because SHA-256 is already the tree's convention for file pinning. + "scripts/asvs/scorecard.py": frozenset({"hashlib"}), "scripts/security/dast_target.py": frozenset({"secrets"}), } From bf6eff17bfc5d383773ab6a9773411d851875c7b Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 1 Aug 2026 15:01:44 -0500 Subject: [PATCH 8/8] fix(asvs): register the corpus pin in the remaining two crypto registries The crypto inventory is stated in FOUR places, not the two I updated: scripts/security/crypto_inventory_check.py INVENTORY (done) docs/ASVS-L2-PHASE0-CHANGES.md section 4 (done) tests/test_key_usage_scope_inventory.py key/not-key split (missing) tests/test_security_static.py out-of-package map (missing) Both now carry it, classified as NOT key material with the reason: a keyless content hash over a build input, not a key, a secret, or a message authenticator. Worth recording rather than quietly fixing. This is the same defect class the whole ASVS effort has been about -- one fact stated in four places, so updating two leaves the other two stale -- and the project's own doc standard says state a load-bearing fact ONCE and link to it. Here the redundancy is load-bearing on purpose (each registry asks a different question: what uses crypto, why, does it hold a key, and is it inside the package), so the fix is not to collapse them. But nothing tells an author that adding one row obliges four edits, and I found out by reding CI twice. 102 tests pass across all five crypto/scorecard suites. --- tests/test_key_usage_scope_inventory.py | 2 ++ tests/test_security_static.py | 2 ++ 2 files changed, 4 insertions(+) diff --git a/tests/test_key_usage_scope_inventory.py b/tests/test_key_usage_scope_inventory.py index c806f72..5b4a5a1 100644 --- a/tests/test_key_usage_scope_inventory.py +++ b/tests/test_key_usage_scope_inventory.py @@ -63,6 +63,8 @@ "WebAuthn credentials": "COSE PUBLIC keys supplied by the authenticator; the engine holds no " "private half and the row already says they are verification material, not a secret", "Config fingerprint": "a keyless content hash for change attribution", + "ASVS corpus pin": "a keyless content hash over a build input (the OWASP ASVS corpus file), " + "not a key, a secret, or a message authenticator", "Engine wheel attestation": "a keyless digest over the installed distribution, verified against " "a recorded value; no key is involved on either side", "AD transport": "a TLS hop whose key material is the OS/directory trust store, not engine-held", diff --git a/tests/test_security_static.py b/tests/test_security_static.py index d1f1df2..60a5e3c 100644 --- a/tests/test_security_static.py +++ b/tests/test_security_static.py @@ -1011,6 +1011,8 @@ def test_xml_import_scanner_sees_indented_imports() -> None: #: BACKLOG #282 widens the gate itself to these roots this pin becomes redundant and should be deleted #: in favour of the gate's inventory. _CRYPTO_SITES_OUTSIDE_THE_PACKAGE = { + # ADR 0156: SHA-256 over the ASVS corpus FILE to pin it to the tagged release. No key. + "scripts/asvs/scorecard.py": frozenset({"hashlib"}), "messagefoundry_webconsole/_security.py": frozenset({"secrets"}), "tee/__main__.py": frozenset({"ssl"}), "tee/anon/keying.py": frozenset({"hashlib"}),