diff --git a/docs/ASVS-ASSESSMENT-METHOD.md b/docs/ASVS-ASSESSMENT-METHOD.md new file mode 100644 index 0000000..13dc557 --- /dev/null +++ b/docs/ASVS-ASSESSMENT-METHOD.md @@ -0,0 +1,161 @@ +# ASVS assessment method + +| | | +|---|---| +| **Standard** | OWASP ASVS **5.0.0**, pinned to release tag `v5.0.0_release` | +| **Target level** | **Level 3** (cumulative — L1 + L2 + L3) | +| **Nature** | Point-in-time, **AI-assisted self-assessment by source review.** Not a certification, accreditation, passed audit, or penetration test. OWASP certifies nobody. | +| **Scope** | Declared positively in §2 | +| **Status** | Method of record from 2026-08-01 (ADR 0156) | + +> **Why this document exists.** ASVS 5.0's assessment chapter is *"deliberately not prescriptive"* and +> supplies **no verdict rubric**. What it does supply is a stability mechanism: disclose a repeatable +> method, declare scope positively, report **every requirement checked rather than exceptions only**, +> and give a written rationale for anything non-applicable or unimplemented. +> +> This document is that disclosure. It exists because the absence of a written rubric measurably cost +> us: three cells (11.7.1, 3.7.3, 5.4.3) changed verdict in one day purely because two assessors +> applied different unwritten rules to the same code. + +--- + +## 1. Verdicts + +**ASVS 5.0 defines three verdicts: verified, exception, and non-applicable-with-rationale.** The +strings *"partially implemented"* and *"not implemented"* appear nowhere in it. Everything below +beyond those three is **our local extension**, and is defined here precisely because the standard +does not define it — an undefined grade is one two assessors will apply differently. + +| Verdict | Meaning | ASVS-native? | +|---|---|---| +| `pass` | The requirement's **verb** is satisfied by a **shipped default**, or by a gate that **refuses to start** when the precondition is absent | yes (*verified*) | +| `fail` | **No implementing control exists in any configuration** | yes (*exception*) | +| `na` | The requirement does not apply to this product on the declared scope. **A written rationale is mandatory** | yes — and the rationale is the one **"must"** in ASVS's assessment chapter | +| `partial` | A control exists but ships off, warns instead of refusing, or covers only part of the in-scope surface | **local extension** — see §1.1 | +| `needs-review` | Examined, but the verdict is genuinely contested or blocked on a decision | **local extension** | +| `unverified` | **Never examined.** Inherited from an earlier assessment and never read against the requirement text | **local extension** | + +### 1.1 The decision procedure — apply in order + +Ambiguity is resolved by taking the **first** rule that matches, not by judgement. + +1. **Does the requirement apply to this product on the declared scope (§2)?** + No → **`na`**, and write the rationale. No rationale, no `na`. +2. **Has this cell been read against the ASVS requirement text at a known commit?** + No → **`unverified`**. *This is not a Pass.* An inherited verdict is a guess. +3. **Does code implementing the requirement's verb exist anywhere in the tree, reachable by any + configuration?** + No → **`fail`**. +4. **Is the verb satisfied by a shipped default, or by a gate that refuses to start when the + precondition is absent?** + Yes → **`pass`**. *"It can be configured" is never a pass. A signed relaxation is never a pass.* +5. **Otherwise** → **`partial`**: the control exists, but it ships off, warns rather than refuses, or + covers part of the surface. +6. **If two assessors following 1–5 disagree** → **`needs-review`**, with the disagreement recorded. + Forcing a premature verdict is what produces flip-flop; parking it is cheaper and honest. + +### 1.2 Worked examples — the ones that actually broke + +These are the real disputes. They are here so the next assessor reaches the same answer. + +| Cell | Verdict | Which rule, and why | +|---|---|---| +| **5.4.3** | `fail` | Rule 3. A scan *hook* exists but its only shipped implementation is `_no_scan`, and there is **no configuration key at all** — an operator must **author** the scanner. Supplying a control is not configuring one. | +| **15.2.5** | `partial` | Rule 5, **not** rule 3. `[sandbox].mode` ships `off`, but `subprocess` mode is real and was verified by executing it. A working control that ships off. | +| **11.7.1** | `fail` | Rule 3. The read-out is all-`None` off Linux and the only refusing branch keys on a **TOML declaration**, never on the measured property. Zero percent of the verb is satisfied by shipped code. | +| **3.7.3** | `fail` | Rule 3. One off-site navigation, a bare 303, no interstitial and no cancel. `oidc_enabled=False` removes the **trigger**, not a control. | +| **10.5.5** | `na` | Rule 1. The requirement is conditional — *"**when using** OIDC back-channel logout"* — and the precondition is false and unreachable by configuration. **Building it would create applicability.** | +| **12.2.2** | `na` | Rule 1. No external-facing services on the declared scope. *Also a scoping error worth remembering: this row spent months scoring 12.3.1's verb.* | +| **12.3.1** | `partial` | Rule 5. Raw TCP and X12 cannot speak TLS in any configuration — but a substantial control exists elsewhere (TLS-by-default store hop with a refusing gate), so partial coverage, not rule 3. | + +### 1.3 Standing tie-breakers + +- **Default to the worse verdict** when uncertain. An unearned pass is the failure this method exists + to prevent. +- **A verdict is only as good as its anchor.** No `pass`, `partial` or `fail` without evidence (§3). +- **Never reason from a paraphrase.** Quote the requirement's `req_description` from the pinned + corpus before judging. Scoring a cell against a remembered or restated verb produced three wrong + verdicts, one of which had stood for three assessment cycles against a requirement ASVS **deleted**. + +--- + +## 2. Scope, declared positively + +Per ASVS's guidance, stated as what **is** included rather than what is excluded. + +**Included:** the MessageFoundry engine, the web console, and the IDE extension, assessed as source, +at a named commit, against **all 345 ASVS 5.0.0 requirements** at **Level 3**. + +**The configuration assessed** — one posture, not a matrix: + +> On-premises single hospital · private network, never internet-facing · SQL Server store · operator +> console behind a TLS-terminating reverse proxy on the hospital LAN · `data_class = phi` under +> `[security].enforcement = enforce` · authentication on · RBAC deny-by-default · data plane bound to +> a NIC so partners can send HL7. + +**One posture column only.** A previous two-posture (A/B) split was retired because cells were scored +under one frame and carried into another, leaving at least one cell's bucket **unrecoverable from the +record**. Where a verdict would differ under a documented opt-in, the default is scored and the delta +is recorded in the cell's residual — never as a second column. + +--- + +## 3. Evidence + +ASVS expects per-requirement, artifact-backed evidence, and states that merely running an +off-the-shelf tool is insufficient. It equally endorses **purpose-written tests** — *"testable using +automation ≠ running an off the shelf tool"* — so the anchors below are squarely within its intent. + +**Every non-`unverified` cell carries at least one anchor**, machine-verified on every commit +(ADR 0156): + +- **Presence anchor** — `path` + `line` + an `expect` **token** that must still resolve. A token + rather than a bare line number, so ordinary edits above it do not thrash every anchor. +- **Absence claim** — a `pattern` that must return nothing **plus a `positive_control` that must + still match.** A grep naming the wrong token returns zero and reads exactly like proof; five + residuals of record died that way, and two of my own greps did in a single day. **An absence claim + without a live positive control is void**, regardless of what the pattern returns. + +**`reviewed_by` and `reviewed_at`** are recorded on every verdict, so staleness is visible and a +verdict can be traced to the pass that set it. + +--- + +## 4. Reporting + +- **Every requirement is reported, not exceptions only** — including `unverified`. ASVS asks for a + summary of all requirements checked; a report that shows only failures hides how much was examined. +- **No document states a count.** The count is **computed** from the scorecard, and prose renders it. + Five documents once asserted three different counts. +- **`unverified` is reported separately from `pass`.** A headline that merges them is not a + measurement — it is an average over guesses. Publish *verified / unexamined*, not a single total, + until the baseline sweep completes. + +--- + +## 5. Pinning + +- The corpus is the **official `v5.0.0_release` asset**, recorded with its **SHA-256**, not a fetch + from `master`. Master is the bleeding-edge branch and a rolling *"latest"* release carries identical + filenames — an unpinned fetch silently moves versions. +- The scorecard records `asvs_version`. **ASVS requirement IDs are not stable across versions** — + bare `1.2.5` is *Architecture* in 4.0.3 and *Encoding and Sanitization* in 5.0.0 — and OWASP's + referencing guidance prefers `v-.
.`. A document-level + version field satisfies that pinning. + +--- + +## 6. What this method is not + +- **Not a certification.** OWASP certifies nobody, and its assessment guidance is written for a + third-party certifying organisation producing a certification report. Applying it to an in-repo + self-assessment is a defensible extension, not a mandate — *"insufficient for certification"* is + not the same as *"insufficient for a self-assessment"*. +- **Not externally validated.** No independent ASVS review, penetration test, or DAST has assessed + this scorecard. +- **Not a guarantee of correctness.** This method makes verdicts *consistent, derived and + drift-detecting*. A wrong verdict recorded carefully is still wrong; adversarial verification + remains the only cure for that. +- **Not prior art.** No open-source project was found maintaining per-requirement ASVS verdicts with + git-versioned evidence anchors — the one comparable tool keeps verdicts in a gitignored database. + Every choice here is justified from the ASVS text directly, because there is no convention to adopt. diff --git a/docs/ASVS-L2-PHASE0-CHANGES.md b/docs/ASVS-L2-PHASE0-CHANGES.md index d770cad..474cfaa 100644 --- a/docs/ASVS-L2-PHASE0-CHANGES.md +++ b/docs/ASVS-L2-PHASE0-CHANGES.md @@ -102,6 +102,7 @@ Update it whenever a crypto dependency, algorithm, or key source changes. | Audit chain | Row-hash chain (tamper-evident): **keyless SHA-256** in the default keyless posture, upgraded to **HMAC-SHA256** — keyed on an HKDF-SHA256-derived (`mefor/audit-chain/v1`) subkey of the store DEK — only when a store key is configured (#190), or on an isolated-module Transit MAC under `cipher_provider=vault_transit` (ADR 0138); both modes hash identical canonical bytes, so keyless deployments and legacy rows still verify. The digest primitive is `audit_row_hash` in `store/store.py` (`hashlib` + `hmac`), shared verbatim by all three backends. **Usage scope:** the HKDF-derived subkey (and, under `vault_transit`, the named Transit audit key) authenticates the **audit row chain only** — an integrity/tamper-evidence key, never a confidentiality key. It is domain-separated from the store DEK it is derived from (`mefor/audit-chain/v1`), decrypts nothing, protects no message body, and is not transport, signing, or token material. **Verification is constant-time and full-walk (ASVS 11.2.4):** `hmac.compare_digest` over `audit_mac_bytes` on every row MAC *and* on the external-anchor head, in `store/store.py`, `store/postgres.py` and `store/sqlserver.py` — the walk never returns early, so verify duration is a function of chain length, not of where a forgery sits (the first divergent row id is still named in the operator-facing result) | `audit_log.row_hash` | Append-only; verified by `messagefoundry audit-verify` | | Config fingerprint ([ADR 0041](adr/0041-load-path-attestation-and-change-attribution.md)) | SHA-256 content digest of a loaded config bundle — path-relative Merkle fold over every loaded file (`*.py` incl `_*.py`, `connections.toml`, `codesets/*`, `environments/*.toml`); `hashlib` in `config/fingerprint.py` | Recorded in the `config_reload` audit detail (not stored as a secret) | Recomputed per reload/startup; binds reviewed-commit → loaded-bytes (integrity/attribution, not confidentiality) | | Engine wheel attestation ([ADR 0041](adr/0041-load-path-attestation-and-change-attribution.md) D3) | SHA-256 over each **loaded** first-party `messagefoundry` module file, compared to the installed wheel's `*.dist-info/RECORD` baseline (a base64 `sha256=` manifest already in the wheel); `hashlib` in `integrity.py` | Drift recorded in the hash-chained `startup_integrity` audit row (not a secret); RECORD baseline read from site-packages metadata | Recomputed at startup + on demand; in-place-tamper tripwire (integrity, not confidentiality). Alert-only by default; `[integrity].fail_closed_on_drift` refuses to start on drift; no-op on an editable install | +| ASVS corpus pin ([ADR 0156](adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md)) | SHA-256 over the **OWASP ASVS 5.0.0 corpus file**, recorded in `[scorecard].corpus_sha256` and recomputed on every verifier run; `hashlib` in `scripts/asvs/scorecard.py`. **Integrity of a build input, not a security control** — no secret, no key, no message authentication, and nothing user- or PHI-derived is hashed. It exists because the corpus was originally fetched from `master` (the bleeding-edge branch, where a rolling "latest" release republishes identical filenames) and matched the tagged `v5.0.0_release` asset only by luck; the digest is now recorded and checked rather than assumed, because ASVS requirement ids are **not stable across versions** (bare `1.2.5` is *Architecture* in 4.0.3 and *Encoding and Sanitization* in 5.0.0), so a corpus that moves silently re-points every id in the scorecard | Not a secret: the digest is committed alongside the corpus it pins | Recomputed on every scorecard verification; a mismatch fails the gate and forces re-verification before any verdict is trusted | | Outbound message signing (opt-in) | Detached JWS (RFC 7515) — RS256/PS256 (RSA) or ES256 (ECDSA P-256), SHA-256; `cryptography` in `transports/signing.py` (ASVS 4.1.5, [ADR 0018](adr/0018-per-message-signatures-accepted-risk.md)) | Operator-supplied PEM **private** signing key per connection (inline via `env()` or a PEM file path; encrypted-key passphrase via `env()`); the **public** key is shared with the partner out-of-band. **Usage scope:** this private key **only** signs this connection's outbound per-message JWS — a message-**authenticity/integrity** key in transit; it is never used for at-rest encryption or session/token material, and the partner holds only the matching **public** verification half | **OFF by default**; per-connection opt-in. `kid` carried in the JWS header so key rotation / a managed provider ([ADR 0019](adr/0019-pluggable-keyprovider-hsm-kms-vault.md)) slots in without a wire change | | DIRECT S/MIME (opt-in, [ADR 0085](adr/0085-direct-hisp-smime-connector.md)) | CMS **sign-then-encrypt** in `transports/direct.py` (core `cryptography` `serialization.pkcs7`): PKCS#7 signature over the body with a **SHA-256** digest, the public-key signature algorithm (RSA / ECDSA) following the loaded signing key type (not pinned to RSA), then a PKCS#7 **envelope** to the partner's recipient cert. The envelope content-encryption cipher is the **`cryptography` pkcs7 library default** — no algorithm is pinned in code | Sender **signing cert** + PEM **private key** (optional `signing_key_password`) and the per-partner **`recipient_cert`**, all operator-supplied files; the recipient cert is trust-verified at construction against an operator `trust_anchor` (one-level direct-issuance check); key/cert mismatch refused. **Usage scope:** the sender signing key signs the CMS body and the partner's `recipient_cert` encrypts the CMS envelope — this material protects the **confidentiality + authenticity of a DIRECT message to one partner in transit**; it is not an at-rest store key and encrypts nothing in the store | **OFF by default** — only when a DIRECT Connection is configured, and its HISP relay host is gated by the **opt-in** `[egress].allowed_direct` allow-list (empty by default = unrestricted; an unlisted host is refused only once the list is populated, or outright when `[security].block_unlisted_outbound` is set). Signing key + recipient certs rotate on the schedule below | | OIDC IdP JWKS verification keys (opt-in, [ADR 0142](adr/0142-federated-sso-oidc-authorization-code-pkce-relying-party-hybrid-ad-backed.md)) | **Public** verifying keys fetched from the IdP JWKS: **RS256/PS256** (RSA, ≥ 2048-bit floor) and **ES256/ES384** (EC P-256/P-384) — rebuilt from each JWK by `cryptography` in [`auth/oidc/jwks.py`](../messagefoundry/auth/oidc/jwks.py); the closed `SignatureAlgorithm` enum forecloses `alg:none` and RS256→HS256 confusion. Bounded, TTL-cached (`DEFAULT_JWKS_TTL_SECONDS`), a 512 KiB body cap, a global min-refetch floor (fetch-amplification bound), and a hard refusal of a duplicate `kid`; a key below the floor is skipped/refused, never merely warned. **Usage scope:** these are **public**, non-secret keys used **only** to verify the IdP's id-token signature at console login — they encrypt nothing and can protect no data; the engine holds no private half. | Fetched from the IdP JWKS URI over the CA-pinned no-redirect opener (row below); held process-local in `JwksCache`, never persisted, never logged | Refetched per TTL / on an unknown `kid` within the amplification bound; rolls when the IdP rotates its signing keys | diff --git a/docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md b/docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md new file mode 100644 index 0000000..4ccd6d5 --- /dev/null +++ b/docs/adr/0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md @@ -0,0 +1,215 @@ +# ADR 0156 — ASVS scorecard as data: a derived count, verified evidence anchors, and a fail-closed drift gate + +**Status:** **Accepted (2026-08-01)** — built and merged the same day; §7 amended at ratification, see below +**Date:** 2026-08-01 +**Supersedes:** — +**Related:** ADR 0155 (DAST) · BACKLOG #205 (risk-acceptance register) · ASVS cell 15.1.3 + +--- + +## Context + +The ASVS L3 score is maintained as **prose**. A dated assessment states a count; later dated documents +supersede it; a risk-acceptance register carries a row per residual; each asserts facts about the code +in sentences. Nothing checks any of it. + +On **2026-08-01** a single re-anchoring session produced the following, all in one day: + +| Observed | Count | +|---|---| +| Times the headline count was re-derived from scratch | 6 | +| Residuals of record that were **factually false** at `HEAD` | 12 | +| Of those, **absence claims** that had silently stopped being true | 5 | +| Cells missing from an enumeration described as *"arithmetic-checked and complete"* | 10 | +| Documents asserting a superseded count as current | 5 | +| Register-vs-scorecard contradictions | 4 | +| The same documentation defect fixed independently by two sessions | 1 | +| Subagent tokens spent on verification | **~23 million** | + +Most of that spend was not discovery. It was **re-deriving facts already known but never durably +recorded**, then re-verifying them because the recorded version had gone quietly false. + +Three failure modes account for all of it: + +1. **Prose asserts facts about code, and the code moves.** A residual says *"no privilege probe of any + kind exists"*; the code grows two; the sentence rots in place. The next session funds work that is + already done, or trusts a control that is not there. +2. **A completeness claim cannot be checked by the thing that makes it.** The dropped-ten defect + survived because the arithmetic closed to 345 and closure was read as proof. **Closure only proves + the four buckets sum — not that every cell landed in one.** No self-consistent count can detect an + absent cell. +3. **The guards that would catch this are blind.** Six `*_doc_drift` test modules exist, one with ~88 + assertions written specifically to stop this class. `git ls-files docs/security/` in the development + repo returns **0** — the documents are vault-only, so every doc-anchored assertion has nothing to + read in the tree where CI runs. **This is ASVS 15.1.3**, currently open, and it is why none of the + above was caught by a machine. + +The lineage convention — dated documents, superseded rather than edited — is **correct for audit** and +is not the problem. The problem is that it is also being used as the *source of truth* for facts that +change under it. + +## Decision + +**Make the scorecard data. Derive the count. Verify the evidence. Fail closed.** + +### 1. One record per requirement, all 345 + +A single structured file — `docs/security/asvs-scorecard.toml`, TOML per the project's config +preference — holds one `[[cell]]` per ASVS 5.0.0 requirement: + +```toml +[[cell]] +id = "12.1.1" +level = 1 # from the held corpus, not typed +verdict = "partial" # pass | partial | fail | na | unverified +posture = "single" +residual = "The SQL Server ODBC PHI hop has no engine-side TLS version floor…" +last_verified = "2026-08-01" +verified_at = "f8d11685" # the commit the evidence was read on + [[cell.evidence]] + path = "messagefoundry/__main__.py" + line = 1978 + expect = "probe_tls_floor" # a token CI asserts is still there +``` + +### 2. The count is computed, never typed + +`scripts/asvs/scorecard.py` sums the verdicts. **No document states a count; documents render it.** +Five documents can no longer assert three counts, because there is only one place a count exists. + +### 3. Completeness is asserted, not claimed + +A test asserts **every one of the 345 ids in the held corpus appears exactly once**, and that no id +outside it appears at all. This is the check whose absence cost ten cells: it would have failed the +moment the enumeration dropped one, rather than closing to 345 and looking correct. + +### 4. Evidence anchors are machine-verified + +For every cell, CI opens each `evidence.path` and asserts `expect` is present at or near `line`. When +the code moves, **the test goes red instead of the document going quietly false.** This is the direct +countermeasure to the twelve false residuals, and specifically to the five false *absence* claims — +which is why an absence claim must record the **search that proved it**, not merely its conclusion: + +```toml + [[cell.absence]] + pattern = "clamav|clamd|ICAP|MpCmdRun|yara" + positive_control = "ScanRejected" # must MATCH, or the search is blind +``` + +A zero result is evidence only if the positive control still hits. **A grep naming the wrong token +returns zero and reads exactly like proof** — that is how five of them survived. + +### 5. `unverified` is a first-class verdict + +Cells inherited from an earlier assessment and never re-read against the requirement text are +`unverified`, **not** `pass`. The renderer reports *verified Pass* and *inherited Pass* separately. + +This makes the largest standing exposure **visible and countable** rather than a caveat in prose: as of +2026-08-01, ~219 Passes have never been checked against the ASVS text, and the one pass that did apply +an adversarial layer caught **three attempted unearned upgrades out of six proposals**. + +### 6. Fail closed, never skip + +The verifier **refuses** when the scorecard is absent but expected, rather than skipping. Skipping is +precisely what 15.1.3 does today and why a green CI proves nothing about these documents. + +### 7. Placement, and how 15.1.3 closes + +| Artifact | Lives in | Why | +|---|---|---| +| Verifier + schema + unit tests (fixture data) | **Development repo** | Generic tooling, no posture data; runs in public CI on every PR | +| `asvs-scorecard.toml` (real verdicts + evidence) | **Vault** | Posture data; `docs/security/`-class | +| Enforcement against the real data | **Vault pre-commit hook + one narrow vault workflow** | See the amendment — the obvious answer was wrong | + +That last row is the fix for **15.1.3**: the guards stop being inspection-only theatre. + +> ### ⚠️ Amendment at ratification (2026-08-01) — §7's original placement was WRONG +> +> This ADR was drafted proposing *"a vault CI job runs the verifier against the real data"*, reasoning +> from the fact that the vault has `ci.yml`, `tests/` and `pyproject.toml`. **Open question 1 asked +> whether that CI actually executes. It does not.** +> +> The GitHub actions API reports **every vault workflow as `disabled_manually`** — CI, Security, +> CodeQL, backlog-hygiene, release, all of them. The last run was **2026-07-27**, and the two vault +> PRs merged on 2026-08-01 both merged with **zero checks**. The estate was switched off at the +> cutover to avoid duplicating public CI, which is a reasonable decision and is not being reversed. +> +> **A CI-only design would have shipped dead.** The built design is therefore: +> +> 1. **A vault pre-commit hook** — the enforcement that works *today*, needs no CI, and fires at +> authoring time, which is when the drift is introduced. +> 2. **One narrow new workflow** (`asvs-scorecard.yml`) — new workflows are active by default even +> though the estate is disabled, so this runs without resurrecting any of it. Stdlib-only, no +> install step, path-filtered, 5-minute cap. It additionally fails if the rendered +> `ASVS-CURRENT.md` has drifted from the data. +> +> **The lesson is the ADR's own thesis applied to itself.** §7 was a confident, plausible statement +> about system state that nobody had checked — exactly the failure mode the rest of this document +> exists to prevent. It was caught only because ratification was gated on answering the open question +> rather than on the argument reading well. + +### 8. A generated entry point + +`ASVS-CURRENT.md` is rendered, never hand-written: the count, the anchor commit, the open cells, and +what awaits an owner decision. The dated lineage stays for audit; a new session reads one generated +page instead of reconstructing state from ten superseding documents. + +## Consequences + +**Good.** + +- A dropped cell becomes **impossible** rather than undetected. +- A rotted evidence claim becomes a **red test** rather than a misleading sentence. +- A re-score becomes *"re-verify the cells whose anchors moved"* instead of *"re-derive all 345."* +- Inherited-versus-verified Pass becomes countable, so the real exposure is legible. +- 15.1.3 closes as a by-product. + +**Costs, stated plainly.** + +- **Populating 345 cells is real work.** Mitigated by generating the skeleton from the held corpus + (ids, levels, chapter/section come free) and marking everything not verified today `unverified` — + which is honest, and is itself the finding. +- **Evidence anchors will break on refactors.** That is the feature; it is also maintenance. `expect` + is a token rather than a line number precisely so ordinary edits do not thrash it. +- **Two homes for one system** (tool public, data vault). Accepted: the alternative is either posture + data in the public repo or a tool nothing runs. +- **This does not make the score correct.** It makes it *consistent, derived, and drift-detecting*. + A wrong verdict recorded carefully is still wrong — adversarial verification remains the only cure + for that, and this ADR does not replace it. + +**Rejected alternatives.** + +- *Keep prose, review harder.* Six documented failure modes in one day survived careful review by + construction: every false residual **read as true**. The project's own standard says the mitigation + must be structural, not diligence. +- *One document to rule them all.* Tried — that is what the current lineage is. It produces five + documents asserting three counts, because a superseding document cannot retract a copy it does not + know about. +- *Put the vault documents in the public repo so the guards see them.* Refused: they are an attacker + roadmap (`SECURITY-DOCS-POLICY.md`). Running the guards in the vault achieves the same end. + +## Status at ratification + +Built and merged 2026-08-01, same day as the decision: + +- **Tool** — `scripts/asvs/scorecard.py` (development repo), stdlib only, no new dependency. +- **Tests** — `tests/test_asvs_scorecard.py`, 17 tests. **Every check is proved to go RED before it is + trusted green**: dropped cell, duplicate cell, an id ASVS retired, a level disagreeing with the + corpus, a moved token, a deleted file, a blind absence search, an absence that became false. + Exit codes verified by hand — **0** clean, **1** findings, **2** could-not-measure on a missing file. +- **Data** — `docs/security/asvs-scorecard.toml` (vault), **all 345 cells** seeded from the held + corpus. 14 carry verified 2026-08-01 evidence; **331 are `unverified`, not `pass`**. The file + therefore does **not** reproduce the published headline and must not be read as doing so — what it + reports today is the state of *verification*, which had never been visible. +- **Entry point** — `docs/security/ASVS-CURRENT.md`, generated, never hand-written. + +## Open questions + +1. ~~**Does the vault CI run `pytest` today?**~~ **ANSWERED 2026-08-01 — no.** Every vault workflow is + `disabled_manually`. §7 was amended accordingly *before* ratification. This is the question that + would have shipped a broken control had it been waved through. +2. **Do ADR and BACKLOG numbers collide across the two repos?** The allocator is per-clone. BACKLOG + spaces are being partitioned; ADR is unresolved and this ADR was allocated in the development repo + deliberately. +3. **Should `ASVS-CURRENT.md` be publishable** in redacted form (count + levels, no residual text) so + adopters see a posture summary without the roadmap? Not decided here. diff --git a/docs/adr/README.md b/docs/adr/README.md index ac1b17a..fd25ef9 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -182,3 +182,4 @@ what is withheld and what you can request. | [0153](0153-collapse-the-posture-gradient-no-data-label-may-allow-a-cleartext-hop.md) | **Collapse the posture gradient — no data label may allow a cleartext hop** — ADR 0092's precedence arm 3 is `not is_phi -> ALLOW`, so an instance whose environment file declares `data_class = "synthetic"` crosses **every** cleartext transport hop silently, with no warning and no audit record. Found by upgrading a dogfood instance whose environment file had been quietly unparseable: the moment it loaded and `data_class = "phi"` took effect, a cleartext MLLP egress that had been crossing without comment became a refusal — nothing about the hop had changed, only a label in a different file, and nothing had reported the guard was off. Deletes that one arm: `insecure_hop_disposition` loses `is_phi` and the precedence becomes loopback -> ALLOW, attested -> ALLOW, `cleartext_accepted` -> WARN, not-`enforcing` -> WARN, else REFUSE. Because the deleted arm returned ALLOW, removing it can only turn a crossing into a WARN or a REFUSE, so **0092's no-loosen rule holds by construction** and an unchanged config is byte-identical unless it relied on the label. Adds `cleartext_accepted` + `cleartext_reason` on **Destination**, deliberately SEPARATE from `tls_hop_attested`: attestation claims *this hop IS secure by means the engine cannot see* and passes silently, the new pair admits *it is NOT, and we accept that* and warns + audits every construction — merging them would leave the audit trail unable to tell a proxy-terminated hop from plaintext PHI on a flat network. **No TLS default is flipped** (an earlier draft proposed it): redundant, since an undeclared cleartext outbound already REFUSES and a refusal naming the hop beats a handshake failure; it hard-fails 14 of 14 inbound MLLP listeners because `_mllp_ssl_context(server=True)` demands a cert before any policy runs, unreachable by posture/attestation/the new flag and not loopback-exempt; and it would silently disarm the four inbound exposed-gates, which early-return when `tls` is truthy. `Tcp()`/`X12()` have no TLS support at all, so for them the declaration is **permanent and structural**, not transitional. Neither `data_class` nor `MEFOR_ALLOW_INSECURE_TLS` is removed — the hop authority stops READING the label, and the variable is unhooked from the authority but survives for the six non-connection cells (store TLS, LDAPS, webhook sink, AI broker) that have no per-connection field to carry an escape. Retains ADR 0148's `[security].enforcement` dial as an arm (its AC-3 requires it on every gate). Scope is the cleartext transport-hop decision ONLY; an explicit out-of-scope table names all four callers of the authority and the three other readers of `HopPosture` | Accepted (2026-07-25) — reworked after adversarial review; owner-ratified; **BUILT 2026-07-28** | | [0154](0154-synchronous-captured-downstream-reply-and-intake-authentication-for-the-inbound-http-listener-adr-0023-deferred-tail.md) | **Synchronous captured-downstream-reply and intake authentication for the inbound HTTP listener** — the ADR 0023 deferred tail: `reply_from` blocks the HTTP turn on a **committed** ADR 0013 `response` row (never an in-flight `DeliveryResponse`), and `intake_auth` (API key / bearer / mTLS subject) adds a peer control behind a posture-keyed gate. Also closes a live hole — `check_http_tls_exposure` returns early on truthy `tls`, so an off-loopback `Http(tls=True)` listener authenticates nobody today | **Accepted (2026-07-31)** — owner-ratified at rev 5; authorises **increment A only** (intake-auth + peer-control gate), which is **built and merged** (2026-08-01, `f2ef0ea9`); sync-reply (increment B) deferred pending a customer | | [0155](0155-dast-dynamic-security-testing-of-the-running-engine.md) | **DAST — dynamic security testing of the running engine** (BACKLOG #318) — no DAST had ever run against this project: every security test was static or in-process, leaving the [Secure_Development_Standards](../Secure_Development_Standards.md) §6.1 *Dynamic* tier row empty. Increment 1 builds a **self-run, authenticated authorization sweep** with **no new dependency**: one `uvicorn` listener on loopback in front of a real Engine + real AuthService, both identities minted over the wire through `POST /auth/login`, and the authorization expectation **derived from the live route table** by a single shared `require*()`-closure walk ([`scripts/security/route_gates.py`](../../scripts/security/route_gates.py), hoisted out of the security doc-drift guard so exactly one derivation of *is this route gated* exists in the tree) rather than a hand-kept list that goes stale the day a route lands. Measured shape: 105 route rows, 100 gated, 87 permission-gated, exactly 5 anonymous. Three passes — **negative** (every gated HTTP row sent with no credential and with an invalid bearer; anything but 401 is a finding), **authorized reach** (how many gated `GET` rows a *privileged* token got past authentication and authorization on — the positive number that stops a wall of 401s reading as *all endpoints protected*), and **viewer BFLA** (anything but a refusal, **including 404**, is a finding, because a 404 on a matched path template means the caller got past authorization into resource lookup). The receipt names what it examined and **fails closed** — below any floor it exits **2 (could not measure)**, never 0 — and records method, path template, status codes, counts and the *relaxed* posture it scanned, never a body, header or token. Two canaries are built from **supported configuration, not source patches** (authentication disabled at the target; the low-privilege identity over-granted while the expectation set stays the viewer's), avoiding the patch-rot failure where a canary silently stops applying; CI runs both **before** the real scan and requires each to exit exactly **1** (findings) *with* its receipt on disk — a 0 (blind), a 2 (could not measure, which is what a neutered canary actually produces) or a crash fails the job and the real scan never runs. The inversion is the design: the sweep and both canaries run as ordinary pytest in the **existing required** test legs, so a change that blinds the detector reds a PR, while the nightly workflow is advisory — **not** a required context (it has no `pull_request` trigger, so it cannot report on a PR) and deliberately **not** `continue-on-error`. **Scope boundary: see the ADR's *Scope boundary* section** — it is stated once there, verbatim, and this row deliberately carries a pointer and no wording of its own. Rejected/deferred: schemathesis (the shipped OpenAPI declares no `securitySchemes` and no per-operation `security`, so `ignored_auth` would pass on every operation having probed none of them; adoption also needs a fifth DEP-1 lock over a ~30-distribution closure that could pull a shipped runtime floor down), ZAP (a SHA-pinned action still pulls a mutable `:stable` image, and the published images do not install the web console), nuclei (template matcher; `pip install nuclei` is an abandoned unrelated package), Dredd/RESTler/CATS, in-process ASGI transport (bypasses the HTTP parser, leaves `request.client` unset), the unauthenticated MLLP/TCP/X12/DICOM ingress plane, the `/ui` console plane, a TLS black-box target, and non-`GET` reach/BFLA | Accepted (2026-07-31) — increment 1 built; advisory, not a required context | +| [0156](0156-asvs-scorecard-as-data-a-derived-count-verified-evidence-anchors-and-a-fail-closed-drift-gate.md) | **ASVS scorecard as data — a derived count, verified evidence anchors, and a fail-closed drift gate** — the ASVS score is maintained as prose, and nothing checks it. One re-anchoring session (2026-08-01) re-derived the headline count **6 times**, found **12 residuals of record factually false at HEAD** (five of them *absence* claims that had silently stopped being true), and found **10 cells missing from an enumeration described as "arithmetic-checked and complete"** — which survived because the arithmetic closed to 345 and closure was read as proof. **Closure only proves the four buckets sum, not that every cell landed in one.** Decision: hold one `[[cell]]` record per requirement (all 345) in `asvs-scorecard.toml`; **compute** the count so no document can state one; assert **every corpus id appears exactly once** (the check whose absence cost ten cells); machine-verify each cell's `evidence` anchor by asserting an expected **token** still resolves, so code movement reds a test instead of rotting a sentence; require an absence claim to record the **search that proved it plus a positive control that must still hit**, because a grep naming the wrong token returns zero and reads exactly like proof; make **`unverified` a first-class verdict** so inherited-versus-verified Pass is countable (~219 Passes have never been read against the requirement text); and **fail closed rather than skip**. Tool + schema + fixture tests live in this repo and run in public CI; the real scorecard lives in the vault with a vault-CI job — which closes **ASVS 15.1.3**, currently open precisely because six `*_doc_drift` modules assert against documents that `git ls-files docs/security/` shows are **not present** in the tree where CI runs. Rejected: *keep prose and review harder* (every false residual **read as true**; the project's own standard says the mitigation must be structural, not diligence), *one document to rule them all* (that is the current lineage, and it produces five documents asserting three counts), and *publish the vault documents so the guards see them* (attacker roadmap, `SECURITY-DOCS-POLICY.md`). Explicitly **does not** make the score correct — only consistent, derived and drift-detecting; adversarial verification remains the only cure for a wrong verdict | **Accepted (2026-08-01)** — built and merged the same day. **§7 was amended at ratification**: it proposed a vault CI job, and the actions API showed every vault workflow `disabled_manually` (last run 2026-07-27; two vault PRs merged that day with zero checks), so a CI-only design would have shipped dead. Built instead as a vault **pre-commit hook** plus one **narrow new workflow**. That §7 was a confident, unchecked claim about system state is the ADR own thesis applied to itself | diff --git a/scripts/asvs/scorecard.py b/scripts/asvs/scorecard.py new file mode 100644 index 0000000..35ad1c2 --- /dev/null +++ b/scripts/asvs/scorecard.py @@ -0,0 +1,462 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 MessageFoundry Organization and contributors +"""ASVS scorecard as data (ADR 0156) — derive the count, verify the evidence, fail closed. + +The score used to live in prose. On 2026-08-01 one re-anchoring session re-derived the headline count +six times, found twelve residuals of record that were false at HEAD (five of them *absence* claims that +had quietly stopped being true), and found ten cells missing from an enumeration that described itself +as "arithmetic-checked and complete". That last one survived because the arithmetic closed to 345 and +closure was read as proof. + +**Closure only proves the four buckets sum. It does not prove every cell landed in one.** That is the +whole reason :func:`check_completeness` exists and asserts against the corpus rather than the total. + +This module is deliberately data-free: it takes the scorecard and the ASVS corpus as paths, so the +public repo can unit-test it against a fixture while the vault runs it against the real posture data +(ADR 0156 §7). It adds no dependency — ``tomllib`` is stdlib. +""" + +from __future__ import annotations + +import argparse +import re +import sys +import tomllib +from collections import Counter +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Final, Literal + +Verdict = Literal["pass", "partial", "fail", "na", "needs-review", "unverified"] + +#: The scoring buckets. ``unverified`` is deliberately first-class (ADR 0156 §5): a cell inherited from +#: an earlier assessment and never re-read against the requirement text is NOT a Pass, and conflating +#: the two is what let ~219 unchecked verdicts hide inside a headline. +VERDICTS: Final[frozenset[str]] = frozenset( + {"pass", "partial", "fail", "na", "needs-review", "unverified"} +) + +#: ASVS 5.0 defines only three verdicts — verified, exception, and non-applicable-with-rationale. The +#: strings "partially implemented" and "not implemented" appear nowhere in it, and the one prominent +#: OSS ASVS tool declines to model partial too. Everything here beyond pass/fail/na is OUR extension, +#: defined in docs/ASVS-ASSESSMENT-METHOD.md §1 — an undefined grade is one two assessors apply +#: differently, which is exactly how 11.7.1, 3.7.3 and 5.4.3 each changed verdict in a single day. +LOCAL_EXTENSION_VERDICTS: Final[frozenset[str]] = frozenset( + {"partial", "needs-review", "unverified"} +) + +#: A verdict that has actually been reached. `unverified` and `needs-review` are NOT among them: +#: the first was never examined, the second was examined and left open on purpose. +DECIDED_VERDICTS: Final[frozenset[str]] = frozenset({"pass", "partial", "fail", "na"}) + +#: How far from the recorded line the expected token may drift before the anchor is considered broken. +#: Anchors name a TOKEN rather than a bare line number precisely so that ordinary edits above a cell's +#: evidence do not thrash every anchor in the file; the window keeps the line number meaningful without +#: making it load-bearing. +ANCHOR_WINDOW: Final[int] = 40 + + +class ScorecardError(Exception): + """A defect in the scorecard itself — malformed, incomplete, or contradicting the corpus.""" + + +@dataclass(frozen=True) +class Anchor: + """A claim that some token exists in the tree, at roughly a known place.""" + + path: str + line: int + expect: str + + +@dataclass(frozen=True) +class Absence: + """A claim that something does NOT exist, plus the proof the search could have seen it. + + A grep naming the wrong token returns zero and reads exactly like proof — that is how five false + absence claims survived for weeks. So an absence claim is only admissible with a + ``positive_control`` that must still match; if the control goes quiet the search has gone blind and + the claim is void, regardless of what the pattern returns. + """ + + pattern: str + positive_control: str + + +@dataclass(frozen=True) +class Cell: + id: str + level: int + verdict: Verdict + residual: str = "" + posture: str = "single" + last_verified: str = "" + verified_at: str = "" + reviewed_by: str = "" + evidence: tuple[Anchor, ...] = () + absence: tuple[Absence, ...] = () + + @property + def is_inherited(self) -> bool: + """A verdict carried from an earlier assessment, never re-read against the requirement text.""" + return self.verdict == "unverified" or not self.last_verified + + +@dataclass +class Findings: + """What a verification pass found. Empty ``problems`` is the only pass condition.""" + + problems: list[str] = field(default_factory=list) + checked_anchors: int = 0 + checked_absences: int = 0 + skipped_anchors: int = 0 + + @property + def ok(self) -> bool: + return not self.problems + + +def corpus_digest(path: Path) -> str: + """SHA-256 of the corpus file, so the pin is checkable rather than asserted. + + The corpus MUST come from the tagged ``v5.0.0_release`` asset. ``master`` is the bleeding-edge + branch and a rolling "latest" release republishes identical filenames, so an unpinned fetch moves + versions silently. Our first corpus was fetched from ``master`` and happened to be byte-identical + to the release — luck, not method, which is why this is now recorded and verified. + """ + import hashlib + + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def load_corpus(path: Path) -> dict[str, int]: + """Map ``req_id`` (V-stripped) to level, from the OWASP ASVS 5.0.0 flat JSON. + + The corpus is the external anchor. Before it was held, every verdict in this project reasoned from + the requirement verb *as paraphrased in our own scorecards* — a closed loop that produced a cell + scored against a requirement ASVS had deleted. + """ + import json + + if not path.is_file(): + raise ScorecardError( + f"ASVS corpus not found at {path} — cannot check completeness without it" + ) + raw: Any = json.loads(path.read_text(encoding="utf-8")) + reqs = raw["requirements"] if isinstance(raw, dict) else raw + return {str(r["req_id"]).lstrip("V"): int(r["L"]) for r in reqs} + + +def load_scorecard(path: Path) -> list[Cell]: + if not path.is_file(): + # Fail closed, never skip (ADR 0156 §6). Skipping is exactly what the doc-drift guards do + # today, and it is why a green CI proves nothing about these documents. + raise ScorecardError( + f"scorecard not found at {path} — refusing to report a pass on a missing file" + ) + data = tomllib.loads(path.read_text(encoding="utf-8")) + cells: list[Cell] = [] + for raw in data.get("cell", []): + verdict = str(raw.get("verdict", "")).lower() + if verdict not in VERDICTS: + raise ScorecardError( + f"cell {raw.get('id')!r}: verdict {verdict!r} not one of {sorted(VERDICTS)}" + ) + # ASVS 5.0's assessment chapter is "should" throughout with exactly ONE "must": a + # non-applicable requirement must be noted with its reason. So an `na` without a rationale is + # not a lax entry — it is the single thing the standard actually requires, omitted. + if verdict == "na" and not str(raw.get("residual", "")).strip(): + raise ScorecardError( + f"cell {raw.get('id')!r}: verdict 'na' requires a written rationale in `residual` — " + "recording the reason for non-applicability is the one MUST in ASVS 5.0's assessment " + "chapter (docs/ASVS-ASSESSMENT-METHOD.md §1)" + ) + cells.append( + Cell( + id=str(raw["id"]), + level=int(raw["level"]), + verdict=verdict, # type: ignore[arg-type] + residual=str(raw.get("residual", "")), + posture=str(raw.get("posture", "single")), + last_verified=str(raw.get("last_verified", "")), + verified_at=str(raw.get("verified_at", "")), + reviewed_by=str(raw.get("reviewed_by", "")), + evidence=tuple( + Anchor(path=str(e["path"]), line=int(e["line"]), expect=str(e["expect"])) + for e in raw.get("evidence", []) + ), + absence=tuple( + Absence(pattern=str(a["pattern"]), positive_control=str(a["positive_control"])) + for a in raw.get("absence", []) + ), + ) + ) + return cells + + +def count(cells: list[Cell]) -> Counter[str]: + """The count is COMPUTED. No document states one; documents render this (ADR 0156 §2).""" + return Counter(c.verdict for c in cells) + + +def check_completeness(cells: list[Cell], corpus: dict[str, int]) -> list[str]: + """Every corpus id appears exactly once, and nothing outside the corpus appears at all. + + **This is the check whose absence cost ten cells.** An enumeration that closes to the right total + can still be missing entries; only comparing against the corpus detects that. + """ + problems: list[str] = [] + seen = Counter(c.id for c in cells) + + for dupe, n in sorted((i, n) for i, n in seen.items() if n > 1): + problems.append( + f"completeness: {dupe} appears {n} times — each requirement gets exactly one cell" + ) + + missing = sorted(set(corpus) - set(seen), key=_sort_key) + if missing: + problems.append( + f"completeness: {len(missing)} requirement(s) in the ASVS corpus have NO cell — " + f"the count cannot be correct while a cell is absent: {', '.join(missing)}" + ) + + unknown = sorted(set(seen) - set(corpus), key=_sort_key) + if unknown: + problems.append( + f"completeness: {len(unknown)} cell(s) are not ASVS 5.0.0 requirement ids " + f"(retired in 5.0, or a typo): {', '.join(unknown)}" + ) + + for c in cells: + want = corpus.get(c.id) + if want is not None and c.level != want: + problems.append(f"{c.id}: level {c.level} but the corpus says L{want}") + return problems + + +def check_anchors(cells: list[Cell], root: Path, findings: Findings) -> None: + """Open every evidence anchor and assert its token still resolves. + + When the code moves, this reds a test — instead of the sentence rotting in place and the next + session funding work that is already done. + """ + for c in cells: + for a in c.evidence: + target = root / a.path + if not target.is_file(): + findings.problems.append(f"{c.id}: evidence path {a.path} does not exist") + continue + lines = target.read_text(encoding="utf-8", errors="replace").splitlines() + lo = max(0, a.line - 1 - ANCHOR_WINDOW) + hi = min(len(lines), a.line + ANCHOR_WINDOW) + findings.checked_anchors += 1 + if a.expect in "\n".join(lines[lo:hi]): + continue + where = " (found elsewhere in the file)" if a.expect in "\n".join(lines) else "" + findings.problems.append( + f"{c.id}: {a.path}:{a.line} no longer contains {a.expect!r} within " + f"±{ANCHOR_WINDOW} lines{where} — the evidence moved or the claim is now false" + ) + + +def check_absences(cells: list[Cell], root: Path, findings: Findings) -> None: + """An absence is proven only when its pattern is quiet AND its positive control still speaks.""" + corpus_files = _python_sources(root) + for c in cells: + for a in c.absence: + findings.checked_absences += 1 + control = _grep_count(a.positive_control, corpus_files) + if control == 0: + findings.problems.append( + f"{c.id}: absence claim is BLIND — its positive control {a.positive_control!r} " + f"matches nothing, so a zero result for {a.pattern!r} proves nothing" + ) + continue + hits = _grep_count(a.pattern, corpus_files) + if hits: + findings.problems.append( + f"{c.id}: absence claim is FALSE — {a.pattern!r} now matches {hits} time(s); " + f"the thing recorded as missing exists" + ) + + +def _python_sources(root: Path) -> list[Path]: + out: list[Path] = [] + for pkg in ("messagefoundry", "messagefoundry_webconsole", "harness", "scripts"): + base = root / pkg + if base.is_dir(): + out.extend( + p + for p in base.rglob("*.py") + if ".venv" not in p.parts and "__pycache__" not in p.parts + ) + return out + + +def _grep_count(pattern: str, files: list[Path]) -> int: + rx = re.compile(pattern) + return sum(1 for f in files if rx.search(f.read_text(encoding="utf-8", errors="replace"))) + + +def _sort_key(cell_id: str) -> tuple[int, ...]: + try: + return tuple(int(p) for p in cell_id.split(".")) + except ValueError: # a malformed id sorts last rather than crashing the report + return (9_999,) + + +def load_meta(path: Path) -> dict[str, str]: + """The ``[scorecard]`` table: version pin, corpus digest, and the assessment anchor commit.""" + data = tomllib.loads(path.read_text(encoding="utf-8")) + return {k: str(v) for k, v in data.get("scorecard", {}).items()} + + +def check_pinning(scorecard: Path, corpus: Path) -> list[str]: + """The scorecard must declare which ASVS version it scores, and pin the corpus by digest. + + **ASVS requirement IDs are not stable across versions** — bare ``1.2.5`` is *Architecture* in + 4.0.3 and *Encoding and Sanitization* in 5.0.0 — and OWASP's referencing guidance prefers + ``v-.
.``. A document-level version field satisfies that + pinning, so a scorecard keyed on bare ids cannot silently re-point when ASVS updates. + """ + problems: list[str] = [] + data = tomllib.loads(scorecard.read_text(encoding="utf-8")) + meta = data.get("scorecard", {}) + + if not str(meta.get("asvs_version", "")).strip(): + problems.append( + "pinning: [scorecard].asvs_version is missing — bare requirement ids re-point across " + "ASVS versions, so the scorecard must say which version its ids mean" + ) + declared = str(meta.get("corpus_sha256", "")).strip() + actual = corpus_digest(corpus) + if not declared: + problems.append( + f"pinning: [scorecard].corpus_sha256 is missing — the corpus digest is {actual}" + ) + elif declared != actual: + problems.append( + f"pinning: corpus digest mismatch — declared {declared[:16]}…, actual {actual[:16]}…. " + "The ASVS corpus changed underneath the scorecard; re-verify before trusting any verdict" + ) + return problems + + +def verify(scorecard: Path, corpus: Path, root: Path) -> Findings: + findings = Findings() + cells = load_scorecard(scorecard) + findings.problems.extend(check_pinning(scorecard, corpus)) + findings.problems.extend(check_completeness(cells, load_corpus(corpus))) + check_anchors(cells, root, findings) + check_absences(cells, root, findings) + return findings + + +def render_current(cells: list[Cell], *, anchor_sha: str) -> str: + """The generated entry point — survey progress FIRST, verdict counts second. + + **Phase 0 of the fix (ADR 0156 / ASSESSMENT-METHOD §4).** A headline count computed over cells + that were never examined is not a measurement; it is an average over guesses, and publishing it is + what made every subsequent first-look read as a *reversal* rather than as *progress*. On + 2026-08-01, 76% of that day's verdict changes were cells nobody had ever opened. + + So this leads with how much of the survey is done, and reports `unverified` separately from + `pass` — which is also what ASVS asks for: a summary of **every requirement checked**, not + exceptions only. + """ + n = count(cells) + total = sum(n.values()) + decided = [c for c in cells if c.verdict in DECIDED_VERDICTS] + examined = [c for c in decided if c.last_verified] + unexamined = total - len(examined) + pct = (100.0 * len(examined) / total) if total else 0.0 + inherited = sum(1 for c in cells if c.verdict in DECIDED_VERDICTS and not c.last_verified) + + lines = [ + "", + "", + "# ASVS 5.0 L3 — current state", + "", + f"**Anchor commit:** `{anchor_sha}` · **Method:** `docs/ASVS-ASSESSMENT-METHOD.md`", + "", + "## Survey progress", + "", + f"**{len(examined)} of {total} requirements have been read against the ASVS text " + f"({pct:.1f}%).** {unexamined} have not.", + "", + "> **There is deliberately no headline score here.** A count over unexamined cells is an", + "> average of guesses. Until the baseline sweep completes, the honest number is the one above:", + "> how much of the survey is done. Verdict counts below cover **examined cells only** unless", + "> stated otherwise.", + "", + "| State | Count | Meaning |", + "|---|---:|---|", + f"| Pass | {n['pass']} | verb satisfied by a shipped default or a refusing gate |", + f"| Partial | {n['partial']} | control exists but ships off, warns, or covers part of the surface |", + f"| Fail | {n['fail']} | no implementing control in any configuration |", + f"| N/A | {n['na']} | does not apply on the declared scope, with a written rationale |", + f"| Needs review | {n['needs-review']} | examined; verdict contested or blocked on a decision |", + f"| **Unverified** | **{n['unverified']}** | **never examined — not a Pass** |", + f"| **Total** | **{total}** | |", + "", + ] + if inherited: + lines += [ + f"⚠️ **{inherited} cell(s) carry a decided verdict with no `last_verified` date** — " + "inherited from an earlier assessment and never re-read. Treat as unexamined.", + "", + ] + lines += [ + "## Open cells", + "", + "| Cell | L | Verdict | Reviewed | Residual |", + "|---|---|---|---|---|", + ] + open_states = {"partial", "fail", "needs-review"} + for c in sorted((c for c in cells if c.verdict in open_states), key=lambda c: _sort_key(c.id)): + seen = c.last_verified or "—" + lines.append(f"| {c.id} | L{c.level} | **{c.verdict}** | {seen} | {c.residual[:150]} |") + return chr(10).join(lines) + chr(10) + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Verify or render the ASVS scorecard (ADR 0156).") + ap.add_argument("--scorecard", type=Path, required=True) + ap.add_argument("--corpus", type=Path, required=True) + ap.add_argument( + "--root", type=Path, default=Path.cwd(), help="tree the evidence anchors point into" + ) + ap.add_argument("--render", type=Path, help="write the generated CURRENT.md here") + # NO --anchor-sha injected by CI. The anchor is the commit the EVIDENCE was read on — a property + # of the assessment, recorded in [scorecard].anchor_commit. Passing ${{ github.sha }} made the + # rendered file differ on every run, so the drift check could never pass: a gate that cannot go + # green is as useless as one that cannot go red, and this one shipped that way. + args = ap.parse_args(argv) + + try: + findings = verify(args.scorecard, args.corpus, args.root) + except ScorecardError as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 # could not measure — never 0, never confused with "clean" + + cells = load_scorecard(args.scorecard) + n = count(cells) + print( + f"scanned {len(cells)} cells " + f"({n['pass']} pass / {n['partial']} partial / {n['fail']} fail / {n['na']} na / " + f"{n['unverified']} unverified); " + f"verified {findings.checked_anchors} evidence anchors and {findings.checked_absences} absence claims" + ) + for p in findings.problems: + print(f" FAIL {p}", file=sys.stderr) + + if args.render and findings.ok: + anchor = load_meta(args.scorecard).get("anchor_commit", "unrecorded") + args.render.write_text(render_current(cells, anchor_sha=anchor), encoding="utf-8") + print(f"rendered {args.render}") + + return 0 if findings.ok else 1 + + +if __name__ == "__main__": # pragma: no cover + raise SystemExit(main()) diff --git a/scripts/security/crypto_inventory_check.py b/scripts/security/crypto_inventory_check.py index 70968d6..590626e 100644 --- a/scripts/security/crypto_inventory_check.py +++ b/scripts/security/crypto_inventory_check.py @@ -321,6 +321,14 @@ # the two ephemeral scan identities it provisions into a store it creates empty in a temp directory # and destroys with it. Not a key and never persisted: a checked-in constant would be strictly # weaker, and the alternative — an operator-supplied credential — is the optional escape hatch only. + # ADR 0156: SHA-256 over the OWASP ASVS corpus FILE, to pin it to the tagged v5.0.0_release + # asset. Integrity of a build input, not a security control: no secret, no key, no message + # authentication, and nothing user- or PHI-derived is hashed. It exists because the corpus was + # originally fetched from `master` (the bleeding-edge branch, where a rolling "latest" release + # republishes identical filenames) and happened to match the release by luck — so the digest is + # recorded and recomputed rather than assumed. Non-cryptographic alternatives were rejected only + # because SHA-256 is already the tree's convention for file pinning. + "scripts/asvs/scorecard.py": frozenset({"hashlib"}), "scripts/security/dast_target.py": frozenset({"secrets"}), } diff --git a/tests/test_asvs_scorecard.py b/tests/test_asvs_scorecard.py new file mode 100644 index 0000000..03a0f39 --- /dev/null +++ b/tests/test_asvs_scorecard.py @@ -0,0 +1,334 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 MessageFoundry Organization and contributors +"""ADR 0156 — the ASVS scorecard verifier. + +Every check here is proved to go RED before it is trusted green. That is not ceremony: this project +has shipped several guards that could not fail, and the defect this tool exists to catch survived +precisely because a total that closed to 345 was read as proof of completeness. + +The tool is data-free by design (ADR 0156 §7), so these tests run against fixtures in the public repo +while the vault runs the same code against the real posture data. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from scripts.asvs.scorecard import ( + Absence, + Anchor, + Cell, + Findings, + ScorecardError, + check_absences, + check_anchors, + check_completeness, + check_pinning, + corpus_digest, + count, + load_corpus, + load_scorecard, + render_current, + verify, +) + +CORPUS = {"1.1.1": 1, "1.1.2": 2, "2.1.1": 3} + + +def _corpus_file(tmp_path: Path, ids: dict[str, int] | None = None) -> Path: + p = tmp_path / "corpus.json" + reqs = [{"req_id": f"V{k}", "L": str(v)} for k, v in (ids or CORPUS).items()] + p.write_text(json.dumps({"requirements": reqs}), encoding="utf-8") + return p + + +def _scorecard_file(tmp_path: Path, body: str) -> Path: + p = tmp_path / "asvs-scorecard.toml" + p.write_text(body, encoding="utf-8") + return p + + +def _cells(*specs: tuple[str, int, str]) -> list[Cell]: + return [Cell(id=i, level=lv, verdict=v) for i, lv, v in specs] # type: ignore[arg-type] + + +# --- completeness: the check whose absence cost ten cells --------------------------------------- + + +def test_completeness_passes_when_every_corpus_id_appears_once() -> None: + cells = _cells(("1.1.1", 1, "pass"), ("1.1.2", 2, "partial"), ("2.1.1", 3, "fail")) + assert check_completeness(cells, CORPUS) == [] + + +def test_completeness_catches_a_dropped_cell() -> None: + """The 2026-08-01 defect, in miniature: a count that sums correctly while a cell is missing.""" + cells = _cells(("1.1.1", 1, "pass"), ("1.1.2", 2, "partial")) # 2.1.1 dropped + problems = check_completeness(cells, CORPUS) + assert any("have NO cell" in p and "2.1.1" in p for p in problems) + + +def test_completeness_catches_a_duplicate_cell() -> None: + cells = _cells( + ("1.1.1", 1, "pass"), ("1.1.1", 1, "partial"), ("1.1.2", 2, "pass"), ("2.1.1", 3, "pass") + ) + assert any("appears 2 times" in p for p in check_completeness(cells, CORPUS)) + + +def test_completeness_catches_an_id_that_is_not_in_the_standard() -> None: + """11.7.2 was scored against ASVS 4.0.3 V8.3.6, deleted in 5.0. This is that class of error.""" + cells = _cells( + ("1.1.1", 1, "pass"), ("1.1.2", 2, "pass"), ("2.1.1", 3, "pass"), ("8.3.6", 2, "partial") + ) + assert any( + "not ASVS 5.0.0 requirement ids" in p and "8.3.6" in p + for p in check_completeness(cells, CORPUS) + ) + + +def test_completeness_catches_a_level_that_disagrees_with_the_corpus() -> None: + cells = _cells(("1.1.1", 3, "pass"), ("1.1.2", 2, "pass"), ("2.1.1", 3, "pass")) + assert any("but the corpus says L1" in p for p in check_completeness(cells, CORPUS)) + + +# --- the count is computed, never typed ---------------------------------------------------------- + + +def test_count_is_derived_from_the_cells() -> None: + cells = _cells(("1.1.1", 1, "pass"), ("1.1.2", 2, "partial"), ("2.1.1", 3, "unverified")) + n = count(cells) + assert (n["pass"], n["partial"], n["unverified"]) == (1, 1, 1) + assert sum(n.values()) == len(cells) + + +# --- evidence anchors ----------------------------------------------------------------------------- + + +def test_anchor_resolves_when_the_token_is_present(tmp_path: Path) -> None: + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text( + "a\nb\ntls_cert_file = None\n", encoding="utf-8" + ) + cells = [ + Cell( + id="1.1.1", + level=1, + verdict="pass", + evidence=(Anchor("messagefoundry/m.py", 3, "tls_cert_file"),), + ) + ] + f = Findings() + check_anchors(cells, tmp_path, f) + assert f.ok and f.checked_anchors == 1 + + +def test_anchor_goes_red_when_the_token_is_gone(tmp_path: Path) -> None: + """The twelve-false-residuals defect: the code moved and the sentence stayed.""" + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text("nothing here\n", encoding="utf-8") + cells = [ + Cell( + id="1.1.1", + level=1, + verdict="pass", + evidence=(Anchor("messagefoundry/m.py", 1, "tls_cert_file"),), + ) + ] + f = Findings() + check_anchors(cells, tmp_path, f) + assert not f.ok and "no longer contains" in f.problems[0] + + +def test_anchor_goes_red_when_the_file_is_gone(tmp_path: Path) -> None: + cells = [ + Cell( + id="1.1.1", + level=1, + verdict="pass", + evidence=(Anchor("messagefoundry/gone.py", 1, "x"),), + ) + ] + f = Findings() + check_anchors(cells, tmp_path, f) + assert not f.ok and "does not exist" in f.problems[0] + + +# --- absence claims: the class this project is worst at ------------------------------------------- + + +def test_absence_holds_when_pattern_is_quiet_and_control_speaks(tmp_path: Path) -> None: + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text( + "class ScanRejected: pass\n", encoding="utf-8" + ) + cells = [ + Cell( + id="1.1.1", level=1, verdict="fail", absence=(Absence("clamav|clamd", "ScanRejected"),) + ) + ] + f = Findings() + check_absences(cells, tmp_path, f) + assert f.ok and f.checked_absences == 1 + + +def test_absence_is_rejected_as_BLIND_when_the_positive_control_matches_nothing( + tmp_path: Path, +) -> None: + """A grep naming the wrong token returns zero and reads exactly like proof. This is the fence.""" + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text("unrelated\n", encoding="utf-8") + cells = [ + Cell(id="1.1.1", level=1, verdict="fail", absence=(Absence("clamav", "ScanRejected"),)) + ] + f = Findings() + check_absences(cells, tmp_path, f) + assert not f.ok and "BLIND" in f.problems[0] + + +def test_absence_is_rejected_as_FALSE_when_the_thing_now_exists(tmp_path: Path) -> None: + """Five residuals of record were absence claims that had silently stopped being true.""" + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text( + "import clamd\nclass ScanRejected: pass\n", encoding="utf-8" + ) + cells = [Cell(id="1.1.1", level=1, verdict="fail", absence=(Absence("clamd", "ScanRejected"),))] + f = Findings() + check_absences(cells, tmp_path, f) + assert not f.ok and "FALSE" in f.problems[0] + + +# --- fail closed, never skip ---------------------------------------------------------------------- + + +def test_missing_scorecard_raises_rather_than_skipping(tmp_path: Path) -> None: + """15.1.3 in one line: the guards skip when the document is absent, so green proves nothing.""" + with pytest.raises(ScorecardError, match="refusing to report a pass on a missing file"): + load_scorecard(tmp_path / "absent.toml") + + +def test_missing_corpus_raises(tmp_path: Path) -> None: + with pytest.raises(ScorecardError, match="cannot check completeness"): + load_corpus(tmp_path / "absent.json") + + +def test_unknown_verdict_is_refused(tmp_path: Path) -> None: + sc = _scorecard_file( + tmp_path, '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "conditional-pass"\n' + ) + with pytest.raises(ScorecardError, match="not one of"): + load_scorecard(sc) + + +# --- end to end ----------------------------------------------------------------------------------- + + +def test_verify_end_to_end_clean(tmp_path: Path) -> None: + (tmp_path / "messagefoundry").mkdir() + (tmp_path / "messagefoundry" / "m.py").write_text("tls_cert_file = None\n", encoding="utf-8") + corpus = _corpus_file(tmp_path) + sc = _scorecard_file( + tmp_path, + f""" +[scorecard] +asvs_version = "5.0.0" +corpus_sha256 = "{corpus_digest(corpus)}" + +[[cell]] +id = "1.1.1" +level = 1 +verdict = "pass" +last_verified = "2026-08-01" +[[cell.evidence]] +path = "messagefoundry/m.py" +line = 1 +expect = "tls_cert_file" + +[[cell]] +id = "1.1.2" +level = 2 +verdict = "unverified" + +[[cell]] +id = "2.1.1" +level = 3 +verdict = "partial" +residual = "ships off" +""", + ) + findings = verify(sc, corpus, tmp_path) + assert findings.ok, findings.problems + + +def test_render_leads_with_survey_progress_not_a_headline_score() -> None: + """Phase 0: a count over unexamined cells is an average of guesses, so it is not the headline.""" + cells = [ + Cell(id="1.1.1", level=1, verdict="pass", last_verified="2026-08-01"), + Cell(id="1.1.2", level=2, verdict="unverified"), + Cell( + id="2.1.1", level=3, verdict="partial", residual="ships off", last_verified="2026-08-01" + ), + ] + out = render_current(cells, anchor_sha="deadbeef") + assert ( + "**2 of 3 requirements have been read against the ASVS text (66.7%).** 1 have not." in out + ) + assert "There is deliberately no headline score here" in out + assert "never examined — not a Pass" in out + assert "deadbeef" in out + + +def test_render_flags_a_decided_verdict_carrying_no_verified_date_as_inherited() -> None: + cells = [Cell(id="1.1.1", level=1, verdict="pass")] # decided, but never dated + out = render_current(cells, anchor_sha="x") + assert "carry a decided verdict with no `last_verified` date" in out + + +# --- the one MUST in ASVS's assessment chapter ---------------------------------------------------- + + +def test_na_without_a_rationale_is_refused(tmp_path: Path) -> None: + """Recording the reason for non-applicability is the only 'must' ASVS 5.0 states.""" + sc = _scorecard_file(tmp_path, '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "na"\n') + with pytest.raises(ScorecardError, match="requires a written rationale"): + load_scorecard(sc) + + +def test_na_with_a_rationale_is_accepted(tmp_path: Path) -> None: + sc = _scorecard_file( + tmp_path, + '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "na"\nresidual = "no WebRTC surface"\n', + ) + assert load_scorecard(sc)[0].verdict == "na" + + +def test_needs_review_is_a_valid_verdict(tmp_path: Path) -> None: + """Parking a contested cell beats forcing a premature verdict — that is what flip-flops.""" + sc = _scorecard_file(tmp_path, '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "needs-review"\n') + assert load_scorecard(sc)[0].verdict == "needs-review" + + +# --- pinning: ids re-point across ASVS versions ---------------------------------------------------- + + +def test_pinning_requires_a_declared_asvs_version(tmp_path: Path) -> None: + corpus = _corpus_file(tmp_path) + sc = _scorecard_file(tmp_path, f'[scorecard]\ncorpus_sha256 = "{corpus_digest(corpus)}"\n') + assert any("asvs_version is missing" in p for p in check_pinning(sc, corpus)) + + +def test_pinning_catches_a_corpus_that_changed_underneath_the_scorecard(tmp_path: Path) -> None: + """bare 1.2.5 is Architecture in 4.0.3 and Encoding in 5.0.0 — a moved corpus re-points ids.""" + corpus = _corpus_file(tmp_path) + sc = _scorecard_file(tmp_path, '[scorecard]\nasvs_version = "5.0.0"\ncorpus_sha256 = "0" \n') + assert any("corpus digest mismatch" in p for p in check_pinning(sc, corpus)) + + +def test_pinning_passes_when_version_and_digest_are_declared(tmp_path: Path) -> None: + corpus = _corpus_file(tmp_path) + sc = _scorecard_file( + tmp_path, + f'[scorecard]\nasvs_version = "5.0.0"\ncorpus_sha256 = "{corpus_digest(corpus)}"\n', + ) + assert check_pinning(sc, corpus) == [] diff --git a/tests/test_key_usage_scope_inventory.py b/tests/test_key_usage_scope_inventory.py index c806f72..5b4a5a1 100644 --- a/tests/test_key_usage_scope_inventory.py +++ b/tests/test_key_usage_scope_inventory.py @@ -63,6 +63,8 @@ "WebAuthn credentials": "COSE PUBLIC keys supplied by the authenticator; the engine holds no " "private half and the row already says they are verification material, not a secret", "Config fingerprint": "a keyless content hash for change attribution", + "ASVS corpus pin": "a keyless content hash over a build input (the OWASP ASVS corpus file), " + "not a key, a secret, or a message authenticator", "Engine wheel attestation": "a keyless digest over the installed distribution, verified against " "a recorded value; no key is involved on either side", "AD transport": "a TLS hop whose key material is the OS/directory trust store, not engine-held", diff --git a/tests/test_security_static.py b/tests/test_security_static.py index d1f1df2..60a5e3c 100644 --- a/tests/test_security_static.py +++ b/tests/test_security_static.py @@ -1011,6 +1011,8 @@ def test_xml_import_scanner_sees_indented_imports() -> None: #: BACKLOG #282 widens the gate itself to these roots this pin becomes redundant and should be deleted #: in favour of the gate's inventory. _CRYPTO_SITES_OUTSIDE_THE_PACKAGE = { + # ADR 0156: SHA-256 over the ASVS corpus FILE to pin it to the tagged release. No key. + "scripts/asvs/scorecard.py": frozenset({"hashlib"}), "messagefoundry_webconsole/_security.py": frozenset({"secrets"}), "tee/__main__.py": frozenset({"ssl"}), "tee/anon/keying.py": frozenset({"hashlib"}),