ADR 0156 — ASVS scorecard as data: derived count, verified evidence anchors, fail-closed drift gate - #120
Open
wshallwshall wants to merge 9 commits into
Open
ADR 0156 — ASVS scorecard as data: derived count, verified evidence anchors, fail-closed drift gate#120wshallwshall wants to merge 9 commits into
wshallwshall wants to merge 9 commits into
Conversation
…sting 23M tokens The ASVS score is maintained as prose and nothing checks it. One re-anchoring session on 2026-08-01 produced, in a single day: 6 re-derivations of the headline count from scratch 12 residuals of record factually FALSE at HEAD 5 of those were ABSENCE claims that had silently stopped being true 10 cells missing from an enumeration called "arithmetic-checked and complete" 5 documents asserting a superseded count as current 4 register-vs-scorecard contradictions ~23M subagent tokens spent verifying Most of that was not discovery. It was re-deriving facts already known but never durably recorded, then re-verifying them because the recorded version had gone quietly false. The dropped-ten defect is the instructive one: it survived because the arithmetic closed to 345 and closure was read as proof. Closure only proves the four buckets sum -- not that every cell landed in one. No self-consistent count can detect an absent cell. Decision: one [[cell]] record per requirement for all 345; the count is COMPUTED so no document can state one; a test asserts every corpus id appears exactly once; each cell's evidence anchor names a TOKEN that CI asserts still resolves, so code movement reds a test instead of rotting a sentence; an absence claim must record the search AND a positive control that must still hit; `unverified` is a first-class verdict so inherited-vs-verified Pass is countable; and the verifier fails closed rather than skipping. Tool and schema here (public CI, fixture data); real scorecard in the vault with a vault-CI job. That placement closes ASVS 15.1.3 as a by-product -- six *_doc_drift modules currently assert against documents that `git ls-files docs/security/` shows are absent from the tree where CI runs, so they are inspection-only theatre. States its costs plainly, including that it does NOT make the score correct -- only consistent, derived and drift-detecting. Number allocated atomically via scripts/coord/alloc.ps1; index row in this commit.
…il-closed (ADR 0156) The tool half of ADR 0156. Stdlib only (tomllib/json/re), no new dependency, and deliberately data-free: it takes the scorecard and the ASVS corpus as paths, so this repo unit-tests it against fixtures while the vault runs the same code against the real posture data. What it enforces, each aimed at a defect observed on 2026-08-01: check_completeness -- every corpus id appears EXACTLY ONCE and nothing outside the corpus appears at all. This is the check whose absence cost ten cells: an enumeration closed to 345 and closure was read as proof. Closure only proves the four buckets sum, not that every cell landed in one. count() -- computed, never typed. Five documents asserting three counts is what this replaces. check_anchors -- each cell's evidence names a TOKEN that must still resolve within a window of the recorded line. Code movement reds a test instead of rotting a sentence. That is the twelve-false-residuals defect. check_absences -- an absence claim is admissible ONLY with a positive control that must still match. A grep naming the wrong token returns zero and reads exactly like proof; five residuals of record died that way. If the control goes quiet the search is BLIND and the claim is void regardless of the pattern. unverified is a first-class verdict, so an inherited Pass cannot hide inside a headline. The renderer reports verified-vs-inherited separately. Fail closed: a missing scorecard EXITS 2 (could not measure), never 0. That distinction is the whole of ASVS 15.1.3, where six *_doc_drift modules skip because their target is not in the tree where CI runs. 17 tests, and every check is proved to go RED before it is trusted green -- dropped cell, duplicate cell, an id ASVS retired, a level disagreeing with the corpus, a moved token, a deleted file, a blind absence search, an absence that became false. Exit codes verified by hand: 0 clean / 1 findings / 2 missing. ruff clean, ruff format clean, mypy clean.
…e was false Accepted 2026-08-01, built and merged the same day. Section 7 originally proposed "a vault CI job runs the verifier against the real data", reasoning from the vault having ci.yml, tests/ and pyproject.toml. Open question 1 asked whether that CI actually executes. It does not. The actions API reports EVERY vault workflow as disabled_manually -- CI, Security, CodeQL, backlog-hygiene, release. Last run 2026-07-27, and the two vault PRs merged on 2026-08-01 both merged with zero checks. The estate was switched off at the cutover to avoid duplicating public CI; that decision stands and is not being reversed here. A CI-only design would have shipped dead. Amended to what was actually built: a vault pre-commit hook (works today, fires at authoring time, when the drift is introduced) plus one narrow new workflow -- new workflows are active by default even though the estate is disabled, so it runs without resurrecting any of it. The lesson is this ADR's own thesis applied to itself: section 7 was a confident, plausible statement about system state that nobody had checked. It was caught only because ratification was gated on answering the open question rather than on the argument reading well.
…tuted away The row was written through a shell double-quoted string, so bash treated `disabled_manually` as a command substitution and replaced it with the empty string. The row shipped reading "every vault workflow (last run 2026-07-27...)" -- the load-bearing word silently deleted. Fitting failure mode for this ADR: the sentence still parsed, still read as plausible, and stated nothing. Caught by grepping for the token rather than for the sentence around it.
wshallwshall
enabled auto-merge (squash)
August 1, 2026 14:10
… a spec
Grounded in a deep-research pass over OWASP's own assessment chapter, NIST
OSCAL, and the one comparable OSS tool. Three findings changed the design.
FINDING 1: "Partial" is not an ASVS verdict. ASVS 5.0 defines exactly three --
verified, exception, and non-applicable-with-rationale. The strings "partially
implemented" and "not implemented" appear NOWHERE in it, and the only prominent
OSS ASVS tool declines to model partial too. Our three-way rubric was a local
extension we invented and never wrote down, which is precisely why 11.7.1, 3.7.3
and 5.4.3 each changed verdict in one day: two assessors, two unwritten rules.
-> docs/ASVS-ASSESSMENT-METHOD.md now states the decision procedure as an
ORDERED rule list with the seven real disputes as worked examples, so the
next assessor reaches the same answer. ASVS's own stability mechanism is a
disclosed repeatable method, not a rubric -- this is that disclosure.
FINDING 2: recording the reason for non-applicability is the ONE "must" in the
whole assessment chapter (everything else is "should"). So an `na` with no
rationale is not a lax entry; it is the single thing the standard requires,
omitted. load_scorecard() now REFUSES it.
FINDING 3: ASVS requirement ids are NOT stable across versions -- bare 1.2.5 is
Architecture in 4.0.3 and Encoding in 5.0.0 -- and master is the bleeding-edge
branch where a rolling "latest" release republishes identical filenames. Our
corpus was fetched from master and turned out byte-identical to the tagged
release: luck, not method. Now pinned by [scorecard].asvs_version plus a
corpus_sha256 the verifier recomputes and compares.
PHASE 0 -- the renderer no longer publishes a headline count. It leads with how
much of the survey is done. 76% of that day's verdict changes were cells nobody
had ever opened, so a count computed over unexamined cells is an average of
guesses, and publishing it is what made every first look read as a REVERSAL
rather than as PROGRESS. Reporting every requirement including `unverified` is
also what ASVS asks for -- a summary of all requirements checked, not exceptions
only.
Also adds `needs-review` (examined, contested, parked deliberately) and
`reviewed_by`. Separating "undecided" from "decided-and-failing" is a cheaper
stability primitive than inventing a grade -- borrowed from the one OSS tool in
this space, which pairs it with reviewed_by/reviewed_at.
OSCAL was evaluated and rejected for this use: ASVS has no OSCAL catalog
(proposed 2020, closed 2021 in a backlog sweep, still none in the 2026 release),
and assessment-results mandatorily imports an assessment-plan which mandatorily
imports an SSP -- a chain a product with no operated system has no source for.
Recorded in the method doc, with Component Definition and CycloneDX declarations
noted as the two unexplored alternatives.
24 tests (17 + 7 new), ruff clean, mypy clean.
The workflow rendered with --anchor-sha ${{ github.sha }} while the committed
entry point carried the code anchor, so the rendered file differed on the anchor
line on EVERY run. `git diff --exit-code` therefore failed unconditionally.
A gate that cannot go green is as useless as one that cannot go red, and this
session has spent all day on the second kind. I shipped the first kind, in the
tool built to prevent exactly this, and it went red on its own first PR.
The anchor is the commit the EVIDENCE was read on -- a property of the
assessment, not of the CI run -- so it belongs in [scorecard].anchor_commit and
is rendered from there. The render is now deterministic: running it twice
produces identical bytes, and the drift check becomes meaningful, catching a
scorecard edit that was not re-rendered.
The ASVS 11.1.3 discovery gate caught scripts/asvs/scorecard.py using hashlib without an inventory entry. The gate is right: it is a discovery gate for exactly this, and it fired on the first commit that introduced new crypto usage. Declared in both places the convention requires -- the scanner INVENTORY and ASVS-L2-PHASE0-CHANGES.md section 4 -- with what it is and, more importantly, what it is NOT: SHA-256 over the ASVS corpus FILE, integrity of a build input, no secret, no key, no message authentication, nothing user- or PHI-derived. Recorded why it exists rather than just that it does: the corpus was originally fetched from master, where a rolling "latest" release republishes identical filenames, and matched the tagged v5.0.0_release asset only by luck. Since ASVS requirement ids are not stable across versions -- bare 1.2.5 is Architecture in 4.0.3 and Encoding in 5.0.0 -- a corpus that moves silently re-points every id in the scorecard, so the digest is now checked rather than assumed.
…ries The crypto inventory is stated in FOUR places, not the two I updated: scripts/security/crypto_inventory_check.py INVENTORY (done) docs/ASVS-L2-PHASE0-CHANGES.md section 4 (done) tests/test_key_usage_scope_inventory.py key/not-key split (missing) tests/test_security_static.py out-of-package map (missing) Both now carry it, classified as NOT key material with the reason: a keyless content hash over a build input, not a key, a secret, or a message authenticator. Worth recording rather than quietly fixing. This is the same defect class the whole ASVS effort has been about -- one fact stated in four places, so updating two leaves the other two stale -- and the project's own doc standard says state a load-bearing fact ONCE and link to it. Here the redundancy is load-bearing on purpose (each registry asks a different question: what uses crypto, why, does it hold a key, and is it inside the package), so the fix is not to collapse them. But nothing tells an author that adding one row obliges four edits, and I found out by reding CI twice. 102 tests pass across all five crypto/scorecard suites.
wshallwshall
disabled auto-merge
August 1, 2026 20:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ASVS score is maintained as prose and nothing checks it. One re-anchoring session on 2026-08-01 produced, in a single day:
Most of that wasn't discovery. It was re-deriving facts already known but never durably recorded, then re-verifying them because the recorded version had gone quietly false.
The instructive one is the dropped ten: it survived because the arithmetic closed to 345 and closure was read as proof. Closure only proves the four buckets sum — not that every cell landed in one. No self-consistent count can detect an absent cell.
What this adds
scripts/asvs/scorecard.py— stdlib only (tomllib/json/re), no new dependency, and deliberately data-free: it takes the scorecard and the ASVS corpus as paths, so this repo unit-tests it against fixtures while the vault runs the same code against real posture data.unverifiedis a first-class verdictOn absence specifically: a grep naming the wrong token returns zero and reads exactly like proof. That's how five residuals stayed false for weeks. So the schema stores the search, not just its conclusion — and if the positive control goes quiet, the claim is void regardless of what the pattern returns.
Tests
17 tests, and every check is proved to go RED before it is trusted green — dropped cell, duplicate cell, an id ASVS retired in 5.0, a level disagreeing with the corpus, a moved token, a deleted file, a blind absence search, an absence that became false.
Exit codes verified by hand: 0 clean · 1 findings · 2 could-not-measure.
ruff check/ruff format --check/mypyall clean.§7 was amended at ratification — the premise was false
The ADR originally proposed "a vault CI job runs the verifier against the real data." Open question 1 asked whether that CI actually executes. It does not — every vault workflow is
disabled_manually, last run 2026-07-27, and both vault PRs that day merged with zero checks.A CI-only design would have shipped dead. Built instead as a vault pre-commit hook plus one narrow new workflow (new workflows are active by default even though the estate is off).
That §7 was a confident, plausible, unchecked claim about system state is this ADR's own thesis applied to itself. It was caught only because ratification was gated on answering the open question rather than on the argument reading well.
What this does not do
It does not make the score correct — only consistent, derived and drift-detecting. A wrong verdict recorded carefully is still wrong; adversarial verification remains the only cure for that, and this ADR doesn't replace it.
The companion vault PR carries the real scorecard and the enforcement.
🤖 Generated with Claude Code