Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 81 additions & 1 deletion docs/ASVS-ASSESSMENT-METHOD.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ These are the real disputes. They are here so the next assessor reaches the same
|---|---|---|
| **5.4.3** | `fail` | Rule 3. A scan *hook* exists but its only shipped implementation is `_no_scan`, and there is **no configuration key at all** — an operator must **author** the scanner. Supplying a control is not configuring one. |
| **15.2.5** | `partial` | Rule 5, **not** rule 3. `[sandbox].mode` ships `off`, but `subprocess` mode is real and was verified by executing it. A working control that ships off. |
| **11.7.1** | `fail` | Rule 3. The read-out is all-`None` off Linux and the only refusing branch keys on a **TOML declaration**, never on the measured property. Zero percent of the verb is satisfied by shipped code. |
| **11.7.1** | `na` | **Rule 1 — the hardest call in this table, and it moved.** The verb is *"full memory encryption is in use"*: a property of the **CPU, firmware and hypervisor**, not of the three software artifacts in §2. Outside the declared scope, so rule 1 fires before rule 3 is ever reached. **The objection this has to answer, because it is a good one:** ADR 0152's rungs 1–2 *do* ship in-engine, so the engine is not silent on this cell. But that code **reports on and gates against** the platform property — it never provides it. Reporting is not implementing (§2's first guard). ⚠️ Previously scored `fail` under rule 3 and cited here as the worked example of one. That reading was not wrong on its own terms; it answered *"does code implement the verb"* without first asking *"is the verb's subject in scope"*, and rule 1 runs first. **This cell has moved four times in eighteen days — it is CLOSED by owner decision (2026-08-02); do not re-derive it.** ⛔ It buys **no** Level 3 claim: see §2.1. |
| **3.7.3** | `fail` | Rule 3. One off-site navigation, a bare 303, no interstitial and no cancel. `oidc_enabled=False` removes the **trigger**, not a control. |
| **10.5.5** | `na` | Rule 1. The requirement is conditional — *"**when using** OIDC back-channel logout"* — and the precondition is false and unreachable by configuration. **Building it would create applicability.** |
| **12.2.2** | `na` | Rule 1. No external-facing services on the declared scope. *Also a scoping error worth remembering: this row spent months scoring 12.3.1's verb.* |
Expand All @@ -86,6 +86,86 @@ Per ASVS's guidance, stated as what **is** included rather than what is excluded
**Included:** the MessageFoundry engine, the web console, and the IDE extension, assessed as source,
at a named commit, against **all 345 ASVS 5.0.0 requirements** at **Level 3**.

**The boundary that phrase implies, stated explicitly because it decides verdicts.** The subject of
this assessment is those three **software artifacts**. It is not the host, the hypervisor, the CPU, the
firmware, or the network the operator deploys onto. Where a requirement's **verb** names a property of
that substrate rather than of the software, the requirement is outside the declared scope and takes
`na` under rule 1 — with the rationale written, as always.

**This is ASVS's own principle, not a local invention.** Verbatim from `0x03-What-is-the-ASVS.md` at the
`v5.0.0` tag:

> "Conversely, ASVS generally excludes requirements that are not directly relevant to the application or
> **where configuration is outside the application's responsibility**. For example, DNS issues are
> typically managed by a separate team or function."

⚠️ **Do not over-read that, and do not reach for the fork clause to do this job.** The same chapter says
organizations are *"strongly encouraged to create an organization- or domain-specific fork that adjusts
requirements"* — but its worked examples of omission are **technology-not-used** (*"omitting irrelevant
sections (e.g., GraphQL, WebSockets, SOAP, if unused)"*), which is the functionality-based shape, and
**forking changes what you are claiming conformance TO**: your tailored ASVS, not stock ASVS 5.0. This
project does **not** fork. It applies rule 1 against a positively-declared scope, which is narrower,
cheaper to defend, and leaves the conformance target unchanged.

Two guards on that boundary, because it is exactly the kind of clause that grows to swallow
inconvenient cells:

* **Shipping code that *reports on* a platform property is not the same as *providing* it, and neither
direction is decisive on its own.** A cell does not become in-scope merely because the engine
observes the substrate; nor does it leave scope merely because the substrate is involved. Ask what
the **verb** requires to be true, and of what.
* ⛔ **This boundary does not shrink the conformance claim's denominator.** A requirement excluded here
is still a requirement OWASP assigns to Level 3. See §2.1.

### 2.1 What an out-of-scope cell does NOT buy

**It does not preserve an unqualified Level 3 claim.** Two facts, both from ASVS 5.0.0 itself:

* **4.0's clause that an organization excluding requirements "may still claim full ASVS compliance"
was DROPPED in 5.0.** The 5.0 text says only that non-applicability must be noted in the report.
There is no longer any standard text saying a documented exclusion preserves a compliance claim, and
a rationale that cites the older wording is citing a superseded standard.
* **OWASP does not certify anyone — but it does retain normative authority over the requirement SET.**
It assigns each requirement to a level. So a Level 3 claim that silently omits a requirement OWASP
places at Level 3 is non-conformant **on OWASP's own terms**, regardless of how well-argued the
exclusion is.

**Therefore:** scoping a cell out is a statement about *what was assessed*, never a statement that the
level was achieved anyway. Any published attestation must say which requirements were excluded, or say
something weaker than "verified at Level 3". Writing `na` in the record and "Level 3 verified" in a
brochure is the failure mode this section exists to prevent.

### 2.2 A count movement is not a posture movement — read the counts accordingly

**The single most misreadable thing this method produces is a change in the bucket totals.** Counts move
for four different reasons, and only one of them means the software got better:

| The count moved because… | Did the posture improve? | What actually happened |
|---|---|---|
| A control was **built or turned on by default** | **Yes** | The verb is now satisfied by shipped code |
| A cell was **read for the first time** (`unverified` → anything) | **No** | The survey advanced. A cell moving `unverified` → `pass` is a *discovery*, not an improvement |
| A **scope boundary was stated** (→ `na`) | **No** | The requirement left the denominator. Identical code, smaller question |
| A **rule was applied more carefully** (re-grade in either direction) | **No** | The assessment got more accurate. Some of these move *down* |

**The worked example, and it is recent.** On 2026-08-02 the fail count went **3 → 2** when 11.7.1 was
scoped out under rule 1. **Zero lines of engine code changed.** A reader comparing only the fail counts
across those two days would conclude a defect was fixed; nothing was. The rationale is on the cell and
the boundary is in §2, but neither is visible to someone reading a total.

**Two obligations follow, and they are cheap:**

1. **Never report a bucket total as a trend without naming which of the four causes moved it.** "Fails
went from 3 to 2" is not a finding. "One cell was scoped out; no code changed" is.
2. **When a count improves, state what would have had to happen for it to mean an improvement, and
whether that happened.** This is the same discipline as a negative control: a number that can only
go one way is not measuring anything.

⚠️ **This cuts against us more often than for us, which is why it is written down.** The survey is
incomplete, so most future movement will come from cause 2 — cells being read for the first time — and
the aggregate will get *worse* before it gets better as `unverified` cells resolve into real verdicts.
**That is the survey working, not a regression**, and it should be reported as such rather than
defended against.

**The configuration assessed** — one posture, not a matrix:

> On-premises single hospital · private network, never internet-facing · SQL Server store · operator
Expand Down
11 changes: 10 additions & 1 deletion docs/Secure_Development_Standards.md
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,16 @@ The project maintains a current evidence set so any claim is backed:
- **Per-project applicability profile** (Appendix A and onward).
- A **claims register** recording each published claim, its wording, and the evidence behind it.

**Attestation posture.** The software is self-attested as NIST SSDF–aligned, tested per NIST SP 800-115, verified against OWASP ASVS 5.0 Level 3, and built to support HIPAA-compliant deployment (controls mapped to NIST SP 800-66 Rev. 2). Third-party validation of the SSDF attestation and the ASVS 5.0 Level 3 assessment raises the weight of these claims. **Attestations are published with releases** so adopters can rely on them; each adopter still performs its own deployment risk assessment (§7.3). None of these is a NIST certificate; displayable certificates (SOC 2, ISO 27001, HITRUST) are a separate, organization-level track.
**Attestation posture.** The software is self-attested as NIST SSDF–aligned, tested per NIST SP 800-115, **assessed against** OWASP ASVS 5.0 **using Level 3 as the target** — an assessment **in progress**, not a completed verification — and built to support HIPAA-compliant deployment (controls mapped to NIST SP 800-66 Rev. 2). Third-party validation of the SSDF attestation and of the ASVS assessment raises the weight of these claims.

> **The ASVS wording above changed on 2026-08-02, and the previous wording was wrong.** It read *"verified against OWASP ASVS 5.0 Level 3."* Three facts make that unsupportable, and they are stated here rather than quietly corrected because the claim was **published**:
> 1. **The survey is incomplete.** A minority of the 345 requirements have been read against the ASVS text at a known commit; the remainder are recorded as *unverified*, which is explicitly **not** a pass. "Verified" asserted a completed verification that had not occurred.
> 2. **Open requirements exist**, including at Level 2 — so the claim was not rescuable by narrowing it to a lower level.
> 3. **At least one Level 3 requirement is scoped out** as a hosting-platform property outside the assessed software. Under ASVS 5.0 that does **not** preserve a Level 3 claim: 4.0's "may still claim full ASVS compliance" clause was **dropped** in 5.0, and OWASP retains normative authority over which requirements sit at which level. A Level 3 claim omitting a Level 3 requirement is non-conformant on OWASP's own terms.
>
> **No accredited Level 3 pathway exists to appeal to.** OWASP certifies no vendor or software and states that any trust mark claiming ASVS compliance is not officially endorsed by it; the one ASVS-based accreditation scheme accredits testing *firms*, and its published scope is Levels 1–2. So "self-attested" is the only honest register available here, and this section now uses it.
>
> Current status is held in the project's private assessment record, which is the count of record; **no figure is restated here**, deliberately, so this page cannot go stale against it. **Attestations are published with releases** so adopters can rely on them; each adopter still performs its own deployment risk assessment (§7.3). None of these is a NIST certificate; displayable certificates (SOC 2, ISO 27001, HITRUST) are a separate, organization-level track.

---

Expand Down
Loading