Skip to content

feat(assurance): claim register, assurance case and regulation mapping (T17) - #41

Open
nick-developer wants to merge 9 commits into
mainfrom
claude/project-thread-nka4jq
Open

nick-developer wants to merge 9 commits into
mainfrom
claude/project-thread-nka4jq

Conversation

@nick-developer

Copy link
Copy Markdown
Owner

Requested by Cod3r · project thread

What and why

Roadmap T17: give customers' privacy, legal and audit teams the evidence they need to approve a relsynth release. New folder docs/assurance/:

  • Claim register claims.yaml (60 claims) and checker tools/claims_check.py, run in the CI job lint. Every claim-like sentence of docs/privacy.md, docs/dp.md, docs/security.md and docs/governance/guide.md is bound to its quote and to propositions (by label, numbered as LaTeX prints them), checks, pytest nodes or measured reports. CI fails when a quote leaves its document, evidence disappears, a new claim is unregistered, a Prop. N anywhere in the docs names an undefined proposition, or a range runs backwards.
  • Assurance case: top claim, sub-claims (structure, measured privacy, DP, governance, traceability), assumptions A1–A5 and defeaters, every leaf citing a register id or proposition (also checked).
  • Regulation mapping against the primary texts: EDPB Guidelines 02/2026 v1.0 (paragraphs), CJEU C-413/23 P (paragraphs), AI Act Art. 10 as amended by Regulation (EU) 2026/1744 (Art. 10(5) moved to Art. 4a), 45 CFR 164.514 and the HHS 2012 guidance, NIST SP 800-226 and SP 800-188. ISO/IEC 27559 and 20889 are mapped by topic only (texts are sold by ISO). Rows a lawyer must decide are marked Lawyer.
  • DPIA template (GDPR Art. 35(7)) and release guidance for internal, partner and public recipients (Five Safes, approval checklist).
  • docs/assurance/evidence/: aggregate result files of T06 and T07 that claims cite (scores and intervals only, no rows or person labels; datasets all marked allowed).

Registering the claims found statements in docs/security.md that 2.0 made false (no formal DP, "privacy checks run by default", five core dependencies, one subprocess, no network access). The guide now says what the code does. A new test confines network access to the service modules that the guide names.

README: one row in the documentation map. The README claims join the register after the 2.0.0 release PR lands, because that PR rewrites the README.

How it was tested

  • ruff check src tests tools bench: all checks passed; python tools/check_spdx.py: ok
  • python tools/claims_check.py: ok: 60 claims traced, 4 documents in scope, 152 propositions
  • pytest tests/unit/test_assurance.py tests/unit/test_security.py: 57 passed. test_assurance covers the register, LaTeX counter numbering (including save-and-restore sections), each kind of drift, citations in the assurance documents, and network confinement
  • Trial merge with the 2.0.0 release branch: the checker reports one real problem there (Props. 951–902 in docs/testing.md), and the coordinator has been told
  • Live servers, science suite and real-database regressions: not affected (no change under src/); CI runs them

Statistics and privacy

  • No new statistical or privacy claim. The documents cite existing propositions and checks.
  • Synthetic draws for the same seed are unchanged (no code change under src/)

Checklist

  • Every commit follows Conventional Commits and is signed off (git commit -s)
  • CHANGELOG.md has entries under Unreleased
  • Documentation updated (docs/assurance/, docs/security.md, docs/privacy.md, README documentation map)
  • No data, credentials or synthetic data fitted to restricted data is committed (evidence files are aggregates)
  • No new dependencies

Generated by Claude Code

Customers' privacy, legal and audit teams need every claim in the
documentation traced to a proof, a test or a measured report (roadmap
T17). tools/claims_check.py checks docs/assurance/claims.yaml: quotes
must still be in their documents, evidence must resolve (proposition
labels numbered as LaTeX prints them, check ids, pytest nodes, report
files), claim-like sentences of the documents in scope must be
registered or exempted with a reason, and Prop. N references in any
Markdown document must name a defined proposition.

tests/unit/test_assurance.py runs the check, tests each kind of drift
it must catch, and confines network access to the service modules the
security guide names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwE13trs5MdpBoE4YyJ
Signed-off-by: Claude <noreply@anthropic.com>
… governance guides

60 claims of docs/privacy.md, docs/dp.md, docs/security.md and
docs/governance/guide.md, each traced to propositions, checks, tests or
the aggregate result files now kept in docs/assurance/evidence/.

Registering them found statements in docs/security.md that 2.0 made
false: 'formal differential privacy is not part of v1.0', 'privacy
checks run by default' (the distance guards are not a privacy check,
docs/privacy.md), five core dependencies (seven), subprocess used once
(four places) and no network access (the service talks to the OIDC
provider and Vault). The guide now says what the code does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwE13trs5MdpBoE4YyJ
Signed-off-by: Claude <noreply@anthropic.com>
…d release guidance

The deliverables of roadmap T17 for customers' privacy, legal and audit
teams:

- assurance-case.md: top claim, sub-claims for structure, measured and
  formal privacy, governance and traceability, assumptions A1-A5 and
  defeaters, every leaf citing the claim register or a proposition;
- regulation-mapping.md: EDPB Guidelines 02/2026 (v1.0, paras. 6-103)
  and EDPS v SRB (C-413/23 P, paras. 71-112), AI Act Art. 10 as amended
  by Regulation (EU) 2026/1744 (Art. 10(5) moved to Art. 4a), 45 CFR
  164.514 and the HHS 2012 guidance, NIST SP 800-226 and SP 800-188,
  and ISO/IEC 27559 and 20889 by topic (texts not available), with
  rows a lawyer must decide marked as such;
- dpia-template.md following GDPR Art. 35(7);
- release-guidance.md for internal, partner and public recipients,
  with the Five Safes and an approval checklist.

The claim checker now also fails when these documents cite a claim id,
a proposition label or a check that does not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwE13trs5MdpBoE4YyJ
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwE13trs5MdpBoE4YyJ
Signed-off-by: Claude <noreply@anthropic.com>
…ward ranges

Sections may name a fixed number in the title ([...; Prop.~950]) while
LaTeX prints the running number; references to the named number are
valid. A reference such as 'Props. 951-902' is now reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwE13trs5MdpBoE4YyJ
Signed-off-by: Claude <noreply@anthropic.com>
@nick-developer nick-developer self-assigned this Oct 31, 2026
test_settings_reference_is_up_to_date fails on main since the runtime
settings (#35) landed after the reference was generated (#39); the
unit jobs of every PR are red until it is regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwE13trs5MdpBoE4YyJ
Signed-off-by: Claude <noreply@anthropic.com>

Copy link
Copy Markdown
Owner Author

The unit jobs failed on every platform with one test: tests/unit/test_settings.py::test_settings_reference_is_up_to_date. The failure doesn't come from this PR. This PR doesn't touch settings, and the same test fails locally on main (a6c4357), because docs/settings-reference.md is missing the five runtime.* settings. Commit 49662bb regenerates the file with python -m relsynth.settings > docs/settings-reference.md, and tests/unit/test_settings.py now passes 34/34. The 2.0.0 release branch regenerates the same file, so whichever PR merges second may need the file regenerated again.


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

After the settings fix, unit (macos-latest, 3.12) failed in a different test, tests/unit/service/test_service_api.py::test_no_source_row_is_persisted. That test is not part of this PR's change, which touches no service code.

The cause is a false positive in the test. Line 120 scans every raw file under the data folder for the 3-byte marker b"Qzx", and that includes the encrypted artefacts (*.enc). Ciphertext is effectively random bytes, so a file of n bytes contains a given 3-byte string with probability of about n / 2²⁴. For artefacts of a few hundred kilobytes, that is a few percent per run. In this run the match was in art_….enc. The decrypted check on line 117 passed.

Proposed fix (not pushed here, because it belongs to the service tests). Line 117 already checks the decrypted content, so skip the ciphertext in the raw scan:

for f in Path(settings.data_dir).rglob("*"):
    if f.is_file() and f.suffix != ".enc":      # ciphertext is checked decrypted above
        assert SECRET_MARK.encode() not in f.read_bytes(), f

A second option is a marker of 16 or more bytes, which removes the chance collision. I can't re-run the job from here, so a maintainer re-run should go green. I'll keep watching this PR.


Generated by Claude Code

test_no_source_row_is_persisted searched every file under the data
folder, including the encrypted artefacts, for a 3-byte marker. Random
ciphertext contains it by chance (macOS unit job on #41); the
artefacts are already checked decrypted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwE13trs5MdpBoE4YyJ
Signed-off-by: Claude <noreply@anthropic.com>
…-nka4jq

Signed-off-by: Claude <noreply@anthropic.com>

# Conflicts:
#	.github/workflows/ci.yml
#	CHANGELOG.md
…ing on Windows

Put README.md in the claim register's scope with 36 entries (RM-01..RM-36).
A README sentence that sums up a guide cites that guide's claim with the new
{claim: ID} evidence kind, which is as strong as the claim it cites.

README fixes found while registering: 'No row is copied' becomes 'Rows are
drawn from a fitted model; only tables you declare mode: copy are copied';
'Privacy checks run by default' becomes copy guards; C700-C706 do not exist,
the 2.0 checks are C600-C606 and the generator propositions are unit-tested.

The checker tests wrote their fixture files in the platform encoding, which
broke on Windows (cp1252); they now write UTF-8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSVqwE13trs5MdpBoE4YyJ
Signed-off-by: Claude <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants