Codex/epistemic consolidation - #64
Merged
Merged
Conversation
Measures cc-framework against the standard set by the flagship sibling repository ghost-ark, records what was found, and plans the work to close the gap. Scope is cc-framework only; no change is proposed to any other repository. The organizing finding: the mathematics is exact and nobody can use it. The LP recovers Frechet-Hoeffding to 1.11e-16 worst case and reproduces the correlation cliff in 20s, yet three sibling projects reimplemented the calculus rather than depend on this repository. One wrote down why -- the most discoverable composition entry point imposes an ROC/detector ontology on events that are deterministic predicates, so a consumer read the source and wrote 214 lines of JavaScript instead. Adds under docs/upgrade/: - BASELINE_MEASUREMENTS.md every number with the command that produced it - FINDINGS_REGISTER.md 20 findings, 5 at severity S1 - COMMITTEE_SCORECARD.md 12 dimensions, each naming what moves it - VERIFICATION_LADDER.md tiers, and what each does not establish - DOWNSTREAM_CONTRACTS.md contracts C1-C7 for the three consumers - EPISTEMIC_UPGRADE_PLAN.md ten workstreams with acceptance gates Measured baseline at 3e22c39: 689 tests pass but only after installing three dependencies the [test] extra does not declare; coverage 69.91% with the report CLI at 0% and core/stats.py at 38%; mypy reports 279 errors in 47 files at its own declared strict scope while CI enforces 7 files; no claim scanner and no coverage gate run anywhere. Also adds scripts/canonicalization_probe.py, an adversarial census over the canonicalization kernel every signed artifact routes through, modelled on ghost-ark's E1 provenance-kernel census. It finds two unintended-kernel classes: two Unicode-distinct keys silently become one, with no exception raised, at top level and nested. Widening the corpus from six hand-probed classes to eleven surfaced the nested case, reproducing E1's lesson that breadth finds more defects. Four positive controls pass, so strictness is not the explanation. The probe also records that five of six number forms diverge from RFC 8785, which makes an independent non-Python verifier impossible until the profile is decided. Provenance of the corpus is census: exact counts, no confidence intervals. The probe exits non-zero while any unintended-kernel class remains, so the S1 findings are falsifiable rather than asserted. Verified: full suite 689 passed / 9 skipped; ruff check and format clean on the new script; all internal document links and anchors resolve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VnVFub3xq2vwPjjz3MUqZV
…greement Delivers upgrade workstream W5. Consumability 2.6 -> 6.8, cross-implementation agreement 0.0 -> 6.5, on delivered evidence. The excavation found that three sibling projects reimplemented this repository's calculus rather than depend on it, and one wrote down why: cc.core.composition_theory operates on ROC point sets and bounds the Youden J statistic, which presumes each guardrail is a classifier with a threshold and an operating curve. Their controls are deterministic refusal rules. They wrote 214 lines of JavaScript instead. What ships: - src/cc/compose/ Named marginals in, sharp interval out. No ROC, Youden, TPR, FPR, threshold or operating-point concept anywhere in the public signature, pinned by a test that greps for that vocabulary so the surface cannot drift back. Detectors reach it through marginal_from_operating_point, which points inward. The result carries the independence baseline, the binding event, the marginal provenance and the non-claims, so a bound over assumed rates cannot serialize without the word "assumed" attached. - conformance/cc-kernel-v1/ 24 accept and 8 reject cases with pinned exact values, a 256-line normative SPEC.md, and a manifest with digests and a 1e-12 tolerance. Every accept case is cross-checked against the finite-atom LP at generation time; the build refuses to write a case the closed form and the LP disagree on. - verifiers/node/cc_compose_verify.mjs A zero-dependency Node implementation written from SPEC.md rather than from the Python. 24/24 accept, 8/8 reject, and it reproduces an external consumer's published numbers. - scripts/differential_compose.py Randomized differential fuzz over both implementations, agreeing on the answer or on the refusal. 23,000 cases across six seeds, zero disagreements. - src/cc/cli/guard.py + cc-guard The inference guards as a stdin/stdout JSON subcommand and as a pure-data decision table, so a non-Python caller needs no Python process. A downstream project asked in writing to route through the post-selection refusal and could not; now it can. Tests assert the table matches the code rule by rule, and that cc-guard agrees with cc.kernel.cliff.cliff_certificate in both directions. - tests/acceptance/ The gate. An external consumer's published four-control result reproduced from cc.compose: interval [0, 0.01], independence baseline 1.2e-5, understatement factor 833x, all three published scenarios, and their sensitivity finding that improving a weak control moves the upper bound by 0.00pp. The fuzzer found a real bug on its first run: countermonotone dependence with exactly one event, where the Python raised IndexError and the Node silently returned NaN. Both were wrong, differently, and the curated corpus had not thought to ask. Both fixed and pinned as reject-countermonotone-one-event. Also fixes finding F-01: build, setuptools and wheel added to the [test] extra, with a CI job that installs .[test] specifically so the extra strangers are told to use is the extra that is tested. Corrects finding F-12, which was wrong. It claimed the claim-boundary manifest was validated by nothing; the original grep covered .github/ and Makefile but not tests/, where the validator is in fact called, and it already checks that every supporting_files path exists. The correction is recorded in place rather than edited away, and the enforcement score is revised from 2.1 to 4.0. Two narrow gaps remain. Honest limits, stated in the artifacts themselves: both implementations were authored in the same project, so a wrong specification yields two implementations wrong together. This is a differential-testing instrument, not an independent replication. The one genuinely non-same-author check is a single external oracle on a single scenario family. Neither the corpus nor the fuzzer covers the constrained LP path. No downstream project has adopted this yet. Verified: suite 689 -> 815 passing, 0 failing, 9 skipped. Coverage 69.91% -> 70.13%; cc/compose/_bounds.py 97.09%, cc/cli/guard.py 97.56%. Closed form agrees with the LP to 3.3e-16 over 600 randomized cases. Ruff check and format clean on all new files. All internal document links and anchors resolve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VnVFub3xq2vwPjjz3MUqZV
Delivers upgrade workstream W3. Canonicalization and provenance integrity 3.8 -> 8.2. All five findings in this dimension are fixed: F-03, F-04, F-05 at severity S1, and F-06, F-07 at S2. A receipt identifies a document only up to the kernel of its canonicalizer. The old kernel had two members it should not have had. It applied NFC normalization to every mapping key and wrote the results into a fresh dict, so two byte-distinct keys sharing a normal form silently became one, with no exception, at top level and nested. The receipt then attested faithfully to a document with a field missing. It also emitted Python repr number forms, which diverge from RFC 8785 on five of six probed forms, so no verifier in another language could agree on the bytes. cc.canonical.v2 implements RFC 8785: ECMAScript Number::toString number forms, UTF-16 code-unit key ordering, no normalization, and unpaired surrogates refused. 15 of 15 probed number forms conform. The remedy is not the one the plan proposed. The plan said normalization must detect collision and fail closed. The better fix, taken instead, was to stop normalizing: RFC 8785 makes normalization the producer's responsibility, and a canonicalizer that mutates content is not a canonicalizer. Two keys differing in Unicode form are two keys, which is what JSON says they are, so there is nothing left to collide. The detection survives as assert_no_confusable_keys, an opt-in producer lint deliberately kept off the hash path -- the original mistake was exactly that a content-altering rule lived inside canonicalization. One declared narrowing of RFC 8785: integers beyond the IEEE-754 safe range are refused rather than emitted, because JCS is defined over doubles and Python's int is unbounded. Emitting bytes that cannot survive a round trip through a conforming parser would produce a receipt nobody else can check. The narrowing is documented rather than left as an undocumented difference. cc.canonical.v1 is retained read-only and remains bug-compatible. Verification dispatches on the profile each receipt declares, so pre-migration receipts stay verifiable; recomputing one under RFC 8785 would report a mismatch for a report that is intact, which is the opposite of what a receipt is for. A test asserts v1 still carries its defects, so it cannot be silently "fixed" out from under history. strict_json_loads refuses repeated keys at any nesting depth and is wired into the report CLI reader, the claim-governance report and artifact readers, and the Merkle transparency-log line reader. Migration was verified, not assumed. Every regenerated capsule artifact was diffed with hashes, hash-derived ids, and the profile identifier scrubbed; all eleven were byte-identical under that scrub. No content, no claim text, and no non-claim changed -- only hashes moved. Two existing tests were rewritten because they asserted the defect as a feature. test_merkle_log_normalizes_unicode_nfc_like_reporting required that a composed and a decomposed spelling produce the SAME leaf hash; under RFC 8785 they are two strings and must not. Its replacement asserts the corrected behavior and records why. One probe intent was corrected: int-vs-float-same-value moved from distinct to equivalent. The original declaration described Python's type system rather than JSON's, which has exactly one number type. Correcting a declaration that was wrong about the domain is legitimate; correcting one to flatter a measured result is not. Recorded in the probe source and the profile document rather than edited away. Honest limits, stated in the artifacts: no fuzzing of the canonicalizer, so the census establishes that the declared classes behave as declared and nothing about the rest of the input space. No cross-language differential on receipts yet -- the conformance corpus covers the composition kernel, but nothing re-canonicalizes a CC report in another language and compares digests, so "cross-language verifiable" describes v2's design rather than a demonstrated result. No external reviewer has attacked either profile. Census under v2: 12 sound, 1 fail-closed, 1 sound-by-rejection; zero unintended-kernel, zero rejection-asymmetry, zero over-discrimination, with four positive controls passing. Verified: suite 816 -> 873 passing, 0 failing, 9 skipped. Capsule replays byte-identically. Node conformance agreement and differential fuzz both still pass. Ruff check and format clean on every touched file. All document links and anchors resolve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VnVFub3xq2vwPjjz3MUqZV
The Evidence Commons needs an instrument that emits evidence cards. This is that instrument, scoped to cc-framework: the emitter and its schema, not an Atlas, a site, or any surface that renders them. A card is a claim plus everything needed to DISAGREE with it: how it was operationalized, the command that reproduces it, what would falsify it, what it assumes, and what it explicitly does not claim. The design constraint is negative. A card is not a trust score, and nothing here computes one. Three labels travel with every card and they are orthogonal: evidence_state local-only | aws-synth-only | aws-live | illustrative verdict pass | fail | unverifiable | not-run publication_state draft | released | superseded | retracted None implies another. A pass that is local-only says nothing about deployed behaviour. A retracted card may still carry a pass: the run happened and the claim was withdrawn anyway. That orthogonality is enforced rather than documented -- tests assert the card exposes no status/score/is_ok/badge/health property, that the site manifest carries no aggregate, that the schema defines none, that render_labels returns all three or raises, and that all 64 label combinations are constructible. If a future change couples two labels, the last of those fails. Two fields most claim registries treat as optional are mandatory here: falsifier and non_claims. Both are now required in the claim-boundary manifest itself, enforced by the validator, so every claim in this repository carries a falsifier because one cannot be added without it. not-run is the default verdict. A card cannot acquire a passing verdict by being written confidently; a pass is reachable only via --run, which executes the commands. The committed cards are always the not-run scaffold, asserted by a test: verdicts are host-specific and a repository shipping someone's laptop results as published evidence is doing the thing this object exists to prevent. unverifiable is not fail, and the distinction earned itself on the first run. make test-kernel exited non-zero because its dependency-install step could not reach the network -- the kernel tests never executed. The first version of the harness mapped any non-zero exit to fail and so reported a failure nobody observed. It now attributes a fail only to pytest exit code 1, whose meaning is documented; everything else is unverifiable with the reason recorded. Per-command outcomes are preserved rather than collapsed, so a card shows [pass=1, unverifiable=1] instead of hiding that the tests passed. Also in this change: - The manifest's supporting_tests_or_commands held bare test paths. A path is not a command, so seven were prefixed with `pytest `. The validator now checks that path-like tokens in commands resolve on disk, closing one of the two residual gaps recorded in F-12. - Under --run, commands execute only if they match a small allowlist. A card generator that shells out arbitrary strings from a data file is a code-execution surface, not an evidence tool. - The git revision records a -dirty suffix when the working tree differs. A card bound to a revision the tree does not match is bound to the wrong thing. - Cards bind supporting files by digest, so editing one makes them stale. That is the feature working, and the ordering is documented. Current state: 8 claims, evidence_state local-only=7 aws-synth-only=1, verdict not-run=8, publication_state draft=8. The single synthetic lane is the enterprise reference, whose card says on its face that emulated evidence is never live evidence. Nothing here is aws-live. Scope note: this repository emits cards and hosts no renderer. Pricing, go-to-market, and venture planning are out of scope for a research repository, matching the boundary the sibling institutional repository enforces with a test. Verified: suite 873 -> 907 passing, 0 failing, 9 skipped. All 8 committed cards validate against schemas/cc.evidence_card.v1.json. Manifest validator clean. Canonicalization census, Node conformance agreement, and the conformance corpus all still pass. Ruff check and format clean on every new and touched file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VnVFub3xq2vwPjjz3MUqZV
Adds docs/research/future-expansion/, a method package for converting a narrative or intuition into research objects that can be tested, replicated, or discarded. - README.md: purpose, operating loop, non-negotiable rules, cited research foundations, contents, mapping onto existing CC-Framework governance surfaces, and a porting manifest for the two sibling project tracks. - source-ledger.md: a worked Week 1 artifact classifying one real narrative source statement by statement, with testable translations, disallowed inferences, the three method heuristics that survive translation, and a properly controlled falsification exercise for its only checkable claims. - eight-week-plan.md: the eight-week schedule with per-week gates, two project tracks, and an appendix mapping each week's artifact onto the claim boundary manifest, non-claims, evidence roles, confirmatory protocol, receipts, and decay semantics already in this repository. - prompt-library.md: eleven prompts that produce research artifacts rather than predictions. Documentation only. Adds no claim to the claim boundary manifest, changes no kernel or evidence behavior, and states explicitly that it is not evidence. Verified with mkdocs build --strict, scripts/check_artifact_boundary.py --static, and the full pytest suite (689 passed, 9 skipped).
…im Observatory A 15-second film about the moment a seductive story is forced to become a test. It is Chamber 00 of the existing Claim Observatory world: the room before the Arrival Hall, where an intuition stops being a feeling and is written down in a form that can lose. Beats: a notebook under warm oracular light; a hard cut to cold as the ember field quantizes into an 84-cell grid of coincidences that would all have felt like proof; four tablets (claim, falsifier, control, non-claim) snapping into a re-centering ledger; a protocol frozen over an empty result slot; the slot filling with INCONCLUSIVE rather than a green tick; the five accents compressing into a ledger rule under the wordmark. - film.html: the film, one self-contained file. Every visual state is a pure function of time via window.__seek(t), so renders are reproducible rather than recorded. No downloaded fonts, no external assets. - render_film.py: deterministic frame capture and H.264/VP9 encode, with a --verdict hook that displays a real verifier result verbatim. - DIRECTORS_CUT.md: beat sheet, color law, voiceover, sound spec, production rules, poster frame, and the two lockups. - README.md: how to watch, render, and place the Ghost-Ark cut. - make film / make film-ghost-ark. Two rules the film holds itself to. It uses no green at any point, because World Bible V3 forbids a green checkmark as final truth; confirmatory cyan carries that role and appears exactly once. And it renders no verdict it was not given: with none injected it falls back to an illustrative INCONCLUSIVE and stamps ILLUSTRATION on its own footage. The repository README gains a hero built on the film's poster frame. The rendered masters follow in the next commit.
… It" Adds the 15.000s master and its poster frame: 1920x1080, 60fps, H.264 High, yuv420p, faststart. The poster is frame 456 - the frozen protocol over an empty result slot, which is the film's emotional center and the frame the repository README now leads with. Also fixes ffmpeg discovery. The Playwright bundle is built only for VP8 screen recording and has no libx264, so the encode failed after a clean 900-frame capture. find_ffmpeg() now probes each candidate for libx264 and prefers one that has it, and --encode-only reuses frames already on disk so an encoder problem never costs another full capture.
The release loop needs somewhere for curiosity to land. This adds that page and the positioning that governs what may be said around it. visual_identity/canonical_page/index.html carries the five required elements: the film with a timecoded transcript, a condensed source ledger, a visible "what this does not establish" strip, one real interactive check, and the invitation to bring one claim that matters. The check is real. It embeds calibration.json from the deterministic claim governance capsule and the SHA-256 recorded in that capsule's manifest, then recomputes the digest in the visitor's browser with WebCrypto. Editing one character fails it closed. Nothing is simulated, no verdict is hard-coded, and the fixture path, the command, and the limitation are printed underneath it: integrity only, not validity and not safety. tests/unit/docs/test_canonical_page_fixture.py guards the drift. If the capsule is regenerated and the page is not updated, the page would show a legitimate artifact failing closed - an honest-looking screen making a false statement. The test fails first instead. docs/product/PUBLIC_POSITIONING.md fixes the lockup, the bio, and the name boundaries, including the strict rule that Ghost-Ark stays out of commercial material and is referenced only as a labelled research case study. It also records what is not sold: no validation, no compliance, no seal of approval. Its own non-claims note that publishing a ninety-day plan is not evidence of executing it, and that no superlative in it is licensed for public use. Also adds a cubits11 cut of the film and moves the written thesis to its flagship wording: a story can start a question, evidence must finish the answer.
…ur own guards Turns scattered artifacts into one program: an inventory of what has actually been built, a graph of how the pieces relate, the concept underneath all of it, and the machinery for making it encounterable. docs/research/epistemic-program/ - inventory.md: every artifact and the question it attempted to answer, with a mandatory limit column. A surprising fraction of this repository exists to prevent a conclusion rather than to produce one. - research-graph.md: the invariant loop (claim, decompose, bind, reproduce, mutate, observe, bound, expose), eight nodes around CLAIM -> EVIDENCE, and where each artifact attaches. Intellectual connection without organizational merger: Ghost-Ark stays separate. - evidence-scope-inflation.md: the core concept. A digest match licenses one inference and readers routinely make three more. - challenges.md: open adversarial challenges, and the result of CH-001. - visibility-manifold.md, distribution-packet.md, claim-intake.md, release-freeze.md: one object into seven surfaces, the first packet, what actually happens when someone brings a claim, and what ships. CH-001 is the substantive finding. Attacking this repository's own drift guard in a detached worktree: forging an evidence artifact and repairing the manifest and page passes the drift guard but is caught by capsule regeneration. Moving the forgery upstream to the declared input and regenerating the whole chain passes everything - 4 + 5 tests green, governance verdict pass, a new valid receipt, and the page displaying BYTES MATCH over a false-positive rate we invented. So the guard chain establishes deterministic reproducibility from declared inputs, which is not measurement. Closing that gap needs an anchor outside the repository, and there isn't one. The finding is fed back into the artifacts rather than only written up: the canonical page's limits strip and limitation text, a new entry in docs/research/NON_CLAIMS.md, and the drift guard's own docstring now state it. Also adds render_check_clip.py, which drives the real page in a browser and captures the ten-second BYTES MATCH -> FAIL CLOSED clip frame by frame, so the clip cannot drift from the artifact it depicts.
Packet 001 listed five posts and contained no copy. This writes them out in publishable form: the demonstration, integrity is not truth, the self-attack, name it before you see it, and the invitation. Each carries a claim, its evidence, and its limit, and each ends at an object a stranger can operate or break rather than at another piece of copy. Draft 03 is the one to lead with once CH-001's write-up and the page's updated limitation are both live.
…, not truth CH-002 asked whether an outside reviewer can distinguish a measured input from an asserted one without trusting the author. The answer is no, for a reason worth stating plainly: any repository-local check is a function of repository contents, and deterministic regeneration lets the author construct a state the check accepts by editing an input and rebuilding. No inversion required. CH-001 is that argument executed. The uncomfortable part is that the exploited property is the one we want. Reproducibility guarantees a change propagates consistently through every artifact, hash, and receipt - for honest and dishonest inputs alike. Also records a cheap partial defence and why it is deliberately not implemented. realized_fpr is exactly 1/24 and the shipped failure matrix has exactly 24 rows, so a derivability check would have caught CH-001's 1/90 forgery. But no column of that matrix has rate 1/24, so what the field denotes is not established. Implementing the check would encode an unverified assumption about its meaning - the same error class this program is about, committed while defending against it. Establish the denotation first. Names the three anchors that would actually close it, ranked by cost, and records that none is implemented.
These predate this branch and are on main, but CI runs `ruff check .` on every push, so the branch inherits a red build for them. All four fixes are mechanical and semantics-preserving: one blank line, an import block split, and two set comparisons flipped from `a <= b` to `b >= a`. `ruff check .` is now clean across the repository.
…thesis
The full suite failed on tests/unit/core/models/test_models_base.py::
test_modelbase_migrate_best_effort. ModelBase.migrate() is documented as never
throwing on arbitrary old dicts; given {'updated_at': ''} it raises, because it
passes values straight into model_validate and updated_at has a timestamp
validator.
Verified pre-existing: reproduces identically on main in a detached worktree,
and src/cc/core/models.py is unmodified on this branch. It is nondeterministic
in CI because the failing example lives in a gitignored .hypothesis database.
Recorded rather than fixed. The obvious repair - swallow invalid values for
known fields - is silent repair, which this repository rejects everywhere else.
The defect is plausibly in the contract, not the code, so the write-up states
both options and recommends narrowing the contract instead. Applying that
unilaterally would be a semantic change to the core model layer made only
because it was inconvenient to this session's build.
# Conflicts: # examples/claim_governance_capsule/expected/cc_report.json # examples/claim_governance_capsule/expected/claim_envelope.json # examples/claim_governance_capsule/expected/claim_governance_audit.json # examples/claim_governance_capsule/manifest.expected.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Why
Verification performed
make test-kernelpassesmake test-releasepassesCompatibility check
Claimenvelope schema or other module namespacesDefinition of Done reference
Which item(s) from the Launch-Hardening Plan does this satisfy?