SameWrite 1.3.1 — fail closed on partial history evidence - #7
Open
ipeterpetrus wants to merge 5 commits into
Open
ipeterpetrus wants to merge 5 commits into
ipeterpetrus wants to merge 5 commits into
Conversation
…ccepted SW-1303-PARTIAL-EVIDENCE. optimize.py read evidence quality from the live sweep only, so the quality each history record carries never entered the decision. A history built from bounded sweeps — carry.py --max-files N, the normal way to keep a large profile affordable — produced a candidate exactly as if the corpus had been swept in full, and a complete live scan masked it. Measured on the released code, and again after the fix: before PARTIAL history, no --accept-partial CANDIDATE, 1 candidate file, exit 0 before PARTIAL history + COMPLETE live scan CANDIDATE, 2 candidate files, exit 0 after both PARTIAL_EVIDENCE, 0 candidate files The module docstring already stated the opposite rule. The guard existed and worked — on one of the two paths evidence can arrive by. The rule, now in one place (effective_quality/worst_quality/emittable/quality_of): a finding may be promoted only when the WORST quality among the evidence ACTUALLY ELIGIBLE to support it is COMPLETE, or is PARTIAL and the caller accepted it. Worst, not majority — one partial observation among nine complete ones still means part of the corpus was never swept. Eligible, not everything — comparable() already excludes other scopes, other workload classes and non-comparable corpora, and a partial record there must not block a run it was never part of. A gate that blocks on evidence a finding never used is not correct, it is merely stuck. --accept-partial authorises PARTIAL and nothing below it. Nobody can say "this bounded population is the one I meant" about evidence that is corrupt, empty, or of unestablished provenance. Legacy records (schema 0/1) have no evidence_quality field, so absence proves nothing about how that sweep was taken: they now read as UNKNOWN rather than COMPLETE. They stay readable and stay visible in reports; they simply cannot carry a promotion. Contract: --json gains effective_evidence_quality and partial_evidence_accepted. No existing field changed meaning, so output_schema_version stays 1. Candidates written from accepted partial evidence record both fields in HYPOTHESIS.md. Exit codes unchanged — normal mode still exits 0, --strict-exit still returns 40. The fix is that no candidate is produced. Held by: a 15-case matrix (A-O) frozen before implementation, five mutants that each go RED when the fix is reversed, a self-check that plants three corruptions in the regression harness and requires the harness itself to fail, and the real integration path asserted end to end. Two pre-existing fixtures in test_optimize.py were legacy-shaped and started failing on the new rule. They now declare their provenance, and two assertions were ADDED to lock the stricter default rather than let the fixture edit hide it. CANONICAL_SKILL_DELTA=0, DEFAULT_MODEL_CONTEXT_DELTA=0. carry.history() untouched. TOTAL_SAVINGS=NOT_PROVEN and WORLD_BEST_CLAIM=NOT_TESTED unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013pnJ85a3DYFU2ZW7CqQKCv
…ce gate
The hotfix diff was handed to an independent reviewer with instructions to make the patched
optimizer promote evidence that should fail closed. It found four ways. Each was reproduced
directly before being accepted, and each now has a regression case and, where it is a rule rather
than a report, a mutant that goes RED when reversed.
schema laundering A schema-1 record carrying evidence_quality: COMPLETE was trusted. Schema
0/1 predates the field, so that claim cannot have come from the sweep — it
comes from a hand-edited file or a migration default. Any record declaring
a schema older than 2 is UNKNOWN whatever it asserts.
retry laundering One run_id saying COMPLETE once and PARTIAL once resolved in favour of the
better claim, because the duplicate was dropped whole. The retry is still
dropped as an observation; its worse quality now survives on the record
that remains.
status contradiction A run could report PARTIAL_EVIDENCE with strict-exit 40 while a ledger
candidate sat on disk. Ledger findings count writes a hook actually denied,
which no sweep bound can make partial, so they stay promotable — but then
the status has to say so. Status now describes what the run produced; the
refused sampled evidence travels in effective_evidence_quality.
stale artifacts A candidate written before 1.3.1 carries no provenance and was reported as
ordinary "existing". It is still never rewritten or deleted, but it is now
named as pre-1.3.1: no evidence provenance, re-review.
One finding was NOT taken: that a partial history should not demote a concentration finding
measured purely from a complete live sweep. True of the measurement, and the gate here is
deliberately broader. For a patch closing a fail-open the conservative direction is the safe one,
--accept-partial is the documented way through, and the cost is written down rather than hidden.
Two test debts this created, fixed rather than papered over: the missing-quality mutant was masked
by the new schema gate firing first, so its oracle now uses a schema-2 record with no quality field
— the only shape that reaches that default; and the pre-existing no-proposal-spam mutant targeted
an emitter block this change rewrote, so it was re-pointed.
560 assertions across fourteen suites, 20 mutants, all green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013pnJ85a3DYFU2ZW7CqQKCv
The complete diff was reviewed independently and blocked. Five HIGH findings, each reproduced here
before being accepted — nothing taken on assertion.
H1 A record could claim COMPLETE while its own counters recorded 5 unreadable files, 2 oversize
lines and a file cap, and still be promoted; so could one with sessions=turns=carry_bytes=0.
derive_quality() now mirrors the producer's rule in carry.accumulate() and a record's quality is
the worst of what it claims and what it can show. Counters that are negative, non-numeric or
boolean make it INVALID. A schema-2 record showing no counters cannot claim completeness.
H2 A torn JSON line was skipped silently, so a sweep that lost evidence still reported COMPLETE —
no history needed to walk through the gate this release adds. scan_full() now counts what it
could not parse, accumulate() degrades the sweep to PARTIAL, and the count travels into the
history record as malformed_lines.
H3 A candidate built from accepted-PARTIAL evidence was written with COMPLETE provenance, because
provenance was stamped by a finding's POSITION in a list and listing-prune is appended after
that boundary. Provenance is now attached where a finding is created, by sampled() and
from_ledger(); no finding infers its source from where it landed.
H4 HOST_BEHAVIOR_SHIFT reported status 30 and strict exit 40 while writing a candidate file. A
population declared not to be one world is not evidence you may promote from, so it joins the
single promotion gate. A status that refuses promotion now leaves nothing on disk, on every path.
H5 The pre-1.3.1 artifact check was a substring search over the whole file: a sentence mentioning
the field satisfied it, an unreadable file counted as current, and invalid UTF-8 escaped the
handler and killed the run. artifact_metadata() parses the header block structurally and returns
PROVEN_CURRENT, PRE_1_3_1_REVIEW_REQUIRED or UNREADABLE_OR_UNVERIFIABLE. Unreadable is never
trusted; nothing is rewritten or deleted.
Three MEDIUM closed with them. candidates_existing keeps its original meaning as bare ids, with the
review states in additive fields, so a consumer treating an id as an id no longer breaks on upgrade.
comparable() picks its anchor from eligible records, so an INVALID arrival can no longer define the
workload and corpus every other record is measured against and then vanish. carry.bounded_paths() is
the single definition of a bounded sample, so the carry sweep and the listing scan stop describing
different corpora under --max-files.
Test debts this created, fixed rather than papered over: four fixtures claimed COMPLETE without the
counters a real sweep always writes, and each now states them with an added assertion locking the new
rule so the fixture edit hides nothing; three mutation patterns moved with the code and were
re-pointed; two mutants were masked by a fix firing earlier than the branch they guard, and their
oracles now isolate that branch; and emit_candidates gained two result classes, so its callers were
updated.
645 assertions across fifteen suites, 29 mutants all RED-on-break, harness self-check 8/8, the
original SW-1303 arms still PARTIAL_EVIDENCE with 0 candidate files, 81/81 mandatory attacks, privacy
5 canary classes x 6 sinks with 0 leaks, AI-VOS readiness 27/27.
CANONICAL_SKILL_DELTA=0, DEFAULT_MODEL_CONTEXT_DELTA=0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013pnJ85a3DYFU2ZW7CqQKCv
…er it broke experiments/aivos/longrun.py unpacked emit_candidates into three values; it now reports two more classes. My local sweep only ran tests/, so CI found it and I did not — the sweep now runs every step CI runs, by exit code. experiments/aivos/readiness.py builds the POSITIVE control: a clean, threshold-crossing trend that must still produce a candidate. Its records claimed COMPLETE without the acquisition counters a real sweep always writes, so the new rule correctly refused to promote them and the control went dark. That is the failure mode a fail-closed patch is supposed to have a control for, and it worked — the fixture now states its counters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013pnJ85a3DYFU2ZW7CqQKCv
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Patch release candidate for
SW-1303-PARTIAL-EVIDENCE. Not 1.4.v1.3.0is not moved and itsreleased runtime is unaffected.
This PR has been through two full independent adversarial reviews. The first one blocked it.
That history is kept below rather than tidied away: the point of a release gate is that it sometimes
catches things.
1. The original defect, before and after
tools/optimize.pyread evidence quality from the live sweep only, and the trend finding had noquality gate at all. A history built from bounded sweeps —
carry.py --max-files N, the normal wayto keep a large profile affordable — was indistinguishable from one swept in full.
The rule now lives in one place: a finding may be promoted only when the worst quality among the
evidence actually eligible to support it is
COMPLETE, or isPARTIALand the caller passed--accept-partial. Worst, not majority. Eligible, not everything.2. The first review blocked this branch
Head
bd7ada5was reviewed on the complete diff and returned 19 hypotheses: 5 HIGH, 3 MEDIUM.Every one was reproduced locally before being accepted — none taken on the reviewer's assertion.
COMPLETEwhile its own counters recorded 5 unreadable files, 2 oversize lines and a file cap, and was promoted; so was one withsessions=turns=carry_bytes=0COMPLETE— no history needed to defeat the new gateeffective_evidence_quality: COMPLETEHOST_BEHAVIOR_SHIFTreported status 30 and strict exit 40 while writing a candidate filecandidates_existingturned from ids into prose while the schema version stayed 13. What closed them
derive_quality()reconstructs the producer's rule fromcarry.accumulate(), the oneplace that knows how a sweep went, and
quality_of()returns the worst of what a record claimsand what it can show. Negative, non-numeric or boolean counters make it
INVALID;sessions==0is
INVALIDorEMPTYon the producer's own terms; a schema-2 record showing no counters cannotclaim completeness and is
UNKNOWN.scan_full()counts the linesjson.loadsrejects,accumulate()aggregates them anddegrades the sweep to
PARTIAL, and the count reaches the history record asmalformed_lines.sampled()andfrom_ledger(). The positionalsampled_countstamping is gone; no finding infers its sourcefrom where it landed in a list.
A status that refuses promotion now leaves nothing on disk, on every path.
artifact_metadata()parses the header block structurally, bounded to 64 KiB and 200lines, returning
PROVEN_CURRENT,PRE_1_3_1_REVIEW_REQUIREDorUNREADABLE_OR_UNVERIFIABLE.OSErrorandUnicodeDecodeErrorboth fail closed. Nothing is rewritten or deleted.candidates_existingis bare ids again; the review states moved tocandidates_existing_review_requiredandcandidates_existing_unverifiable.comparable()decides eligibility first and anchors only on eligible records, using thederived quality rather than the claimed one.
carry.bounded_paths()is the single definition of a bounded sample, used by both thecarry sweep and the listing scan.
4. Coverage
Two frozen matrices, both written with their expected outcomes before implementation: the
original A–O partial-evidence matrix, and a second one for the five blockers. 29 mutants, each
RED when the fix it guards is reversed and GREEN on the real implementation. A harness self-check
that plants three corruptions in the regression harness and requires the harness itself to fail.
Positive controls are part of the gate, because a fail-closed patch can look excellent by refusing
everything: a clean COMPLETE population still promotes, an accepted PARTIAL population still
promotes, and unrelated invalid evidence still does not poison a valid run.
5. Test debts this created, fixed rather than papered over
Four fixtures claimed
COMPLETEwithout the acquisition counters a real sweep always writes. Eachnow states them, with an added assertion locking the new rule so the fixture edit hides nothing.
Three mutation patterns moved with the code and were re-pointed. Two mutants were masked by a fix
firing earlier than the branch they guard, and their oracles now isolate that branch.
emit_candidates()gained two result classes, so its callers were updated — including one inexperiments/that CI caught and my local sweep had missed, which is why that sweep now runs everystep CI runs, by exit code.
6. Scope
CANONICAL_SKILL_DELTA=0,DEFAULT_MODEL_CONTEXT_DELTA=0— no file underskills/orhooks/istouched.
carry.history()'s append protocol is unchanged and the concurrency suites still pass.DEFECT-ENV-EMPTY-ANSWERis out of scope: it was attributed to the machine's global agentinstructions, not to SameWrite. The deferred LOW findings from the earlier acceptance review are
untouched except where legacy quality handling was inseparable from correct partial-evidence
semantics.
TOTAL_SAVINGS=NOT_PROVENandWORLD_BEST_CLAIM=NOT_TESTEDare unchanged.🤖 Generated with Claude Code
https://claude.ai/code/session_013pnJ85a3DYFU2ZW7CqQKCv