Skip to content

feat(gov): prove every rule reaches the runtime that obeys it - #25

Merged
indykish merged 9 commits into
mainfrom
feat/m06-harness-robustness
Aug 27, 2026
Merged

feat(gov): prove every rule reaches the runtime that obeys it#25
indykish merged 9 commits into
mainfrom
feat/m06-harness-robustness

Conversation

@indykish

@indykish indykish commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Intent

Close the delivery gaps between a written rule and the agent bound by it, and stop charging every session for context that enforces nothing. Four gaps came out of a read-only audit of the instruction and skill surface across Claude Code, Codex, and OpenCode in a consuming repository; each one let a written rule govern nothing.

Spec: docs/v1/done/M06_001_P1_GOV_CLI_HARNESS_ROBUSTNESS.md

What was broken

Gap Evidence
Pack files drift in the checkout that authors them, unchecked dispatch/write_rust.md carried 1,776 bytes against a 9,971-byte pack source — the entire Rust error discipline missing from the copy this repository's agents, evals/dispatch/coverage.sh, and the rule ledger read. dispatch/write_go.md: 1,204 against 1,751.
The DOC READ recorder belonged to no pack audits/doc-read.sh appeared in no managed_files row, so in a consumer the command the operating model names was absent: a runtime read hook invoked nothing and pre-commit had nothing to compare.
RULE ERR-RS was enforced by review alone Rust had no deterministic façade at all — no dispatch/write_rust.sh, no leaf.
One boundary, three cadences The prose said the declared verify.* set runs "always"; the same suites then ran in the hooks and again inside a bare orly gate.

Three prose claims were also simply false: dispatch/verify.md and docs/VERIFY_TIERS.md said no lane needs live Postgres or Redis while the config declares an integration command; the skill chain called the integration skill "never skipped" while dispatch/verify.md called it conditional; and SOUL.md credited orly gate verify with enforcing the banned-word list, which docs.language reports without failing by design (src/criteria.ts:150-160).

What changed

§1 — orly verify gains packs.sources.current. planFiles deliberately skips writing a managed file into the checkout that owns its source, so that copy is maintained by hand and no install corrects it. The new check asks managedContent (now exported, so one definition decides what bytes belong at a target) what each pack would ship and compares. A target the checkout does not carry stays silent — a consumer-only path is not drift. Both drifted copies are regenerated.

§2 — universal.authoring ships the recorder and the library it sources. The test installs into a fresh repository and runs the pair there, because a missing library fails at run time and would pass any copied-file check.

§3 — Rust gets a deterministic façade. dispatch/write_rust.sh runs audits/rust-error.sh over staged *.rs and blocks the two clauses a machine can decide from the text: a map_err whose closure stringifies its own binding, and a crate declaring a public Error with no pub type Result<…> beside it. The binding is what makes the first decidable — to_string() on the cause destroys the source() chain, the same call on a path is correct. source() shape and whether a context addition is warranted stay judgment; a gate that guesses at those is one people learn to ignore.

§4 — Two cadences, one boundary. A Section lane (CONFORM plus the lane covering the surface touched, reported as a Section claim) and the milestone boundary, where the full declared set runs once. CHORE(close) names orly gate pr; the bare chain pays the fast tier twice for the same verdict.

§5 — persona costs voice, not enforcement. Five engineering clauses were hiding inside the opt-in persona pack, so dropping the voice silently dropped symlink-edit routing, dotfiles backup, vault naming, sibling-repo precedent, and the read-the-reference mandate. They move to the always-on blocks; only the machine-local checkout paths stay behind. Four packaged skill descriptions shrink under a 320-character bound — every host renders skill metadata into a fixed budget and truncates when it overflows.

Effect on a consuming repository

Measured by rendering agentsfleet's pack selection at this commit:

rules WITH persona.indy:     37,166 B  (~9.3k tok)
rules WITHOUT persona.indy:  28,211 B  (~7.1k tok)
committed today (0.7.1):     35,436 B

Dropping persona.indy is now a lossless 8,955-byte cut. It was not before this branch.

Verification

Lane Result
make audit (declared conform) all checks passed
bun test src (declared verify.unit) 160 pass, 0 fail — baseline 148, delta +12
make install-evals (packed-tarball distribution) 23 passed, 0 failed
bash evals/dispatch/run.sh 43 passed, 0 failed
bash evals/dispatch/coverage.sh ALL CHECKS PASSED
gitleaks detect no leaks found, 576 commits scanned
orly gate pr every criterion green

Session notes

  1. The DOC READ gate fired on me mid-branch — I regenerated dispatch/write_rust.md after last reading it, so the recorded read no longer matched the content and pre-commit refused. Working as designed, and the first evidence that §2's recorder is load-bearing rather than decorative.
  2. Two of my own edits cited a pack-owned file from an always-on clause; reference closure and the existing pack-filter test caught both before they could break an install for a repository without that pack. Same defect class the branch exists to close.
  3. /orly-write-integration-test: N/A — no module boundary with real input/output changed. The distribution lane (make install-evals) is this repository's integration tier and ran green.
  4. Follow-up, deliberately out of scope: the consuming repository's own hook cadence, harness rows, and pack selection land in its branch once this version is published.

🤖 Generated with Claude Code


Added after review of the lifecycle wiring (§6–§8)

Indy asked for the fixes to land in the engine rather than per-project, since orly is what he builds every project with. Three more defects surfaced there.

§6 — The generated hooks refused the commits they exist to protect. orly gate work judged branch shape and tree cleanliness, and the generated pre-commit hook runs exactly that. A commit hook's tree is dirty by construction — that is what is being committed — and the operating model itself prescribes committing a new spec on the default branch. A fresh orly init could not commit its own installation; the only way past was --no-verify, which Hard Safety forbids outright. Never seen in agentsfleet because it hand-wrote its hooks with --accept-dirty.

Each gate now asks only what it can answer honestly at its own cadence:

work    the declared conform command, no git state     pre-commit
verify  spec dimensions, docs language, fast verify.*  pre-push
pr      branch, tree, pushed, spec criteria, slow      CHORE(close)

conform is its own tier rather than riding in verify, so the chain runs it once. orly gate pr skips the fast tier because its own git.pushed criterion proves HEAD is the commit pre-push already graded.

§7 — CONFORM has to cost seconds now that it runs at every commit. This repository's own declared conform was make audit, which contains the bun test src its verify.unit also declares — so orly gate ran the unit suite twice. Split into make conform (four deterministic rule gates, ~2s), with make audit unchanged for pre-push and CI.

§6.4 — The dispatch stopped judging files the engine wrote. Staging a fresh install put orly's own 444-line audits/logging.sh in front of orly's own 350-line cap — a finding no consumer can act on, because the next orly update overwrites the file. dispatch_managed_paths filters the install record out of the staged set.

§8 — dispatch/lifecycle.md gains a stage-by-stage table naming the command each stage runs and the gate that fires it.

Verification (re-run)

Lane Result
make conform (declared conform) all checks passed, ~2s
make audit (full invariance chain) all checks passed
bun test src (declared verify.unit) 161 pass, 0 fail — baseline 148, delta +13
make install-evals 23 passed, 0 failed
orly gate pr every criterion green

Session notes (continued)

  1. A dispatch sweep — one command running every façade the repository carries, so a shipped leaf fires without hand-wiring — was built and then removed. The rule ledger correctly refused it: a runner declares no scope, so nothing it carries can fire, and it was the one speculative addition in the branch. Wiring a façade into a repository's conform stays the repository's call.
  2. The size cap fired at 38,039 bytes against 37,888 while adding §6–§8, exactly as SOUL.md says it should: adding a rule means making room. Rationale tails were cut, triggers kept.
  3. /orly-write-integration-test: N/A — no module boundary with real input/output changed. make install-evals is this repository's integration tier and ran green.

indykish and others added 8 commits August 27, 2026 13:57
M06_001 opens on four delivery gaps a cross-runtime instruction audit
found: pack files that drift in the checkout that reads them, a DOC READ
recorder that belongs to no pack, a Rust error standard no machine
decides, and a verification cadence the prose calls "always" where it
means "before the boundary".

Test Baseline: unit=148 integration=0

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
install skips writing a managed file into the checkout that owns its
source, so the copy living beside the source is maintained by hand and no
install ever corrects it. Both halves of that had already happened:
dispatch/write_rust.md carried 1,776 bytes against a 9,971-byte source —
the entire Rust error discipline missing from the copy this repository's
agents, its dispatch-coverage audit, and its rule ledger read — and
dispatch/write_go.md 1,204 against 1,751.

orly verify gains packs.sources.current, which asks managedContent (now
exported, so one definition decides what bytes belong at a target) what
each pack would ship and compares it to what is there. A target the
checkout does not carry stays silent: a consumer-only path is not drift.

Both drifted copies are regenerated from their sources.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The operating model tells every agent to record a triggered read with
`bash audits/doc-read.sh log <path>`, and pre-commit compares that record
against the staged diff. The rule shipped through universal.authoring;
the script belonged to no pack at all. In a consumer the command the rule
names was simply absent — a runtime read hook invoked nothing, the check
had nothing to compare, and the proof-line stayed a self-report about
itself.

universal.authoring now carries the recorder and the library it sources.
The test installs into a fresh repository and RUNS the pair there, since
a missing library fails at run time and would pass any copied-file check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rust had no deterministic façade at all, so RULE ERR-RS — written, cited
by the doc-read map, called mandatory by the operating model — was
enforced by review alone.

dispatch/write_rust.sh runs audits/rust-error.sh over staged *.rs and
blocks the two clauses a machine can decide from the text: a map_err
whose closure stringifies its OWN binding (to_string()/format! on the
cause destroys the source() chain, while the same call on a path is
correct — the binding is what makes it decidable), and a crate declaring
a public Error type with no `pub type Result<...>` beside it.

The other two clauses stay judgment on purpose. Whether source() returns
its own kind needs the Display body read beside it, and whether a context
addition is warranted is a call-site question; a gate that guesses at
those is one people learn to ignore.

One scanner emits both signals so the #[cfg(test)] carve-out is
implemented once and the halves cannot disagree about what a test module
is. Carve-outs match audits/logging.sh exactly, so one Rust scope answers
to both leaves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cadence prose said the declared verify.* set runs "always", which
reads as every done-claim. The same suites then ran again in the hooks and
a third time inside a bare `orly gate` — the comprehensive tier's
wall-clock spent answering a question the scoped lane already answered.

VERIFY now states two cadences: a Section lane (CONFORM plus the lane
covering the surface the Section touched, reported as a Section claim)
and the milestone boundary, where the full declared set runs once.
CHORE(close) names `orly gate pr` — the bare chain pays the fast tier
twice on the way to the same verdict.

Three claims that were false are corrected: dispatch/verify.md and
docs/VERIFY_TIERS.md still said no lane needs live Postgres or Redis while
the config declares an integration command; the skill chain called the
integration skill "never skipped" while dispatch/verify.md called it
conditional; and SOUL.md credited `orly gate verify` with enforcing the
banned-word list, which docs.language reports without failing by design.

Each claim is now pinned by a render test, since none of it is visible to
a determinism check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
persona.indy is opt-in and five engineering clauses were hiding inside
it, so a repository dropping the voice silently dropped symlink-edit
routing, dotfiles backup, vault naming, sibling-repo precedent, and the
read-the-reference mandate along with it — 9,575 rendered bytes charged
to every session, of which only some was voice.

The clauses move to the always-on blocks. The vault line folds into the
credentials rule it half-duplicated. Only the machine-local checkout
paths stay behind, which is what a pack named for a person is for. A
render test asserts the split so it cannot quietly reverse.

Four packaged skill descriptions shrink under a 320-character bound.
Every host renders skill metadata into a fixed budget and truncates the
set when it overflows — Codex says so out loud — and a truncated
description is a skill the model can no longer route to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every Dimension DONE, rubric graded against real output, spec moved to
done/.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

The hook orly generates for pre-commit runs `orly gate work`, which judged
branch shape and tree cleanliness. A commit hook's tree is dirty by
construction — that is what is being committed — and the operating model
itself says a new spec is committed on the default branch. So the hook
failed twice over on a normal commit, a fresh `orly init` could not commit
its own installation, and the only way past was `--no-verify`, which Hard
Safety forbids outright.

Each gate now asks only what it can answer honestly at its own cadence:

  work    the declared conform command, no git state    (pre-commit)
  verify  spec dimensions, docs language, fast verify.*  (pre-push)
  pr      branch, tree, pushed, spec criteria, slow      (CHORE(close))

conform becomes its own tier rather than riding in verify, so the chain
runs it once. `orly gate pr` skips the fast tier because its own
git.pushed criterion proves HEAD is the commit pre-push already graded.

Two consequences followed. CONFORM now runs at every commit, so this
repository's own declared conform — `make audit`, which contains the
`bun test src` its verify.unit also declares — splits into `make conform`
(four deterministic rule gates, about two seconds) with `make audit`
keeping the full chain for pre-push and CI. And the dispatch stops judging
files the engine wrote: staging a fresh install put orly's own 444-line
audits/logging.sh in front of orly's own 350-line cap, a finding no
consumer can act on because the next update overwrites the file.

dispatch/lifecycle.md gains a stage-by-stage table of what runs where.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@indykish
indykish merged commit 8d52ecb into main Aug 27, 2026
4 checks passed
@indykish
indykish deleted the feat/m06-harness-robustness branch August 27, 2026 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants