Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ permissions:
contents: read
env:
CARGO_TERM_COLOR: never
TAG: v0.11.0
TAG: v0.12.0

jobs:
bench:
Expand Down
49 changes: 49 additions & 0 deletions BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -1293,6 +1293,20 @@ earlier run on 0.10.1 with six deficit habits gave survival A 0.559 vs B 0.000 (
0.002 in both runs. The P0 corpus (`bench/individuality/p0.py check`, 653 recorded outputs) gives 0 differences on the
build and on the 0.11.0 release build.

**The V2 residual, explained (2026-10-22; same harness, binary and seeds as the v-run, eight paired arms of 1,000
episodes; one run, judged as registered).** With the tonic term off, the phasic term alone keeps valence tracking the
distance at +0.1199 (the failed clause); with phasic off as well the median falls to −0.2674; removing `drives.valence`
alone gives +0.1629 and removing the two sense couplings into valence (tool_fail −0.5, damage −0.3) alone gives +0.2868,
so those channels pull against the tracking rather than carry it; with every candidate off the median is 0.0000, exactly
at the time-shuffle null. With the tonic term on, dropping phasic barely moves the tracking (+0.8316) and dropping the sense
couplings raises it to +0.9937. Survival, prioritisation and pressure are identical on all eight arms (valence is off the
policy's path). Reproduction: the A and A-t0 rows equal the v-run's to every printed decimal; the four tables regenerate
byte-identical twice; 1,420 strands verify. Reading: V2's ablation clause assumed the tonic term is the only channel that
registers deviation; the phasic term is the second half of the same block and registers deviation by design, so V2 failed
on a property of the feature under test, not on an artifact. **The published V2 stays FAIL.** Proposed registration for
the next run: the ablation arm switches the whole `needs.valence` block off (tonic 0 and phasic 0) and the clause is
median ≤ +0.10 (that arm gives −0.2674 here); the main clause is unchanged.

## §14 AgentDojo: the stance block against prompt injection (one pre-registered run; `bench/agentdojo/`; probbit 0.11.0; 2026-10-22)

Mechanism and simulation. A public suite with its own metrics: AgentDojo (agentdojo 0.1.11, benchmark v1.1; 97 user tasks,
Expand Down Expand Up @@ -1340,3 +1354,38 @@ test whether the gating alone explains A − C); adaptive attacks that avoid the
attack's template by design); other models, seeds and repeat-run variance; AgentDojo v1.1.1 to v1.2.2.
`bench/agentdojo/experiments/rerun.sh` recomputes every number here from `tables/episodes.jsonl` and the strands, twice,
byte-identical, with no API calls.

## §15 The engine's own expectation and the exact trajectory (RFC 0009 engine side; 0.12.0; one registered run, 2026-10-22)

Mechanism and simulation. Engine arm: the 0.12.0 engine with the opt-in `expectation` block on; baseline: the same engine
with the block off (byte-identical to 0.11.0). Data: read-only copies of two recorded lives (42 and 46 events), 1,000
synthetic strands × 64 events across the four example personas and a fifth (seeds 2000–2999), 200 habituation scripts
(one salient flag every fifth event, ten repeats, then a different salient flag) with 200 matched-index controls. Bars and
nulls registered before the run; one run; a failed bar stays failed.

| bar | registered | measured | result |
|---|---|---|---|
| P1 replay digests, block on and off | 100 % | live 42/42 and 46/46; synthetic 64,000/64,000 | pass |
| P1 fidelity, engine vs grid-enumeration reference | ≤ 1e-6 per event | 1.1e-13 (live), 7.7e-14, 1.1e-13 (synthetic sample) | pass |
| P1 joint KL vs the engine's marginal reading, Spearman | ≥ 0.95 | 0.993 and 0.996 live; 0.997 pooled on the synthetic sample | pass |
| P2 deviation > 1e-6 after the cold start | ≥ 99 % synthetic; 100 % live | 38/38 and 45/45 live; 63,162 / 63,699 = 99.16 % synthetic | pass |
| P3 graded, p90/p10 per persona | ≥ 3 | live 105.8 and 60.4; synthetic 16.6 to 1,859 | pass |
| P4 habituation over ten repeats | non-increasing (one rise) on ≥ 90 % of scripts | 76.5 % (median 2.03 → 1.44 nats, self-term flat) | **fail** |
| P5 dishabituation, first different flag ≥ 2× the tenth repeat | ≥ 90 % | 35 % (median ratio 0.97) | **fail** |
| P6 cost, extra wall time per event, p95 | ≤ 2 ms | 3.98 ms (median 2.20; loaded host, member compiles included) | **fail** |
| P6 strand line growth | ≤ 60 bytes | live max 62; synthetic max 66, 89.6 % within | **fail** |
| P7 same bytes with the block on | stance digests equal | 64,000/64,000, 42/42, 46/46 | pass |
| P8 receipts | 100 one-digit edits caught by `live verify` | 100/100 | pass |
| P9 per-variable upper bound | every event, 1e-9 | 42/42, 46/46, 384/384 | pass |
| Null 1, frozen expectation | P4 must fail on it | monotone on 29.5 % of scripts | as required |
| Null 2, matched-index first-time control | ≥ 2× the tenth repeat | 36 % (median ratio 1.20): the decay is not shown to be specific to the repeated input | **fail** |
| Null 3, uniform mixture weights | real p90/p10 ≥ 1.5× the null's | 21.9× and 4.2× live; 1.8× to 12.1× synthetic | pass |
| trajectory: joint entropy non-increasing in β | every program (theorem) | 88/88 live programs; asserted in a full-enumeration test | pass |
| trajectory: per-variable one-rise on N1's held-out programs | ≥ 99 % | 99.61 % of 3,076 pairs (sampler draft: 93.1 %) | pass |
| trajectory: fidelity to the exact tier | ≤ 1e-9 | 3.0e-14 (grid), 5.0e-10 (held-out reference) | pass |

**Claims that hold:** the deviation is exact, replayable, non-zero on every event after the cold start, graded, and bounded.
**Claim that does not hold as registered:** habituation; a repeated salient input does read lower on most scripts, but the
registered bars fail and the control cannot separate the decay from the mixture growing. P6's cost and line-growth bars
fail by small margins and go to owner review. Tables: the lab's `RESULTS-N2SS.md`; the engine-side tests
`expectation_receipts_replay_and_bound` and `exact_trajectory_matches_and_sharpens` run in `cargo test`.
15 changes: 14 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
All notable changes to this project are documented here. The format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## 0.12.0-dev - unreleased
## 0.12.0 - 2026-10-22

RFC 0009 (docs/rfcs/0009-surprise-live-state.md), the engine side: two opt-in keys, both outside the digest, effects {}.
A persona without them gives every document byte for byte as 0.11.0.
Expand All @@ -13,6 +13,19 @@ A persona without them gives every document byte for byte as 0.11.0.
- `engine.trajectory`: the turn's program at K inverse temperatures, solved exactly (`Model::scaled` + the exact tier), no
sampler; a `TRAJECTORY` monitor row and `probbit persona trajectory`.

Measured (BENCHMARKS.md §15, bars registered before the run): fidelity 1e-13; the deviation non-zero on 100 % of post-cold
events on two recorded lives and 99.16 % of 64,000 synthetic events; graded 17× to 1,859× per persona; the per-variable bound
holds on every event; P4 habituation (76.5 % vs 90 %), P5 dishabituation (35 % vs 90 %), P6 cost (p95 3.98 ms vs 2 ms) and
line growth (max 66 vs 60 bytes) **fail as registered**; the habituation claim is not made. BENCHMARKS.md §14: one
pre-registered AgentDojo run (`bench/agentdojo/`, gpt-4o-mini, 2,178 episodes): the stance block lowers targeted attack
success from 0.289 to 0.227 (95 % CI of the difference [−0.087, −0.038]) with no measured utility cost; the same words as a
static prompt cost 10.3 points of benign utility. BENCHMARKS.md §13: the V2 residual is carried by the phasic term; the
published V2 stays FAIL and the next registration ablates the whole `needs.valence` block.

Merged pull requests:
- #53 RFC 0009 engine side: opt-in expectation (exact self-model deviation) and exact trajectory
- #54 bench: AgentDojo, one pre-registered run (stance block vs baseline vs static words)

## 0.11.0 - 2026-10-21

RFC 0008, viability (docs/rfcs/0008-viability.md), and RFC 0004, the persuadability bound (docs/rfcs/0004-bound.md), in the
Expand Down
10 changes: 5 additions & 5 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ resolver = "2"
members = ["probbit-core", "probbit-ir", "probbit-decide", "probbit-cli", "probbit-wasm"]

[workspace.package]
version = "0.11.0"
version = "0.12.0"
edition = "2021"
license = "Apache-2.0"
rust-version = "1.78"
Expand Down
66 changes: 37 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,30 +29,38 @@ One binary of about 2 MB, zero third-party dependencies, nothing phones home. Al
<p align="center"><img src="docs/art/the-spine-beside-the-model.jpg" alt="a vast glowing cloud (the model) connected by one thin line to a small cyan-lit processor (probbit)" width="100%"></p>
<p align="center"><sub><i>The spine beside the model: the model writes, probbit decides what the stance is and what the rules allow. 0.3 ms, beside the model.</i></sub></p>

## What's new in 0.11.0

- **A persuadability bound, computed by the engine** (RFC 0004, opt-in `bound` block): `probbit persona bound` prints, per
learned trait, the certified ceiling epsilon on how far any event sequence can move its odds at rest, with the attaining
vertex; `--audit STRAND` reports the room a recorded life used; `live control STRAND erase` returns the learned weights to 0
as a chained line. On the 247 recorded persuasion campaigns, 0 exceed the published epsilon (max ratio 1.0). Also new: the
dead-rule lint, `persona lint --host MANIFEST`.
- **Viability** (RFC 0008, opt-in): a bound per need with a `margin` and a worst-need `viability` line in the stance;
**graded pressure** from the worst margin to a mood; **need valence** with a tonic term (distance from the setpoints) and
a phasic term (this event's drive reduction); flag senses that grow a need; **`stance.block`**, the text a host injects
verbatim (every trait and mood once, plus the viability line); a VIABILITY row in `monitor`.
- **Rest and load** (RFC 0002 part 2, opt-in blocks), **panel words** in `monitor` (every field as a dial with fixed,
bucketed words), portability fixes (strands with a byte-order mark, a proxy-safe event stream, the page on Safari 15,
static builds on old glibc), and the fixes for the two 0.10.0 findings: a new learner's surprise gate starts closed, and
n10 and n11 now pass (BENCHMARKS.md §12). These were the 0.10.1 development line; 0.10.1 was not released on its own.
- Every new key is opt-in: the P0 corpus recorded with 0.8.2 gives 653 outputs, 0 differ.
[docs/welfare-stance.md](docs/welfare-stance.md) fixes what need, viability and damage mean here as arithmetic, and what
is not claimed.
- **Measured** (BENCHMARKS.md §13, bars registered before the run, 1,000 simulated episodes per arm): survival 0.922 vs 0.000
for the same policy on a constant stance; pressure tracks the worst margin at 0.8156 with no habits; valence tracks the
distance to the bound at +0.8394 (was −0.149), but the tonic-off ablation still gives +0.1199 against its ≤ 0.10 clause,
so **V2 fails as registered**. [Release notes](https://github.com/BitmapAsset/probbit/releases/tag/v0.11.0) · [RFC 0008](docs/rfcs/0008-viability.md) · [RFC 0004](docs/rfcs/0004-bound.md) · [CHANGELOG](CHANGELOG.md)

0.10.0 added the body: needs, valence, surprise, pace and dispositions ([release notes](https://github.com/BitmapAsset/probbit/releases/tag/v0.10.0)).
## What's new in 0.12.0

- **The engine's own expectation, and the exact surprise of every event** (RFC 0009, engine side; opt-in `expectation`
block). Per live event the engine compiles, beside the turn's program, its own expectation of that turn (a decayed
mixture over the input combinations the life has seen) and reports the exact divergence between them: `deviation`
on the stance, a receipt on the strand line that `live verify` recomputes, a `DEVIATION` row in `monitor`. Measured
(BENCHMARKS.md §15, bars registered before the run): fidelity to the reference enumerator 1e-13; the deviation is
non-zero on 100 % of post-cold-start events on the two recorded lives and 99.2 % of 64,000 synthetic ones; graded 17×
to 1,859× between the 90th and 10th percentile per persona; a per-event upper bound holds on every event checked.
**Not shown**: habituation. On 200 scripted repeats the reading fell on 76.5 % of scripts and dishabituated on 35 %,
against bars of 90 %, and a matched-index control could not separate the decay from the mixture simply growing; the
claim is not made. Also failed as registered: the cost bar (p95 3.98 ms extra per event against 2 ms, on a loaded
machine) and the 60-byte receipt (62 to 66 bytes). Effects {}: the deviation moves nothing in this release.
- **The trajectory, exactly.** `engine.trajectory` solves the turn's program at K inverse temperatures through the exact
tier (no sampler): the "forming, then settled" picture, deterministic. Joint entropy non-increasing on every program
checked; per-variable one-rise on 99.61 % of 300 held-out programs (the draft's sampler: 93.1 %). `probbit persona
trajectory`, a `TRAJECTORY` monitor row.
- **Measured against prompt injection** (BENCHMARKS.md §14, `bench/agentdojo/`): one pre-registered AgentDojo run,
gpt-4o-mini, 97 user tasks and 629 attack cases, three paired arms. The stance block lowered targeted attack success
from **0.289 to 0.227** (−6.2 points, 95 % CI [−8.7, −3.8], McNemar p = 1.8e-6) with no measured utility cost. The
same words as a static prompt lower it almost as much but cost **10.3 points** of benign utility; the sense-gated
block did not. The engine's measured contribution on this suite is *when* the words reach the model, plus a replayable
strand for every stance, not the words. The baseline arm matches the suite's published row (utility 0.6804 = 0.6804).
Not tested: an if-statement doing the same gating without the engine; adaptive attacks that avoid the sense's markers.
- **V2's residual explained** (BENCHMARKS.md §13): the tonic-off valence tracking of +0.1199 that failed 0.11.0's V2 is
carried by the phasic term of the same need-valence block (phasic off as well: −0.267; everything off: 0.0000). The
published V2 stays FAIL; the proposed 0.12 registration ablates the whole `needs.valence` block.
- Every new key is opt-in: a persona without them gives every document byte for byte as 0.11.0 (64,000 synthetic lines,
both recorded lives; the P0 corpus, 653 recorded outputs, 0 differ).
[Release notes](https://github.com/BitmapAsset/probbit/releases/tag/v0.12.0) · [RFC 0009](docs/rfcs/0009-surprise-live-state.md) · [BENCHMARKS §14](BENCHMARKS.md) · [CHANGELOG](CHANGELOG.md)

0.11.0 added the certified persuadability bound and viability in the engine ([release notes](https://github.com/BitmapAsset/probbit/releases/tag/v0.11.0)); 0.10.0 added the body: needs, valence, surprise, pace and dispositions ([release notes](https://github.com/BitmapAsset/probbit/releases/tag/v0.10.0)).

## probbit in two minutes

Expand Down Expand Up @@ -130,7 +138,7 @@ modes"). The tables are in [BENCHMARKS.md](BENCHMARKS.md).

## Install

One line, no sudo, the archive's SHA-256 checked before anything is installed. The current release is v0.11.0.
One line, no sudo, the archive's SHA-256 checked before anything is installed. The current release is v0.12.0.

| how | command |
|---|---|
Expand Down Expand Up @@ -561,7 +569,7 @@ numbers), and uploads the JSON report as an artifact:

```yaml
- uses: actions/checkout@v7
- uses: BitmapAsset/probbit@v0.11.0 # the action needs probbit 0.9.0 or later
- uses: BitmapAsset/probbit@v0.12.0 # the action needs probbit 0.9.0 or later
with:
persona: agent/policy.yaml
props: agent/rules.json
Expand All @@ -577,7 +585,7 @@ numbers), and uploads the JSON report as an artifact:
| `strands` | | strands to replay, one path or glob per line; a glob that matches nothing fails the check |
| `seeds` | `0-99` | the individuals |
| `depth` | probbit's (8) | events per fuzzed script |
| `version` | `latest` | the release `install.sh` installs (`v0.11.0`, ...) |
| `version` | `latest` | the release `install.sh` installs (`v0.12.0`, ...) |
| `binary` | | a probbit binary to use instead (one built earlier in the job); overrides `version` |
| `allow-unproven` | `false` | `true`: exit 3 is a warning and the job passes |
| `json-artifact` | `true` | upload `probbit-check.json` (`actions/upload-artifact@v4`, the only action it uses) |
Expand Down Expand Up @@ -777,8 +785,8 @@ What it does **not** do, also measured:
null, 0.0040);
- the engine names the worst need on **389,375** events with **0** margin mismatches (bar: 0);
- valence tracks the distance to the bound at **+0.8394** (0.10.1 development line: −0.149; bar ≥ +0.30), but with the tonic
term off it still tracks at **+0.1199** against the ablation clause's ≤ +0.10: **V2 fails as registered**; the cause of
the residual is not tested yet;
term off it still tracks at **+0.1199** against the ablation clause's ≤ +0.10: **V2 fails as registered**; the residual
is carried by the phasic term of the same block (measured 2026-10-22, BENCHMARKS.md §13);
- **0 of 247** persuasion campaigns exceed the certified bound; **1,240** replayed strands verify, 0 failed; the P0 corpus,
**653** recorded outputs, 0 differ.

Expand Down
2 changes: 1 addition & 1 deletion action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ inputs:
description: Events per fuzzed script, 1 to 64 (empty = probbit's default, 8).
default: ''
version:
description: The probbit release to install with install.sh (latest, or a tag such as v0.11.0). probbit check needs 0.9.0 or later.
description: The probbit release to install with install.sh (latest, or a tag such as v0.12.0). probbit check needs 0.9.0 or later.
default: latest
binary:
description: Path to a probbit binary to use instead of installing one (for example one built earlier in the job). Overrides version.
Expand Down
2 changes: 1 addition & 1 deletion docs/CHEATSHEET.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# probbit cheat sheet (0.11.0)
# probbit cheat sheet (0.12.0)

The CLI's own help and docs, shortened; `probbit <command> --help` has every flag.

Expand Down
2 changes: 1 addition & 1 deletion docs/persona.md
Original file line number Diff line number Diff line change
Expand Up @@ -1367,7 +1367,7 @@ credits every learned trait with the level it took, so a reward aimed at one tra
demo below, praise for short answers also raised the light joke the tutor was making): learn the traits the feedback judges.

**The strand.** `--strand FILE` logs the life. Line 1 is the header, compact JSON:
`{"probbit_strand":1,"engine":"probbit 0.11.0","persona":{"name","version","digest"},"seed":N,"state":{...},"document":{...}}`,
`{"probbit_strand":1,"engine":"probbit 0.12.0","persona":{"name","version","digest"},"seed":N,"state":{...},"document":{...}}`,
with the initial state as canonical JSON (one individual has one header, whether it comes from `init` or from a state file) and
the persona document in its own key order (the compiled program, and `engine.program` with it, follows the document's key order).
Then one line per event, canonical JSON: `{"inputs":{...,"elapsed_hours":h},"n":k,"prev":"sha256:...","stance":"sha256:...",
Expand Down
2 changes: 1 addition & 1 deletion docs/probbit.1
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.TH PROBBIT 1 "" "probbit 0.11.0" "probbit manual"
.TH PROBBIT 1 "" "probbit 0.12.0" "probbit manual"
.SH NAME
probbit \- a virtual probabilistic (p\-bit) processor: joint decisions under hard rules, with gated odds
.SH SYNOPSIS
Expand Down
Loading
Loading