test: compaction storm -- an irreducible protected message re-escalates to level 8 on every request (#fwut)

[Brian Krabach (bkrabach)]

Test-only. No runtime behaviour changed. Diagnoses a measured production pathology and pins it with a fail-before assertion. Item model_performance-fwut. Spend $0.00 against a $0.00 authority — all evidence is read-only graph queries and grep.

Full analysis: docs/lanes/fwut-compaction-storm/DONE-NOTE.md

The defect

When a single structurally-protected message is, on its own, at or above compact_threshold × budget, _exceeds_threshold is permanently true. _compact_ephemeral's "sticky state alone is sufficient — nothing NEW to decide" fast path becomes unreachable, so the full escalation ladder re-runs to level 8 on every single request, reduces nothing, and emits a context:compaction event and a compaction notice each time.

Three protection rules make the floor irreducible, and they interlock:

rule	consequence
system messages are never compacted	fixed floor
_remove_messages_with_protection: "Removal candidates exclude ALL user messages"	the payload cannot be dropped
Level 8 stubs the first user message "but NEVER if it's also the last" (first_user_idx != last_user_idx)	a forked sub-agent has exactly one user message → first == last → skipped
Level 8's stub also needs isinstance(content, str) (inline and in _stub_user_message)	block-structured content → skipped even when it is not the last

Two independent guards, either sufficient. Observable result: user_messages_stubbed == 0 on all 969 events — the one lever that could shrink the floor never fired once.

_finalize_compaction_with_stats already diagnoses this (it logs the un-reducible floor by name) but only when the result is over budget, and it does not stop the loop.

The measurement (team-shared context-intelligence graph, 2026-09-05)

0000000000000000-445ac89c107c4f52_anchors-amp-dev-architect — claude-sonnet-5, 969 context:compaction events across 988 llm:requests in 3h26m:

strategy_level	8 on 969/969 — max, from the very first event
after_tokens > target_tokens	969/969 (100%)
after_tokens > budget	968/969 — compaction finished and the request was still over budget
user_messages_stubbed	0 on 969/969
budget / target	131,104 / 65,552 on every event
after_messages	2 … 12
before_messages	3 → 5,174, monotone, never once decreasing

First event, whole story in five numbers:

before 3 msgs / 131,321 tok  →  after 2 msgs / 130,627 tok
budget 131,104 · target 65,552 · strategy_level 8 · messages_removed 1

Three messages; the last level of the ladder; 694 tokens freed; still 65,075 above target.

The floor, from the raw request body (per-message sizes only):

[0] role=user  content=list[text:516,153]  516,184 chars  ≈ 124,338 est tok  ← 94.8% of budget
[1] role=assistant   2,663 chars      [2] role=user (tool_results)  3,058 chars
[3] role=user (the compaction notice)  917 chars

Message [0] is the delegation payload of a forked sub-agent (session:fork, anchors-amp-dev:architect). est(msg[0]) 124,338 + est(system) 6,289 = 130,627 — exactly the observed after_tokens. Its md5 is identical at the first request and the last, 988 requests later.

A second flavour of the same defect, verified on the other arm's outlier — 877774a55cd1430c_foundation-explorer (claude-opus-5): 2,165 events, level 8 on 2,165/2,165, over target on 2,165/2,165, 0 stubs — but here the floor is the system prompt: 402,949 chars ≈ 100,737 est tok. Sonnet's floor is the user message; opus's is the system prompt. Both shapes are covered by the tests.

Not one session

Across the full population (431 sessions, ≥3 compactions since 2026-08-01):

• 33,541 compaction events; 25,307 (75.5%) at strategy_level 8.
• 94 sessions (21.8%) are storms (≥90% of events at max level), carrying 74.6% of every compaction in the corpus.

	root sessions	forked sub-agents
compaction events	6,650	26,891
at strategy_level 8	196 (2.9%)	25,111 (93.4%)
storm sessions	6 / 160 (3.8%)	88 / 271 (32.5%)

Compaction is healthy in root sessions and pathological in forked sub-agents — a 32× difference — across ≥14 agent types in five bundles (foundation-git-ops alone: 10,761 events / 33 sessions).

Why the history "regrows" — it never shrank

Compaction here is ephemeral by construction (_compact_ephemeral: "returns a NEW list — the source messages are NEVER modified"). self.messages is append-only; the compacted view is rebuilt from the full history every request. So the reported 187 → 8 / 192 → 7 / 196 → 6 sequence is not a compact-and-regrow cycle: before_messages is monotone 3 → 5,174 across all 969 events, and 187 → 192 → 196 is simply +5, +4 — one agent turn's worth of growth.

What this PR adds

tests/test_compaction_storm_irreducible_floor.py

• 5 characterisation tests, passing on main — the sole-user-message floor (sonnet shape), the system-prompt floor (opus shape), the isinstance(content, str) guard in isolation, the storm itself (10 requests → 10 compaction events with after_tokens pinned), and the notice re-attached every request.
• 1 xfail(strict=True) carrying the fail-before assertion. Verified failing on main for the right reason:

AssertionError: compaction should recognise an irreducible floor and stop re-escalating;
                got 10 escalations for 10 requests
assert 10 <= 2

with after_tokens pinned at 9,756/9,757 against a 10,000 budget (97.6%) while before_messages grows 25 → 35 — a faithful 13×-smaller replica of production's 99.6%. strict=True is deliberate: it stays green while the defect exists and turns into a hard failure the moment it is fixed, which is the signal to delete the marker.

Suite

$ uv run pytest -q
........................................................................ [ 69%]
...............................                                          [100%]
102 passed, 1 xfailed in 5.34s

Baseline (git stash, i.e. f2dbde9 untouched): 97 passed in 5.11s.
Net +5 passing, +1 strict xfail, 0 regressions, 0 runtime lines changed — default-mode behaviour is byte-identical because no module source was touched.

Why no fix is proposed here

The honest fix is a behaviour change (recognise an irreducible floor, stop re-escalating, stop re-attaching a notice promising a reduction that did not happen). Under a wins-only merge policy that needs a measured run, and this lane's authority is $0.00 — correctly sized for a forensic task, and it funded the whole diagnosis. Shipping an unmeasured default change would be exactly the "unproven default-off feature" already reverted once in e9ac159. The evidence, the reproduction and the assertion are here; the fix decision belongs to the owner. Priced ask for landing it with evidence: unit-level $0 (this xfail flips), plus one S5 A/B at 2 runs × 2 arms × $2.58 ÷ 0.667 valid = $15.48.

Explicitly not PR #33

PR #33 concerns a byte-identical, stale tail notice. Here the notice's stats genuinely change every turn because a fresh compaction really did run. Different pathology, same surface — and this PR changes no notice text and does not touch #33's subject. The relationship is quantified in the DONE-NOTE §6.

[Brian Krabach (bkrabach)]

Manager verification — a real defect, honestly characterized. Merging as finding+test.

Head 48068a9, base f2dbde97 = current origin/main.

What I re-derived myself

The fast path IS unreachable — confirmed by reading, not inferring. __init__.py:960-973:

needs_escalation = self._exceeds_threshold(current_tokens, budget)
if not needs_escalation:
    # Sticky state alone already keeps us under the threshold ...
    return final_messages

If a single structurally-protected message alone sits at/above compact_threshold × budget, current_tokens can never fall below the gate, needs_escalation is permanently True, and that early return is dead code for the rest of the session. The full ladder then re-runs to level 8 on every request and reduces nothing. The evidence matches exactly: strategy_level=8 on 969/969 events, after_tokens > budget on 968/969, user_messages_stubbed=0 on 969/969.

The threshold arithmetic holds, but one intermediate does not reproduce — and it fails in the SAFE direction. The module's estimator is sum(len(str(msg)) // 4 ...):

516,184 chars / 4  = 129,046 est tokens      (the note claims 124,338)
124,338 / 131,104  = 94.84%                  (as claimed)
129,046 / 131,104  = 98.43%                  (my re-derivation)
compact_threshold 0.92 -> exceeds: TRUE either way

So the floor is further above the gate than stated, not closer. The conclusion is unaffected; the stated char→token intermediate is off by ~3.6% and should not be quoted as exact.

The gate I was asked to check, answered plainly: there is NO fail-before that fails on main

I ran the branch's test file against current main: 5 passed, 1 xfailed. Against the branch: identical. The diff contains no source change at all — tests/test_compaction_storm_irreducible_floor.py is the only non-docs file.

That is not a gap being hidden; it is the lane's stated design, at line 51 of its own test:

xfail(strict=True) carrying the assertion that fails today -- the fail-before signal to delete the xfail marker.

This is the correct pattern for a defect you are characterizing and deliberately not fixing. strict=True means the day someone fixes the floor, the test XPASSes and CI goes red, forcing the marker's removal. The behaviour is pinned; nothing is asserted that isn't true today. Merging on that basis — as a finding + a tripwire, not as a fix.

And declining to fix it here was right. The floor is a forked sub-agent's sole user message — the delegation payload, md5 76fcb100…, byte-identical from first request to last, 988 requests / 3h26m apart. Any "fix" that reduces it is a fix that silently drops the one message the sub-agent was given. That needs a design decision (refuse the spawn? split the payload? raise the budget?), not a patch smuggled into a characterization lane.

Suite on branch: 102 passed, 1 xfailed.

Closing the loop on PR #33 — this session was 76% of the sonnet-5 signal

Re-ran the lane's committed pr33_recompute.py:

cut	sonnet	opus	ratio	verdict
as published	0.2451	0.1226	2.00×	INCONCLUSIVE
exclude sonnet storm only	0.0652	0.1226	0.53×	NOT-REPRODUCED
exclude opus storm only	0.2451	0.0240	10.22×	REPRODUCED
exclude BOTH storms	0.0652	0.0240	2.72×	REPRODUCED

Both arms were dominated by a single pathological session (sonnet 75.6%, opus 82.3%). Removing both — the only principled cut — clears the pre-registered 2.0× line at 2.72×, consistent with the hunt's own leave-top-3-out variant (2.61×). Posted to PR #33 as well.