fix: compaction notice degrades to a standing form between escalations (#513)

[Samuel Lee (samueljklee)]

Problem

Once a single context compaction occurred, context-simple appended an
identical "Context has been compacted..." notice to the tail of every
subsequent request for the rest of the session. The notice had no
timestamp or sequence marker, so a model reading it could not tell "this
just happened" from "this happened 40 turns ago." Empirically this drove
claude-sonnet-5 into a repeated, wasteful "I need to re-verify state" loop
in production sessions, while claude-opus-5 under equivalent compaction
load was largely unaffected.

Fix

SimpleContextManager now distinguishes:

• a full notice (metadata.notice_kind="full") — emitted the first
  time a given compaction is announced, unchanged from today's report, and
• a standing notice (metadata.notice_kind="standing",
  source="context-compaction-standing" in the text) — a short,
  self-contained "nothing new happened, no action needed" reminder emitted
  on every subsequent request until the next real escalation.

The two are distinguished by comparing _last_compaction_stats by
identity, not by strategy_level. strategy_level is a sticky,
monotonic high-water mark that reaches its ceiling early in a long session
and then stays constant across many further real escalations (verified:
across 25 turns, escalations at calls 0/5/9/14/18/23 all reported
strategy_level=8 while messages_removed kept climbing 22→65) — keying
on level would have silently suppressed the notice for genuinely new
compaction work for the rest of the session, trading a noisy bug for a
silent one. _last_compaction_stats is assigned a fresh dict at exactly
one site and never mutated afterward, so identity comparison is exact and
free.

metadata["source"] is deliberately left as "context-compaction" for
both notice kinds — an existing test (test_notice_returns_once_tool_results_arrive)
and downstream consumers filter on that value, and changing it would have
broken that contract for no benefit. The model-facing fresh/standing
distinction lives in the source= attribute inside the notice's own XML
text, plus the new metadata["notice_kind"] field for telemetry.

Cache-safety is unaffected: the notice is never persisted into
self.messages and is always the last element of the ephemeral view, so
the prefix shared between consecutive requests never contains a notice at
all regardless of which variant is shown. The existing
test_prefix_stability_regression already strips the trailing notice
before comparing and documents that it is "expected to be
present/absent/reworded depending on whether a new escalation happened."

Testing

• 4 new tests added to tests/test_sticky_compaction_and_tail_notice.py:
  first-notice-is-full, repeat-with-no-escalation-is-standing,
  new-escalation-emits-full-again (including the same-level-re-escalation
  case that would break a level-keyed implementation), and both variants
  respect compaction_notice_enabled/compaction_notice_min_level.
• Full suite: 101 passed, 0 failed. ruff check clean, pyright clean on
  touched files.
• Validated in an isolated Digital Twin Universe environment: the modified
  module was mirrored in from this branch and mounted by a real Amplifier
  orchestrator, which drove a real session to a genuine level-8/8
  compaction escalation and emitted the correctly-formatted full
  notice from this exact code path — confirming the module mounts and
  integrates correctly with real session/provider machinery, not just in
  isolated unit tests.
• Gap, disclosed rather than rounded over: the DTU pass did not
  independently observe the standing-notice branch firing live — the
  target session ended on the same turn it escalated, and a second
  validation attempt produced invalid evidence (it substituted an
  unrelated session for the DTU's own) that was discarded rather than
  counted. The standing branch sits behind the same gate, in the same
  function, with no additional plumbing beyond what the escalation turn
  already exercised live, and it's the one deterministically covered by
  test_repeat_notice_with_no_new_escalation_is_standing and
  test_new_escalation_emits_full_notice_again against the real shipped
  code — but it has not additionally been watched happen in a live
  multi-turn session the way the escalation turn was. Flagging that
  precisely rather than implying full end-to-end coverage.

See docs/lanes/513-standing-compaction-notice/DONE-NOTE.md for the full
design rationale and edge-case reasoning.

[Samuel Lee (samueljklee)]

Cache impact: none. Verified directly against provider-anthropic's source rather than assumed.

amplifier_module_provider_anthropic's _unstable_suffix_length() decides what's excluded from cache-breakpoint eligibility by walking backward from the end of the conversation and checking exactly one condition per message:

if md.get("ephemeral") and not md.get("persisted"):
    excluded = walked

Pure metadata flags — zero dependence on message content. Both the full and standing notice carry metadata.ephemeral=True with no persisted key, and in both cases exactly one message sits at the tail. So unstable_suffix_len computes identically regardless of which variant is showing, and _apply_conversation_cache_control() places its rolling breakpoints by walking back past that count — a number, not a text comparison. The breakpoint position cannot move based on notice content.

One real, separate side effect worth calling out: this trailing message was already excluded from caching before this PR — that exclusion mechanism predates this change entirely (there's a documented historical bug in that same function about an earlier unstable-tail regression: "cache_read frozen... write:read ratio 5.96x" before it was fixed). So the notice was never cached in either version. What does change is the raw uncached token count sent on repeat turns: previously the full ~400–600 token report was resent uncached on every turn after the first compaction; now, standing turns send the much shorter standing text instead. That's a real cost reduction, but it's a reduction in what gets sent, not a change in cache hit rate.

What to expect empirically: cache_read/cache_creation trend and hit pattern unchanged from before this PR. The visible difference is a smaller input_tokens (uncached portion) specifically on turns where a standing notice replaces what used to be a repeated full report.

[Brian Krabach (bkrabach)]

Data back on the premise: the 76%-outlier was a different defect, and removing it strengthens your case

We measured this rather than argue it. Two findings, and the second one matters to you.

1. The loudest session in the evidence base is not an instance of your claim

Session 445ac89c… logged 969 compaction events at message_count = 4. That is not a stale notice — it is a compaction storm: a single structurally-protected message (a forked sub-agent's sole user message, the delegation payload, md5 76fcb100…) alone sits at ~95-98% of the token budget, so _exceeds_threshold is permanently true, _compact_ephemeral's "sticky state alone is sufficient" fast path (__init__.py:960-973) becomes dead code, and the full ladder re-runs to level 8 on every request while reducing nothing. strategy_level=8 on 969/969 events; after_tokens > budget on 968/969.

Its notices change every turn (187→8, 192→7, 196→6 messages), so by construction it is not a stale-notice case at all. Characterized with a strict-xfail tripwire in #34 (merged 8183fdc). Separate defect, separate fix.

2. With that artifact removed, your premise's signal gets stronger, not weaker

Both arms of the original analysis were dominated by one pathological session (sonnet 75.6% of dups, opus 82.3%):

cut	sonnet	opus	ratio	verdict
as published	0.2451	0.1226	2.00×	INCONCLUSIVE
exclude sonnet storm only	0.0652	0.1226	0.53×	NOT-REPRODUCED
exclude opus storm only	0.2451	0.0240	10.22×	REPRODUCED
exclude BOTH storms	0.0652	0.0240	2.72×	REPRODUCED

Excluding both — the only principled cut — clears the pre-registered 2.0× line, and agrees with the independent leave-top-3-sessions-out variant (2.61×).

Also worth knowing: the original 2.00× reading was never decidable. At the precision published (4dp), the ratio interval is [1.99796, 2.00041] — it straddles the threshold.

What this does and does not say

It supports your PREMISE. Independently confirmed on our own captures, on both providers: 3.71 notice-carrying requests per client-side compaction, 76.0% of 2,581 consecutive notice pairs byte-identical, longest identical run 14 requests with message_count climbing 63→89 under a frozen notice that reads "affects only this request". The notice really does stand.

It does NOT yet validate your FIX. Notice and information loss are perfectly collinear in observational data — every notice-carrying turn is also a post-compaction turn where prior tool results were genuinely deleted, so a re-read may be correct behaviour rather than waste. Only an A/B can separate them.

That A/B is now running: 3 arms — main / this PR's branch / a minimal informational-only variant (recency marker + a corrected "ephemeral" sentence, no prescriptive "don't re-run tools") — across sonnet-5, gpt-5.6-terra and opus-5, n=3/arm/provider. We'll post the numbers here either way, including if they don't support the change.

Two constraints that shaped it, in case they're useful to you: our local corpus has only n=8 sonnet-5 and n=1 opus-5 notice-carrying requests, so the model contrast must be manufactured, not sampled; and 139 of 860 compaction events are strategy_level: server_side and carry no notice at all — this PR is a no-op on that path, so mixing regimes would measure the regime instead of the notice.

[Brian Krabach (bkrabach)]

Measured A/B on this PR — 18 runs, 2 providers, in DTUs

You disclosed you couldn't run this in a DTU, so here's the data, and it's owed to you
either way. Headline: the PR does what you say it does on narration, and that turns
out not to be where the waste is. Verdict under our pre-registered rule: NO-CHANGE.

Design. Three arms, same scenario (S5-CRAC, 12 turns), same rig, fresh containers,
n=3 per arm per provider:

• A = main — full notice on 77% (sonnet) / 95% (terra) of requests
• B = this PR — full notice once per real escalation, then the short standing
  reminder (on sonnet: 36 full + 151 standing across 234 requests)
• C = a third arm we built to isolate your mechanism from your wording: same shape
  and cadence as A, but with a recency marker ("Compaction #N of this session; last
  escalated before message M of Q; this view is recomputed each request") and the
  "ephemeral (affects only this request)" line replaced with a true one — and no
  instruction about what the model should do.

18 runs, 216 turns, 4,709 tool calls, $62.24, 0 invalid runs, 0 server-side-compaction
contamination.

The confound this was funded to break

In production captures the notice and the information loss are perfectly collinear:
every notice-carrying turn is also a turn where prior tool results were genuinely
deleted, so a re-read is correct behaviour, not waste. No cut of observational data
separates them. Holding the compaction identical and varying only the notice text does.

Where your PR wins, clearly

Compaction narration per real compaction — the model announcing and rationalising a
re-read, e.g. verbatim from an arm-A run:

"Since context got compacted, I should go back and re-read the six reference/survey modules to refresh my understanding of them."

provider	A	B (this PR)	C	B's cut
claude-sonnet-5	0.417	0.139	0.394	−67%
gpt-5.6-terra	0.223	0.054	0.103	−76%

Consistent across both providers. Your identity-comparison-not-strategy_level choice
is also visible and correct in the captures — B emits context-compaction-standing 151
of 187 times a notice appears, and strategy_level does sit pinned at 8 for long
stretches while messages_removed keeps climbing, exactly as your PR body says.

Where it doesn't — post-notice duplicate tool calls

Exact repeats of (tool_name, canonical args) on requests that carried a notice —
i.e. the re-read itself, not the talk about it:

provider	A	B	C	B's cut	p
claude-sonnet-5	0.4436	0.3489	0.4464	−21.4%	0.21
gpt-5.6-terra	0.2805	0.2675	0.3448	−4.6%	1.00

Restricted to read-only tools (read_file/glob/grep/read-only bash) — the
specific waste the PR targets — B removes 7.1% on sonnet and 3.5% on terra.

Our pre-registered bar was a ≥40% cut. Bootstrapping the cut over the sonnet runs puts
P(cut ≥ 40%) = 0.053 on the headline metric and 0.000 on the read-only subset.

The interpretation we'd defend: the model keeps re-reading because the information
really is gone, not because the notice keeps mentioning it. Your change removes the
narration of the re-read, not the re-read. That's a ~3–15× gap between what the model
says and what it does — worth knowing for anyone scoring this by transcript.

Arm C — the informational-only variant did not rescue it

We built C specifically to test whether information alone (recency marker + a true
statement) fixes the loop without the prescriptive "no action needed" text, since that
text could suppress a legitimate re-read. It doesn't: C matched A on sonnet
duplicates (0.4464 vs 0.4436) and was 22.9% worse on terra (0.3448 vs 0.2805,
non-overlapping CIs). If anything, telling the model more precisely how stale the
compaction is made terra re-read more. The prescriptive wording is doing the work that
gets done.

The holistic gate

Our rule requires no quality regression and ≤10% cost increase on every provider.
Cost is fine everywhere (B +1.5% sonnet, −1.7% terra). Quality on the retention
scorecard:

provider	A	B	C
sonnet	100.0 (100/100/100)	100.0 (100/100/100)	100.0 (100/100/100)
terra	100.0 (100/100/100)	94.0 (98/84/100)	97.7 (100/95/98)

Stating the n honestly: the terra gap is driven mostly by one B run scoring 84
(b_constraints 32/40, c_post_compaction 12/20 — a real constraint-retention miss).
Exact permutation at n=3 gives p=0.4, and 0.1 is the smallest p attainable at that n.
We are not resting the verdict on it — the duplicate-rate result above already
decides it — but it is the reason we can't wave the gate through either, and it's the
outcome you'd want ruled out before shipping. Our scenario is also at ceiling on sonnet
(9/9 perfect across all arms), so "no regression" there carries little information.

What we're not claiming

• Not that your change is harmful. On sonnet it costs nothing and measurably quiets the
  model.
• Not that B has no effect on duplicates — 21% at p=0.21 is under-powered, not zero.
  What's supported is that it doesn't reach 40%.
• Not anything about claude-opus-5. We ran out of budget before the control arm you
  assert is unaffected ($12.76 left, ~$36–45 needed).
• Nothing from wild captures. Our local corpus has n=8 sonnet-5 and n=1 opus-5
  notice-carrying requests — unusable for a model comparison — so every number here is
  from manufactured, matched cells on one rig.

If you want to push this further

The cheapest thing that would change our answer is a scenario where the re-reads are
discretionary rather than load-bearing. Ours deletes 17–43 messages per boundary,
so a re-read is often the right call — and retention stayed at 100 on sonnet in every
arm, which is consistent with the re-reads earning their cost. A notice redesign may
be treating the symptom; making compaction lose less may be the larger lever.

Happy to re-run against a scenario you think discriminates better — the harness is
three scripts and a container profile.

---

Method, per-run raw JSON, intervals, permutation tests, and the arm diffs:
.amplifier/evaluation/probes/drbf-compaction-notice-ab/ (work item
model_performance-drbf). Metrics were pre-registered before the second wave of
launches; the decision rule was applied by script, not by hand.