Skip to content

feat(compaction): hoist system messages into a stable cache-friendly prefix - #124

Open
tishachawla-jg wants to merge 1 commit into
mainfrom
compaction-cache-prefix
Open

feat(compaction): hoist system messages into a stable cache-friendly prefix#124
tishachawla-jg wants to merge 1 commit into
mainfrom
compaction-cache-prefix

Conversation

@tishachawla-jg

Copy link
Copy Markdown
Collaborator

Summary

Improves cost compaction by attacking the real cost driver: the prompt-cache discount. Cached prefix tokens bill far cheaper than fresh ones, and a volatile value ahead of the static instructions busts the whole discount.

_compact_messages (in src/tokenops/control/integration.py) already deduped non-system messages but pinned system messages in place. It now hoists all system messages into a stable leading prefix — static instructions first, volatile conversation after — preserving relative order within the system block and within the tail.

This closes the gap between what the context_compaction policy already documents ("move volatile values below the static prefix") and what the live rewrite actually did.

Behavior

  • No-op when system messages are already first (the common case), so most callers see no change.
  • When a system message was interleaved later, it moves into the prefix. Honest tradeoff, noted in the docstring: the first such reorder busts the old cache once, then the new stable order caches; net win when the system block is stable across calls.
  • Dedup of non-system messages unchanged (keeps first occurrence).

Tests

  • New: test_compact_hoists_system_into_stable_prefix, test_compact_is_noop_when_system_already_first.
  • Full suite: 227 passed, 2 skipped, no regressions.

Not in this PR (follow-ups)

🤖 Generated with Claude Code

…prefix

Cost compaction's biggest lever is the prompt-cache discount: cached
prefix tokens bill far cheaper than fresh ones, and a single volatile
value ahead of the static instructions busts the whole discount.

_compact_messages already deduped non-system messages but pinned
system messages in place. Now it hoists all system messages into a
stable leading prefix (static instructions first, volatile
conversation after), preserving relative order within the system
block and the tail. No-op when system is already first, which is the
common case.

Closes the gap between the context_compaction policy's stated
technique ("move volatile values below the static prefix") and the
live rewrite. Adds unit tests for the hoist and the no-op path; full
suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant