[insterbridge]

Problem

Prompt caching collapses on large conversations. With a ~400k-token context the
gateway reported cache_read of only ~8.8k — about 1.5% of the body.

Cause: the client's own system prompt is folded into block 2 of the
canonical 3-block system array, and that block carries the cache_control
breakpoint. Harnesses vary their system prompt per turn — omp injects live
context (date, cwd, counters) — so block 2 changed on every request and
invalidated every message segment behind it.

Measured on a live session, 1.82 MB conversation:

	before	after
reusable prefix	26,433 bytes	1,810,729 bytes
loss	98.5%	0.07%

Fix

Two changes, both small:

1. buildStartPlanSystem no longer folds the client text into the system
   blocks, so they are byte-stable across turns.
2. applyStartPlanSystem attaches that text as a user turn, inserted before
   the final message.

Placement is the subtle part. applyAnthropicCacheControl marks the last
non-system message's last block. Appending the volatile text after that point
makes the breakpoint itself vary, so every request writes a fresh cache entry
and reads nothing — I measured cache_read=0 with a perfectly stable history
before moving it. Inserting it ahead of the marked turn anchors the breakpoint
on append-only history.

Verification

Against the live gateway on glm-5.3-flash, identical 20-turn history, client
system prompt deliberately changed between the two calls:

write:  input_tokens=15351  cache_read=256
vary:   input_tokens=60     cache_read=15552

Full prefix reused. The instructions still reach the model.

Tests pin both properties:

• differing client prompts produce byte-identical system blocks
• the cache breakpoint never lands on the volatile instructions

bun test: 1121 pass, 0 fail. tsc --noEmit: clean.

Tradeoff

Client system instructions now arrive as a user turn rather than a system
block, so they may carry slightly less authority. The alternative is keeping
them in the system block and losing the cache for the whole conversation.

[TriDefender]

I'm not sure what will changes will happen to agentic turns if you pass the harness instructions in as user context, did you try that before hand? also the current system-prompt injected carried filepath locations of the proxy program file (which is how zcode structured their system prompt) this might add more confusion regarding r/w operations

[TriDefender]

Thanks for investigating this and sharing the gateway measurements. The cache loss is plausible: if a harness changes its system prompt on every turn, a prefix cache can lose reuse for the conversation that follows, including when the harness talks directly to an upstream service. Our proxy also changes the prompt structure, so its handling of block boundaries and cache markers can affect how much is reusable.

I’m concerned about adopting this particular fix without further validation:

1. Instruction semantics. Start-plan already injects ZCode’s system instructions. Moving the client harness’s system prompt into a user message changes its role and can put its tool, permission, and output rules in conflict with the ZCode instructions. Reaching the model as text does not establish that the harness still behaves as intended.
2. Growing conversations. The reported live test varies the client system prompt over an identical history. In a real append-only session, the inserted message is reconstructed before the current last message, so it moves between requests. For example, the wire histories can look like H, B1, M1 and then H, M1, assistant response, B2, M2. They diverge immediately after H, even if B1 and B2 are identical. Could we measure cache reads across several actual growing turns, with both stable and changing client prompts, rather than extrapolate from identical history?
3. Tool-call pairing. Inserting before the last message can place the extra user message between an assistant tool_use and its tool_result. Whether the gateway merges, accepts, or rejects that sequence needs verification; a unit test should cover it as well.

Could we also compare representative harness behavior before and after the change, especially conflicting tool/permission instructions, and test the relevant gateway paths? The cache problem deserves attention, but these compatibility tradeoffs need to be understood before merging.

[TriDefender]

If your harness is changing system_prompt every turn then you will get cache misses no matter you use this proxy or not.

[insterbridge]

You are right on all three points, and I verified the last two in code rather than arguing from the earlier measurement. Two of them were real defects in the patch as submitted.

Point 3 — tool-call pairing (confirmed defect). Inserting before the last message put the extra user turn between an assistant tool_use and its tool_result:

2:assistant  text+tool_use
3:user       # Additional instructions
4:user       tool_result

That breaks the required adjacency. Fixed.

Point 2 — growing conversations (confirmed defect, and the more serious one). You were exactly right that splicing before the current last message makes the block move. Real append-only turns produced:

turn1: ... | u:a | a:b | u:# Additional | u:c
turn2: ... | u:a | a:b | u:c | a:d | u:# Additional | u:e

The prefix diverges right after a:b every turn, so the identical-history test I ran was measuring the wrong thing. My original measurement was therefore not evidence for the real workload.

Current approach. Append the client text after the history instead, and move the cache breakpoint off it: applyAnthropicCacheControl now skips a synthesized instruction tail and marks the last real conversation turn.

turn1: ... | u:a | a:b | u[BP]:c | u:# Additional
turn2: ... | u:a | a:b | u:c | a:d | u[BP]:e | u:# Additional
tool : ... | a:R+tool_use | u[BP]:tool_result | u:# Additional

That is append-only, keeps tool adjacency, and the breakpoint sits on history rather than on volatile text.

Point 1 — instruction semantics. I do not have evidence that moving harness instructions from system to user preserves tool, permission, and output behavior, and I should not have shipped it as though the cache result settled that question. Your framing is right: reaching the model as text is not the same as the harness still behaving as intended. The filepath-location concern is also fair — the injected system prompt carries proxy paths, and instructions arriving as user context alongside that invites exactly the confusion you describe.

I cannot produce the measurements you asked for right now: both models on this account have hit 1005 exceed quota limit, so I have no live gateway numbers for growing turns, tool-call sequences, or before/after harness behavior. I would rather say that than extrapolate again.

Two options, your call:

1. I finish the validation you asked for once quota resets — growing turns with stable and changing prompts, a tool-call sequence through the gateway, and a harness comparison on conflicting instructions — and report the numbers before asking for review again.
2. Close this PR and reopen it as an issue describing the cache loss with the measurements I do have, leaving the instruction-semantics design to you.

Your third comment is the part I keep coming back to: if the harness rewrites its system prompt every turn, the miss happens upstream too. That suggests the more useful fix might be on the harness side, or scoped to only the volatile part of the prompt, rather than relocating all client instructions. I would like your read on which of those is worth pursuing.

[insterbridge]

Closing this. Your review was right on the points that mattered, and the more I measured the less the proposed fix looked like the right shape.

Summary of what the investigation actually established:

The proxy-side bug is real. Upstream's issue #71 fix folds the client system prompt into block 2, which carries the cache breakpoint. On a live 1.82 MB session that left 26,433 bytes reusable — 1.44%. Keeping the client text out of those blocks raised it to 1,810,729 bytes, 98.76%. Those are measurements from real traffic, not a synthetic harness.

But the fix I proposed is not the right one. Two defects you identified were real, and I confirmed both in code: inserting before the last message wedged a user turn between tool_use and tool_result, and it made the inserted block move as the conversation grew. I fixed both by appending after the history and skipping the volatile tail when placing the breakpoint.

And your third comment is the one that settles it. With a stable system prompt, growing turns reuse 12,416 tokens every turn (input_tokens 14/24/34). With a system prompt that changes each turn, the same conversation re-bills ~10.5k on turns 1 and 2 before recovering to 99.2% by turn 3. The miss originates in what the harness sends, not in the proxy. Moving the text around inside the proxy does not change that.

Instruction semantics I never validated. I have no evidence that relocating harness instructions from system to user preserves tool, permission, or output behavior, and the filepath concern you raised is a fair reason to be cautious about it independently.

I am keeping the cache-stable-system-block change locally, since it measurably helps my setup, but I agree it should not land upstream on this evidence. Thanks for the careful review — points 2 and 3 in particular saved a bad merge.