Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
6fd6fad
fix(openai-native): send body prompt_cache_key for cache affinity
myk1yt Oct 30, 2026
0f52a75
fix(cost): account cache writes per TTL class instead of one aggregat…
myk1yt Oct 30, 2026
b647ec7
fix(cost): bill the TTL split on the telemetry recompute and in tier …
myk1yt Oct 30, 2026
025ff9c
test(cost): correct the mixed-split test label to match its arithmetic
myk1yt Oct 30, 2026
282ee30
test(openai-native): cover the 12h rate under each service tier
myk1yt Oct 30, 2026
65ac862
feat(api): emit an opt-in prompt cache TTL gated by catalog capability
myk1yt Oct 30, 2026
bed36d8
fix(api,cost): hash long cache keys, clamp TTL splits, harden the TTL…
myk1yt Oct 30, 2026
072cdc7
fix(api,core): key prompt enhancement off the task id, not the handle…
myk1yt Oct 30, 2026
980308e
Merge remote-tracking branch 'origin/main' into fix/cache-ttl-accounting
myk1yt Oct 31, 2026
5891cdc
merge: bring in #2260 cache TTL accounting at 980308e8a (main@51b7f579f)
myk1yt Oct 31, 2026
923478c
merge: bring in main@070ee98eb (#2233 provider-settings move)
myk1yt Oct 31, 2026
64eaa41
Merge remote-tracking branch 'origin/main' into fix/cache-ttl-accounting
myk1yt Nov 17, 2026
3bb6dc1
fix(providers): dedupe vertex cacheWrite1hTokens and pin 1M-tier 1h c…
myk1yt Nov 17, 2026
fcc9db0
Merge branch 'fix/cache-ttl-accounting' into feat/cache-ttl-knob
myk1yt Nov 17, 2026
c6db408
test(caching): cover TTL accept/reject paths across gateway, vertex c…
myk1yt Nov 17, 2026
8d5912b
docs: land ADR-0006 prompt-cache TTL program and three session reports
myk1yt Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 75 additions & 0 deletions docs/architecture/adr/0006-prompt-cache-ttl-program.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# ADR-0006: Prompt-cache TTL program — catalog-embedded per-TTL pricing, opt-in knob, fail-closed capability gates

> RECONSTRUCTED 2026-10-31: the original file (written 261031 ~01:58) was lost in a docs-folder deletion incident (~01:40-02:10 window). This copy is rebuilt verbatim from the session transcript. Report folders are session artifacts; a live backup also exists at C:/Users/k1yt/ZooCode_docs_rebuild/.

## Status

Proposed (2026-10-31). Deciders: @myk1yt (program owner), upstream maintainers for merge. Design produced by a 3-seat double-blind architect swarm; per-seat plans in `docs/261031_0001_session_caching-diet-wave/architect-report-ttl-adr-{mlarge,glm,deep}.md`, reconciliation in `ttl-adr-pick.md` (session artifacts, not committed). Supersedes nothing.

## Context

Zoo-Code emits only bare `{ type: "ephemeral" }` cache markers — no TTL anywhere (`src/api/providers/anthropic.ts:67`, four `src/api/transform/caching/*.ts` transforms, `openai.ts:121,150`, `minimax.ts:85`, Bedrock `{ cachePoint: { type: "default" } }` at `bedrock.ts:1040` + `base-strategy.ts:53-55`). Per-TTL usage splits (`usage.cache_creation.{ephemeral_5m_input_tokens, ephemeral_1h_input_tokens}`; OpenAI `cache_write_tokens_details.{ttl_30m,ttl_12h}_tokens`) are typed in the pinned SDKs (`messages.d.ts:166-174`; AWS `CacheTTL` enum) but read nowhere — grep over `src/` returns zero hits. `src/shared/cost.ts:50` prices writes at one rate.

Economics (verified by the 261030 caching validation swarm): Anthropic 5m write = 1.25x base input, 1h write = 2.0x. Shipping a TTL knob without per-TTL write splits under-reports every 1h write by **37.5%** (50% for OpenAI's 2.5x 12h class). Break-even for 1h is a single avoided re-write per hour (premium 0.75xP vs saving 1.15xP per expiry event; ~0.65 events/hour). Bedrock hard-fails (`ValidationException`) on a TTL a model does not support.

## Decision Drivers

- Billing correctness (a knob that misprices is worse than no knob; the repo already under-reports DeepSeek-style shape bugs once — PR #2252).
- Single source of truth for price data (catalog), not derived multipliers.
- Zero behavior change by default (byte-identical requests unless the user opts in).
- Hard provider failures are unacceptable (Bedrock gate must fail closed).
- Mutation-diff budget (`MAX_CHANGED_LINES=500` per package) and per-PR reviewability.

## Decision

Adopt **Option A — catalog-embedded per-TTL write prices + a provider-settings opt-in knob** — executed in a staged order so the split-accounting half lands **before** any request can emit 1h:

1. **PR-C5a (accounting only, zero request-shape change):** `ModelInfo` gains optional `cacheWritesPrice1h`/`cacheWritesPrice12h` (populated from a single `deriveCacheWritesPrice1h(inputPrice)` ratio helper, never retyped per entry); usage chunks gain **count-split fields** (`cacheWriteTokens1h`, `cacheWriteTokens12h`) accumulated from the vendor's reported split, with `writes5m = total - 1h - 12h` clamped at 0; `cost.ts` gains optional trailing split params (backward-compatible; existing 55 call sites untouched).
2. **PR-C5b (mechanism):** `promptCacheTtl?: "5m" | "1h"` added to `baseProviderSettingsShape` (provider-settings, **not** global-settings — `ApiHandlerOptions = Omit<ProviderSettings, "apiProvider">`, `src/shared/api.ts:14`, means a global key is invisible to handlers). Emission gated by a catalog capability array `promptCacheTtlSupport?: ("5m"|"1h")[]`; unset knob emits **no `ttl` key at all** (not `"5m"`).
3. **PR-C6 (settings/UI):** the knob ships last and rides the provider-profile round-trip (`apiConfiguration`), satisfying the Persisted Setting Checklist in AGENTS.md with `awsUsePromptCache`/`anthropicBeta1MContext` as precedents; UI disables 1h for models without capability.
4. **PR-C7 (Bedrock per-model gate):** capability comes from the catalog flag on resolved base-model ids (never ARN/custom-id string inference); both cache-point emitters (`base-strategy.ts:54`, `bedrock.ts:1040`) are patched together; requests carry one uniform TTL per request, making AWS's longer-before-shorter ordering hazard structurally unreachable.

Invariants binding all PRs: **I1** price from reported counts, never from the requested TTL, never summing two shapes of the same quantity; **I2** absent capability metadata means 5m-only (fail closed); **I3** default request bytes unchanged; **I4** one uniform TTL per request.

## Alternatives

- **Option B — multiplier table / overrides map outside the catalog (fastest, ~260-280 lines):** dismissed. It creates a second source of truth for write prices; this repo has already lived the drift (Moonshot `cacheWritesPrice: 0 "free for limited time"` staleness, MiMo precedent). Rejected 3/3 seats.
- **Option C — full staging behind a temporary `experiments` flag:** dismissed as shipped design. The flag is a throwaway surface with its own removal cost; C5b's knob is already the opt-in, so a second gate adds nothing. Its *ordering* (accounting first) is adopted into this decision.
- **Global-settings knob location (mlarge variant):** dismissed; provider-settings route verified independently by two seats with the `ApiHandlerOptions` citation.
- **Static Bedrock 1h model-id allowlist const:** dismissed; duplicates capability the catalog already expresses (drift surface), though the repo does carry the `BEDROCK_1M_CONTEXT_MODEL_IDS` precedent for simpler cases.

## Consequences

Positive: the 37.5% under-report becomes unrepresentable before the knob can ship; C5a is a pure reporting fix mergeable on its own; capability and price live in the catalog where pricing reviewers already look; Bedrock cannot be sent an unsupported TTL from the shipped code path.

Negative: ~51 ungated catalog lines across `packages/types/src/providers/{anthropic,bedrock,openai}.ts` must be arithmetically reviewed by hand; `Task.ts` accumulator plumbing (~8 sites) is touched (a missed site silently falls back to the lump rate — mitigated by the catalog-invariant and split tests); a per-provider knob means users with many profiles set TTL per profile, not once globally; OpenRouter pass-through exposes a single `cache_write` figure, so its 1h pricing keeps the documented fallback rather than a split.

## Risks and Mitigations

- **Two Bedrock emitters diverge (mixed TTL request -> 400):** both-emitter test asserting every `cachePoint` carries the ttl (C7 plan).
- **Silent fallback re-hides the bug for a forgotten catalog entry:** invariant test — every Anthropic/Bedrock entry with `supportsPromptCache: true` gets a defined 1h price/capability.
- **Dynamic catalogs (OpenRouter fetcher, custom ARNs) lack capability data:** fail-closed to 5m; if the invariant cannot be enforced, C5a re-specs to omit 1h attribution rather than misprice.
- **Pinned SDK does not actually return the split on some path:** C5a reads via optional chain; if the Anthropic split proves unverifiable without a live call, gate C5b behind one CEO-key probe (same protocol as PR-C8) and ship C5a's OpenAI 12h half only.
- **`applyLongContextPricing` forgetting the new field (>200k-context 1h writes at half rate):** dedicated long-context split test.

## Validation

- The 37.5% tripwire test: 1,000 1h-write tokens at `inputPrice 3.0` must price **0.006**, not 0.00375 (`src/utils/__tests__/cost.spec.ts`).
- Default-parity: knob unset => serialized request contains no `"ttl"` key (snapshot/grep assertions per provider spec).
- Fail-closed gate: `promptCacheTtl:"1h"` + non-capable model => no `ttl` emitted; Bedrock Nova-class row asserted against `ValidationException` shape.
- Settings round-trip: saved `"1h"` survives `getStateToPostToWebview()`; unset stays `undefined` (default applied by consumers).
- Mixed-shape non-additivity fixtures (DeepSeek lesson, PR #2252) on every split reader.

## Revisit Triggers

- AWS publishes/changes the per-model 1h list, or a live probe returns `ValidationException` for 2+ allowlisted models -> re-pin or defer C7 permanently (Bedrock stays 5m).
- Vendors reprice TTL classes away from the 1.25x/2.0x/2.5x ratios -> catalog edit; if per-request mixed TTLs are ever wanted, the uniform-TTL invariant (I4) must gain ordering enforcement first.
- `packages/types` enters the mutation-gate `PACKAGE_CONFIGS` -> re-check C5a line budget. (UPDATE 261031: C3 verified `types` IS in PACKAGE_CONFIGS at scripts/stryker-diff.mjs:40-44 — C5a's catalog lines DO count against the gate; coder measured own 217/500, cumulative 281/500.)
- PR-C8's live Kimi/Moonshot probe returns per-TTL write fields -> extend C5a's readers with the Moonshot lever rather than a new design.

## References

- `docs/261030_0005_session_diet-caching-validation/FINAL-PLAN.md` (K5/K8 rows, DO-NOT list)
- Architect seat reports + pick record (session folder `docs/261031_0001_session_caching-diet-wave/`)
- PR #2252 (wire-shape lesson: fixtures must match vendor schema), PR #2260 (C5a implementation)
- ADR-0003, ADR-0005 (repo ADR numbering/format precedent)
94 changes: 94 additions & 0 deletions src/api/providers/__tests__/anthropic-vertex.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { VERTEX_1M_CONTEXT_MODEL_IDS } from "@roo-code/types"

import { AnthropicVertexHandler } from "../anthropic-vertex"
import { MAX_TIMEOUT_MS } from "../utils/abort-signal"
import { calculateApiCostAnthropic } from "../../../shared/cost"
import { asyncStreamFrom, collectStream } from "../../../test-utils/stream"
import { makeCreateMessageMetadata } from "../../../test-utils/api"
import type { ApiStreamChunk } from "../../../api/transform/stream"
Expand Down Expand Up @@ -1108,6 +1109,77 @@ describe("VertexHandler", () => {
)
})

it("completePrompt carries ttl 1h on the user marker when the model is capable", async () => {
handler = new AnthropicVertexHandler({
apiModelId: "claude-3-5-sonnet-v2@20241022",
vertexProjectId: "test-project",
vertexRegion: "us-central1",
promptCacheTtl: "1h",
})

// The Vertex catalog carries no TTL capability yet, so inject one to
// exercise the emission path this PR adds.
const model = handler.getModel()
vitest.spyOn(handler, "getModel").mockReturnValue({
...model,
info: { ...model.info, promptCacheTtlSupport: ["5m", "1h"] },
})

const mockCreate = vitest.fn().mockResolvedValue({ content: [{ type: "text", text: "ok" }] })
const client = handler["client"] as unknown as { messages: { create: unknown } }
client.messages.create = mockCreate

await handler.completePrompt("Test prompt")

const body = mockCreate.mock.calls[0][0]
const controls: Array<Record<string, unknown>> = []
const walk = (node: unknown) => {
if (Array.isArray(node)) {
node.forEach(walk)
return
}
if (node && typeof node === "object") {
for (const [key, child] of Object.entries(node as Record<string, unknown>)) {
if (key === "cache_control" && child && typeof child === "object") {
controls.push(child as Record<string, unknown>)
}
walk(child)
}
}
}
walk(JSON.parse(JSON.stringify(body)))
expect(controls).toHaveLength(1)
expect(controls[0]).toEqual({ type: "ephemeral", ttl: "1h" })
})

it("completePrompt emits the plain ephemeral literal when the knob is unset", async () => {
handler = new AnthropicVertexHandler({
apiModelId: "claude-3-5-sonnet-v2@20241022",
vertexProjectId: "test-project",
vertexRegion: "us-central1",
})

const mockCreate = vitest.fn().mockResolvedValue({ content: [{ type: "text", text: "ok" }] })
const client = handler["client"] as unknown as { messages: { create: unknown } }
client.messages.create = mockCreate

await handler.completePrompt("Test prompt")

// Byte-parity with the pre-knob literal (ADR-0006 I3): no ttl key anywhere.
expect(JSON.stringify(mockCreate.mock.calls[0][0])).not.toContain("ttl")
expect(mockCreate).toHaveBeenCalledWith(
expect.objectContaining({
messages: [
{
role: "user",
content: [{ type: "text", text: "Test prompt", cache_control: { type: "ephemeral" } }],
},
],
}),
undefined,
)
})

it("should handle API errors for Claude", async () => {
handler = new AnthropicVertexHandler({
apiModelId: "claude-3-5-sonnet-v2@20241022",
Expand Down Expand Up @@ -1378,6 +1450,28 @@ describe("VertexHandler", () => {
expect(model.betas).toContain("context-1m-2025-08-07")
})

it("bills a 1M-context 1h cache write at the tier-derived rate, not the 200K rate", () => {
// Review pin: the 1M merge carries the tier's TTL rate, but the vertex
// catalog declares no explicit cacheWritesPrice1h, so the effective 1h
// rate is derived from the applied input price (2x per the shared
// resolver). If the tier merge regressed to the entry-level input price,
// the derived 1h rate would drop from 12.0 to 6.0 (50% under-billing).
const handler = new AnthropicVertexHandler({
apiModelId: "claude-sonnet-4-6",
vertexProjectId: "test-project",
vertexRegion: "us-central1",
vertex1MContext: true,
})

const model = handler.getModel()
const { totalCost } = calculateApiCostAnthropic(model.info, 0, 0, 100_000, 0, 100_000)

// Derived tier 1h rate 12.0: (12.0 / 1_000_000) * 100_000 = 1.2
// The entry-derived rate 6.0 would bill 0.6 (50% under).
expect(totalCost).toBeCloseTo(1.2, 9)
expect(totalCost).not.toBeCloseTo(0.6, 9)
})

it("should enable 1M context for Claude Opus 4.7 when beta flag is set", () => {
const handler = new AnthropicVertexHandler({
apiModelId: "claude-opus-4-7",
Expand Down
26 changes: 26 additions & 0 deletions src/api/providers/__tests__/anthropic.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -310,6 +310,32 @@ describe("AnthropicHandler", () => {
const body = mockCreate.mock.calls[mockCreate.mock.calls.length - 1]?.[0]
expect(JSON.stringify(body)).not.toContain("ttl")
})

it("emits no ttl key for a 1h knob on a model without 1h capability", async () => {
// Every catalog model now advertises 1h (1h cache writes are GA on all
// active Claude models), so the reject path is exercised by resolving a
// 5m-only info — the shape a dynamic or stale catalog would produce.
// The knob must fail closed here too (ADR-0006 I2), not only in ttl.spec.ts.
const staleCatalogHandler = new AnthropicHandler({
...mockOptions,
promptCacheTtl: "1h",
})

const model = staleCatalogHandler.getModel()
vitest.spyOn(staleCatalogHandler, "getModel").mockReturnValue({
...model,
info: { ...model.info, promptCacheTtlSupport: ["5m"] },
})

await collectStream(staleCatalogHandler.createMessage(systemPrompt, threeTurnMessages()))

const body = mockCreate.mock.calls[mockCreate.mock.calls.length - 1]?.[0]
expect(JSON.stringify(body)).not.toContain("ttl")

const controls = requestCacheControls()
expect(controls).toHaveLength(3)
expect(controls.every((control) => control.ttl === undefined)).toBe(true)
})
})

it("should include 1M context beta header for Claude Sonnet 4.6 when enabled", async () => {
Expand Down
19 changes: 19 additions & 0 deletions src/api/providers/__tests__/bedrock.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1159,6 +1159,25 @@ describe("AwsBedrockHandler", () => {
expect(model.info.outputPrice).toBe(22.5)
})

it("carries the tier's own cacheWritesPrice1h when the 1M context tier applies", () => {
// Review pin: the 1M merge overrode inputPrice but must also carry the
// tier's TTL rate. The entry-level 1h rate was derived from the 200K input
// price, so a fallback to it bills a 1M-context 1h write at half the rate.
const handler = new AwsBedrockHandler({
apiModelId: "anthropic.claude-sonnet-4-6",
awsAccessKey: "test",
awsSecretKey: "test",
awsRegion: "us-east-1",
awsBedrock1MContext: true,
})

const model = handler.getModel()

// Tier input price 6.0 => 1h rate must be 2x that, not the 200K entry's 6.0.
expect(model.info.cacheWritesPrice1h).toBe(12.0)
expect(model.info.cacheWritesPrice1h).not.toBe(6.0)
})

it("should use default context window when awsBedrock1MContext is false for Claude Sonnet 4", () => {
const handler = new AwsBedrockHandler({
apiModelId: BEDROCK_1M_CONTEXT_MODEL_IDS[0],
Expand Down
52 changes: 52 additions & 0 deletions src/api/providers/__tests__/zoo-gateway.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,24 @@ const DEFAULT_MODEL_CATALOG = vitest.hoisted(() => ({
cacheReadsPrice: 0.1,
description: "Claude 3.5 Haiku",
},
// The runtime gateway catalog never sets `promptCacheTtlSupport`, so this row
// exists only to inject the capability the accept path needs. It is kept
// separate from the shared haiku row so the reject-path tests keep a model
// with no TTL capability.
"anthropic/claude-3.5-sonnet": {
maxTokens: 8192,
contextWindow: 200000,
supportsImages: true,
supportsPromptCache: true,
// `as const` keeps the literal a "1h" tuple; a plain ["1h"] widens to
// string[] and fails the ModelRecord index signature.
promptCacheTtlSupport: ["1h" as const],
inputPrice: 3,
outputPrice: 15,
cacheWritesPrice: 3.75,
cacheReadsPrice: 0.3,
description: "Claude 3.5 Sonnet",
},
}))

vitest.mock("../fetchers/modelCache", () => ({
Expand Down Expand Up @@ -421,6 +439,40 @@ describe("ZooGatewayHandler", () => {
expect(addCacheBreakpoints).toHaveBeenCalled()
})

it("does not emit a ttl for a gateway model with no TTL capability", async () => {
// The runtime gateway catalog carries no TTL capability, so a 1h knob must
// fail closed (ADR-0006 I2) even though the model is prompt-cache capable.
const { addCacheBreakpoints } = await import("../../transform/caching/vercel-ai-gateway")
const handler = new ZooGatewayHandler({
...mockOptions,
zooGatewayModelId: "anthropic/claude-3.5-haiku",
promptCacheTtl: "1h",
})

await handler.createMessage("prompt", [{ role: "user", content: "Hi" }]).next()

expect(addCacheBreakpoints).toHaveBeenCalled()
const lastCall = vitest.mocked(addCacheBreakpoints).mock.calls.at(-1)
expect(lastCall).toBeDefined()
expect(lastCall![2]).toBeUndefined()
})

it("passes the resolved ttl for a gateway model with 1h TTL capability", async () => {
// The runtime catalog never sets `promptCacheTtlSupport`, so the accept path is
// only reachable with an injected capability. The dedicated catalog row keeps
// the reject-path model capability-free.
const { addCacheBreakpoints } = await import("../../transform/caching/vercel-ai-gateway")
const handler = new ZooGatewayHandler({
...mockOptions,
zooGatewayModelId: "anthropic/claude-3.5-sonnet",
promptCacheTtl: "1h",
})

await handler.createMessage("prompt", [{ role: "user", content: "Hi" }]).next()

expect(vitest.mocked(addCacheBreakpoints).mock.calls.at(-1)![2]).toBe("1h")
})

it("yields tool_call_partial chunks when streaming tool calls", async () => {
mockCreate.mockImplementation(async () =>
asyncStreamFrom([
Expand Down
Loading