diff --git a/docs/architecture/adr/0006-prompt-cache-ttl-program.md b/docs/architecture/adr/0006-prompt-cache-ttl-program.md new file mode 100644 index 0000000000..d419515a5b --- /dev/null +++ b/docs/architecture/adr/0006-prompt-cache-ttl-program.md @@ -0,0 +1,75 @@ +# ADR-0006: Prompt-cache TTL program — catalog-embedded per-TTL pricing, opt-in knob, fail-closed capability gates + +> RECONSTRUCTED 2026-10-31: the original file (written 261031 ~01:58) was lost in a docs-folder deletion incident (~01:40-02:10 window). This copy is rebuilt verbatim from the session transcript. Report folders are session artifacts; a live backup also exists at C:/Users/k1yt/ZooCode_docs_rebuild/. + +## Status + +Proposed (2026-10-31). Deciders: @myk1yt (program owner), upstream maintainers for merge. Design produced by a 3-seat double-blind architect swarm; per-seat plans in `docs/261031_0001_session_caching-diet-wave/architect-report-ttl-adr-{mlarge,glm,deep}.md`, reconciliation in `ttl-adr-pick.md` (session artifacts, not committed). Supersedes nothing. + +## Context + +Zoo-Code emits only bare `{ type: "ephemeral" }` cache markers — no TTL anywhere (`src/api/providers/anthropic.ts:67`, four `src/api/transform/caching/*.ts` transforms, `openai.ts:121,150`, `minimax.ts:85`, Bedrock `{ cachePoint: { type: "default" } }` at `bedrock.ts:1040` + `base-strategy.ts:53-55`). Per-TTL usage splits (`usage.cache_creation.{ephemeral_5m_input_tokens, ephemeral_1h_input_tokens}`; OpenAI `cache_write_tokens_details.{ttl_30m,ttl_12h}_tokens`) are typed in the pinned SDKs (`messages.d.ts:166-174`; AWS `CacheTTL` enum) but read nowhere — grep over `src/` returns zero hits. `src/shared/cost.ts:50` prices writes at one rate. + +Economics (verified by the 261030 caching validation swarm): Anthropic 5m write = 1.25x base input, 1h write = 2.0x. Shipping a TTL knob without per-TTL write splits under-reports every 1h write by **37.5%** (50% for OpenAI's 2.5x 12h class). Break-even for 1h is a single avoided re-write per hour (premium 0.75xP vs saving 1.15xP per expiry event; ~0.65 events/hour). Bedrock hard-fails (`ValidationException`) on a TTL a model does not support. + +## Decision Drivers + +- Billing correctness (a knob that misprices is worse than no knob; the repo already under-reports DeepSeek-style shape bugs once — PR #2252). +- Single source of truth for price data (catalog), not derived multipliers. +- Zero behavior change by default (byte-identical requests unless the user opts in). +- Hard provider failures are unacceptable (Bedrock gate must fail closed). +- Mutation-diff budget (`MAX_CHANGED_LINES=500` per package) and per-PR reviewability. + +## Decision + +Adopt **Option A — catalog-embedded per-TTL write prices + a provider-settings opt-in knob** — executed in a staged order so the split-accounting half lands **before** any request can emit 1h: + +1. **PR-C5a (accounting only, zero request-shape change):** `ModelInfo` gains optional `cacheWritesPrice1h`/`cacheWritesPrice12h` (populated from a single `deriveCacheWritesPrice1h(inputPrice)` ratio helper, never retyped per entry); usage chunks gain **count-split fields** (`cacheWriteTokens1h`, `cacheWriteTokens12h`) accumulated from the vendor's reported split, with `writes5m = total - 1h - 12h` clamped at 0; `cost.ts` gains optional trailing split params (backward-compatible; existing 55 call sites untouched). +2. **PR-C5b (mechanism):** `promptCacheTtl?: "5m" | "1h"` added to `baseProviderSettingsShape` (provider-settings, **not** global-settings — `ApiHandlerOptions = Omit`, `src/shared/api.ts:14`, means a global key is invisible to handlers). Emission gated by a catalog capability array `promptCacheTtlSupport?: ("5m"|"1h")[]`; unset knob emits **no `ttl` key at all** (not `"5m"`). +3. **PR-C6 (settings/UI):** the knob ships last and rides the provider-profile round-trip (`apiConfiguration`), satisfying the Persisted Setting Checklist in AGENTS.md with `awsUsePromptCache`/`anthropicBeta1MContext` as precedents; UI disables 1h for models without capability. +4. **PR-C7 (Bedrock per-model gate):** capability comes from the catalog flag on resolved base-model ids (never ARN/custom-id string inference); both cache-point emitters (`base-strategy.ts:54`, `bedrock.ts:1040`) are patched together; requests carry one uniform TTL per request, making AWS's longer-before-shorter ordering hazard structurally unreachable. + +Invariants binding all PRs: **I1** price from reported counts, never from the requested TTL, never summing two shapes of the same quantity; **I2** absent capability metadata means 5m-only (fail closed); **I3** default request bytes unchanged; **I4** one uniform TTL per request. + +## Alternatives + +- **Option B — multiplier table / overrides map outside the catalog (fastest, ~260-280 lines):** dismissed. It creates a second source of truth for write prices; this repo has already lived the drift (Moonshot `cacheWritesPrice: 0 "free for limited time"` staleness, MiMo precedent). Rejected 3/3 seats. +- **Option C — full staging behind a temporary `experiments` flag:** dismissed as shipped design. The flag is a throwaway surface with its own removal cost; C5b's knob is already the opt-in, so a second gate adds nothing. Its *ordering* (accounting first) is adopted into this decision. +- **Global-settings knob location (mlarge variant):** dismissed; provider-settings route verified independently by two seats with the `ApiHandlerOptions` citation. +- **Static Bedrock 1h model-id allowlist const:** dismissed; duplicates capability the catalog already expresses (drift surface), though the repo does carry the `BEDROCK_1M_CONTEXT_MODEL_IDS` precedent for simpler cases. + +## Consequences + +Positive: the 37.5% under-report becomes unrepresentable before the knob can ship; C5a is a pure reporting fix mergeable on its own; capability and price live in the catalog where pricing reviewers already look; Bedrock cannot be sent an unsupported TTL from the shipped code path. + +Negative: ~51 ungated catalog lines across `packages/types/src/providers/{anthropic,bedrock,openai}.ts` must be arithmetically reviewed by hand; `Task.ts` accumulator plumbing (~8 sites) is touched (a missed site silently falls back to the lump rate — mitigated by the catalog-invariant and split tests); a per-provider knob means users with many profiles set TTL per profile, not once globally; OpenRouter pass-through exposes a single `cache_write` figure, so its 1h pricing keeps the documented fallback rather than a split. + +## Risks and Mitigations + +- **Two Bedrock emitters diverge (mixed TTL request -> 400):** both-emitter test asserting every `cachePoint` carries the ttl (C7 plan). +- **Silent fallback re-hides the bug for a forgotten catalog entry:** invariant test — every Anthropic/Bedrock entry with `supportsPromptCache: true` gets a defined 1h price/capability. +- **Dynamic catalogs (OpenRouter fetcher, custom ARNs) lack capability data:** fail-closed to 5m; if the invariant cannot be enforced, C5a re-specs to omit 1h attribution rather than misprice. +- **Pinned SDK does not actually return the split on some path:** C5a reads via optional chain; if the Anthropic split proves unverifiable without a live call, gate C5b behind one CEO-key probe (same protocol as PR-C8) and ship C5a's OpenAI 12h half only. +- **`applyLongContextPricing` forgetting the new field (>200k-context 1h writes at half rate):** dedicated long-context split test. + +## Validation + +- The 37.5% tripwire test: 1,000 1h-write tokens at `inputPrice 3.0` must price **0.006**, not 0.00375 (`src/utils/__tests__/cost.spec.ts`). +- Default-parity: knob unset => serialized request contains no `"ttl"` key (snapshot/grep assertions per provider spec). +- Fail-closed gate: `promptCacheTtl:"1h"` + non-capable model => no `ttl` emitted; Bedrock Nova-class row asserted against `ValidationException` shape. +- Settings round-trip: saved `"1h"` survives `getStateToPostToWebview()`; unset stays `undefined` (default applied by consumers). +- Mixed-shape non-additivity fixtures (DeepSeek lesson, PR #2252) on every split reader. + +## Revisit Triggers + +- AWS publishes/changes the per-model 1h list, or a live probe returns `ValidationException` for 2+ allowlisted models -> re-pin or defer C7 permanently (Bedrock stays 5m). +- Vendors reprice TTL classes away from the 1.25x/2.0x/2.5x ratios -> catalog edit; if per-request mixed TTLs are ever wanted, the uniform-TTL invariant (I4) must gain ordering enforcement first. +- `packages/types` enters the mutation-gate `PACKAGE_CONFIGS` -> re-check C5a line budget. (UPDATE 261031: C3 verified `types` IS in PACKAGE_CONFIGS at scripts/stryker-diff.mjs:40-44 — C5a's catalog lines DO count against the gate; coder measured own 217/500, cumulative 281/500.) +- PR-C8's live Kimi/Moonshot probe returns per-TTL write fields -> extend C5a's readers with the Moonshot lever rather than a new design. + +## References + +- `docs/261030_0005_session_diet-caching-validation/FINAL-PLAN.md` (K5/K8 rows, DO-NOT list) +- Architect seat reports + pick record (session folder `docs/261031_0001_session_caching-diet-wave/`) +- PR #2252 (wire-shape lesson: fixtures must match vendor schema), PR #2260 (C5a implementation) +- ADR-0003, ADR-0005 (repo ADR numbering/format precedent) diff --git a/src/api/providers/__tests__/anthropic-vertex.spec.ts b/src/api/providers/__tests__/anthropic-vertex.spec.ts index f4860e9914..bae29755f1 100644 --- a/src/api/providers/__tests__/anthropic-vertex.spec.ts +++ b/src/api/providers/__tests__/anthropic-vertex.spec.ts @@ -8,6 +8,7 @@ import { VERTEX_1M_CONTEXT_MODEL_IDS } from "@roo-code/types" import { AnthropicVertexHandler } from "../anthropic-vertex" import { MAX_TIMEOUT_MS } from "../utils/abort-signal" +import { calculateApiCostAnthropic } from "../../../shared/cost" import { asyncStreamFrom, collectStream } from "../../../test-utils/stream" import { makeCreateMessageMetadata } from "../../../test-utils/api" import type { ApiStreamChunk } from "../../../api/transform/stream" @@ -1108,6 +1109,77 @@ describe("VertexHandler", () => { ) }) + it("completePrompt carries ttl 1h on the user marker when the model is capable", async () => { + handler = new AnthropicVertexHandler({ + apiModelId: "claude-3-5-sonnet-v2@20241022", + vertexProjectId: "test-project", + vertexRegion: "us-central1", + promptCacheTtl: "1h", + }) + + // The Vertex catalog carries no TTL capability yet, so inject one to + // exercise the emission path this PR adds. + const model = handler.getModel() + vitest.spyOn(handler, "getModel").mockReturnValue({ + ...model, + info: { ...model.info, promptCacheTtlSupport: ["5m", "1h"] }, + }) + + const mockCreate = vitest.fn().mockResolvedValue({ content: [{ type: "text", text: "ok" }] }) + const client = handler["client"] as unknown as { messages: { create: unknown } } + client.messages.create = mockCreate + + await handler.completePrompt("Test prompt") + + const body = mockCreate.mock.calls[0][0] + const controls: Array> = [] + const walk = (node: unknown) => { + if (Array.isArray(node)) { + node.forEach(walk) + return + } + if (node && typeof node === "object") { + for (const [key, child] of Object.entries(node as Record)) { + if (key === "cache_control" && child && typeof child === "object") { + controls.push(child as Record) + } + walk(child) + } + } + } + walk(JSON.parse(JSON.stringify(body))) + expect(controls).toHaveLength(1) + expect(controls[0]).toEqual({ type: "ephemeral", ttl: "1h" }) + }) + + it("completePrompt emits the plain ephemeral literal when the knob is unset", async () => { + handler = new AnthropicVertexHandler({ + apiModelId: "claude-3-5-sonnet-v2@20241022", + vertexProjectId: "test-project", + vertexRegion: "us-central1", + }) + + const mockCreate = vitest.fn().mockResolvedValue({ content: [{ type: "text", text: "ok" }] }) + const client = handler["client"] as unknown as { messages: { create: unknown } } + client.messages.create = mockCreate + + await handler.completePrompt("Test prompt") + + // Byte-parity with the pre-knob literal (ADR-0006 I3): no ttl key anywhere. + expect(JSON.stringify(mockCreate.mock.calls[0][0])).not.toContain("ttl") + expect(mockCreate).toHaveBeenCalledWith( + expect.objectContaining({ + messages: [ + { + role: "user", + content: [{ type: "text", text: "Test prompt", cache_control: { type: "ephemeral" } }], + }, + ], + }), + undefined, + ) + }) + it("should handle API errors for Claude", async () => { handler = new AnthropicVertexHandler({ apiModelId: "claude-3-5-sonnet-v2@20241022", @@ -1378,6 +1450,28 @@ describe("VertexHandler", () => { expect(model.betas).toContain("context-1m-2025-08-07") }) + it("bills a 1M-context 1h cache write at the tier-derived rate, not the 200K rate", () => { + // Review pin: the 1M merge carries the tier's TTL rate, but the vertex + // catalog declares no explicit cacheWritesPrice1h, so the effective 1h + // rate is derived from the applied input price (2x per the shared + // resolver). If the tier merge regressed to the entry-level input price, + // the derived 1h rate would drop from 12.0 to 6.0 (50% under-billing). + const handler = new AnthropicVertexHandler({ + apiModelId: "claude-sonnet-4-6", + vertexProjectId: "test-project", + vertexRegion: "us-central1", + vertex1MContext: true, + }) + + const model = handler.getModel() + const { totalCost } = calculateApiCostAnthropic(model.info, 0, 0, 100_000, 0, 100_000) + + // Derived tier 1h rate 12.0: (12.0 / 1_000_000) * 100_000 = 1.2 + // The entry-derived rate 6.0 would bill 0.6 (50% under). + expect(totalCost).toBeCloseTo(1.2, 9) + expect(totalCost).not.toBeCloseTo(0.6, 9) + }) + it("should enable 1M context for Claude Opus 4.7 when beta flag is set", () => { const handler = new AnthropicVertexHandler({ apiModelId: "claude-opus-4-7", diff --git a/src/api/providers/__tests__/anthropic.spec.ts b/src/api/providers/__tests__/anthropic.spec.ts index 4f15564323..415b6f8bfa 100644 --- a/src/api/providers/__tests__/anthropic.spec.ts +++ b/src/api/providers/__tests__/anthropic.spec.ts @@ -310,6 +310,32 @@ describe("AnthropicHandler", () => { const body = mockCreate.mock.calls[mockCreate.mock.calls.length - 1]?.[0] expect(JSON.stringify(body)).not.toContain("ttl") }) + + it("emits no ttl key for a 1h knob on a model without 1h capability", async () => { + // Every catalog model now advertises 1h (1h cache writes are GA on all + // active Claude models), so the reject path is exercised by resolving a + // 5m-only info — the shape a dynamic or stale catalog would produce. + // The knob must fail closed here too (ADR-0006 I2), not only in ttl.spec.ts. + const staleCatalogHandler = new AnthropicHandler({ + ...mockOptions, + promptCacheTtl: "1h", + }) + + const model = staleCatalogHandler.getModel() + vitest.spyOn(staleCatalogHandler, "getModel").mockReturnValue({ + ...model, + info: { ...model.info, promptCacheTtlSupport: ["5m"] }, + }) + + await collectStream(staleCatalogHandler.createMessage(systemPrompt, threeTurnMessages())) + + const body = mockCreate.mock.calls[mockCreate.mock.calls.length - 1]?.[0] + expect(JSON.stringify(body)).not.toContain("ttl") + + const controls = requestCacheControls() + expect(controls).toHaveLength(3) + expect(controls.every((control) => control.ttl === undefined)).toBe(true) + }) }) it("should include 1M context beta header for Claude Sonnet 4.6 when enabled", async () => { diff --git a/src/api/providers/__tests__/bedrock.spec.ts b/src/api/providers/__tests__/bedrock.spec.ts index 0d8e5a784a..f21f0cd1e4 100644 --- a/src/api/providers/__tests__/bedrock.spec.ts +++ b/src/api/providers/__tests__/bedrock.spec.ts @@ -1159,6 +1159,25 @@ describe("AwsBedrockHandler", () => { expect(model.info.outputPrice).toBe(22.5) }) + it("carries the tier's own cacheWritesPrice1h when the 1M context tier applies", () => { + // Review pin: the 1M merge overrode inputPrice but must also carry the + // tier's TTL rate. The entry-level 1h rate was derived from the 200K input + // price, so a fallback to it bills a 1M-context 1h write at half the rate. + const handler = new AwsBedrockHandler({ + apiModelId: "anthropic.claude-sonnet-4-6", + awsAccessKey: "test", + awsSecretKey: "test", + awsRegion: "us-east-1", + awsBedrock1MContext: true, + }) + + const model = handler.getModel() + + // Tier input price 6.0 => 1h rate must be 2x that, not the 200K entry's 6.0. + expect(model.info.cacheWritesPrice1h).toBe(12.0) + expect(model.info.cacheWritesPrice1h).not.toBe(6.0) + }) + it("should use default context window when awsBedrock1MContext is false for Claude Sonnet 4", () => { const handler = new AwsBedrockHandler({ apiModelId: BEDROCK_1M_CONTEXT_MODEL_IDS[0], diff --git a/src/api/providers/__tests__/zoo-gateway.spec.ts b/src/api/providers/__tests__/zoo-gateway.spec.ts index a9beaada1c..09460f099c 100644 --- a/src/api/providers/__tests__/zoo-gateway.spec.ts +++ b/src/api/providers/__tests__/zoo-gateway.spec.ts @@ -67,6 +67,24 @@ const DEFAULT_MODEL_CATALOG = vitest.hoisted(() => ({ cacheReadsPrice: 0.1, description: "Claude 3.5 Haiku", }, + // The runtime gateway catalog never sets `promptCacheTtlSupport`, so this row + // exists only to inject the capability the accept path needs. It is kept + // separate from the shared haiku row so the reject-path tests keep a model + // with no TTL capability. + "anthropic/claude-3.5-sonnet": { + maxTokens: 8192, + contextWindow: 200000, + supportsImages: true, + supportsPromptCache: true, + // `as const` keeps the literal a "1h" tuple; a plain ["1h"] widens to + // string[] and fails the ModelRecord index signature. + promptCacheTtlSupport: ["1h" as const], + inputPrice: 3, + outputPrice: 15, + cacheWritesPrice: 3.75, + cacheReadsPrice: 0.3, + description: "Claude 3.5 Sonnet", + }, })) vitest.mock("../fetchers/modelCache", () => ({ @@ -421,6 +439,40 @@ describe("ZooGatewayHandler", () => { expect(addCacheBreakpoints).toHaveBeenCalled() }) + it("does not emit a ttl for a gateway model with no TTL capability", async () => { + // The runtime gateway catalog carries no TTL capability, so a 1h knob must + // fail closed (ADR-0006 I2) even though the model is prompt-cache capable. + const { addCacheBreakpoints } = await import("../../transform/caching/vercel-ai-gateway") + const handler = new ZooGatewayHandler({ + ...mockOptions, + zooGatewayModelId: "anthropic/claude-3.5-haiku", + promptCacheTtl: "1h", + }) + + await handler.createMessage("prompt", [{ role: "user", content: "Hi" }]).next() + + expect(addCacheBreakpoints).toHaveBeenCalled() + const lastCall = vitest.mocked(addCacheBreakpoints).mock.calls.at(-1) + expect(lastCall).toBeDefined() + expect(lastCall![2]).toBeUndefined() + }) + + it("passes the resolved ttl for a gateway model with 1h TTL capability", async () => { + // The runtime catalog never sets `promptCacheTtlSupport`, so the accept path is + // only reachable with an injected capability. The dedicated catalog row keeps + // the reject-path model capability-free. + const { addCacheBreakpoints } = await import("../../transform/caching/vercel-ai-gateway") + const handler = new ZooGatewayHandler({ + ...mockOptions, + zooGatewayModelId: "anthropic/claude-3.5-sonnet", + promptCacheTtl: "1h", + }) + + await handler.createMessage("prompt", [{ role: "user", content: "Hi" }]).next() + + expect(vitest.mocked(addCacheBreakpoints).mock.calls.at(-1)![2]).toBe("1h") + }) + it("yields tool_call_partial chunks when streaming tool calls", async () => { mockCreate.mockImplementation(async () => asyncStreamFrom([