diff --git a/.github/workflows/test-drift.yml b/.github/workflows/test-drift.yml index 0451767e..ed9d304e 100644 --- a/.github/workflows/test-drift.yml +++ b/.github/workflows/test-drift.yml @@ -3,6 +3,20 @@ on: schedule: - cron: "0 6 * * *" # Daily 6am UTC pull_request: + # DELIBERATE SCOPE — this filter lists DRIFT-GRADING code, not the production + # files the drift legs grade. `drift-live-pr` is a ~30-minute job that burns + # real provider credits on every trigger, and its question is "does THIS diff + # introduce provider-format divergence in the drift harness", not "does the + # mock still emit field X". Adding a graded production file (src/responses.ts, + # src/ws-responses.ts, …) would, by the same logic, pull in nearly every file + # under src/ and run the live leg on almost every PR. + # + # Per-field emission properties are gated by the ALWAYS-ON unit lane instead: + # test-unit.yml has no `paths:` filter, so `pnpm test` runs on every PR and + # hard-fails on a dropped/null/empty value (e.g. the `encrypted_content` + # value pins in src/__tests__/responses.test.ts and ws-responses.test.ts). + # If you came here because a drift assertion "never triggers on the code it + # guards": check the unit lane first — that is where the value pin lives. paths: - "src/agui-types.ts" - "src/__tests__/drift/agui-schema.drift.ts" diff --git a/CHANGELOG.md b/CHANGELOG.md index dfd4df7b..ddcf4e9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ### Added +- **Reasoning `encrypted_content` on the Responses API.** aimock now synthesizes an opaque (base64) `reasoning.encrypted_content` blob on the TERMINAL reasoning item (`response.output_item.done` and non-streaming `output[]`). Emitted only when the request wants it: `include: ["reasoning.encrypted_content"]` OR a stateless request (`store: false` / ZDR). Stored / opted-out replays stay byte-identical. A request carrying the EXPLICIT `include: ["reasoning.encrypted_content"]` opt-in, against a reasoning-CAPABLE model, now gets a reasoning item even when the fixture declares NO `reasoning` summary — a `summary: []` item that exists purely to carry the blob, matching what real OpenAI returns when summaries were never requested. This is the shape a fixture recorded from the real stateless agent-framework flow has, so gating the item on a declared summary starved the exact case the feature exists to serve. The synthesized item leads `output[]` / takes `output_index` 0 and shifts the rest, emits `output_item.added` → `output_item.done` with no summary-text events (there is no summary part to describe), and is suppressed for non-reasoning models (gpt-4o etc.), which have no reasoning channel at all. Note the two gates differ in width on purpose: `store: false` still attaches the blob to a reasoning item that a declared summary already produces, but it does NOT synthesize an item where the fixture declares none — creating an output item that did not previously exist is a bigger behavior change than adding a field to one already on the wire, so it takes the gate that is directly observable in the request. `agent-framework-openai` >= 1.11.0 sends `include` and never `store`, so the narrower gate costs the feature nothing. This lets `agent-framework-openai` >= 1.11.0 replay reasoning-paired tool calls on a stateless reasoning + multi-tool chain (related: microsoft/agent-framework#7233). Applies to both the HTTP and WebSocket Responses transports. Known limitations: the blob is withheld from the in-progress `added` item, which is aimock's own simplification rather than upstream parity — real OpenAI populates `added` too (recorded captures carry a shorter blob there beside an empty summary, re-encrypted by `done`), but the field is `anyOf: [string, null]` and non-required, so omitting it is contract-legal and no known consumer requires it; and aimock skips inbound reasoning items, so it cannot reproduce OpenAI's `invalid_encrypted_content` rejection of a mismatched blob/id. - **OpenRouter chat / LLM router simulation.** Requests whose original path starts with `/api/v1/` (point any OpenAI SDK at a `baseURL` ending `/api/v1`) are detected as OpenRouter and shaped to match real OpenRouter bytes: a `gen-` id prefix (a fixture `id` override still wins), a top-level `provider` (default = the winning model slug's author, fixture-overridable via `provider`), both `finish_reason` and `native_finish_reason` on every choice/delta, an always-present `system_fingerprint` and `service_tier` (null by default), `message.reasoning`, and a rich `usage` with a **fixture-scriptable** `cost` + `cost_details` (emitted only when a fixture supplies a cost — never fabricated), plus `is_byok` / `prompt_tokens_details` / `completion_tokens_details` when overridden. Callers on the plain OpenAI `/v1/...` base are byte-for-byte unchanged. - **`models[]` fallback (router failover) simulation.** When the request body carries `models: [...]`, aimock walks `[model, ...models]` in order and serves the first fixture returning a NON-error response; a `429`/`503` error fixture on a candidate simulates a runtime provider failure and falls through to the next. The winning slug is echoed back as the top-level `model`, so a test asserts failover by reading `response.model`. (Deliberate non-goal: an unknown/invalid model is a fixture miss, not OpenRouter's up-front invalid-model 400.) - **OpenRouter discovery endpoints** `GET /api/v1/models` (catalog synthesized from loaded chat fixtures), `GET /api/v1/key`, and `GET /api/v1/credits`, matching the real response shapes. diff --git a/src/__tests__/drift/openai-responses.drift.ts b/src/__tests__/drift/openai-responses.drift.ts index 6d5231cf..47d3fc42 100644 --- a/src/__tests__/drift/openai-responses.drift.ts +++ b/src/__tests__/drift/openai-responses.drift.ts @@ -7,12 +7,20 @@ import { describe, it, expect, beforeAll, afterAll } from "vitest"; import { createServer, type ServerInstance } from "../../server.js"; import type { Fixture } from "../../types.js"; -import { extractShape, triangulate, compareSSESequences, formatDriftReport } from "./schema.js"; +import { + extractShape, + triangulate, + compareSSESequences, + formatDriftReport, + type SSEEventShape, + type ShapeDiff, +} from "./schema.js"; import { openaiResponsesNonStreamingShape, openaiResponsesTextEventShapes, openaiResponsesToolCallEventShapes, openaiResponsesReasoningEventShapes, + openaiResponsesEncryptedReasoningEventShapes, } from "./sdk-shapes.js"; import { resolveLiveModel, @@ -558,6 +566,25 @@ describe("OpenAI Responses API reasoning drift", () => { expect(reasoningAdded, "no output_item.added with type=reasoning").toBeDefined(); }); + // NOTE: encrypted_content emission is ALSO enforced by the unit + WS suites + // (responses.test.ts / ws-responses.test.ts), which hard-fail on a dropped, + // null or empty blob and run in the ALWAYS-ON `pnpm test` lane (test-unit.yml + // has no `paths:` filter). That lane — not this file — is what gates a PR that + // touches src/responses.ts; `test-drift.yml`'s PR `paths:` filter deliberately + // scopes the live PR leg to drift-grading code (see the comment on that + // filter). + // + // The drift-side anchor is the shape pin in sdk-shapes.ts, split by opt-in: + // `openaiResponsesReasoningEventShapes` (no blob) grades the opted-OUT legs, + // `openaiResponsesEncryptedReasoningEventShapes` (blob) grades the opted-IN + // leg below. Pinning the blob unconditionally made every opted-out leg report + // `item.encrypted_content` critical forever — a finding that no code change + // could clear, so it gated nothing. + // + // A SHAPE pin alone is not enough in either direction, so `gradeEncryptedBlob` + // below adds the value-level grading. See its docstring for what shape + // comparison structurally cannot see. + it("reasoning event shapes include item_id, output_index, summary_index", async () => { const res = await httpPost(`${reasoningInstance.url}/v1/responses`, { model: "gpt-4o-mini", @@ -613,26 +640,36 @@ describe("OpenAI Responses API reasoning drift", () => { expect((partDoneData.part as { type: string; text: string }).text).toBe(REASONING_TEXT); }); - it("reasoning event shapes triangulate against SDK expectations", async () => { - const sdkEvents = openaiResponsesReasoningEventShapes(); - - const res = await httpPost(`${reasoningInstance.url}/v1/responses`, { - model: "gpt-4o-mini", - input: [{ role: "user", content: "Think carefully" }], - stream: true, - }); - - expect(res.status).toBe(200); - - const mockEvents = parseTypedSSE(res.body); - const mockSSEShapes = mockEvents.map((e) => ({ + // DELTA-KEY ISOLATION. The collector derives each diff's delta `id` from its + // `path` (`parseDriftBlock`: `id: path`) and `computeDelta` keys findings by + // `provider::id` ONLY — `DriftEntry.scenario` rides along in the report but + // NEVER enters the key. So two findings that differ only by scenario collapse + // last-write-wins: verified against the real `computeDelta`, an opted-OUT + // finding at `item.encrypted_content` sitting on the base report demotes a + // NEW-in-head opted-IN finding at the same path from `block[]` to `advisory[]` + // — i.e. it stops failing the required check. Prefixing every path this file + // reports with its scenario + event type keeps the two legs' delta keys + // disjoint without touching the shared delta machinery that every other drift + // surface depends on. + const scopePaths = (diffs: ShapeDiff[], prefix: string): ShapeDiff[] => + diffs.map((d) => ({ ...d, path: `${prefix}:${d.path}` })); + + // Grade a reasoning SSE body against the SDK shape variant for the request + // that produced it. `scenario` keeps the opted-out and opted-in legs' drift + // keys distinct so the delta gate attributes a finding to the right variant. + // + // Since reasoning is not available on gpt-4o-mini via real API, the SDK shape + // is used as both "expected" and "real" for shape validation. + const triangulateReasoningEvents = ( + sdkEvents: SSEEventShape[], + body: string, + scenario: string, + ) => { + const mockSSEShapes = parseTypedSSE(body).map((e) => ({ type: e.type, dataShape: extractShape(e.data), })); - // Triangulate reasoning-specific events against SDK shapes. - // Since reasoning is not available on gpt-4o-mini via real API, we - // use SDK shapes as both "expected" and "real" for shape validation. for (const sdkEvent of sdkEvents) { const mockEvent = mockSSEShapes.find((m) => m.type === sdkEvent.type); if (!mockEvent) { @@ -640,17 +677,155 @@ describe("OpenAI Responses API reasoning drift", () => { continue; } - const diffs = triangulate(sdkEvent.dataShape, sdkEvent.dataShape, mockEvent.dataShape); - const report = formatDriftReport( - `OpenAI Responses Reasoning:${sdkEvent.type}`, - diffs, - "openai-responses", + const context = `${scenario}:${sdkEvent.type}`; + const diffs = scopePaths( + triangulate(sdkEvent.dataShape, sdkEvent.dataShape, mockEvent.dataShape), + context, ); + const report = formatDriftReport(context, diffs, "openai-responses"); expect( diffs.filter((d) => d.severity === "critical"), report, ).toEqual([]); } + }; + + /** + * VALUE-level grading of the terminal reasoning item's `encrypted_content`. + * + * A shape pin structurally CANNOT see a degenerate value, in either direction + * (all four cases below were reproduced locally against this suite): + * + * - opted IN, `encrypted_content: null` → kind "null" vs the pin's "string", + * and schema.ts's real-vs-mock check deliberately exempts null-vs-other + * ("Allow null vs other type (optional fields)") → no diff, leg GREEN. + * - opted IN, `encrypted_content: ""` → still kind "string" → no diff, + * leg GREEN. + * - opted OUT, blob present → a field in mock but not in the pin + * grades `info` ("MOCK EXTRA FIELD"), and these legs fail only on + * `critical` → leg GREEN. + * - opted IN, blob absent → the ONE case the shape pin does + * catch (critical). + * + * The blob is the payload agent-framework-openai >= 1.11.0 replays verbatim on + * its stateless path (microsoft/agent-framework#7233), so a null or empty blob + * is exactly as broken as an absent one, and a blob handed to a request that + * never opted in is drift too. + * + * Emitted through `formatDriftReport` with the `openai-responses` surface + * marker so a violation lands as a ROUTED drift finding (collector → exit 2 → + * delta gate) instead of a bare `expect` failure the collector could only + * quarantine (exit 5). + */ + const gradeEncryptedBlob = (body: string, expectBlob: boolean, scenario: string) => { + const label = expectBlob ? "opted-in" : "opted-out"; + const path = `${scenario}:response.output_item.done:item.encrypted_content(value)`; + const diffs: ShapeDiff[] = []; + + const terminal = parseTypedSSE(body) + .filter( + (e) => + e.type === "response.output_item.done" && + (e.data as { item?: { type?: string } }).item?.type === "reasoning", + ) + .at(-1); + const item = terminal + ? (terminal.data as { item: { encrypted_content?: unknown } }).item + : undefined; + + if (item === undefined) { + // Folded into the diff list (not a bare `expect`) so a missing terminal + // item is a routed critical finding rather than a quarantined failure. + diffs.push({ + path, + severity: "critical", + issue: "LLMOCK DRIFT — no terminal response.output_item.done with type=reasoning", + expected: "reasoning item", + real: "reasoning item", + mock: "", + }); + } else { + const blob = item.encrypted_content; + const observed = blob === undefined ? "" : JSON.stringify(blob); + if (expectBlob && (typeof blob !== "string" || blob.length === 0)) { + diffs.push({ + path, + severity: "critical", + issue: + "LLMOCK DRIFT — opted-in request must carry a NON-EMPTY encrypted_content string " + + "(agent-framework replays it verbatim; null/empty is as broken as absent)", + expected: "non-empty string", + real: "non-empty string", + mock: observed, + }); + } else if (!expectBlob && blob !== undefined) { + diffs.push({ + path, + severity: "critical", + issue: + "LLMOCK DRIFT — opted-out request must carry NO encrypted_content " + + "(over-emission hands the blob to a consumer that never asked for it)", + expected: "", + real: "", + mock: observed, + }); + } + } + + const report = formatDriftReport( + `${scenario} (${label} encrypted_content value)`, + diffs, + "openai-responses", + ); + + expect( + diffs.filter((d) => d.severity === "critical"), + report, + ).toEqual([]); + }; + + it("reasoning event shapes triangulate against SDK expectations", async () => { + // Opted OUT: no `include`, no `store: false` — the terminal item must carry + // NO encrypted_content, so this leg is graded against the no-blob variant. + const res = await httpPost(`${reasoningInstance.url}/v1/responses`, { + model: "gpt-4o-mini", + input: [{ role: "user", content: "Think carefully" }], + stream: true, + }); + + expect(res.status).toBe(200); + + triangulateReasoningEvents( + openaiResponsesReasoningEventShapes(), + res.body, + "OpenAI Responses Reasoning", + ); + // Over-emission check: a blob here means the gate leaked it to a request + // that never opted in. Invisible to the shape pin (grades `info`). + gradeEncryptedBlob(res.body, false, "OpenAI Responses Reasoning"); + }); + + it("opted-in reasoning event shapes carry a non-empty encrypted_content", async () => { + // Opted IN via `include` — the exact request agent-framework-openai + // >= 1.11.0 sends on its stateless-replay path. This is the leg that GATES + // the emission: drop `encrypted_content` in responses.ts and the terminal + // item reports `item.encrypted_content` critical here. The shape pin catches + // ABSENCE; `gradeEncryptedBlob` catches null/empty, which the pin cannot. + const res = await httpPost(`${reasoningInstance.url}/v1/responses`, { + model: "gpt-4o-mini", + input: [{ role: "user", content: "Think carefully" }], + stream: true, + include: ["reasoning.encrypted_content"], + }); + + expect(res.status).toBe(200); + + triangulateReasoningEvents( + openaiResponsesEncryptedReasoningEventShapes(), + res.body, + "OpenAI Responses Encrypted Reasoning", + ); + gradeEncryptedBlob(res.body, true, "OpenAI Responses Encrypted Reasoning"); }); }); diff --git a/src/__tests__/drift/sdk-shapes.ts b/src/__tests__/drift/sdk-shapes.ts index c634bec4..7fe9eaed 100644 --- a/src/__tests__/drift/sdk-shapes.ts +++ b/src/__tests__/drift/sdk-shapes.ts @@ -401,7 +401,19 @@ export function openaiResponsesToolCallEventShapes(): SSEEventShape[] { ]; } -export function openaiResponsesReasoningEventShapes(): SSEEventShape[] { +/** + * Reasoning SSE event shapes, parameterized on the encrypted-reasoning opt-in. + * + * `emitEncrypted` mirrors `responses.ts`'s own + * `buildReasoningOutputItem({ emitEncrypted })` gate: real OpenAI (and aimock) + * carry `encrypted_content` on the terminal item ONLY when the request opted in + * (`include: ["reasoning.encrypted_content"]` or `store: false`). Pinning it + * unconditionally would report critical drift against every opted-OUT leg, + * whose absent blob is correct behavior — so the two variants are exposed as + * separate shape functions and each drift leg pins the one matching the request + * it actually sends. + */ +function reasoningEventShapes(emitEncrypted: boolean): SSEEventShape[] { return [ { type: "response.output_item.added", @@ -464,12 +476,37 @@ export function openaiResponsesReasoningEventShapes(): SSEEventShape[] { type: "reasoning", id: "rs_abc123", summary: [{ type: "summary_text", text: "Step by step..." }], + // Real OpenAI carries the encrypted reasoning blob on the terminal + // item when the request opts in (include / store:false). aimock emits + // it there too. Anchored on `done` only — `added` is opportunistic in + // real OpenAI and aimock deliberately omits it, so pinning `added` + // would flag an intentional, legal shape choice as drift. + ...(emitEncrypted ? { encrypted_content: "gAAAAAB..." } : {}), }, }), }, ]; } +/** + * Opted-OUT reasoning shapes: no `include`, no `store: false`. The terminal item + * carries NO `encrypted_content` — its absence is correct, not drift. + */ +export function openaiResponsesReasoningEventShapes(): SSEEventShape[] { + return reasoningEventShapes(false); +} + +/** + * Opted-IN reasoning shapes: the request asked for + * `include: ["reasoning.encrypted_content"]` (or went stateless with + * `store: false`), so the terminal item MUST carry the blob. This is the variant + * that gates the emission — drop the blob in `responses.ts` and the opted-in + * drift leg reports `item.encrypted_content` critical. + */ +export function openaiResponsesEncryptedReasoningEventShapes(): SSEEventShape[] { + return reasoningEventShapes(true); +} + export function openaiResponsesNonStreamingShape(): ShapeNode { return extractShape({ id: "resp_abc123", diff --git a/src/__tests__/responses.test.ts b/src/__tests__/responses.test.ts index 1139fdbf..46c8d54e 100644 --- a/src/__tests__/responses.test.ts +++ b/src/__tests__/responses.test.ts @@ -1878,6 +1878,549 @@ describe("reasoning_summary_text.done includes item_id", () => { }); }); +// ─── reasoning.encrypted_content (microsoft/agent-framework#7233) ──────────── +// +// Real OpenAI populates an opaque `encrypted_content` blob on the reasoning item +// when the request opts in via `include: ["reasoning.encrypted_content"]` OR is +// stateless (`store:false`) — the path used by agent-framework-openai >= 1.11.0. +// aimock synthesizes a deterministic placeholder so that client can replay the +// reasoning item instead of hard-failing the reasoning + multi-tool follow-up +// request. +// +// aimock carries the blob on the TERMINAL (`output_item.done`) reasoning item and +// deliberately omits it from the in-progress `added` item. That is aimock's own +// CHOICE, not upstream parity — real OpenAI DOES populate `added`, but the field +// is `anyOf: [string, null]` and non-required, so `done` is where consumers must +// read it and omitting it on `added` is legal. + +describe("reasoning.encrypted_content emission", () => { + // Mirror of `syntheticEncryptedReasoning` in responses.ts — pins the id→blob + // relationship so a non-deterministic (e.g. Math.random) blob is caught. + const expectedBlob = (id: string): string => + Buffer.from(`aimock-encrypted-reasoning:${id}`).toString("base64"); + + function reasoningItem( + events: SSEEvent[], + kind: "added" | "done", + ): { id: string; encrypted_content?: string } | undefined { + const ev = events.find( + (e) => + e.type === `response.output_item.${kind}` && + (e.item as { type?: string })?.type === "reasoning", + ); + return ev?.item as { id: string; encrypted_content?: string } | undefined; + } + + // For an OPTED-IN request, grades ONLY what is contractual about the blob on + // the in-progress `added` item: if the field is there it must be a well-formed, + // id-derived blob — absence and presence are BOTH legal. + // + // Real OpenAI populates `added` opportunistically (see the section header and + // the note on `buildReasoningOutputItem` in responses.ts) and + // `encrypted_content` is `anyOf: [string, null]` and non-required, so aimock + // omitting it there is aimock's own CHOICE, not the provider contract. The + // drift contract in this same tree already draws the line exactly here: + // `sdk-shapes.ts` anchors the blob on `done` ONLY, explicitly because "pinning + // `added` would flag an intentional, legal shape choice as drift", and + // `gradeEncryptedBlob` grades the terminal item alone. Pinning the ABSENCE here + // instead made 9 tests fail any future change that moved aimock TOWARD upstream + // parity. + // + // It deliberately does not assert presence either, so aimock's current omission + // keeps passing. Nothing real is lost by relaxing it: the property that matters + // is the blob reaching the TERMINAL item, which every call site pins directly + // via `expect(done.encrypted_content).toBe(expectedBlob(done.id))`, so putting + // the blob on `added` INSTEAD of `done` still fails. + // + // For an OPTED-OUT request, absence on `added` genuinely IS the contract (no + // blob may appear anywhere) and stays asserted at those call sites directly. + const expectAddedBlobIsContractLegal = (item: { + id: string; + encrypted_content?: string; + }): void => { + if (!("encrypted_content" in item)) return; + expect(typeof item.encrypted_content, "encrypted_content on `added` must be a string").toBe( + "string", + ); + // Present means it must be the SAME id-derived blob — a degenerate or + // mismatched value there is as broken as a wrong one on `done` + // (cf. `gradeEncryptedBlob` in the drift suite). + expect(item.encrypted_content).toBe(expectedBlob(item.id)); + }; + + const reasoningTextFixture: Fixture = { + match: { userMessage: "textreason" }, + response: { content: "AAPL vs MSFT", reasoning: "Look up AAPL, then MSFT." }, + }; + const reasoningToolFixture: Fixture = { + match: { userMessage: "toolreason" }, + response: { toolCalls: [{ name: "get_stock", arguments: "{}" }], reasoning: "Call the tool." }, + }; + const reasoningContentToolFixture: Fixture = { + match: { userMessage: "bothreason" }, + response: { + content: "done", + toolCalls: [{ name: "get_stock", arguments: "{}" }], + reasoning: "Think, then call.", + }, + }; + // `blocks` forces the positional NEW branch of buildContentWithToolCalls* (both + // streaming and non-streaming), which builds its own reasoning item. + const reasoningBlocksFixture: Fixture = { + match: { userMessage: "blocksreason" }, + response: { + content: "done", + toolCalls: [{ name: "get_stock", arguments: "{}" }], + reasoning: "Think, then call.", + blocks: [ + { type: "toolCall", name: "get_stock", arguments: "{}" }, + { type: "text", text: "done" }, + ], + }, + }; + + // --- Builder-level: placement + determinism --- + + it("builder omits encrypted_content everywhere when the flag is off", () => { + const events = buildTextStreamEvents("result", "gpt-5-test", 100, "thinking..."); + expect(reasoningItem(events, "added")!.encrypted_content).toBeUndefined(); + expect(reasoningItem(events, "done")!.encrypted_content).toBeUndefined(); + }); + + it("builder emits the blob on the done item, deterministic by id", () => { + // trailing `true` = emitEncryptedReasoning + const events = buildTextStreamEvents( + "result", + "gpt-5-test", + 100, + "thinking...", + undefined, + undefined, + true, + ); + const done = reasoningItem(events, "done")!; + expect(done.encrypted_content).toBe(expectedBlob(done.id)); // pins presence + determinism + expectAddedBlobIsContractLegal(reasoningItem(events, "added")!); + }); + + // --- HTTP streaming: every builder + gate triggers + completed.output --- + + async function streamReasoning( + fixture: Fixture, + userMessage: string, + body: Record, + ): Promise { + instance = await createServer([fixture]); + const res = await post(`${instance!.url}/v1/responses`, { + model: "gpt-5-test", + input: [{ role: "user", content: userMessage }], + stream: true, + ...body, + }); + expect(res.status).toBe(200); + return parseResponsesSSEEvents(res.body); + } + + const OPT_IN = { include: ["reasoning.encrypted_content"] }; + + // One case per streaming builder: buildTextStreamEvents, + // buildToolCallStreamEvents, buildContentWithToolCallsStreamEvents (legacy + + // blocks branches). Each must carry the blob on the done reasoning item. + const streamingCases: [string, Fixture, string][] = [ + ["text (buildTextStreamEvents)", reasoningTextFixture, "textreason"], + ["tool-only (buildToolCallStreamEvents)", reasoningToolFixture, "toolreason"], + [ + "content+tool legacy (buildContentWithToolCallsStreamEvents)", + reasoningContentToolFixture, + "bothreason", + ], + [ + "content+tool blocks (buildContentWithToolCallsStreamEvents)", + reasoningBlocksFixture, + "blocksreason", + ], + ]; + + for (const [label, fixture, msg] of streamingCases) { + it(`streaming ${label} carries the blob on done when opted in`, async () => { + const events = await streamReasoning(fixture, msg, OPT_IN); + const done = reasoningItem(events, "done")!; + expect(done, "no done reasoning item").toBeDefined(); + expect(done.encrypted_content).toBe(expectedBlob(done.id)); + expectAddedBlobIsContractLegal(reasoningItem(events, "added")!); + }); + } + + it("streaming reflects the blob in response.completed.output", async () => { + const events = await streamReasoning(reasoningTextFixture, "textreason", OPT_IN); + const completed = events.find((e) => e.type === "response.completed") as SSEEvent & { + response: { output: { type: string; id: string; encrypted_content?: string }[] }; + }; + const outputReasoning = completed.response.output.find((o) => o.type === "reasoning"); + expect(outputReasoning, "no reasoning item in completed.output").toBeDefined(); + // Pin the blob INDEPENDENTLY, against the id-derived expectation — NOT against + // the `output_item.done` event's item. `buildResponsePreamble` pushes that very + // object into `prefixOutputItems`, so the two are the SAME REFERENCE and + // comparing them is `undefined === undefined` the moment emission stops: the + // assertion passed with the feature entirely off. + expect(outputReasoning!.encrypted_content).toBe(expectedBlob(outputReasoning!.id)); + }); + + it("streaming emits the blob for a stateless request (store:false) without include", async () => { + const events = await streamReasoning(reasoningTextFixture, "textreason", { store: false }); + const done = reasoningItem(events, "done")!; + expect(done.encrypted_content).toBe(expectedBlob(done.id)); + }); + + // --- HTTP streaming LEAK direction: the same builders must stay silent --- + // + // The positive cases above only prove the blob CAN be emitted; they pass just as + // well if a call site hardcodes the flag ON. A leak is the worse failure for a + // mock: it silently changes response bytes for every consumer that never opted + // in (no `include`, no `store:false`), so the stored / opted-out replay is no + // longer byte-identical. + // + // `handleResponses` threads the gate into each streaming builder from a SEPARATE + // call site (buildTextStreamEvents, buildToolCallStreamEvents and + // buildContentWithToolCallsStreamEvents), so one negative per case is required — + // with only the text negative present, hardcoding the flag on at either of the + // other two sites left the whole suite green. + for (const [label, fixture, msg] of streamingCases) { + it(`streaming ${label} omits the blob when the request did not opt in`, async () => { + const events = await streamReasoning(fixture, msg, {}); + const done = reasoningItem(events, "done"); + expect(done, "no done reasoning item").toBeDefined(); + // Pin genuine key ABSENCE, not merely a falsy value. + expect(done!).not.toHaveProperty("encrypted_content"); + expect(reasoningItem(events, "added")!).not.toHaveProperty("encrypted_content"); + }); + } + + // --- Non-streaming: every builder path (incl. the positional blocks branch) --- + + async function nonStreamReasoning( + fixture: Fixture, + userMessage: string, + extra: Record = OPT_IN, + ): Promise<{ type: string; encrypted_content?: string; id: string }[]> { + instance = await createServer([fixture]); + const res = await post(`${instance!.url}/v1/responses`, { + model: "gpt-5-test", + input: [{ role: "user", content: userMessage }], + stream: false, + ...extra, + }); + expect(res.status).toBe(200); + const body = JSON.parse(res.body) as { + output: { type: string; encrypted_content?: string; id: string }[]; + }; + return body.output; + } + + const nonStreamingCases: [string, Fixture, string][] = [ + ["text (buildOutputPrefix)", reasoningTextFixture, "textreason"], + ["tool-only (buildToolCallResponse)", reasoningToolFixture, "toolreason"], + ["content+tool legacy (buildOutputPrefix)", reasoningContentToolFixture, "bothreason"], + [ + "content+tool blocks (buildContentWithToolCallsResponse)", + reasoningBlocksFixture, + "blocksreason", + ], + ]; + + for (const [label, fixture, msg] of nonStreamingCases) { + it(`non-streaming ${label} carries the blob when opted in`, async () => { + const output = await nonStreamReasoning(fixture, msg); + const r = output.find((o) => o.type === "reasoning")!; + expect(r, "no reasoning output item").toBeDefined(); + expect(r.encrypted_content).toBe(expectedBlob(r.id)); + }); + } + + // --- Non-streaming LEAK direction: the same builders must stay silent --- + // + // The positive cases above only prove the blob CAN be emitted; they pass just + // as well if a builder emits it unconditionally. A leak is the worse failure + // for a mock: it silently changes response bytes for every consumer that never + // opted in (no `include`, no `store:false`), so the stored/opted-out replay is + // no longer byte-identical. One negative per builder site — buildOutputPrefix + // (text + content+tool legacy), buildToolCallResponse, and the `blocks` branch + // of buildContentWithToolCallsResponse — each pinning genuine key ABSENCE, not + // merely a falsy value. + for (const [label, fixture, msg] of nonStreamingCases) { + it(`non-streaming ${label} omits the blob when the request did not opt in`, async () => { + const output = await nonStreamReasoning(fixture, msg, {}); + const r = output.find((o) => o.type === "reasoning")!; + expect(r, "no reasoning output item").toBeDefined(); + expect(r).not.toHaveProperty("encrypted_content"); + }); + } + + // --- Summary-LESS fixtures: the blob must still reach an opted-in client ---- + // + // The cases above all declare a `reasoning` summary in the fixture, which is + // what makes the reasoning item exist in the first place. A fixture RECORDED + // from the real stateless agent-framework flow typically has NO summary + // (OpenAI returns `summary: []` unless summaries are explicitly requested), so + // gating the item on a declared summary means the very flow this feature + // exists to serve gets no item and therefore no blob. Real OpenAI, for a + // reasoning-capable model, still returns a reasoning item carrying the blob. + // + // Emission stays gated on the SAME opt-in predicate, so a stored / opted-out + // replay of a summary-less fixture is byte-identical to before (no reasoning + // item at all), and on the requested model's reasoning capability, so a + // gpt-4o replay does not grow an item the real provider would never send. + + const noSummaryTextFixture: Fixture = { + match: { userMessage: "nosumtext" }, + response: { content: "AAPL is up" }, + }; + const noSummaryToolFixture: Fixture = { + match: { userMessage: "nosumtool" }, + response: { toolCalls: [{ name: "get_stock", arguments: "{}" }] }, + }; + const noSummaryContentToolFixture: Fixture = { + match: { userMessage: "nosumboth" }, + response: { content: "checking", toolCalls: [{ name: "get_stock", arguments: "{}" }] }, + }; + const noSummaryBlocksFixture: Fixture = { + match: { userMessage: "nosumblocks" }, + response: { + content: "checking", + toolCalls: [{ name: "get_stock", arguments: "{}" }], + blocks: [ + { type: "toolCall", name: "get_stock", arguments: "{}" }, + { type: "text", text: "checking" }, + ], + }, + }; + + const noSummaryCases: [string, Fixture, string][] = [ + ["text", noSummaryTextFixture, "nosumtext"], + ["tool-only", noSummaryToolFixture, "nosumtool"], + ["content+tool legacy", noSummaryContentToolFixture, "nosumboth"], + ["content+tool blocks", noSummaryBlocksFixture, "nosumblocks"], + ]; + + for (const [label, fixture, msg] of noSummaryCases) { + it(`non-streaming ${label} with NO declared summary carries an empty-summary item + blob when opted in`, async () => { + const output = await nonStreamReasoning(fixture, msg); + const r = output.find((o) => o.type === "reasoning") as + | { id: string; summary: unknown[]; encrypted_content?: string } + | undefined; + expect(r, "no reasoning output item").toBeDefined(); + expect(r!.summary).toEqual([]); + expect(r!.encrypted_content).toBe(expectedBlob(r!.id)); + // Real OpenAI leads the output array with the reasoning item. + expect(output[0].type).toBe("reasoning"); + }); + + it(`streaming ${label} with NO declared summary carries an empty-summary item + blob when opted in`, async () => { + const events = await streamReasoning(fixture, msg, OPT_IN); + const added = reasoningItem(events, "added") as + | { id: string; summary: unknown[]; encrypted_content?: string } + | undefined; + const done = reasoningItem(events, "done") as + | { id: string; summary: unknown[]; encrypted_content?: string } + | undefined; + expect(added, "no added reasoning item").toBeDefined(); + expect(done, "no done reasoning item").toBeDefined(); + expect(added!.summary).toEqual([]); + expect(done!.summary).toEqual([]); + expect(done!.encrypted_content).toBe(expectedBlob(done!.id)); + expectAddedBlobIsContractLegal(added!); + }); + + // The whole blast-radius argument: nothing changes for a client that did not + // opt in. Pins the ABSENCE of any reasoning item, streaming and not. + it(`${label} with NO declared summary emits NO reasoning item when the request did not opt in`, async () => { + const output = await nonStreamReasoning(fixture, msg, {}); + expect(output.some((o) => o.type === "reasoning")).toBe(false); + + const events = await streamReasoning(fixture, msg, {}); + expect(reasoningItem(events, "added")).toBeUndefined(); + expect(reasoningItem(events, "done")).toBeUndefined(); + expect(events.some((e) => e.type.startsWith("response.reasoning_summary"))).toBe(false); + }); + } + + it("streaming a summary-less item emits added → done with NO summary-text events between", async () => { + const events = await streamReasoning(noSummaryTextFixture, "nosumtext", OPT_IN); + // An empty summary has no parts and no deltas: emitting a + // reasoning_summary_part/text event for it would describe a part that does + // not exist in the item's `summary: []`. + expect(events.filter((e) => e.type.startsWith("response.reasoning_summary"))).toEqual([]); + const seq = events + .filter((e) => (e.item as { type?: string })?.type === "reasoning") + .map((e) => e.type); + expect(seq).toEqual(["response.output_item.added", "response.output_item.done"]); + }); + + it("streaming a synthesized reasoning item keeps output_index aligned with completed.output", async () => { + const events = await streamReasoning(noSummaryContentToolFixture, "nosumboth", OPT_IN); + const completed = events.find((e) => e.type === "response.completed") as SSEEvent & { + response: { output: { type: string; id: string }[] }; + }; + // Reasoning must LEAD the output array and every other item shifts by one. + expect(completed.response.output.map((o) => o.type)).toEqual([ + "reasoning", + "message", + "function_call", + ]); + // Every output_item.done event's output_index must equal that item's slot in + // the final output array — an inserted leading item that failed to shift the + // sequencing would desync here. + const dones = events.filter((e) => e.type === "response.output_item.done"); + for (const d of dones) { + const id = (d.item as { id: string }).id; + expect(d.output_index).toBe(completed.response.output.findIndex((o) => o.id === id)); + } + }); + + it("does NOT synthesize a reasoning item for a non-reasoning model", async () => { + // gpt-4o never emits a reasoning channel, so an opted-in replay against it + // must not grow an item the real provider would never send (aimock#254). + instance = await createServer([noSummaryTextFixture]); + const res = await post(`${instance!.url}/v1/responses`, { + model: "gpt-4o", + input: [{ role: "user", content: "nosumtext" }], + stream: false, + ...OPT_IN, + }); + expect(res.status).toBe(200); + const body = JSON.parse(res.body) as { output: { type: string }[] }; + expect(body.output.some((o) => o.type === "reasoning")).toBe(false); + }); + + // --- The 2x2 matrix: which trigger may CREATE an item vs only decorate one --- + // + // Two gates, deliberately different widths: + // blob on an item being emitted anyway → `include` OR `store: false` + // SYNTHESIZING an item the fixture never declared → `include` ONLY + // + // Creating an output item that did not previously exist is a bigger behavior + // change than adding a field to one already on the wire, so it takes the gate + // that is directly observable in the request rather than the inferred + // `store: false` one. `agent-framework-openai` >= 1.11.0 sends `include` and + // never `store`, so the motivating client is unaffected by the narrowing. + + it("row 3: store:false + a DECLARED summary still carries the blob (non-streaming, unchanged)", async () => { + const output = await nonStreamReasoning(reasoningTextFixture, "textreason", { store: false }); + const r = output.find((o) => o.type === "reasoning")!; + expect(r, "no reasoning output item").toBeDefined(); + expect(r.encrypted_content).toBe(expectedBlob(r.id)); + }); + + // ROW 4 — the whole point of the narrowing, and the row that must never + // silently regress: `store: false` must NOT conjure a reasoning item for a + // fixture that declares no summary. Asserted on both transports' HTTP paths and + // for a reasoning-CAPABLE model, so the model gate cannot be what makes it pass. + it("row 4: store:false + NO declared summary synthesizes NOTHING (non-streaming)", async () => { + const output = await nonStreamReasoning(noSummaryTextFixture, "nosumtext", { store: false }); + expect(output.some((o) => o.type === "reasoning")).toBe(false); + // Guard against the model gate being the reason this passes: the SAME + // fixture and model DO produce the item under the explicit `include` opt-in. + const optedIn = await nonStreamReasoning(noSummaryTextFixture, "nosumtext"); + expect(optedIn.some((o) => o.type === "reasoning")).toBe(true); + }); + + it("row 4: store:false + NO declared summary synthesizes NOTHING (streaming)", async () => { + const events = await streamReasoning(noSummaryTextFixture, "nosumtext", { store: false }); + expect(reasoningItem(events, "added")).toBeUndefined(); + expect(reasoningItem(events, "done")).toBeUndefined(); + const completed = events.find((e) => e.type === "response.completed") as SSEEvent & { + response: { output: { type: string }[] }; + }; + expect(completed.response.output.some((o) => o.type === "reasoning")).toBe(false); + }); + + it("row 4 holds for every builder path (tool-only, content+tool legacy, blocks)", async () => { + for (const [, fixture, msg] of noSummaryCases) { + const output = await nonStreamReasoning(fixture, msg, { store: false }); + expect( + output.some((o) => o.type === "reasoning"), + `${msg} synthesized on store:false`, + ).toBe(false); + const events = await streamReasoning(fixture, msg, { store: false }); + expect( + reasoningItem(events, "done"), + `${msg} synthesized on store:false (stream)`, + ).toBeUndefined(); + } + }); + + // --- `include` SELECTIVITY: an UNRELATED include value is not an opt-in ----- + // + // Every leak negative above sends NO `include` at all, so together they pin the + // predicate's presence/absence but NOT its selectivity. Broadening + // `requestIncludesEncryptedReasoning` from + // `include.includes("reasoning.encrypted_content")` to any non-empty `include` + // left the ENTIRE suite green (verified by mutation), and a real client sending + // `include: ["file_search_call.results"]` then received a blob it never asked + // for — the same byte-level over-emission the per-call-site negatives close, + // but per include VALUE rather than per call site. + // + // `file_search_call.results` is a genuine member of the SDK's + // `ResponseIncludable` union (openai/resources/responses/responses.d.ts: + // `'file_search_call.results' | 'message.input_image.image_url' | + // 'computer_call_output.output.image_url' | 'reasoning.encrypted_content'`) + // with nothing to do with reasoning, so this exercises a request a client + // actually sends rather than a fabricated string. + // + // `store` is deliberately absent: `requestWantsEncryptedReasoning`'s other + // branch fires on `store: false`, so sending it would make the assertion pass + // for the wrong reason. + const UNRELATED_INCLUDE = { include: ["file_search_call.results"] }; + + it("streaming omits the blob for a non-empty include that lacks the reasoning value", async () => { + instance = await createServer([reasoningTextFixture]); + const res = await post(`${instance!.url}/v1/responses`, { + model: "gpt-5-test", + input: [{ role: "user", content: "textreason" }], + stream: true, + ...UNRELATED_INCLUDE, + }); + expect(res.status).toBe(200); + // NOWHERE in the raw bytes — stronger than an item-level check, and it also + // covers `response.completed.output` and any future carrier of the field. + expect(res.body).not.toContain("encrypted_content"); + const events = parseResponsesSSEEvents(res.body); + expect(reasoningItem(events, "done"), "no done reasoning item").toBeDefined(); + expect(reasoningItem(events, "done")!).not.toHaveProperty("encrypted_content"); + expect(reasoningItem(events, "added")!).not.toHaveProperty("encrypted_content"); + }); + + it("non-streaming omits the blob for a non-empty include that lacks the reasoning value", async () => { + instance = await createServer([reasoningTextFixture]); + const res = await post(`${instance!.url}/v1/responses`, { + model: "gpt-5-test", + input: [{ role: "user", content: "textreason" }], + stream: false, + ...UNRELATED_INCLUDE, + }); + expect(res.status).toBe(200); + expect(res.body).not.toContain("encrypted_content"); + const body = JSON.parse(res.body) as { output: { type: string }[] }; + expect( + body.output.some((o) => o.type === "reasoning"), + "no reasoning output item", + ).toBe(true); + }); + + // The SYNTHESIS gate is the same predicate, so its selectivity needs the same + // negative: an unrelated include value must not conjure an item either. + it("a non-empty include that lacks the reasoning value synthesizes NOTHING", async () => { + const output = await nonStreamReasoning(noSummaryTextFixture, "nosumtext", UNRELATED_INCLUDE); + expect(output.some((o) => o.type === "reasoning")).toBe(false); + const events = await streamReasoning(noSummaryTextFixture, "nosumtext", UNRELATED_INCLUDE); + expect(reasoningItem(events, "done")).toBeUndefined(); + // Guard against the model gate being the reason this passes: the SAME fixture + // and model DO produce the item under the explicit opt-in. + const optedIn = await nonStreamReasoning(noSummaryTextFixture, "nosumtext"); + expect(optedIn.some((o) => o.type === "reasoning")).toBe(true); + }); +}); + // ─── Bug fix: multi-fco after single item_reference ───────────────────────── describe("multi-fco after single item_reference", () => { diff --git a/src/__tests__/ws-responses.test.ts b/src/__tests__/ws-responses.test.ts index eda04698..f0ac8e92 100644 --- a/src/__tests__/ws-responses.test.ts +++ b/src/__tests__/ws-responses.test.ts @@ -50,6 +50,19 @@ const toolReasoningFixture: Fixture = { }, }; +// Combined content+toolCalls fixture that also carries reasoning. Distinct match +// key so it drives the WS dispatch's content+toolCalls branch +// (buildContentWithToolCallsStreamEvents) — the third streaming builder, which +// the text-only "think" and tool-only "tool-reason" fixtures never reach. +const contentToolReasoningFixture: Fixture = { + match: { userMessage: "both-reason" }, + response: { + content: "Here you go.", + toolCalls: [{ name: "get_weather", arguments: '{"city":"NYC"}' }], + reasoning: "Let me reason, then call the tool.", + }, +}; + // Combined content+toolCalls fixture carrying an ORDERED `blocks` array placing // the tool call BEFORE the text. On the WebSocket /v1/responses surface this // must yield the function_call output item at a LOWER output_index than the @@ -74,6 +87,7 @@ const allFixtures: Fixture[] = [ reasoningFixture, capabilityReasoningFixture, toolReasoningFixture, + contentToolReasoningFixture, wsBlocksToolFirstFixture, ]; @@ -784,3 +798,223 @@ describe("WebSocket /v1/responses reasoning capability gating", () => { ws.close(); }); }); + +// ─── WS reasoning.encrypted_content (microsoft/agent-framework#7233) ───────── +// +// The gate must be honored over the WebSocket Responses transport too — the WS +// dispatch rebuilds the request field-by-field, so `include` / `store` and the +// emitted blob need transport-level coverage (deleting the threading otherwise +// leaves every test green). + +describe("WebSocket /v1/responses encrypted reasoning", () => { + const expectedBlob = (id: string): string => + Buffer.from(`aimock-encrypted-reasoning:${id}`).toString("base64"); + + function createMsg(userContent: string, extra: Record): string { + return JSON.stringify({ + type: "response.create", + model: "gpt-4.1", // non-strict keeps reasoning for non-reasoning models (see above) + input: [{ role: "user", content: userContent }], + ...extra, + }); + } + + async function collectUntilCompleted(ws: { + waitForMessages: (n: number) => Promise; + }): Promise { + const maxEvents = 50; + let events: WSEvent[] = []; + for (let count = 1; count <= maxEvents; count++) { + events = parseEvents(await ws.waitForMessages(count)); + if (events[events.length - 1].type === "response.completed") return events; + } + throw new Error( + `response.completed never arrived within ${maxEvents} events ` + + `(last: ${events[events.length - 1]?.type})`, + ); + } + + const reasoningDone = (events: WSEvent[]) => + events.find( + (e) => + e.type === "response.output_item.done" && + (e.item as { type?: string })?.type === "reasoning", + )?.item as { id: string; encrypted_content?: string } | undefined; + + it("emits the blob on the done item when opted in via include (content+reasoning)", async () => { + instance = await createServer(allFixtures); + const ws = await connectWebSocket(instance.url, "/v1/responses"); + ws.send(createMsg("think", { include: ["reasoning.encrypted_content"] })); + + const events = await collectUntilCompleted(ws); + const done = reasoningDone(events); + expect(done, "no done reasoning item").toBeDefined(); + expect(done!.encrypted_content).toBe(expectedBlob(done!.id)); + + // Also reflected in the terminal completed.output. + const completed = events.find((e) => e.type === "response.completed"); + const output = ( + completed!.response as { output: { type: string; encrypted_content?: string }[] } + ).output; + const outReasoning = output.find((o) => o.type === "reasoning"); + expect(outReasoning!.encrypted_content).toBe(done!.encrypted_content); + + ws.close(); + }); + + it("emits the blob for a tool-only reasoning response when opted in", async () => { + instance = await createServer(allFixtures); + const ws = await connectWebSocket(instance.url, "/v1/responses"); + ws.send(createMsg("tool-reason", { include: ["reasoning.encrypted_content"] })); + + const events = await collectUntilCompleted(ws); + const done = reasoningDone(events); + expect(done!.encrypted_content).toBe(expectedBlob(done!.id)); + + ws.close(); + }); + + // The WS dispatch threads the gate into each streaming builder separately, so + // the content+toolCalls branch needs its own case — the text and tool-only + // cases above both stay green if that one call site drops the flag. + it("emits the blob for a content+toolCalls reasoning response when opted in", async () => { + instance = await createServer(allFixtures); + const ws = await connectWebSocket(instance.url, "/v1/responses"); + ws.send(createMsg("both-reason", { include: ["reasoning.encrypted_content"] })); + + const events = await collectUntilCompleted(ws); + const done = reasoningDone(events); + expect(done, "no done reasoning item").toBeDefined(); + expect(done!.encrypted_content).toBe(expectedBlob(done!.id)); + + ws.close(); + }); + + it("emits the blob for a stateless request (store:false) without include", async () => { + instance = await createServer(allFixtures); + const ws = await connectWebSocket(instance.url, "/v1/responses"); + ws.send(createMsg("think", { store: false })); + + const events = await collectUntilCompleted(ws); + const done = reasoningDone(events); + expect(done!.encrypted_content).toBe(expectedBlob(done!.id)); + + ws.close(); + }); + + // --- LEAK direction: every WS gate call site must stay silent --- + // + // The positive cases above only prove the blob CAN be emitted; they pass just as + // well if a call site hardcodes the flag ON, which silently changes the bytes + // every consumer that never opted in receives. The WS dispatch threads the gate + // into each streaming builder from a SEPARATE call site, so one negative per + // branch is required — with only the text negative present, hardcoding the flag + // on in either the content+toolCalls or the tool-only branch left the entire + // suite green. + const wsLeakCases: [string, string][] = [ + ["text (buildTextStreamEvents)", "think"], + ["tool-only (buildToolCallStreamEvents)", "tool-reason"], + ["content+toolCalls (buildContentWithToolCallsStreamEvents)", "both-reason"], + ]; + + for (const [label, msg] of wsLeakCases) { + it(`omits the blob for ${label} when the request does not opt in`, async () => { + instance = await createServer(allFixtures); + const ws = await connectWebSocket(instance.url, "/v1/responses"); + ws.send(createMsg(msg, {})); + + const events = await collectUntilCompleted(ws); + const done = reasoningDone(events); + expect(done, "no done reasoning item").toBeDefined(); + // Pin genuine key ABSENCE, not merely a falsy value. + expect(done!).not.toHaveProperty("encrypted_content"); + + ws.close(); + }); + } + + // `include` SELECTIVITY. The leak cases above send NO `include` at all, so they + // pin the predicate's presence/absence but not its selectivity: broadening the + // gate from `include.includes("reasoning.encrypted_content")` to any non-empty + // `include` left the whole suite green. `file_search_call.results` is a genuine + // member of the SDK's `ResponseIncludable` union with nothing to do with + // reasoning, so a real client sending it must not receive a blob. The WS + // dispatch reads `include` off the parsed frame itself, so this needs its own + // transport-level case. No `store` — that is the gate's OTHER branch. + it("omits the blob for a non-empty include that lacks the reasoning value", async () => { + instance = await createServer(allFixtures); + const ws = await connectWebSocket(instance.url, "/v1/responses"); + ws.send(createMsg("think", { include: ["file_search_call.results"] })); + + const events = await collectUntilCompleted(ws); + const done = reasoningDone(events); + expect(done, "no done reasoning item").toBeDefined(); + expect(done!).not.toHaveProperty("encrypted_content"); + // NOWHERE in the frames, which also covers completed.output. + expect(JSON.stringify(events)).not.toContain("encrypted_content"); + + ws.close(); + }); + + // A fixture with NO declared `reasoning` summary — the shape a capture from the + // real stateless agent-framework flow has — must still reach an opted-in WS + // client with a summary-less reasoning item carrying the blob. Needs a + // reasoning-CAPABLE model (the shared `createMsg` above deliberately uses + // gpt-4.1, for which no reasoning item is synthesized at all). + function createReasoningModelMsg(userContent: string, extra: Record): string { + return JSON.stringify({ + type: "response.create", + model: "gpt-5", + input: [{ role: "user", content: userContent }], + ...extra, + }); + } + + for (const [label, msg] of [ + ["text", "hello"], + ["tool-only", "weather"], + ] as [string, string][]) { + it(`${label} fixture with NO declared summary still carries an empty-summary item + blob`, async () => { + instance = await createServer(allFixtures); + const ws = await connectWebSocket(instance.url, "/v1/responses"); + ws.send(createReasoningModelMsg(msg, { include: ["reasoning.encrypted_content"] })); + + const events = await collectUntilCompleted(ws); + const done = reasoningDone(events) as + | { id: string; summary: unknown[]; encrypted_content?: string } + | undefined; + expect(done, "no done reasoning item").toBeDefined(); + expect(done!.summary).toEqual([]); + expect(done!.encrypted_content).toBe(expectedBlob(done!.id)); + // No summary parts exist on the item, so no summary events describe them. + expect(events.some((e) => e.type.startsWith("response.reasoning_summary"))).toBe(false); + + ws.close(); + }); + + it(`${label} fixture with NO declared summary emits NO reasoning item when opted out`, async () => { + instance = await createServer(allFixtures); + const ws = await connectWebSocket(instance.url, "/v1/responses"); + ws.send(createReasoningModelMsg(msg, {})); + + const events = await collectUntilCompleted(ws); + expect(reasoningDone(events)).toBeUndefined(); + + ws.close(); + }); + + // The narrowing, over WS: `store: false` widens the BLOB gate but not the + // SYNTHESIS gate, so a summary-less fixture still yields no reasoning item. + // The WS dispatch computes both gates itself, so this needs its own case. + it(`${label} fixture with NO declared summary synthesizes NOTHING on store:false`, async () => { + instance = await createServer(allFixtures); + const ws = await connectWebSocket(instance.url, "/v1/responses"); + ws.send(createReasoningModelMsg(msg, { store: false })); + + const events = await collectUntilCompleted(ws); + expect(reasoningDone(events)).toBeUndefined(); + + ws.close(); + }); + } +}); diff --git a/src/responses.ts b/src/responses.ts index 5d0149ed..d34a4f53 100644 --- a/src/responses.ts +++ b/src/responses.ts @@ -38,6 +38,7 @@ import { strictNoMatchMessage, strictNoMatchLogLine, } from "./helpers.js"; +import { isReasoningModel } from "./model-utils.js"; import { matchFixtureDiagnostic, recordMatchOptions } from "./router.js"; import { writeErrorResponse, delay, calculateDelay } from "./sse-writer.js"; import { createInterruptionSignal } from "./interruption.js"; @@ -74,6 +75,12 @@ interface ResponsesRequest { temperature?: number; max_output_tokens?: number; response_format?: { type: string; [key: string]: unknown }; + // Additional output data requested by the caller, e.g. + // "reasoning.encrypted_content" (gates encrypted-reasoning emission). + include?: string[]; + // Server-side storage flag. `false` (ZDR / stateless replay) is one of the + // triggers for encrypted reasoning; see `requestWantsEncryptedReasoning`. + store?: boolean; [key: string]: unknown; } @@ -292,6 +299,8 @@ export function buildTextStreamEvents( reasoning?: string, webSearches?: string[], overrides?: ResponseOverrides, + emitEncryptedReasoning = false, + synthesizeSummarylessReasoning = false, ): ResponsesSSEEvent[] { const { respId, created, events, prefixOutputItems, nextOutputIndex } = buildResponsePreamble( model, @@ -299,6 +308,8 @@ export function buildTextStreamEvents( reasoning, webSearches, overrides, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); const { events: msgEvents, msgItem } = buildMessageOutputEvents( @@ -331,6 +342,8 @@ export function buildToolCallStreamEvents( reasoning?: string, webSearches?: string[], overrides?: ResponseOverrides, + emitEncryptedReasoning = false, + synthesizeSummarylessReasoning = false, ): ResponsesSSEEvent[] { const { respId, created, events, prefixOutputItems, nextOutputIndex } = buildResponsePreamble( model, @@ -338,6 +351,8 @@ export function buildToolCallStreamEvents( reasoning, webSearches, overrides, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); const fcOutputItems: object[] = []; @@ -418,10 +433,146 @@ export function buildToolCallStreamEvents( return events; } +/** + * Whether the incoming Responses request should receive encrypted reasoning. + * + * Two observed triggers, gated on EITHER: + * - `include: ["reasoning.encrypted_content"]` — the explicit opt-in, and what + * `agent-framework-openai` >= 1.11.0 auto-appends on its stateless-replay path + * (it never sends `store`), so this is the branch that fires for real clients. + * - `store: false` — captured responses carry the blob when not server-side + * stored (ZDR) and omit it when stored; kept as a secondary trigger for + * clients that go stateless without opting in via `include`. + * + * Stored / opted-out replays stay byte-identical. Exported so the WebSocket + * Responses transport can share one gate. + */ +export function requestWantsEncryptedReasoning(req: ResponsesRequest): boolean { + if (requestIncludesEncryptedReasoning(req)) return true; + return req.store === false; +} + +/** + * Whether the request carries the EXPLICIT `include` opt-in — a strictly + * narrower gate than `requestWantsEncryptedReasoning`, which also fires on + * `store: false`. + * + * This is the gate for SYNTHESIZING a reasoning item that the fixture does not + * declare (see `shouldSynthesizeBlobOnlyReasoning`), and it is deliberately the + * narrow one. Attaching a field to an item aimock was already emitting is a much + * smaller step than conjuring a whole output item that did not previously exist, + * so the larger behavior change gets the gate that is directly observable in the + * request rather than the inferred `store: false` one. `agent-framework-openai` + * >= 1.11.0 auto-appends `include` on its stateless-replay path and never sends + * `store` at all, so the motivating client is fully served by this branch — the + * narrowing costs the feature nothing. + * + * Exported so the WebSocket Responses transport shares one gate. + */ +export function requestIncludesEncryptedReasoning(req: ResponsesRequest): boolean { + const include = req.include; + return Array.isArray(include) && include.includes("reasoning.encrypted_content"); +} + +/** + * Synthetic stand-in for OpenAI's opaque `reasoning.encrypted_content`. aimock + * has no real ciphertext; it emits a base64 placeholder derived from the + * reasoning id (stable for a given item, opaque like the real field). Consumers + * store it (e.g. as agent-framework `protected_data`) and replay it verbatim; + * aimock matches on content and ignores the echoed blob. Without it, + * agent-framework-openai >= 1.11.0 hard-fails a reasoning + multi-tool chain on + * the follow-up request. See microsoft/agent-framework#7233. + * + * aimock skips inbound reasoning items entirely, so it cannot reproduce + * OpenAI's `invalid_encrypted_content` rejection of a mismatched blob/id — a + * known replay-only limitation. + */ +function syntheticEncryptedReasoning(reasoningId: string): string { + return Buffer.from(`aimock-encrypted-reasoning:${reasoningId}`).toString("base64"); +} + +/** + * Build a Responses `reasoning` output item. `summaryText` controls the summary + * (present for the terminal `done` / non-streaming item, omitted for the + * in-progress `added` item); `emitEncrypted` controls the blob INDEPENDENTLY of + * the summary, so a blob with an empty summary is expressible. aimock emits the + * blob only on the terminal item and omits it on `added`: real OpenAI populates + * `added` opportunistically, but the field is `anyOf: [string, null]` and + * non-required, so consumers should read it at `done` and treat `added` as + * best-effort — omitting there is legal and harmless. + */ +function buildReasoningOutputItem( + reasoningId: string, + opts: { summaryText?: string; emitEncrypted?: boolean } = {}, +): Record { + const item: Record = { + type: "reasoning", + id: reasoningId, + summary: + opts.summaryText === undefined ? [] : [{ type: "summary_text", text: opts.summaryText }], + }; + if (opts.emitEncrypted) { + item.encrypted_content = syntheticEncryptedReasoning(reasoningId); + } + return item; +} + +/** + * Whether to synthesize a summary-LESS reasoning item that exists purely to + * carry the encrypted blob. + * + * Every reasoning item aimock emits otherwise originates in a fixture's declared + * `reasoning` summary text. But a fixture RECORDED from the real stateless + * agent-framework flow typically has NO summary — OpenAI returns `summary: []` + * unless summaries are explicitly requested — so keying the item on a declared + * summary starves the exact flow this feature exists to serve: fully opted in, + * yet no reasoning item and therefore no blob. Real OpenAI, for a + * reasoning-capable model, still returns a reasoning item (`summary: []`) with + * the blob populated. So synthesize one, gated on: + * + * - the EXPLICIT `include` opt-in ONLY (`requestIncludesEncryptedReasoning`), + * which is NARROWER than the `requestWantsEncryptedReasoning` gate that + * controls the blob itself. A `store: false` request whose fixture declares no + * summary therefore gets NO reasoning item, exactly as before this change. + * Two reasons: (1) creating an output item that did not previously exist is a + * bigger behavior change than adding a field to an item already being + * emitted, so it earns the gate that is directly observable in the request + * rather than the inferred one; (2) the `store: false` trigger's supporting + * capture evidence is not verifiable from this repo, and a larger change must + * not be stacked on an unverifiable premise. `agent-framework-openai` >= + * 1.11.0 sends `include` and never `store`, so nothing is lost. + * - the REQUESTED model's reasoning capability (aimock#254) — emitting a + * reasoning item for gpt-4o would be LESS faithful, not more, since that + * model has no reasoning channel at all. Unlike `resolveReasoningForModel`, + * which fails open for a declared summary (a recorded summary is evidence the + * model did reason), there is nothing to preserve here: a synthesized item on + * a non-reasoning model would be pure fabrication, so this gate is hard. + * + * Note this is INDEPENDENT of `emitEncryptedReasoning`: when a fixture DOES + * declare a summary, the blob still rides on it under either trigger (including + * `store: false`), unchanged. + */ +function shouldSynthesizeBlobOnlyReasoning( + reasoning: string | undefined, + model: string | undefined, + synthesizeSummarylessReasoning: boolean, +): boolean { + if (reasoning) return false; // a declared summary already produces the item + if (!synthesizeSummarylessReasoning) return false; // no explicit `include` opt-in + return isReasoningModel(model); +} + +/** + * `reasoning` is `undefined` for a synthesized blob-only item (see + * `shouldSynthesizeBlobOnlyReasoning`). In that case the item's `summary` is + * `[]`, so the summary-part / summary-text events are SKIPPED entirely — they + * would describe a part that does not exist on the item — leaving a coherent + * `output_item.added` → `output_item.done` pair with nothing between. + */ function buildReasoningStreamEvents( - reasoning: string, - model: string, + reasoning: string | undefined, chunkSize: number, + emitEncryptedReasoning = false, ): ResponsesSSEEvent[] { const reasoningId = generateId("rs"); const events: ResponsesSSEEvent[] = []; @@ -429,56 +580,53 @@ function buildReasoningStreamEvents( events.push({ type: "response.output_item.added", output_index: 0, - item: { - type: "reasoning", - id: reasoningId, - summary: [], - }, + item: buildReasoningOutputItem(reasoningId), }); - events.push({ - type: "response.reasoning_summary_part.added", - item_id: reasoningId, - output_index: 0, - summary_index: 0, - part: { type: "summary_text", text: "" }, - }); - - for (let i = 0; i < reasoning.length; i += chunkSize) { - const slice = reasoning.slice(i, i + chunkSize); + if (reasoning !== undefined) { events.push({ - type: "response.reasoning_summary_text.delta", + type: "response.reasoning_summary_part.added", item_id: reasoningId, output_index: 0, summary_index: 0, - delta: slice, + part: { type: "summary_text", text: "" }, }); - } - events.push({ - type: "response.reasoning_summary_text.done", - item_id: reasoningId, - output_index: 0, - summary_index: 0, - text: reasoning, - }); + for (let i = 0; i < reasoning.length; i += chunkSize) { + const slice = reasoning.slice(i, i + chunkSize); + events.push({ + type: "response.reasoning_summary_text.delta", + item_id: reasoningId, + output_index: 0, + summary_index: 0, + delta: slice, + }); + } - events.push({ - type: "response.reasoning_summary_part.done", - item_id: reasoningId, - output_index: 0, - summary_index: 0, - part: { type: "summary_text", text: reasoning }, - }); + events.push({ + type: "response.reasoning_summary_text.done", + item_id: reasoningId, + output_index: 0, + summary_index: 0, + text: reasoning, + }); + + events.push({ + type: "response.reasoning_summary_part.done", + item_id: reasoningId, + output_index: 0, + summary_index: 0, + part: { type: "summary_text", text: reasoning }, + }); + } events.push({ type: "response.output_item.done", output_index: 0, - item: { - type: "reasoning", - id: reasoningId, - summary: [{ type: "summary_text", text: reasoning }], - }, + item: buildReasoningOutputItem(reasoningId, { + summaryText: reasoning, + emitEncrypted: emitEncryptedReasoning, + }), }); return events; @@ -536,6 +684,8 @@ function buildResponsePreamble( reasoning?: string, webSearches?: string[], overrides?: ResponseOverrides, + emitEncryptedReasoning = false, + synthesizeSummarylessReasoning = false, ): PreambleResult { const respId = overrides?.id ?? responseId(); const created = overrides?.created ?? Math.floor(Date.now() / 1000); @@ -567,8 +717,15 @@ function buildResponsePreamble( }, }); - if (reasoning) { - const reasoningEvents = buildReasoningStreamEvents(reasoning, model, chunkSize); + if ( + reasoning || + shouldSynthesizeBlobOnlyReasoning(reasoning, model, synthesizeSummarylessReasoning) + ) { + const reasoningEvents = buildReasoningStreamEvents( + reasoning, + chunkSize, + emitEncryptedReasoning, + ); events.push(...reasoningEvents); const doneEvent = reasoningEvents.find( (e) => @@ -725,15 +882,26 @@ function buildFunctionCallOutputEvents( // ─── Non-streaming response builders ──────────────────────────────────────── -function buildOutputPrefix(content: string, reasoning?: string, webSearches?: string[]): object[] { +function buildOutputPrefix( + content: string, + model: string, + reasoning?: string, + webSearches?: string[], + emitEncryptedReasoning = false, + synthesizeSummarylessReasoning = false, +): object[] { const output: object[] = []; - if (reasoning) { - output.push({ - type: "reasoning", - id: generateId("rs"), - summary: [{ type: "summary_text", text: reasoning }], - }); + if ( + reasoning || + shouldSynthesizeBlobOnlyReasoning(reasoning, model, synthesizeSummarylessReasoning) + ) { + output.push( + buildReasoningOutputItem(generateId("rs"), { + summaryText: reasoning, + emitEncrypted: emitEncryptedReasoning, + }), + ); } if (webSearches && webSearches.length > 0) { @@ -780,10 +948,19 @@ function buildTextResponse( reasoning?: string, webSearches?: string[], overrides?: ResponseOverrides, + emitEncryptedReasoning = false, + synthesizeSummarylessReasoning = false, ): object { return buildResponseEnvelope( model, - buildOutputPrefix(content, reasoning, webSearches), + buildOutputPrefix( + content, + model, + reasoning, + webSearches, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, + ), overrides, ); } @@ -794,14 +971,20 @@ function buildToolCallResponse( reasoning?: string, webSearches?: string[], overrides?: ResponseOverrides, + emitEncryptedReasoning = false, + synthesizeSummarylessReasoning = false, ): object { const output: object[] = []; - if (reasoning) { - output.push({ - type: "reasoning", - id: generateId("rs"), - summary: [{ type: "summary_text", text: reasoning }], - }); + if ( + reasoning || + shouldSynthesizeBlobOnlyReasoning(reasoning, model, synthesizeSummarylessReasoning) + ) { + output.push( + buildReasoningOutputItem(generateId("rs"), { + summaryText: reasoning, + emitEncrypted: emitEncryptedReasoning, + }), + ); } if (webSearches && webSearches.length > 0) { for (const query of webSearches) { @@ -835,6 +1018,8 @@ export function buildContentWithToolCallsStreamEvents( webSearches?: string[], overrides?: ResponseOverrides, blocks?: FixtureBlock[], + emitEncryptedReasoning = false, + synthesizeSummarylessReasoning = false, ): ResponsesSSEEvent[] { const { respId, created, events, prefixOutputItems, nextOutputIndex } = buildResponsePreamble( model, @@ -842,6 +1027,8 @@ export function buildContentWithToolCallsStreamEvents( reasoning, webSearches, overrides, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); // The output items assembled in emission order (after any reasoning / @@ -944,6 +1131,8 @@ function buildContentWithToolCallsResponse( webSearches?: string[], overrides?: ResponseOverrides, blocks?: FixtureBlock[], + emitEncryptedReasoning = false, + synthesizeSummarylessReasoning = false, ): object { if (blocks && blocks.length > 0) { // NEW PATH: the non-streaming `output[]` array is positionally observable, @@ -953,12 +1142,16 @@ function buildContentWithToolCallsResponse( // path's ordering for the same `blocks` fixture. const ordered = resolveFixtureBlocks(blocks); const output: object[] = []; - if (reasoning) { - output.push({ - type: "reasoning", - id: generateId("rs"), - summary: [{ type: "summary_text", text: reasoning }], - }); + if ( + reasoning || + shouldSynthesizeBlobOnlyReasoning(reasoning, model, synthesizeSummarylessReasoning) + ) { + output.push( + buildReasoningOutputItem(generateId("rs"), { + summaryText: reasoning, + emitEncrypted: emitEncryptedReasoning, + }), + ); } if (webSearches && webSearches.length > 0) { for (const query of webSearches) { @@ -987,7 +1180,14 @@ function buildContentWithToolCallsResponse( } // LEGACY PATH: message item first, then function_call items — unchanged. - const output = buildOutputPrefix(content, reasoning, webSearches); + const output = buildOutputPrefix( + content, + model, + reasoning, + webSearches, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, + ); for (const tc of toolCalls) { output.push(buildFunctionCallOutputItem(tc)); } @@ -1085,6 +1285,19 @@ export async function handleResponses( completionReq._endpointType = "chat"; completionReq._context = getContext(req); + // Emit a synthetic `reasoning.encrypted_content` only when the request wants it + // (opted in via `include`, or stateless `store: false` — mirrors real OpenAI). + // This is what agent-framework-openai >= 1.11.0 sends on stateless-replay + // requests; without the blob it hard-fails a reasoning + multi-tool chain. + // See microsoft/agent-framework#7233. + const emitEncryptedReasoning = requestWantsEncryptedReasoning(responsesReq); + // Synthesizing a reasoning item the fixture never declared is a bigger step + // than adding a field to one already being emitted, so it takes the NARROWER + // explicit-`include` gate: a `store: false` request with a summary-less fixture + // keeps emitting no reasoning item at all. See + // `shouldSynthesizeBlobOnlyReasoning`. + const synthesizeSummarylessReasoning = requestIncludesEncryptedReasoning(responsesReq); + const testId = getTestId(req); const { fixture, skippedBySequenceOrTurn } = matchFixtureDiagnostic( fixtures, @@ -1261,6 +1474,8 @@ export async function handleResponses( response.webSearches, overrides, response.blocks, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); res.writeHead(200, { "Content-Type": "application/json" }); res.end(JSON.stringify(body)); @@ -1274,6 +1489,8 @@ export async function handleResponses( response.webSearches, overrides, response.blocks, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); const interruption = createInterruptionSignal(fixture); const completed = await writeResponsesSSEStream(res, events, { @@ -1319,6 +1536,8 @@ export async function handleResponses( effReasoning, response.webSearches, overrides, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); res.writeHead(200, { "Content-Type": "application/json" }); res.end(JSON.stringify(body)); @@ -1330,6 +1549,8 @@ export async function handleResponses( effReasoning, response.webSearches, overrides, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); const interruption = createInterruptionSignal(fixture); const completed = await writeResponsesSSEStream(res, events, { @@ -1375,6 +1596,8 @@ export async function handleResponses( effReasoning, response.webSearches, overrides, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); res.writeHead(200, { "Content-Type": "application/json" }); res.end(JSON.stringify(body)); @@ -1386,6 +1609,8 @@ export async function handleResponses( effReasoning, response.webSearches, overrides, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); const interruption = createInterruptionSignal(fixture); const completed = await writeResponsesSSEStream(res, events, { diff --git a/src/ws-responses.ts b/src/ws-responses.ts index acef2f13..8dfb2248 100644 --- a/src/ws-responses.ts +++ b/src/ws-responses.ts @@ -10,6 +10,8 @@ import type { ChatCompletionRequest, Fixture } from "./types.js"; import { matchFixtureDiagnostic } from "./router.js"; import { responsesToCompletionRequest, + requestIncludesEncryptedReasoning, + requestWantsEncryptedReasoning, buildTextStreamEvents, buildToolCallStreamEvents, buildContentWithToolCallsStreamEvents, @@ -172,8 +174,18 @@ async function processMessage( stream: parsed.stream, temperature: parsed.temperature, max_output_tokens: parsed.max_output_tokens, + include: (parsed as { include?: string[] }).include, + store: (parsed as { store?: boolean }).store, }; + // Gate encrypted-reasoning emission identically to the HTTP transport so the + // agent-framework#7233 stateless-replay path works over WebSocket too. + const emitEncryptedReasoning = requestWantsEncryptedReasoning(responsesReq); + // Synthesizing a reasoning item the fixture never declared takes the NARROWER + // explicit-`include` gate, transport-identically to HTTP — a `store: false` + // request with a summary-less fixture still gets no reasoning item. + const synthesizeSummarylessReasoning = requestIncludesEncryptedReasoning(responsesReq); + const completionReq = responsesToCompletionRequest(responsesReq); completionReq._endpointType = "chat"; const contextHeader = defaults.upgradeHeaders?.["x-aimock-context"]; @@ -283,6 +295,8 @@ async function processMessage( response.webSearches, extractOverrides(response), response.blocks, + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); const interruption = createInterruptionSignal(fixture); @@ -326,6 +340,8 @@ async function processMessage( ), response.webSearches, extractOverrides(response), + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); const interruption = createInterruptionSignal(fixture); const completed = await sendEvents( @@ -370,6 +386,8 @@ async function processMessage( ), response.webSearches, extractOverrides(response), + emitEncryptedReasoning, + synthesizeSummarylessReasoning, ); const interruption = createInterruptionSignal(fixture); const completed = await sendEvents(