feat(responses): synthesize reasoning.encrypted_content when requested

.github/workflows/test-drift.yml

@@ -3,6 +3,20 @@ on:
   schedule:
     - cron: "0 6 * * *" # Daily 6am UTC
   pull_request:
+    # DELIBERATE SCOPE — this filter lists DRIFT-GRADING code, not the production
+    # files the drift legs grade. `drift-live-pr` is a ~30-minute job that burns
+    # real provider credits on every trigger, and its question is "does THIS diff
+    # introduce provider-format divergence in the drift harness", not "does the
+    # mock still emit field X". Adding a graded production file (src/responses.ts,
+    # src/ws-responses.ts, …) would, by the same logic, pull in nearly every file
+    # under src/ and run the live leg on almost every PR.
+    #
+    # Per-field emission properties are gated by the ALWAYS-ON unit lane instead:
+    # test-unit.yml has no `paths:` filter, so `pnpm test` runs on every PR and
+    # hard-fails on a dropped/null/empty value (e.g. the `encrypted_content`
+    # value pins in src/__tests__/responses.test.ts and ws-responses.test.ts).
+    # If you came here because a drift assertion "never triggers on the code it
+    # guards": check the unit lane first — that is where the value pin lives.
     paths:
       - "src/agui-types.ts"
       - "src/__tests__/drift/agui-schema.drift.ts"

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 ### Added
 
+- **Reasoning `encrypted_content` on the Responses API.** aimock now synthesizes an opaque (base64) `reasoning.encrypted_content` blob on the TERMINAL reasoning item (`response.output_item.done` and non-streaming `output[]`). Emitted only when the request wants it: `include: ["reasoning.encrypted_content"]` OR a stateless request (`store: false` / ZDR). Stored / opted-out replays stay byte-identical. A request carrying the EXPLICIT `include: ["reasoning.encrypted_content"]` opt-in, against a reasoning-CAPABLE model, now gets a reasoning item even when the fixture declares NO `reasoning` summary — a `summary: []` item that exists purely to carry the blob, matching what real OpenAI returns when summaries were never requested. This is the shape a fixture recorded from the real stateless agent-framework flow has, so gating the item on a declared summary starved the exact case the feature exists to serve. The synthesized item leads `output[]` / takes `output_index` 0 and shifts the rest, emits `output_item.added` → `output_item.done` with no summary-text events (there is no summary part to describe), and is suppressed for non-reasoning models (gpt-4o etc.), which have no reasoning channel at all. Note the two gates differ in width on purpose: `store: false` still attaches the blob to a reasoning item that a declared summary already produces, but it does NOT synthesize an item where the fixture declares none — creating an output item that did not previously exist is a bigger behavior change than adding a field to one already on the wire, so it takes the gate that is directly observable in the request. `agent-framework-openai` >= 1.11.0 sends `include` and never `store`, so the narrower gate costs the feature nothing. This lets `agent-framework-openai` >= 1.11.0 replay reasoning-paired tool calls on a stateless reasoning + multi-tool chain (related: microsoft/agent-framework#7233). Applies to both the HTTP and WebSocket Responses transports. Known limitations: the blob is withheld from the in-progress `added` item, which is aimock's own simplification rather than upstream parity — real OpenAI populates `added` too (recorded captures carry a shorter blob there beside an empty summary, re-encrypted by `done`), but the field is `anyOf: [string, null]` and non-required, so omitting it is contract-legal and no known consumer requires it; and aimock skips inbound reasoning items, so it cannot reproduce OpenAI's `invalid_encrypted_content` rejection of a mismatched blob/id.
 - **OpenRouter chat / LLM router simulation.** Requests whose original path starts with `/api/v1/` (point any OpenAI SDK at a `baseURL` ending `/api/v1`) are detected as OpenRouter and shaped to match real OpenRouter bytes: a `gen-` id prefix (a fixture `id` override still wins), a top-level `provider` (default = the winning model slug's author, fixture-overridable via `provider`), both `finish_reason` and `native_finish_reason` on every choice/delta, an always-present `system_fingerprint` and `service_tier` (null by default), `message.reasoning`, and a rich `usage` with a **fixture-scriptable** `cost` + `cost_details` (emitted only when a fixture supplies a cost — never fabricated), plus `is_byok` / `prompt_tokens_details` / `completion_tokens_details` when overridden. Callers on the plain OpenAI `/v1/...` base are byte-for-byte unchanged.
 - **`models[]` fallback (router failover) simulation.** When the request body carries `models: [...]`, aimock walks `[model, ...models]` in order and serves the first fixture returning a NON-error response; a `429`/`503` error fixture on a candidate simulates a runtime provider failure and falls through to the next. The winning slug is echoed back as the top-level `model`, so a test asserts failover by reading `response.model`. (Deliberate non-goal: an unknown/invalid model is a fixture miss, not OpenRouter's up-front invalid-model 400.)
 - **OpenRouter discovery endpoints** `GET /api/v1/models` (catalog synthesized from loaded chat fixtures), `GET /api/v1/key`, and `GET /api/v1/credits`, matching the real response shapes.

src/__tests__/drift/openai-responses.drift.ts

@@ -7,12 +7,20 @@
 import { describe, it, expect, beforeAll, afterAll } from "vitest";
 import { createServer, type ServerInstance } from "../../server.js";
 import type { Fixture } from "../../types.js";
-import { extractShape, triangulate, compareSSESequences, formatDriftReport } from "./schema.js";
+import {
+  extractShape,
+  triangulate,
+  compareSSESequences,
+  formatDriftReport,
+  type SSEEventShape,
+  type ShapeDiff,
+} from "./schema.js";
 import {
   openaiResponsesNonStreamingShape,
   openaiResponsesTextEventShapes,
   openaiResponsesToolCallEventShapes,
   openaiResponsesReasoningEventShapes,
+  openaiResponsesEncryptedReasoningEventShapes,
 } from "./sdk-shapes.js";
 import {
   resolveLiveModel,
@@ -558,6 +566,25 @@ describe("OpenAI Responses API reasoning drift", () => {
     expect(reasoningAdded, "no output_item.added with type=reasoning").toBeDefined();
   });
 
+  // NOTE: encrypted_content emission is ALSO enforced by the unit + WS suites
+  // (responses.test.ts / ws-responses.test.ts), which hard-fail on a dropped,
+  // null or empty blob and run in the ALWAYS-ON `pnpm test` lane (test-unit.yml
+  // has no `paths:` filter). That lane — not this file — is what gates a PR that
+  // touches src/responses.ts; `test-drift.yml`'s PR `paths:` filter deliberately
+  // scopes the live PR leg to drift-grading code (see the comment on that
+  // filter).
+  //
+  // The drift-side anchor is the shape pin in sdk-shapes.ts, split by opt-in:
+  // `openaiResponsesReasoningEventShapes` (no blob) grades the opted-OUT legs,
+  // `openaiResponsesEncryptedReasoningEventShapes` (blob) grades the opted-IN
+  // leg below. Pinning the blob unconditionally made every opted-out leg report
+  // `item.encrypted_content` critical forever — a finding that no code change
+  // could clear, so it gated nothing.
+  //
+  // A SHAPE pin alone is not enough in either direction, so `gradeEncryptedBlob`
+  // below adds the value-level grading. See its docstring for what shape
+  // comparison structurally cannot see.
+
   it("reasoning event shapes include item_id, output_index, summary_index", async () => {
     const res = await httpPost(`${reasoningInstance.url}/v1/responses`, {
       model: "gpt-4o-mini",
@@ -613,44 +640,192 @@ describe("OpenAI Responses API reasoning drift", () => {
     expect((partDoneData.part as { type: string; text: string }).text).toBe(REASONING_TEXT);
   });
 
-  it("reasoning event shapes triangulate against SDK expectations", async () => {
-    const sdkEvents = openaiResponsesReasoningEventShapes();
-
-    const res = await httpPost(`${reasoningInstance.url}/v1/responses`, {
-      model: "gpt-4o-mini",
-      input: [{ role: "user", content: "Think carefully" }],
-      stream: true,
-    });
-
-    expect(res.status).toBe(200);
-
-    const mockEvents = parseTypedSSE(res.body);
-    const mockSSEShapes = mockEvents.map((e) => ({
+  // DELTA-KEY ISOLATION. The collector derives each diff's delta `id` from its
+  // `path` (`parseDriftBlock`: `id: path`) and `computeDelta` keys findings by
+  // `provider::id` ONLY — `DriftEntry.scenario` rides along in the report but
+  // NEVER enters the key. So two findings that differ only by scenario collapse
+  // last-write-wins: verified against the real `computeDelta`, an opted-OUT
+  // finding at `item.encrypted_content` sitting on the base report demotes a
+  // NEW-in-head opted-IN finding at the same path from `block[]` to `advisory[]`
+  // — i.e. it stops failing the required check. Prefixing every path this file
+  // reports with its scenario + event type keeps the two legs' delta keys
+  // disjoint without touching the shared delta machinery that every other drift
+  // surface depends on.
+  const scopePaths = (diffs: ShapeDiff[], prefix: string): ShapeDiff[] =>
+    diffs.map((d) => ({ ...d, path: `${prefix}:${d.path}` }));
+
+  // Grade a reasoning SSE body against the SDK shape variant for the request
+  // that produced it. `scenario` keeps the opted-out and opted-in legs' drift
+  // keys distinct so the delta gate attributes a finding to the right variant.
+  //
+  // Since reasoning is not available on gpt-4o-mini via real API, the SDK shape
+  // is used as both "expected" and "real" for shape validation.
+  const triangulateReasoningEvents = (
+    sdkEvents: SSEEventShape[],
+    body: string,
+    scenario: string,
+  ) => {
+    const mockSSEShapes = parseTypedSSE(body).map((e) => ({
       type: e.type,
       dataShape: extractShape(e.data),
     }));
 
-    // Triangulate reasoning-specific events against SDK shapes.
-    // Since reasoning is not available on gpt-4o-mini via real API, we
-    // use SDK shapes as both "expected" and "real" for shape validation.
     for (const sdkEvent of sdkEvents) {
       const mockEvent = mockSSEShapes.find((m) => m.type === sdkEvent.type);
       if (!mockEvent) {
         expect.fail(`Mock missing reasoning event type: ${sdkEvent.type}`);
         continue;
       }
 
-      const diffs = triangulate(sdkEvent.dataShape, sdkEvent.dataShape, mockEvent.dataShape);
-      const report = formatDriftReport(
-        `OpenAI Responses Reasoning:${sdkEvent.type}`,
-        diffs,
-        "openai-responses",
+      const context = `${scenario}:${sdkEvent.type}`;
+      const diffs = scopePaths(
+        triangulate(sdkEvent.dataShape, sdkEvent.dataShape, mockEvent.dataShape),
+        context,
       );
+      const report = formatDriftReport(context, diffs, "openai-responses");
 
       expect(
         diffs.filter((d) => d.severity === "critical"),
         report,
       ).toEqual([]);
     }
+  };
+
+  /**
+   * VALUE-level grading of the terminal reasoning item's `encrypted_content`.
+   *
+   * A shape pin structurally CANNOT see a degenerate value, in either direction
+   * (all four cases below were reproduced locally against this suite):
+   *
+   *   - opted IN, `encrypted_content: null` → kind "null" vs the pin's "string",
+   *     and schema.ts's real-vs-mock check deliberately exempts null-vs-other
+   *     ("Allow null vs other type (optional fields)") → no diff, leg GREEN.
+   *   - opted IN, `encrypted_content: ""`   → still kind "string" → no diff,
+   *     leg GREEN.
+   *   - opted OUT, blob present             → a field in mock but not in the pin
+   *     grades `info` ("MOCK EXTRA FIELD"), and these legs fail only on
+   *     `critical` → leg GREEN.
+   *   - opted IN, blob absent               → the ONE case the shape pin does
+   *     catch (critical).
+   *
+   * The blob is the payload agent-framework-openai >= 1.11.0 replays verbatim on
+   * its stateless path (microsoft/agent-framework#7233), so a null or empty blob
+   * is exactly as broken as an absent one, and a blob handed to a request that
+   * never opted in is drift too.
+   *
+   * Emitted through `formatDriftReport` with the `openai-responses` surface
+   * marker so a violation lands as a ROUTED drift finding (collector → exit 2 →
+   * delta gate) instead of a bare `expect` failure the collector could only
+   * quarantine (exit 5).
+   */
+  const gradeEncryptedBlob = (body: string, expectBlob: boolean, scenario: string) => {
+    const label = expectBlob ? "opted-in" : "opted-out";
+    const path = `${scenario}:response.output_item.done:item.encrypted_content(value)`;
+    const diffs: ShapeDiff[] = [];
+
+    const terminal = parseTypedSSE(body)
+      .filter(
+        (e) =>
+          e.type === "response.output_item.done" &&
+          (e.data as { item?: { type?: string } }).item?.type === "reasoning",
+      )
+      .at(-1);
+    const item = terminal
+      ? (terminal.data as { item: { encrypted_content?: unknown } }).item
+      : undefined;
+
+    if (item === undefined) {
+      // Folded into the diff list (not a bare `expect`) so a missing terminal
+      // item is a routed critical finding rather than a quarantined failure.
+      diffs.push({
+        path,
+        severity: "critical",
+        issue: "LLMOCK DRIFT — no terminal response.output_item.done with type=reasoning",
+        expected: "reasoning item",
+        real: "reasoning item",
+        mock: "<absent>",
+      });
+    } else {
+      const blob = item.encrypted_content;
+      const observed = blob === undefined ? "<absent>" : JSON.stringify(blob);
+      if (expectBlob && (typeof blob !== "string" || blob.length === 0)) {
+        diffs.push({
+          path,
+          severity: "critical",
+          issue:
+            "LLMOCK DRIFT — opted-in request must carry a NON-EMPTY encrypted_content string " +
+            "(agent-framework replays it verbatim; null/empty is as broken as absent)",
+          expected: "non-empty string",
+          real: "non-empty string",
+          mock: observed,
+        });
+      } else if (!expectBlob && blob !== undefined) {
+        diffs.push({
+          path,
+          severity: "critical",
+          issue:
+            "LLMOCK DRIFT — opted-out request must carry NO encrypted_content " +
+            "(over-emission hands the blob to a consumer that never asked for it)",
+          expected: "<absent>",
+          real: "<absent>",
+          mock: observed,
+        });
+      }
+    }
+
+    const report = formatDriftReport(
+      `${scenario} (${label} encrypted_content value)`,
+      diffs,
+      "openai-responses",
+    );
+
+    expect(
+      diffs.filter((d) => d.severity === "critical"),
+      report,
+    ).toEqual([]);
+  };
+
+  it("reasoning event shapes triangulate against SDK expectations", async () => {
+    // Opted OUT: no `include`, no `store: false` — the terminal item must carry
+    // NO encrypted_content, so this leg is graded against the no-blob variant.
+    const res = await httpPost(`${reasoningInstance.url}/v1/responses`, {
+      model: "gpt-4o-mini",
+      input: [{ role: "user", content: "Think carefully" }],
+      stream: true,
+    });
+
+    expect(res.status).toBe(200);
+
+    triangulateReasoningEvents(
+      openaiResponsesReasoningEventShapes(),
+      res.body,
+      "OpenAI Responses Reasoning",
+    );
+    // Over-emission check: a blob here means the gate leaked it to a request
+    // that never opted in. Invisible to the shape pin (grades `info`).
+    gradeEncryptedBlob(res.body, false, "OpenAI Responses Reasoning");
+  });
+
+  it("opted-in reasoning event shapes carry a non-empty encrypted_content", async () => {
+    // Opted IN via `include` — the exact request agent-framework-openai
+    // >= 1.11.0 sends on its stateless-replay path. This is the leg that GATES
+    // the emission: drop `encrypted_content` in responses.ts and the terminal
+    // item reports `item.encrypted_content` critical here. The shape pin catches
+    // ABSENCE; `gradeEncryptedBlob` catches null/empty, which the pin cannot.
+    const res = await httpPost(`${reasoningInstance.url}/v1/responses`, {
+      model: "gpt-4o-mini",
+      input: [{ role: "user", content: "Think carefully" }],
+      stream: true,
+      include: ["reasoning.encrypted_content"],
+    });
+
+    expect(res.status).toBe(200);
+
+    triangulateReasoningEvents(
+      openaiResponsesEncryptedReasoningEventShapes(),
+      res.body,
+      "OpenAI Responses Encrypted Reasoning",
+    );
+    gradeEncryptedBlob(res.body, true, "OpenAI Responses Encrypted Reasoning");
   });
 });

src/__tests__/drift/sdk-shapes.ts

@@ -401,7 +401,19 @@ export function openaiResponsesToolCallEventShapes(): SSEEventShape[] {
   ];
 }
 
-export function openaiResponsesReasoningEventShapes(): SSEEventShape[] {
+/**
+ * Reasoning SSE event shapes, parameterized on the encrypted-reasoning opt-in.
+ *
+ * `emitEncrypted` mirrors `responses.ts`'s own
+ * `buildReasoningOutputItem({ emitEncrypted })` gate: real OpenAI (and aimock)
+ * carry `encrypted_content` on the terminal item ONLY when the request opted in
+ * (`include: ["reasoning.encrypted_content"]` or `store: false`). Pinning it
+ * unconditionally would report critical drift against every opted-OUT leg,
+ * whose absent blob is correct behavior — so the two variants are exposed as
+ * separate shape functions and each drift leg pins the one matching the request
+ * it actually sends.
+ */
+function reasoningEventShapes(emitEncrypted: boolean): SSEEventShape[] {
   return [
     {
       type: "response.output_item.added",
@@ -464,12 +476,37 @@ export function openaiResponsesReasoningEventShapes(): SSEEventShape[] {
           type: "reasoning",
           id: "rs_abc123",
           summary: [{ type: "summary_text", text: "Step by step..." }],
+          // Real OpenAI carries the encrypted reasoning blob on the terminal
+          // item when the request opts in (include / store:false). aimock emits
+          // it there too. Anchored on `done` only — `added` is opportunistic in
+          // real OpenAI and aimock deliberately omits it, so pinning `added`
+          // would flag an intentional, legal shape choice as drift.
+          ...(emitEncrypted ? { encrypted_content: "gAAAAAB..." } : {}),
         },
       }),
     },
   ];
 }
 
+/**
+ * Opted-OUT reasoning shapes: no `include`, no `store: false`. The terminal item
+ * carries NO `encrypted_content` — its absence is correct, not drift.
+ */
+export function openaiResponsesReasoningEventShapes(): SSEEventShape[] {
+  return reasoningEventShapes(false);
+}
+
+/**
+ * Opted-IN reasoning shapes: the request asked for
+ * `include: ["reasoning.encrypted_content"]` (or went stateless with
+ * `store: false`), so the terminal item MUST carry the blob. This is the variant
+ * that gates the emission — drop the blob in `responses.ts` and the opted-in
+ * drift leg reports `item.encrypted_content` critical.
+ */
+export function openaiResponsesEncryptedReasoningEventShapes(): SSEEventShape[] {
+  return reasoningEventShapes(true);
+}
+
 export function openaiResponsesNonStreamingShape(): ShapeNode {
   return extractShape({
     id: "resp_abc123",