feat(chat): migrate the chat protocols onto the pipeline - #476
Draft
Menci wants to merge 77 commits into
Draft
Conversation
…ols share The first commit of the chat migration. Four source protocols, ten translation directions between them, and one array of stages every chain runs — this is the space they run over and the first three entries of that array. The key layout says two things the context object could not. Each protocol has its own request and response key, so a translation consumes the source and provides the target — and `compose` then refuses a chain that could re-enter its own protocol, which is `ctx.targetApi === <self>` made structural. And `ingress.*` is what the client asked for, so `wantsStream` is four disjoint keys rather than one: a translated request must not inherit the target protocol's answer to a question the client never asked. That guard turns out to be two things wearing one expression. "Do not run on a re-entered request" is now structural and no stage asks it. "Which protocol was chosen" stays a live question, because a vendor normalizer genuinely needs to know — it becomes a declared fact read through `needs`, which is visibility rather than removal. So the five guards are not deleted by the migration; they change what they read. The rewrites themselves are transcribed, not redesigned: they already worked. What is new is that a stage returns the record it hands on instead of assigning to `ctx.payload`. Today's code already writes its map conditionally, which is the convention that keeps a 49-message conversation costing three objects, so the tests assert identity rather than equality — a payload no flag touched comes back as the same object, and a flag that fires but changes nothing changes nothing. Gemini is source-only. Nothing in `packages/translate/src/` targets it, so by the ruling it contributes one pipeline rather than two: a source role and no target role.
The chat interceptors were written against route.candidate, the key the selector split replaced. They reach for the live candidate only to read its enabled flags, and the selector carries those as data — so they now read what travels in the record and the tests need no candidate stub at all. The shared chat stages arrive with them: resolveChatCandidates enumerates, narrows to the wires a source protocol can reach, and orders by affinity, which can refuse outright when a turn's own state requires two upstreams at once.
The reference chat chain, and what it establishes for the three families that follow: the edge decides whether the client sees the frames or the one object they add up to, which is where the stream-to-value collection belongs — the upstream speaks SSE whatever the client asked for, so folding is the edge's own work rather than a second reading. The shared chat resolver narrows to the wires a source protocol can reach and orders what is left by affinity, which can refuse a turn whose carried state needs two upstreams at once. The interceptors run as ordinary stages between the fork and the ending. Metering reads the upstream's own usage off its own events as they pass, so a streaming turn settles from the promise its last chunk resolves rather than in the stage. The route still runs the existing surface.
The chain composed and nothing had run it. Doing so found the edge handing its SSE view up without move(), which the handover gate refuses — a stream the run never took ownership of would have escaped the record's own guarantee. The five cases are the ones only running can state: frames written out when the client asked to stream, the same frames folded into one object when it did not, the upstream's forwardable headers kept and content-length dropped, a refusal carrying the upstream's own status and words, and a dial nobody answered failing over to the next candidate rather than ending the run.
The chain declared the services affinity needs and then read the client's own payload straight out of the record, so the per-candidate rewrite was lost — a turn carrying state for one upstream would have been sent to another unchanged. The ending now asks for what affinity materialized, and records the candidate that answered so a follow-up turn comes back to it.
A chat run's three extra services are all the live half of resolution: the candidate, and the payload affinity materializes for it. Neither can enter the record — move() would freeze the provider's own models cache, and materializing is per candidate — so the resolver keeps them and the stage that dials asks back by selector, which is the shape the shared services already use. The seam splits rather than being wrapped. Building the chat context over an already-built one would have minted a second dump accumulator, and the body can only be taken once: takeRequestBody empties what it is given, so the second call would hand the dump an empty buffer. What is shared is now the options and the services, and each family builds the one context its run has.
Gemini contributes one pipeline rather than two, because nothing translates into it, and it has no wire of its own: the ending translates the turn out to Chat Completions, dials, and translates the answer back, metering on the dialect the upstream actually spoke. Only that first wire is built — the Messages and Responses ones, `:countTokens`, this family's interceptors and the affinity egress are named in the header as absent rather than left implied. Running the chain settled three things it had wrong. The stream handed to the client is moved into the record like every other fact; the turn that is translated is the one affinity materialized for the candidate, and the candidate that answered is marked for the next turn; and the metered reading is converted to billing metrics rather than cast to them, since a cast leaves a TokenUsage where UsageQuantities is declared and the usage row it writes then carries no metric at all.
A translation is two declarations: it consumes the source protocol's request key and provides the target's, and coming back it consumes the target's response key and provides the source's. Saying it that way retires the runtime test. The source key is gone below the handoff, so a stage needing it cannot be placed there and compose says so at assembly — which is ctx.targetApi === <self> deleted and made structural. And nothing below knows a translation happened: the target chain sees its own protocol's keys and only those, so one chain serves a native request and a translated one. A refusal is handed to the pair as the upstream's own bytes, with the headers that actually came back rather than a synthesized set — that is what lets a context-window error become the shape Claude Code reads for auto-compaction. An answer that arrived as a value has no pair to map it, so the handoff fails rather than handing the client another protocol's object under its own key.
…oken categories What an upstream reports is per token category and what is billed is per metric name; the two are different shapes and a cast between them typechecks. The usage row was therefore written with no metrics at all — the recorder looked for input_tokens and found input. Converting through the same function every other caller uses fixes it, and the chain now asserts the quantities rather than only that a row was written.
The chain rendered failures through the shared envelope, which is the OpenAI families' shape: a Gemini SDK reads error.status — the Google-RPC name — and would have found it undefined, and error.code carries the HTTP status the generic shape has no field for. The mapping the replaced surface used already existed next door; it is now a renderer the edge calls, so the two entry points that build this envelope agree on one definition of it.
/v1/messages served through the chain Chat Completions established: an edge that writes Anthropic's own named SSE events when the client asked to stream and reassembles them into one message when it did not, settlement above the fork, affinity-ordered candidates, and a native ending that dials callMessages. Three things the replaced serve/attempt/respond surface said are said here rather than left implied. The turn that goes out is the one affinity materialized for the candidate, and the candidate that answered is marked for the next turn. The beta flags keep their typed transport path — read off the inbound headers and dropped from what the provider is handed, so no header allowlist can admit them and no other source protocol can leak them in. And a dial that never connected is a failure value the fork moves past rather than a throw that ends the run. Only the native wire is built. The translated ones, count_tokens and this family's own interceptors are named in the header as absent rather than left implied. What the header cannot say is that a pre-upstream refusal now renders in the shared error envelope rather than Anthropic's, since renderErrorEnvelope is what the edge writes; an upstream that refused in its own words is still handed on in them.
Builds `/v1/responses`' own wire as a pipeline, on the chain Chat Completions established: the edge that writes SSE frames or the one response object, settlement above the fork, candidate narrowing ordered by affinity, failover, and an ending that dials `callResponses`. Two things this family states that its siblings do not. A provider answers with the branch it actually ran, and one of them is not a stream — a compaction is a single envelope that carries its own counts — so it rides at the response key's value arm rather than at a key of its own or lowered into synthesized frames. And the client's stream is terminated by `[DONE]` here rather than by whatever the upstream's stream ended on, which is what the transport reads to know the turn is over; the ending stops reading at the turn's terminal event, and a stream that never states one ends the run with the sentence the replaced surface used. Scope is one wire and one transport. The WebSocket entry, `/v1/responses/compact`, the two translated wires, this family's interceptors and the stored-items membrane are each a step of their own and are named in the file header rather than implied by absence. Tests run the chain: frames written out, frames folded into one response, an envelope served as itself, a refusal answered in the upstream's own status and words, and a dial nobody answered failed over to the next candidate.
… the upstream said Three families rendered a gateway-synthesized refusal through the OpenAI envelope. A Gemini client reads error.status — the Google-RPC name — and a Messages client reads a top-level type and a request_id; neither shape carries those, so both would have found the fields missing. The rule the envelope already encoded stays: an upstream that refused in its own words is handed on in them, because those words are already what its client reads. Only a refusal the gateway itself produced has no body to forward, and that is the one case where the *client's* protocol decides the shape. That distinction is now named once and each protocol supplies its own envelope. A turn whose carried state needs two upstreams at once answers 400, as every family did before the shared stage invented a 409 for it. The three chat chains assemble alongside the six that came before them.
The seam wrote an SSE comment as keepalive, which is invisible to any client by design — but Anthropic defines a ping event and its clients read one, so the two are not the same wire byte. A family that has its own idle frame now names it alongside the frames it is answering with, and Messages exports the one it writes so the route that serves that chain cannot forget it.
The interceptors rewrite the request as a fact and the ending read the payload back from the resolver, so all three stages mutated a copy nobody sent. Deleting every one of them left the suite green. Affinity's per-candidate payload now enters the record below the fork, where there is a payload to speak of — each candidate has its own, and re-running the suffix is what produces the next. Everything between that stage and the dial rewrites a fact, and the ending sends the fact. Four comments claimed the fact space deletes the ctx.targetApi guards. The record retracts that: the claim rested on splitting the interceptor array by role, which was rejected — 「这里没有 source 和 target 之分！这是同一个数组！」 — and with one array the guards remain. What changed is that what they test is declared rather than ambient, which is visibility, not deletion.
…teway made A gateway-synthesized refusal reached the client as type: 'api_error' — which says the gateway broke rather than that the request did — and Responses lost the param and code that name which field was at fault and why routing failed. A protocol's envelope carries more than a status and a sentence, and only whoever refused knows those, so the refusal now carries what it would write and the edge renders it. The upstream's own body still wins wherever there is one. Which refusal it was travels with it: a turn that cannot be routed, a model no wire reaches, and a model no upstream has are three different things to say.
Time to first token was written in one place no stage reaches, so every chat run took the neutral branch and recorded no TTFT at all. It is measured where the token is — the only place that can tell a generated frame from the envelope around it. A rate can depend on the service tier and on how much input there was. Both are selector coordinates rather than quantities, and dropping them left the pricer with an empty selector, so neither axis could ever apply. The reading now goes through the same measurement the other families use and carries both.
…ng reads The shared stages' header described a dispatch stage that picks a wire. No such stage was built — every chain dials its own protocol — so what it now describes is the payload entering the record below the fork, and what will decide a wire once the translated ones land. The selector is one statement about a candidate and was written twice; the chat resolver uses the definition the shared one already had. ChatNarrowing.source was declared and never read. A handoff declared it needs the upstream's headers and then supplied [] when they were absent, which would have hidden the assembly error that absence is.
The ending read its payload back from the resolver while everything between the fork and the dial rewrites the request as a fact, so a rewrite would have gone nowhere. Affinity's per-candidate payload now enters the record below the fork, where there is one payload to speak of, and the ending sends the fact. Time to first token was written in one place no stage on this chain reaches, so every Messages run took the neutral branch and recorded no TTFT at all. It is measured where the token is — the only place that can tell a generated frame from the envelope around it. A rate can depend on the service tier and on how much input there was. Both are selector coordinates rather than quantities, and dropping them left the pricer with an empty selector, so neither axis could ever apply. The reading now goes through the same measurement the other families use and carries both. A stream that ran out before message_stop was served as a whole answer, and frames an upstream wrote after it went out to the client. The meter now stops at the terminal event and fails a stream that never reached one, which is what the protocol's own reassembly already did for the folded shape alone. ChatNarrowing.source goes with the key it was declared under.
…e its first token The ending read its payload back from the resolver while everything between the fork and the dial rewrites the request as a fact, so a rewrite would have gone nowhere. Affinity's per-candidate payload now enters the record below the fork, where there is one payload to speak of, and the ending sends the fact. Time to first token was written in one place no stage on this chain reaches, so every Responses run took the neutral branch and recorded no TTFT at all. It is measured where the token is — the only place that can tell a generated frame from the envelope around it. The two readings this chain already had right — pricing facts beside the quantities, and a stream that fails rather than answering short — were carried by no test, which is how the other chains came to disagree with them. They have one each now. ChatNarrowing.source goes with the key it was declared under.
…tates The ending read its turn back from the resolver while everything between the fork and the dial rewrites the request as a fact, so a rewrite would have gone nowhere. Affinity's per-candidate turn now enters the record below the fork, where there is one turn to speak of, and the ending translates the fact. Time to first token was written in one place no stage on this chain reaches, so every Gemini run took the neutral branch and recorded no TTFT at all. It is measured where the token is, on the dialect the upstream spoke — the same one the usage beside it is read on. A rate can depend on the service tier and on how much input there was. Both are selector coordinates rather than quantities, and dropping them left the pricer with an empty selector, so neither axis could ever apply. The reading now goes through the same measurement the other families use and carries both. A wire that closed cleanly had not thereby finished the turn: with no finish reason on any choice, nothing that came out of the translation said the answer was over, and a client streaming was served those frames as a whole answer. The folded shape already refused them, through the protocol's own reassembly; both shapes now do, because the ending stops at the turn's terminal frame and fails a stream that ran out before one. ChatNarrowing.source goes with the key it was declared under.
The meter read to exhaustion, so an upstream that dropped mid-turn was served as a whole answer and anything it sent after its terminator was forwarded. The streaming path is where this has to be caught: nothing folds those frames, so the collector's own check never sees them. The terminator is written out before the read stops, because it is what a client reads as the end. The first test written for this passed with the guard deleted — it drove the non-streaming path, where the collector throws the same sentence for its own reasons. It now drains the stream, and both halves of the guard are covered.
# Conflicts: # packages/gateway/src/data-plane/pipeline/serve.ts
The chain existed and was tested against stub providers, but nothing reached it: the route still built a chat context and ran the onion, so every property the chain states was stated only under test. The handler is now the prologue and epilogue the other migrated families use — read the ingress, open the chat prologue against a scratchpad store, hand the payload over, and write what the run answered with. Two decisions stay at the entry because only it can make them, and both are read before any stage can rewrite what they are read from: whether the client asked to stream, which the run has to be opened knowing, and whether it asked to be shown the usage chunk metering asks the upstream for either way. A body that is not JSON never enters a pipeline at all — there is no model to resolve and no attempt to make — so it is answered in this protocol's own 400 envelope. Wiring is what states what the chain does not carry yet, and two of them are written down in failing tests rather than papered over. The affinity egress never runs, so a turn hands back no `reasoning_opaque` and the next one cannot be pinned to the upstream that served this one; `http_test.ts` says so twice and is left saying it. Six of the nine interceptors are not stages — the usage-chunk request, both usage normalizers and the three vendor normalizers — a refusal that reached no upstream leaves the streamed-usage key unwritten, and the meter reports what was billed without reporting whether the stream reached its terminator. Each of those is the chain's own to close.
The chain existed and was tested against stub providers, but nothing reached it. The generate entry is now the prologue and epilogue the other migrated families use; `/v1/messages/count_tokens` stays where it is, because it is a second operation over this protocol rather than another wire under its chain. A stream is handed to the seam with this protocol's own idle frame. Anthropic defines a `ping` event and its clients read one, so an idle connection is held open with that rather than with an SSE comment no client sees — the chain names the frame and the route passes it on. A body that is not JSON is answered in Anthropic's own envelope, and a body carrying `anthropic_beta` or `betas` is refused as before, both now recorded as the gateway refusals they are. Wiring states what the chain does not carry yet. None of this family's five interceptors is a stage, and the first of them is the one `http_test.ts` catches: Claude Code's one-token model probe is answered by dialling an upstream instead of by the gateway. The web-search shim, the billing-attribution scrub, the reasoning and role rewrites, and the affinity egress are absent with it; so is the mid-stream `error` event `respond.ts` wrote into a client's own stream. A refusal that reached no upstream leaves the streamed-usage key unwritten, and the meter reports what was billed without reporting whether the stream reached its terminator.
The chain existed and was tested against stub providers, but nothing reached it. Both generate actions are now the prologue and epilogue the other migrated families use; `:countTokens` stays where it is, because it is a second operation over this protocol rather than another wire under its chain. Gemini carries the model in the path rather than the body, so the id the run resolves against is the one the route already split off the action segment, and whether the turn streams is the action itself rather than a field. A body that is not JSON is answered in the Google-RPC envelope this protocol's clients read. Wiring states what the chain does not carry yet, and `http_test.ts` catches the first: the chain has one wire, so a candidate reachable only over Messages or Responses is dialled on Chat Completions anyway and the turn fails. None of the four Gemini interceptors is a stage, the thought-signature rewrite and the affinity egress on the way out are absent with them, and a stream that ends without a terminal event ends the run as a throw rather than as the Google-RPC envelope `respond.ts` wrote into the client's own stream. A refusal that reached no upstream leaves the streamed-usage key unwritten, and the meter reports what was billed without reporting whether the stream reached its terminator.
…sal settle at all Two defects the non-chat families had already been through, both found by wiring the routes. Every chat chain handed its usage up as a bare list, so each handler adapted it with failed: false — a stream that stopped before its terminator was recorded as a turn that produced what it said it would. The meters now report both together, which is what the seam has taken since the non-chat families moved, and the four adapters delete. A refusal never provided the streamed-usage key at all, so settlement's still-reading test read undefined and deferred a turn that had no stream to wait for: the run wrote no performance row. Refusing now says there is nothing still to read, which is the same statement an ending makes when it did not stream.
…ages Chat Completions had three of its interceptors as stages; the other three families had none, so their chains ran nothing between the fork and the ending. Gemini's thought suppression was among the missing, which is a client that never opted in being shown thought parts. Thirteen stages, one per (interceptor, protocol) pair because the payload types differ, and one rule each: the role fold, the flag reading and the key removal are written once, and a protocol contributes only the walk over its own items. Two of them speak about the response — Gemini's thought suppression and the Responses cache-token fold — and say so as a response-direction declaration, carrying what they read on the way down in the stage's own closure, because a response-side `needs` can only name what the ending provides. What the interceptor form did by mutation is a rewrite now: the record is frozen, so Gemini's three strippers cannot delete a field in place. Every rule writes conditionally, so a turn it does not touch comes back by identity and the layer costs what it actually changed. A `ctx.targetApi` guard is gone wherever array position now says the same thing. Four interceptors stay in the interceptor form. The Claude Code probe answers with the Messages family's own response facts rather than rewriting a payload, and the Responses compact shim, the Responses server-tool shim and the Messages web-search shim each drive a turn of their own. Composing the stages into each family's chain is the next step and is not here.
… chain runs it The per-rule tests say what one stage does; nothing said that the array a family's `pipeline.ts` will hold assembles at all. `compose` refuses declarations that do not line up, so running the whole array between stages that declare what the real neighbours declare — an edge that needs the answer, the headers and the billed set on the way up, and an ending that answers with all three — is that check. It also pins the one ordering that lives between stages rather than inside one: the reasoning sentinel is the gateway's canonical form, and a vendor normalizer can only put it on the wire in the vendor's shape because it runs after the stage that wrote it.
Opening a chat run built the context from one options object and then opened the prologue without the recording those options had started, so every pipelined chat turn over HTTP wrote a record holding its metadata and no events — a key with retention configured paid for a recording of nothing. The WebSocket entry passed its own sink and was unaffected, which is why the gap survived: the two entries share this function and only one of them used what it returns.
Compaction is a second operation over this protocol, not another wire under the generate pipeline. Everything above the ending it shares with generation, stage for stage — the stored-items membrane, the narrowing, the four request rules — because a compaction is routed, pinned and rewritten exactly as a turn is. The ending is where they part, and it is where the action pivot went. Two wires. An upstream whose own endpoint compacts is dialled with `compact` and answers one envelope, expanded into the events the stateful half reads so the turn's items are stored under an id this gateway minted. An upstream with no compaction wire — every Messages and Chat Completions candidate, and any Responses candidate an operator opted in with `responses-compact-shim` — gets a stage that rewrites the turn into the compactor's own, hands into the ordinary generate fork below it, and packs the summary into an envelope of this gateway's own on the way back. Nothing needs an action to travel in the record: one wire dials `compact`, the other dials `generate` under a stage that folds the answer. The rewrite itself stays in one place. `summarizationTurnFor` is lifted out of the interceptor and shared, so the simulated compaction is byte-identical whichever entry asked for it, and the interceptor form keeps serving the WebSocket generate path that still reaches it. Two deliberate differences from the replaced entry. A summarization that closed no assistant text is now a candidate the fork moves past rather than a throw that ends the request — the next upstream may well summarize. And the verdict a compaction carries in its own `status` is folded into the accounting at the edge, which is where the completed resource comes into existence and therefore the one reader that can state it. Stated gap: the server-tool shim is still only an interceptor, so a compaction whose caller declared a server tool is summarized without the ReAct loop around it — the same gap the generate chain already states.
Every chat entry — four families' generate, both count-token operations, the compaction, and the WebSocket transport — serves through a chain now, so the serve/attempt/respond layers below them have no caller. Seventy-five files go, including all four attempt modules, the three interceptor registries, and every rule that became a stage. iterate-candidates.ts goes with them: the fork is a stage, and the loop it replaced was the last thing holding that shape. What stays is what a chain still reaches: the Claude Code probe's recognition and frames, the web-search request shaping the count-token chain runs, the compaction shim's summarization, SourceStreamState for the socket, and the server-tool shim — the one piece whose behaviour has not been ported, kept so that porting it does not start from nothing.
Both count-token chains composed no settlement, so the stage that writes what a run billed was simply absent from them. The row they write is empty either way today — nothing here is billable — which is why nothing caught it. The ruling is about the reason rather than the outcome: an upstream that began charging for the operation would provide a non-empty set and nothing else would change. A chain with no settlement stage has nowhere for that to go, and writing "count_tokens is exempt" hard-codes today's commercial arrangement into the architecture.
…t passed Four comments described the branch mid-migration: the shared stages still said nothing picks a wire yet, and the Responses hydration still named an entry layer that catches a throw the chain now answers with. Two test comments claimed more than assembly delivers. `compose` refuses a wire's own rule placed below a handoff that consumed its key, and it walks one array — the same rule left above the fork is invisible to it. What keeps a wire's rule off a translated turn is that the rule lives in the wire's chain, and an interceptor array states the order of what sits above the fork.
Six of the deleted interceptors had no successor, and one of them was holding up billing. Nothing forced `stream_options.include_usage` any more, so an OpenAI-compatible upstream sent no usage chunk, the meter's reading stayed undefined and the attempt was billed an entity with no quantities — every Chat Completions turn, and every Messages, Responses or Gemini turn that reached an upstream over this wire. All six belong to the wire rather than to the source chain: each speaks about one upstream's Chat Completions endpoint, so a turn that leaves for another protocol must not carry them and a turn that arrives from one must. They keep the order the onion ran them in, which is the same order in both directions — the usage chunk is asked for above everything, and coming back the vendor dialects have the first say, so the cache-bucket fold reads cache fields under OpenAI's names and the carrier split reads usage the fold has already settled. What the interceptor form said with `ctx.targetApi !== 'chat-completions'` is now said by position. The cache-bucket fold is one rule over a protocol's own field names, which the Responses stage of the same name reads too, and the two stream helpers every response-direction rule needed are written once beside it. Each rule fails a test of its own when reverted and is caught when dropped from the wire, and a streaming turn through a routed endpoint is checked to bill non-zero against an upstream that reports usage only when it is asked to.
The interceptor array was split by role: a rule a wire owns sat in its source chain, so a turn arriving over a translation ran one rule of its target's set instead of the whole thing. A DeepSeek upstream reached as responses via chat completions got neither the reasoning rewrite nor the cache-key strip, both of which that upstream answers 400 without. The old surface ran the whole target array on a translated body, and the human rejected the role split that would have justified anything else — 「这是同一个 数组」. A rule about a wire now lives in that wire's chain, which is also what the four dropped guards said: each keyed on the target, not on whether the turn had been translated. The test that proves it is the one nothing had: a turn arriving from another protocol, scrubbed by a rule its own source chain never ran.
…erved Two defects with one root between them: what a chat turn hands across a boundary was the record's own value. A record is deep-frozen, and a provider's interceptors shape their payload in place — Copilot writes `copilot_cache_control` onto individual messages. Spreading a frozen fact gives a fresh top level over frozen children, so every Copilot chat turn answered 502 with `Cannot add property copilot_cache_control, object is not extensible`, raised where nothing could explain it. Building an attempt's body is now one boundary rather than three copies of the same three lines, and the copy it makes goes all the way down. The dump was the other half. `stream.frame` had no producer on any chat chain, so a key with retention configured paid for records holding stage boundaries and no content, and the dashboard's collected view had nothing to replay. The tee sits at the family's edge, where the frames are still protocol frames and are already the source protocol's — what the client read. A non-streaming turn folds those same frames into one value, so recording there is what puts the stream in the record beside the body assembled from it. The frame test is what found the 502: nothing had run a Copilot chat turn through a pipeline and read the answer.
…eration Counting reaches the same provider and runs the same interceptors, so it carried the same defect: a Copilot `/v1/messages/count_tokens` turn 502'd where the initiator rule wrote into a nested field of a frozen payload. The chain's own claim is that what it measures is what generation would send. Building its body the way generation builds it is what makes that true of the dial as well as of the rules.
`Failure` carries three things a client could be shown and each edge decided for itself which to reach for, so four of six read the upstream's body and the protocol's default and stepped over the envelope in between. Nothing is currently lost — only the Chat Completions and Responses refusals write one — but the field is part of the shape every edge receives, and the next refusal to carry one on Messages or Gemini would have disappeared silently. The order is a property of the type rather than of any family: the refusing party's own words first, then the gateway's own account of a refusal it made, then the protocol's rendering of a status and a sentence. Stated once, beside the type that raises the question.
…n that serves it The compact shim ran on every Responses turn in the replaced surface, and only its compact half was carried onto the chain. So a `POST /v1/responses` whose input ended in a `compaction_trigger` was forwarded verbatim — to a translator that models no such item on a Messages or Chat Completions candidate — and a compaction this gateway had written reached an upstream as a blob it has no key for, losing the summarized conversation it stood for. Both halves are stages now, and both chains compose them: a compaction the shim wrote is issued through one entry and echoed back into the other, so neither half belongs to one of them. `expandShimCompactions` puts a blob of ours back into the items it encoded, and `summarizeForCompaction` — the stage the compaction chain already simulated with, now gated by what each chain reads as the ask — answers a `compaction_trigger` with an envelope of this gateway's own. Engagement is one reading, `simulatesCompaction`, taken by both endings and by both new stages: the operator's `responses-compact-shim` opt-in, or a candidate with no Responses endpoint at all. Where it does not hold, the trigger and the blob travel on to the upstream that answers them itself.
The onion's removal kept this file for "the compaction shim's summarization" — what a chain still reaches — but `withResponsesCompactShim` came along with it and has had no caller since. Now that both of its halves are stages, keeping it would leave the engagement rule and the envelope fold stated twice, free to drift from the one the chains run. What stays is the substance: the vendored compactor prompt and handoff prefix with their extraction evidence, the inbound expansion, the summarization turn, the summary reading and the envelope. The file's header says where the rules that use them run. Its tests follow the same split the rest of the migration took: the orchestration is checked where it happens, on the two chains, and what is written down here is the shape of the turn the compactor is sent and of the envelope it comes back in.
Fifty-nine files cover the chat families and two of them reach a route. That gap hid a total outage: every Copilot turn answered 502 because the record's frozen payload met a provider that writes into the body it is handed, and only a route test saw it. This states, against a Copilot upstream so the provider's own boundary chain runs, what the direct-call tests cannot: a streamed turn arrives as well-formed Chat Completions SSE ending in the sentinel, a collected turn arrives as one assembled body, an upstream's refusal arrives in the upstream's own words and status, a served turn writes exactly one usage row carrying the tokens the upstream reported, and a candidate reachable only over /messages still answers the client in the protocol it asked in. Each row was checked by breaking what it names: dropping the done frame from the SSE render, inverting the edge's stream/collect branch, synthesizing the refusal envelope over the upstream's body, billing empty quantities, and handing the target wire's frames up without the pair's mapping.
The same statement the Chat Completions route file makes, for the protocol Claude Code speaks, and against a Copilot upstream so the provider's Messages boundary chain runs on a body it is free to write into. Two things are this family's own and are said rather than assumed: the turn ends with `message_stop` and not a transport sentinel, and every frame the client reads carries the SSE event name of the event its data states. The answer's text is found by block kind, because the edge writes the turn's affinity state back as a leading redacted_thinking block. Each row was checked by breaking what it names: dropping message_stop from the client's render, inverting the edge's stream/collect branch, synthesizing the refusal envelope over the upstream's body, billing empty quantities, and handing the target wire's frames up without the pair's mapping.
A translated wire runs the client's body through a translator, and a translator refuses input it would otherwise coerce or silently drop. Nothing caught that refusal, so it escaped the run: a Gemini turn carrying a `functionResponse` part in model content answered 500 with an internal-error envelope and a stack trace, naming the gateway as the party at fault for a request that was simply not serveable over that wire. The replaced surface rendered it as a 400 in the caller's own protocol, which is what `translatorInputErrorResult` was for. The handoff is the one place every translation happens, so it answers there. Writing only a status and a sentence is enough — each family already renders a failure in its own protocol at its own edge, which is the only place that knows what that protocol's clients read. Answering rather than throwing also makes the verdict a candidate's rather than the run's: the same body may translate cleanly for the next candidate, whose wire is another protocol, and failover re-runs the suffix to find out. That is something the replaced surface had no way to express.
The same statement the other two route files make, for the one family with no wire of its own. Every row here is already a translated turn, so the last one adds the second pair — the same request answered over a Messages-only candidate — because "the client gets its own protocol back" has to hold for whichever wire the picker landed on. This family ends a turn differently from either OpenAI-shaped protocol: no sentinel, and the last frame is the one whose candidate states a finishReason. The refusal row also states what this family decided deliberately — the upstream's own object and status reach the client rather than being quoted back inside a Google-RPC envelope that cannot express either. Each row was checked by breaking what it names: dropping the terminal event from the client's render, inverting the edge's stream/collect branch, synthesizing the refusal envelope over the upstream's body, billing empty quantities on the wire the turn was dialled on, and narrowing the target picker so the Messages pair is unreachable.
`renderMessagesFailure` and `renderGeminiFailure` rendered a pre-stream `ChatServeFailure` into the old execute surface's `ExecuteResult`. Narrowing now states its own refusal and the edge renders it, and the failed-upstream list those two appended is appended by `resolveChatCandidates` — so both, and the result builders under them, describe an architecture that is gone. `translatorInputErrorResult` went the same way, one step behind: the handoff now answers a translator's refusal itself. Its test kept two properties worth keeping — the Anthropic envelope's shape and its key order, which a client comparing bytes against Anthropic-direct depends on — so the test moves to the function that writes that envelope today rather than being deleted with the one that used to.
The two dials this family makes were the last building their own body: a spread of the record's request, the candidate's model stamped on it, the addressed id dropped. That gives a provider a fresh top level over frozen children, and a provider shapes the body it is handed in place — Copilot marks individual messages for caching — so the first nested write throws and the client is answered 502 from the point in the stack least able to explain it. Both go through `bodyForAttempt`, as generation and measurement already do on the other families. What each dial still says for itself is what is specific to it: the compaction endpoint takes neither `store` nor `stream`. Copilot's Responses boundary happens to be written functionally today, so the route this adds says the turn reaches Copilot's own endpoint and comes back — what it rewrote on the way out included. What pins the ownership is a stage-level turn at each dial, dialling a provider that writes into a nested node the way the boundaries that already do.
The note explaining why the web-search shim is not registered on this boundary pointed at the gateway's `messagesInterceptors`, which no longer exists. What it was reaching for survives the rewrite: the shim is the gateway's work rather than a provider boundary's, and this provider opts into it by flag.
The refusal rows already here send a turn that asked for one object. The riskier half is the turn that asked for SSE: the seam has a stream to open and must not, because nothing was ever generated and an error written into a 200 stream reads to every one of these clients as a turn that succeeded and said nothing. Verified against the current upstreams — the client is answered with the upstream's own object under its own status on all three — and each row was checked by making its family's edge render a failure as frames whenever the client asked to stream.
Four things the migration made true and nothing had written down. The compact shim moved out of `interceptors/` — nothing in it is an interceptor any more, and the rules that use it are stages in both chains that compose it. The Qwen normalizer's interceptor form went entirely: the stage beside it is what the wire runs, and its flag gate is now asserted where the stage is tested rather than only where the interceptor was. `renderResponsesFailure` and `responsesInputErrorResult` followed the two that went with the other families, and `openAiErrorResult` under them. With them gone, `ChatServeFailure` had two variants nobody could produce — narrowing states a model that is missing or unsupported as its own refusal now, failed upstreams and all — so the type is the one shape that still travels as a throw, and the affinity alias that extracted it is the type itself. The two Responses edges render through `renderFailure` like every other.
Five constants encoded a vendor fact with nothing to check it against. Each now carries the primary source it was verified from. One of them was wrong. The billing-attribution stage said the block carries a per-call `cch=<hash>` and that Anthropic's endpoint reads it to bill against the user's plan tier. Reading the vendor's own shipped build (2.1.226): `cch` is the fixed literal `00000` and is emitted only on the first-party and Vertex paths, so on the third-party path this stage serves there is no `cch=` at all — what varies per call is the fingerprint suffix on `cc_version`, derived from the first user message. The line pattern was always what carried the load. And no source anywhere says the endpoint bills from it; what the build shows is a client-attribution signal, so the comment now says that much and no more. The rest were confirmed. DeepSeek documents `prompt_tokens` as exactly hit + miss. The stripped Gemini part fields are unforwardable for a reason Google states outright — a `fileUri` names a file the API will not hand back. The stripped tool capabilities turn out to be precisely the `Tool` schema minus `functionDeclarations`, which is worth saying because it tells the next reader how to re-derive the list rather than guess at it.
…the wire it lands on
Two halves of the same rule, both found by review after the first pass fixed
only the cases it happened to look at.
A refusal crossing a translation kept the object it came in. That object is the
*target* protocol's — an OpenAI `{error:{type,code}}` reaching a client reading
Anthropic, or reaching Gemini, which has no wire of its own and so gets a
foreign envelope on every refusal there is. A Gemini SDK reads `error.status`,
which an OpenAI envelope has not got, and `error.code` as a number where it
would find a string. The replaced surface forwarded an upstream body only when
it was already Google-RPC shaped and otherwise wrote its own envelope around the
upstream's words; that is what this restores, at the handoff, where the fact
that a translation happened is known. A pair that does rewrite still carries its
object across, because by then it is in the client's own protocol.
The other half is the Chat Completions wire. `disableReasoningOnForcedToolChoice`
and `stripPromptCacheKey` sat in the source chain, so a Messages, Responses or
Gemini turn dialled over that wire got neither — the same defect fixed for the
other two families, in the one file that fix did not touch. Both keep their
place ahead of every vendor dialect: the canonical sentinel is emitted before a
vendor spells it, and the field an upstream would reject is gone before a vendor
rewrites what is left.
Three tests asserted the behaviour being corrected here. They were written
against the pipeline's own interim state rather than against the surface it
replaces, so they are restated rather than deleted.
…ning four rules twice The compaction chain ran four Responses wire rules above its ending, and both wires below re-run them. Idempotent, so nothing was wrong on the wire — but the run record showed each stage twice, and it was the wire-placement fix applied to one file and not its neighbour. Three comments were saying less than the truth. The Responses and Messages headers described their unported shims as "still only in the interceptor form", which reads as "runs by the old mechanism"; there is no interceptor array left, so it runs not at all, and a turn declaring a hosted tool reaches the upstream unshimmed. The handoff's refusal helper claimed assembly catches a family that misnames its streamed-usage key — assembly cannot, since the handoff is a wire's first stage and nothing above it in that composition needs the key; the runner's provides check on the dial is what catches it. And the shared error envelope explained its own test backwards. None of these changes behaviour except the de-duplication. What they change is whether the next reader is told the truth about where the gaps are.
Three modules the pipeline left behind, each with no caller but its own test. `traverseTranslation` threaded a translate trip around an inner attempt and invoked the pair's `apiError` hook on a refusal — which is `handOff`, stated as declarations instead of as a wrapper. `providerStreamResultToExecuteResult` turned a provider's stream into an `ExecuteResult`; each wire's ending reads one into facts now. `respondGemini` was that family's whole respond layer. Only one function survived any of them: the Google-RPC envelope this family's route answers an unknown action with, which `renderGeminiError` already writes. So the route builds it there, and nothing is left that needs a respond layer. The three tests go with them. What they covered is covered where the behaviour now lives — the pair's refusal rewrite in the translated-wire rows, and the Google-RPC envelope in the Gemini route rows.
Two adjacent comments made overlapping claims about the same two lines. What matters is the one that has already been violated: building the options twice takes the body twice, and the second read is empty.
A run record is read for what the client got, and the dashboard's collected view is assembled by replaying exactly those frames. Three chat edges tee them into the record; the two Responses edges did not, so `/v1/responses` and `/v1/responses/compact` recorded stage boundaries and no content at all — the shape of a turn with none of it. The replaced surface recorded them from `respond.ts`. The tee goes at the edge, above the client-egress wrapper, because that is the last place the frames are this protocol's own and already carry everything every layer of the egress rewrites into them. One tee covers both shapes the generate edge hands out — the SSE body and the object the fold assembles — because both read the same iterable, which is also why a non-streaming turn records. Not the events framing. The WebSocket transport reads those frames and writes each one itself, and it already records what it wrote; teeing here as well would enter every frame twice and make the replay show the turn twice. That is the division `recordSentPayloadBytes` already draws for the bytes, so it is drawn the same way for the frames, and the transport's count is now fenced by a test. The compaction edge tees above the stateful half, where the frames carry the ids its resource is assembled under. They never reach a client there, so the record is the only place they survive at all.
…used on generation `:countTokens` reaches Messages through a pair of its own rather than a handoff, because what crosses the boundary is one measurement and not frames. The translator it runs is the same one, and it refuses input it would otherwise have to coerce or silently drop — but nothing caught that refusal here, so it escaped the run: a body carrying a `functionResponse` part in model content answered 500 with an internal-error envelope and a stack trace, naming the gateway as the party at fault for a request that was simply not measurable. Generation was fixed at the handoff; the replaced surface answered both actions alike, in Gemini's own Google-RPC envelope with a 400. So the pair answers it, exactly as the handoff does: a status and a sentence, no envelope, because the edge above renders a failure in this protocol's own words and is the only place that knows what a Gemini client reads. Answering rather than throwing also makes the verdict a candidate's rather than the run's. The three keys it answers with are what "no upstream was measured" looks like on a counting chain — the refusal, nothing billed, no upstream headers. There is no streamed reading among them, and that is the whole difference from the generating chains: a measurement never opens a stream, so this chain carries no such key for a refusal to have to settle.
The edge teed the frames into the record for the SSE transport and stood down for the WebSocket one, which teed its own — so the family's edge branched on how a transport frames, for a question that is not about framing. The stated reason was that only a transport knows which frames reached the socket, but the WebSocket tee recorded on the way *in* to its writer, not after it, so the two recorded the same frames and only the placement differed. One tee, at the edge, for both. A WebSocket turn is the same frames rendered differently, which is the rule the whole design rests on, and reading is what records — so a transport that stopped early still records exactly what it took. The test that made this safe to collapse is the one that counts: recorded frames against the messages the socket actually sent. Re-adding the second tee fails it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #475. Draft until its base lands.
Every chat entry serves through a pipeline: four families'
generate, both count-tokenoperations, the compaction, and the WebSocket transport. The onion below them is deleted.
The fact space
Four source protocols, ten translation directions between them, and one shared array of
stages that every chain runs. Two things the key layout says that the old context object
could not:
consumes the source and provides the target, so which protocol a stage is looking at is
a declared need rather than an ambient field.
ingress.*is what the client asked for. It survives the switch to whatever protocolthe upstream turned out to speak, which is why
wantsStreamis four disjoint keys.Gemini is source-only — nothing translates into it — so by the ruling it contributes one
pipeline rather than two.
Where a wire is chosen
A candidate that can serve a request may speak a different protocol than the client did, so
the wire is picked per candidate by the chain's last stage, which hands into the chain for
it. Failover re-runs the whole suffix including that stage, so the next candidate re-picks
with no mechanism of its own — which is what "only the last stage may name a target" buys.
All nine wires are built. A translated one is
handOff(pair)followed by the targetprotocol's own ending, so one chain serves a native request and a translated one and nothing
below the handoff knows which it is. A rule that speaks about a wire lives in that wire's
chain rather than in a source chain, which is how a translated turn gets shaped for the
protocol it is going to instead of the one it came from.
What only wiring exposed
The chains were tested against their declarations long before any route ran one, and every
defect below survived that. They are one lesson rather than a list of accidents: a stage
tested through its declaration is tested against what it says, and these are all things a
declaration cannot say. Four of them are about a seam — between a source chain and a wire,
between the record and a provider, between a run and its recording, between a translator and
the run around it.
interceptors shape their payload in place, down to nested nodes. A shallow spread hands them
a fresh top level over frozen children, so
Cannot add property copilot_cache_control, object is not extensiblecame back from the point in the stack least able to explain it.Every Chat Completions turn, every Messages turn carrying
cache_control— essentially allClaude Code traffic — and everything dialled over those wires. Fixed at one boundary,
bodyForAttempt, so a wire cannot express the dial any other way.the belief that every rule had been ported. Every Chat Completions turn billed zero, and so
did every other family's turn over that wire.
been left in the source chains.
stream.framehad no producer on any chatchain, so a key with retention configured paid for records holding stage boundaries and
nothing else, and the dashboard's collected view had nothing to replay.
the context from an options object and then opened without the recording those options had
started. The WebSocket entry passed its own sink, which is why the gap survived.
the caller's fault; the replaced surface said so in the caller's own protocol. The handoff
now answers it as a 400 there — and as a value, so failover can try a candidate whose wire
is a protocol that can carry the body, which the replaced surface had no way to express.
narrowing.refusenever provided thestreamed-usage key, so settlement's still-reading test read
undefinedand deferred a turnthat had no stream to wait for.
flag after being resumed past the terminal frame, and the client-output wrapper returns at
that event, so the resumption never came.
TokenUsagepass forUsageQuantities.suppress-thought-partsamong them,so thought parts reached clients that never asked for them.
down and none wrote it back, so a follow-up turn could not be pinned.
copy rather than the record.
count_tokensskipped settlement, against a ruling that had been explicitly ratified.And what only re-reading exposed
A three-way review against the design documents and the transcript found four more, every one of
them the same defect as an entry above, in a file that entry's fix did not touch:
over a translation ran neither — the fix the other two families got, missing from the third.
Collapsing that also removed a second tee in the WebSocket transport, which would otherwise have
recorded every frame twice.
of its own, so every Gemini refusal was reaching its client in a protocol that client cannot
read: a Google SDK reads
error.status, which an OpenAI envelope has not got, anderror.codeas a number where it would find a string.
/v1beta/…:countTokenslet a translator's refusal escape as a 500, where generation answersit as a 400 — the same fix, on the operation beside the one that got it.
The more useful lesson than the list above it: a fix applied to the family it was found in is half
a fix. Each of these was already understood, written down, and left undone somewhere else.
What closed the gap
Route-level coverage. Before it, of the 81 test files under the chat directory two reached the
real app at all — the Responses WebSocket transport's, and one that asks the target picker a
question through it; everything else called pipeline helpers directly, and no HTTP chat route was
driven end to end by anything. Chat Completions, Messages and Gemini now have six rows each — a streamed turn,
a collected one, an upstream refusal, a refusal on a request that asked to stream, a usage row
with real quantities, and a turn dialled over a translated wire — run against a Copilot upstream
so the provider's own interceptor chain executes, and each verified by mutating the mechanism it
covers and watching exactly that row go red. Responses has one such row rather than six, which is
the thinnest of the four and is said here rather than left to be found.
The freeze defect is the measure of what that is worth: reverting the copy to a shallow spread
turns every Chat Completions row red, the four that expect a 200 reporting
expected 502 to be 200.What is deleted
78 files: all four
attempt.ts, the serve and respond layers, the three interceptorregistries, and every rule that became a stage.
iterate-candidates.tsgoes with them — thefork is a stage, and that loop was the last thing holding the old shape. With the chains
serving, the replaced surface's refusal renderers, its translator-error builders and the
result builders under them went too, and
ChatServeFailureshrank to the one shape that stilltravels as a throw.
What stays is what a chain still reaches: the Claude Code probe's recognition and frames, the
web-search request shaping the count-token chain runs, the compaction shim's summarization,
and
SourceStreamStatefor the socket.Known gaps, stated rather than implied
The code is kept deliberately so that porting it does not start from nothing — deleting it
would turn an unreachable feature into a lost one.
ctx.targetApiguards are enforced by position, not structurally:composewalks one array, and a source chain and the wire chain it hands into are separate
compositions joined by
into.24-rulings.mdrecords what the check actually covers.stream.endhas no producer,await drain()has no coverage, anddump.failed()isnever called on a failed run.
than a 502 or a 500, on every chat entry.
is answered as this protocol's own envelope carrying that text as the message, where the
replaced surface forwarded the bytes with the upstream's own content type. The status is
still the upstream's. This follows the ruling the branch already made for a body a family
cannot parse: a gateway that cannot read the answer says so in words its client can read.
Deferred<T>— implemented in feat(pipeline): add the fact-record pipeline core #474 and used by the runner's teardown, but settlementstill reaches the background scheduler rather than providing a deferred fact.
Two decisions left open
Both are recorded in
24-rulings.md§13a rather than settled here.is broken there, and the reason is structural rather than lucky: interceptor chains exist
only for the three chat protocols. The general fix is to copy at the provider contract
boundary rather than at each family's wire, which reaches across feat(gateway): serve the non-chat families through the pipeline #475 and the provider
packages.
cache-bucket fold and the vendor field renames change what the client is shown but not what
is billed. Faithful to the surface this replaced and consistent across the families, but
nobody has ruled on it.