feat(gateway): serve the non-chat families through the pipeline - #475
Draft
Menci wants to merge 51 commits into
Draft
feat(gateway): serve the non-chat families through the pipeline#475Menci wants to merge 51 commits into
Menci wants to merge 51 commits into
Conversation
The first half of migrating the non-chat families: the space every family's pipeline extends, the services a run is given, and the two stages every family's serve pipeline needs. Neither shared stage names a family's own key. Branching is a capability of the framework and what something branches on is a concept in the domain, so a family hands in the two domain-shaped things — how to narrow a candidate, and how to read an attempt's outcome — and the stages stay written against the shared space. That is what lets them compose into a pipeline over any family's larger space with no variance question to lose, since assembly reasons over declarations and declarations are strings. `resolveCandidates` carries both traits, and it is what found the modelling error corrected in the commit below it: refusing a request no upstream can serve means answering with a key the descend path never carries. The families themselves follow. Nothing is routed through this yet, so the gateway's behaviour is unchanged.
The first family, and the one worth doing first: rerank already has a canonical contract — parse, render, serialize, and a usage reading — so it proves the shared stages on real business without needing a protocol written first. Four stages, and the shape every family will repeat. `emitRerank` is the edge: it renders the canonical answer back into whichever of the four protocols the client spoke, which is why the source protocol is an ingress fact and not a request one — it has to survive the switch to whatever the upstream turned out to speak. `callRerankUpstream` is the ending: it dials, parses the body, and provides both the answer and what the call is billable for. Two behaviours change, both because the architecture has no passthrough. The same-protocol path used to forward the upstream's bytes unread; it is now parsed and re-serialized like the translating path, so one code path serves both and the dump shows a parsed answer either way. And an upstream that refused is a value rather than a forwarded Response, so an earlier stage can fail over it. The client's headers reach the upstream from the record rather than from a live request object, which is what lets the dump show what was there to be filtered. The test that matters most is the one that records a hole. `callRerankUpstream` reads `ingress.http.headers`, and the derived entry contract does not mention it: a stage whose only trait is `return` declares no request side, by ruling, so assembly cannot see what an ending stage reads — and every family's ending stage reads something. The consequence is concrete: a caller omitting that key gets a runtime failure at the deepest stage instead of an assembly error, which is the thing the entry contract exists to prevent. The type layer still catches it at the definition site, so it is a gap and not a break. Nothing is routed here yet; the route table still runs the passthrough serve.
The simplest family, and the one that shows the shape with nothing else in the way: one protocol, no translation and no stream, so the array is exactly the four stages every family has. The narrowing is a constant rather than a function of the request, because an embeddings request carries nothing a candidate could be incompatible with. `packages/protocols/src/embeddings` was five lines of index signature — a passthrough shape, not a contract — so the real one is written here: the request's four input arms, `encoding_format`, `dimensions` and `user`, and the response's embeddings, model and usage, each cited against the OpenAI specification. `encoding_format` is why the canonical response holds numbers rather than whatever the upstream wrote. Both official OpenAI SDKs send `encoding_format: base64` when their caller did not choose one, so base64 is the common case on the wire; an upstream that ignores the field and answers with float arrays hands such a client a body it decodes as base64 and turns into noise. Parsing whatever arrived and writing what the client asked for is what closes that, which is why the encoding is an ingress fact and stays put. A base64 embedding is the vector's float32 elements packed little-endian, and reading it into numbers is exact: every float32 is a float64, so a vector survives any number of trips. Two behaviours change. A field outside the protocol is now named in a 400 rather than forwarded, because the request schema is `additionalProperties: false` and both gateways compared against — LiteLLM and copilot-api — draw the same line; this is the one decision worth a second look, since vLLM and Jina document supersets of the endpoint. And an upstream that reports no usage now bills an entity with no quantities, which is how "called and reported nothing" is said, rather than no row at all. The test records what assembly cannot see, and this family shows it twice over: a return-only stage declares no request side, so neither `ingress.http.headers` nor `request.embeddings.canonical` reaches the derived entry contract — and unlike rerank, whose edge needed the payload for rendering, nothing here puts the payload in the contract by accident. The route still runs the passthrough serve. The old handler parses and serializes through the new contract so the two agree until it is deleted.
Five families at once, each with a real protocol contract where it had none.
`completions`, `embeddings`, `images` and `audio` ran on `passthrough-serve.ts`,
which forwards a body it has not parsed — and the architecture has no such
concept, so three of them needed a contract written before they could have a
pipeline at all. `packages/protocols/src/{audio,images,completions,embeddings}/`
carries those now, cited to the OpenAI specification.
`alpha-search` is shaped differently and is not forced into the others' shape: it
has no upstream model and no candidate list, executing locally through the
configured search provider or relaying a selected provider's response. That is an
ending, not a missing family — the same distinction ruling 5 settled for HTTP,
where a provider that does not ultimately speak plain HTTP supplies its own.
Nothing is routed here yet: the route table still mounts the passthrough handlers,
so the gateway's behaviour is unchanged. The wiring, and deleting the passthrough
helpers, is one commit once every family is sound.
Two defects the families found in what they were built on are recorded in the
Pull Request and fixed in the commits that follow. Both were found by building on
the core rather than by reading it, which is what this layer is for.
`failover` declared that it consumes and provides `response.http.body` for every family. Three of the six never produce one — rerank, embeddings and images read their answer to the end, so there is nothing still open by the time the fork sees it — and the runner checks `provides` at handover. Those pipelines composed cleanly and would have thrown on the first real request, at the deepest stage, naming a key their author had never written down. It cannot be a fixed key in either direction: claiming one a family never produces throws, and staying silent about one it does produce and hands up throws the other way. Which keys carry a resource is a statement only the family can make, so it makes it — alongside the failure predicate it already hands in, for the same reason. Branching is the framework's; what something branches on, and what it owns, are the domain's. Found by the completions family, which verified it against a throwaway pipeline rather than reasoning about it. The three affected families' own tests asserted `entryNeeds` and nothing else, so none of them caught it — the same shape of untested claim the core's review found earlier, appearing again one layer up. The test added here asserts the declaration for both kinds of family rather than the absence of the bug, because the absence of the bug is what the other tests were already asserting when it was present.
Settlement is a stage now, and it sits above `failover` so a run bills once however many candidates it tried. Repetition passes through the stage that observes usage, not through this one. It is unconditional. A run that measured rather than generated still writes, and its row simply names no billed entity — emptiness is observed rather than declared, which is why the word "unknown" appears nowhere: the situations are concrete and the list is open. The write is scheduled rather than awaited, because a transient repository failure must not turn an already-flowing upstream response into a 502. `BillableEntity` gained the pricing inputs a rate can need beyond the quantities. Absent is a real reading and not a missing one — most families price on the quantities alone — and making it required broke five families that correctly have none. It also caught a regression I had introduced: the rerank migration dropped the `inputTokens` pricing fact that `settleRerank` passed, so a rerank rate depending on input size would have priced against nothing. Restored, and the reading now travels with the entity it prices rather than being recomputed at the write.
A `ModelCandidate` is two things wearing one type: the selection — which upstream, which model row, which flags — which is data, and the handles — the provider instance, the fetcher, the models cache — which are live. The six families put the whole thing in the record, so `move()` deep-froze all three handles and the SWR cache refresh the provider does on its own schedule broke. The architecture ruled this out before any code existed, and I did not apply it: "Where something is chosen per attempt, the resolver is the service and the selector is a fact. A per-upstream transport is not a fact and is not pinned at the prologue either; what is injected is the thing that resolves one, and what travels is the identifier it resolves from." So `route.candidate` becomes `route.attempt`, carrying the upstream id, the model id and a snapshot of the flags — snapshotted rather than referenced, because the record must show what was true when the attempt was made rather than what the row says now. `resolveAttempt` is a service and the ending stages ask it for the thing that dials. It also fixes the same rule failing the other way: a candidate in the record is walked by the dump encoder, so a run's dump was serializing the provider instance. Nothing caught it because every family test builds a candidate literal with no live handles, and no route mounts a pipeline, so the suite never runs one against a real provider. The test added here asserts the property in both directions — the selector reaches nothing live, and the candidate would have frozen it.
`writeSettlement` was written and composed into nothing, so no migrated family recorded a usage row or a performance sample. Every serve pipeline now carries it, above the fork, so a run bills once however many candidates it tried. Adding it proved its own absence: the completions tests immediately failed with "Repo not initialized", because until now nothing in six families ever reached a write. That error was the fix working. The entry contracts gained `ingress.http.headers`, which five of the six omitted. `compose` cannot derive it — a stage whose only trait is `return` declares no request side, and every family's ending is one — so the hand-written type is the only thing covering for that, and it was covering wrong. One test here is worth reading twice. I first asserted that a run reaching no upstream still records a performance sample; it does not, because `recordPerformance` returns early without an attempt's telemetry and there is no attempt. That matches the replaced surface exactly, so the expectation was invented and the stage was right. The test now says what happens and why.
The two streaming families marked their body with `Object.assign(body, {
[Symbol.asyncDispose]: … })`, which was how a resource was claimed before
ownership stopped being a detection. The runner reads `isOwned` now, so a body
marked that way was invisible: `failover` declared it consumes one, `drain()`
existed, and neither could see anything. A losing attempt's connection stayed
open and the winner's was never drained.
`own(body, release)` makes the claim, and the fact's type says `Owned` rather
than `AsyncDisposable` — the language puts `Symbol.asyncDispose` on every async
generator and on no `ReadableStream`, so a structural type admits an iterator
that is not a resource and rejects the body that is.
The test asserts the property rather than the absence of the bug: a losing 429's
body and a winning stream, the answer handed back before anything is drained, and
the winner drained when the caller says so. Verified by mutation — dropping the
brand from `own()` fails exactly that test and nothing else.
Both families rendered an answer and never said what status it was, so an upstream 429, a resolver's 404 and a 400 all reached the client as a 200 carrying an error envelope. That is not a difference the no-passthrough ruling asks for: a client is not owed the upstream's exact bytes, but it is owed the truth about what happened, and the replaced surface forwarded the status. The edge provides `response.http.status` on both arms — the upstream's own when it refused, the gateway's own when the resolver refused before dialling, and 200 for an answer. The three tests drive the pipeline rather than reading its declarations, which is what the two families were missing: their whole suites asserted `entryNeeds` and nothing else, so a family that could not express a status at all passed. Verified by mutation — pinning the failure arm back to 200 fails both refusal tests.
… unnamed embeddings model The pipeline's shared resolver phrased its own refusal, which changed what a client is told: five families spell the endpoint they could not serve, while rerank spells out which part of the request no candidate could satisfy. The sentence moves to the family as `Narrowing.unsupported`, taking the reasons `reject` gave so a family can use them or ignore them. Copilot's /embeddings answers without the top-level `model` the schema marks required, so parsing refused a body the replaced surface served. The parser now takes the model the request named and completes the record with it.
…'s edge The pipeline dropped the upstream's response headers, so vendor traces, quota state and retry-after stopped reaching clients on the routes it took over. The attempt hands them up unfiltered and the edge decides what a client may see, which keeps both in a dump. A refusal that never dialled has none to carry, so the shared resolver answers with an empty list and no family repeats it. serveThrough now reads the status and headers off the exit facts. A family's render owns the bytes and their media type and nothing else, which is also why content type stays blocked from forwarding. The assembly test builds every family, including the four whose pipeline is built from the request and would otherwise never be composed by a test.
A stage logger wrote to the dump record and to whatever sink the services carried, and the prologue carried none — so on every run without retention configured, a warning or an error reached nothing at all. A failed usage write was the case that showed it. The sink writes warnings and errors to the console the rest of the gateway uses. Debug and info describe one request's progress and would bury the request log at a line per stage, so the threshold is fixed here; a dump still records every level when one is open.
…ot faults Nothing in the domain throws. A refused connection, a timeout or a reset is an outcome the fork has to see so it can try the next candidate, so each ending catches what the platform raised and hands it up as its family's failure. A dial that reached no upstream bills nothing and carries no headers; only the performance row records that the attempt happened. Attribution moves ahead of the dial. It was written from the call's result, so an attempt that never returned left the previous candidate's context in place and misattributed the row. A 2xx body a JSON protocol cannot read is likewise a failure value rather than an unhandled parse error: the gateway that cannot read an answer has not served the request, and the upstream still counts as called having reported nothing. The body reader images already had is now shared, since three families need it.
Both handlers still ran the passthrough scaffold: they read the body, hand-rolled the multipart and JSON edit shapes into a provider request, picked a candidate by endpoint capability and relayed whatever the upstream sent. The images protocol already owns both readings and the family already has a pipeline, so what is left here is a prologue and an epilogue — parse, hand over, write the answer. Reading the body moves to the contract, which reports a malformed request by throwing; the handler turns that into the same 400 envelope the scaffold wrote, with the same sentences. The two endpoints differ only in which parser reads the bytes, so the rest is one shared function.
…r reads them Workspace packages sort after the relative imports, and the two handlers wired to their pipelines had them first. Nothing else changes.
…gs independent
An upstream that refused in its own words is handed on in them. The envelope was
written once for OpenAI-shaped families and applied everywhere, which turned a
rerank client's `{message}` into `{error:{message}}` — a shape its SDK does not
read. What distinguishes the two cases is whether an upstream answered at all,
not which shape it answered in, so that is what the renderer now tests.
Rerank reads usage before results. They are independent readings of one body and
one failing is no reason to discard the other, so an answer this gateway cannot
model still bills for what the upstream metered.
A same-protocol answer is rendered back out unchanged, so results it carries
that the canonical form does not model are no longer a refusal — only a
translation needs to read them. A translation that fails on an answer that
parsed is 502: the upstream answered and the gateway cannot say it in the
client's protocol.
Three tests described the passthrough surface that /v1/embeddings and /v2/rerank no longer run through, and two of them asserted behaviour the design replaces: a 2xx body a JSON protocol cannot read was forwarded verbatim with the upstream's own status. A gateway that never read the answer cannot claim to have served the request, so it refuses in its own words and the upstream still counts as called having reported nothing. The rest are kept as they were, against the pipeline: a failed usage write leaves the answer alone and is still reported, and the last candidate's refusal reaches the client with its status, its headers and its own words.
Whether a request streams is written in the request — `stream: true` in a JSON body, a form field in a multipart upload — and the body can only be read once. The prologue read it internally and took the answer as a parameter, so a family that learns it from the body had no order in which to call it. Guessing `false` is not harmless: the abort controller a streaming run cancels its upstream with is minted from that flag, so a client that disconnected would stop cancelling anything. Reading the ingress is now its own step, and the run opens once the handler knows what was asked for — which also lets the requested model reach the dump through the context that stamps it rather than a second call afterwards.
…t ends A streaming family's answer is the stream, and the seam could only build a response from bytes it already had. It now takes frames as an answer of its own shape: the response is staged on the context first, because hono's SSE helper constructs the response itself and headers passed to a constructor would be lost. A stream's usage arrives with its last chunk, which is after the run has answered — the fact carrying it is a promise, and settling from it is the epilogue's job. Settlement's write is extracted so both paths do the same thing: the stage settles what the ending had already read, and the epilogue settles what only the drained stream could say.
Completions and audio were the only families whose edge handed the upstream's headers on unread, so a body the gateway re-serialized carried the upstream's own content-length and content-encoding — a client would have been told the wrong length for bytes it was actually given. Both now filter as the other families do, and both exits declare the key the seam reads off them.
The release was scheduled the moment the run returned, so for a streaming family it consumed the very frames the client was waiting for — one connection has one reader, and the two raced for every chunk. The terminal [DONE] was the frame most often lost. Reading the frames out to the client is what releases the body they came from, so the drain now waits for that to finish, in a finally so a client that stopped reading still leaves nothing open. A buffered answer was serialized from facts the run already held and still releases at once. Completions serves through its pipeline; the streaming half settles from the usage its last chunk carries.
The reader ran until the upstream closed, so an upstream that holds the connection open past transcript.text.done held the client's stream open with it. Returning at the terminal event closes the read, which cancels the upstream — the behaviour the replaced surface had. The route stays on its existing surface. Migrating it needs a decision this change cannot make: srt and vtt are parsed into cues and rendered back, which the protocol states is not byte-identical to what the upstream sent, and the replaced surface forwarded those documents unchanged.
Two families' endings dialled outside the try the other three had, so a refused connection ended the run as a fault: no failover to the next candidate, no performance row, and a 500 where the replaced surface answered 502. Attribution moves ahead of the dial with it, so an attempt that never returned names its own candidate. A streaming answer returned hono's response without finalizing the gateway context, and finalize is what writes the dump — so a streamed request recorded nothing at all. The upstream's frames are recorded where they are read, before the edge decides which of them the client sees. The prologue's dump sink was a stub that discarded every event while still telling the runner to accumulate them, so a key with retention configured paid for a recording nobody could read. It is absent until it has somewhere to go.
A request carrying stream: true reaches the upstream, and the pipeline cannot read what comes back — the images stream is not carried as facts, so the answer became a 502 after the call had already been made and charged for. The replaced surface forwarded those frames. This is the same blocker audio has: a response shape the fact space does not model yet. The pipeline, its tests and the protocol contracts stay; what is withdrawn is the route, until the shape it cannot carry is carried.
…t two families Completions and audio wrapped an upstream's error body in a second envelope, so a client read an escaped JSON document where the message should have been. They now hand the upstream's own object on, as the other three families do. Three ingress keys were declared and never provided or read, under a comment claiming every request recorded them. The method, path and body are the dump's to record from the request context; what a family actually hands over is the headers, because every ending forwards what a provider may of them. Rerank's attempt module had no referrers left after the pipeline took the route. Two comments cited design documents that are not in the repository.
The family produced a billed set and composed no settlement, so nothing it did was ever written. What stopped it was a request-side need on serve.model that the stage never read: declaring it put a resolved model in the entry contract of a family that resolves none. Settlement reads what came back and nothing on the way down, which is what the declaration now says.
…arries Audio's pipeline had no behavioural coverage: its test file established the entry contract and one property of the runner's ownership, and nothing ran a stage. It now serves a transcription, an upstream refusal, and a stream whose upstream keeps talking after the terminal event — the last fails if the reader does not stop there. The dump's log-line test named itself after structured fields and asserted every part of the line except them; removing the fields from the encoder left it green.
Menci
changed the title
The deployment apps compile the gateway under their own lib settings, where Headers takes [string, string][] and the inferred string[][] does not fit. Both platform typechecks failed while the package's own passed, so the mismatch only appeared under pnpm run verify.
A stage had no way to begin work without reaching past the record for a scheduler, and anything it started that way escaped the run's own accounting. Deferral is now a property of the value: a helper marks a promise, the marked value enters the record like any other, and teardown waits for what this run began. Declared rather than sniffed. Testing for a then method would make a fact that happens to hold a promise indistinguishable from one the run owes work to, and it would be quietly awaited instead of reported. Per run and not per root, because a discarded branch's facts are private to it and a root-level sweep could never see them. Teardown has a deadline and exceeding it is an error a reader can act on, since a value that never settles would otherwise hold teardown open forever.
…gain `srt` and `vtt` were parsed into cues and rendered back through a writer of our own, which the type itself admitted was not byte-identical to what the upstream sent: that writer terminates every cue with a blank line, numbers SubRip from 1 and pads every fraction to three digits, so a document following none of those conventions came back changed. The replaced surface forwarded them untouched. The canonical transcription now holds the bytes that arrived and the renderer hands them straight back — for the three document renderings, and for any body the reading could not open at all. What is read out of the bytes (the transcript, the cues, the upstream's own object) is what the record shows and what usage is measured from, so a reading that fails costs telemetry and never the answer. The subtitle writer has no caller left and is gone with it. Usage becomes a reading of its own. A report under a discriminator this protocol does not name is still no report, but one it does name and cannot read now throws, so a caller can say an upstream metered something unreadable rather than record it as though it had metered nothing.
Both images endpoints accept `stream: true` and answer with SSE, and this protocol could read only the single JSON body — so a gateway carrying it had nothing to parse a streamed answer into. The stream is now the protocol's own: `parseImagesStream` reads the upstream's body into events, `imagesStreamEventToSSEFrame` writes them back, and `isImagesTerminalEvent` says when the image is done. The terminal is the completed event and there is no `[DONE]` — the two endpoints name their terminal apart (`image_generation.completed`, `image_edit.completed`), so the suffix is what the reading matches, which also serves an endpoint added later. An event's name is written twice on the wire, as the SSE label and as `type` in the payload, and the two are reconciled at the boundary so every reader takes it off the payload. `imagesRequestWantsStream` reads the flag the client sent. It admits both the boolean and the string, because a multipart edit carries every field as the text the client typed.
…nts through the pipeline A request carrying `stream: true` reached the upstream and was charged for, and the ending stage then had nothing to read it with: the images fact space held a parsed JSON body and nothing else, so the answer became a synthesized 502 after the image had already been paid for. That is why the route was withdrawn to its existing surface, which forwarded those frames. The stream is now carried the way completions and audio carry theirs. `response.images.canonical` holds a stream, a value or a failure at one key — telling them apart is reading a value — and `response.images.streamedUsage` holds what the upstream will have reported by the time the events run out. The ending reads the SSE when the client asked to stream and the upstream answered with one (an upstream that ignores the flag still answers the single body the other arm reads), meters usage off the events as they pass, and stops at the completed event, failing the answer if the body ends without one. The upstream's body is now owned and the fork declares it, so a losing attempt is released. The edge renders SSE frames or the JSON object, and `writeSettlement` is told when a run is still reading, so a streaming turn settles once — from the promise, in the epilogue — rather than twice. Both endpoints route through `serveThrough` again. What the endpoint decides is which contract reads the body; after that both hand the same canonical request to the same run.
The route was the last one left on the replaced surface, held there by one question: that surface forwarded a transcription's document untouched, and the pipeline parsed it into cues and wrote it back. Carrying the document settles it, and three more of the replaced surface's statements had to follow it across. A 2xx body is the answer whether or not this endpoint could read it. Nothing is serialized from the parse any more, so a body no reading could open is still what goes back — with a warning and a request-only usage row, never a 502 and never a failover to a candidate that would answer no differently. The two readings are independent for the same reason: an upstream whose usage block this gateway cannot model still had its transcript carried, and one whose document it could not open is still recorded as having metered nothing. A stream states its own outcome in its last event. An upstream that stops before `transcript.text.done` answered 200 and then did not finish, which is a failed request however much of the transcript reached the client, so the promise the ending stage hands up now carries that alongside what was billed. The handler writes its own epilogue rather than going through `serveThrough`, on two counts that belong to the seam: `Rendered.contentType` has no value for an upstream that declared no media type, and `DeferredUsage` cannot carry a stream's outcome. Fold both into the seam and this handler is `serveThrough` again.
The ending returns at the completed event so an upstream that holds the connection open does not hold the client's stream open with it, and nothing observed that: every other case in the suite reads an upstream that closes its own body, where returning and running to EOF look the same.
…r both
A dump record had exactly one shape: the turn's two edges, `{request, response}`.
A pipelined endpoint has no such pair to record — what happened is the run, every
stage and both directions, which `@floway-dev/pipeline` already encodes as an
NDJSON event stream. There was nowhere for that stream to go.
A stored record is now either shape and `shape` is what a reader dispatches on.
The storage below is untouched: a run's stream is one more gzipped body file
under `dumps/v1/`, carried by the response descriptor with `type: 'run'`, staged
in the spilled-files registry before it is written and pointed at by a row
written last — so retention, the sweep and the files-before-row ordering apply to
it without knowing it exists. The row's request-headers column, which is NOT
NULL and means nothing for a run, holds the empty list; nothing reads it back
because `get` dispatches on the descriptor first.
`DumpMetadata` stays common — the dashboard lists both kinds together — so the
attribution hooks and the assembly that fills it move out of the edge
accumulator into `DumpAttribution`, which both shapes will stamp.
A stream that stops short is recorded as a turn that produced what it said it would: the deferred settlement passes failed=false unconditionally, so a neutral performance row and a dump success are written either way. It affects every streaming family. Three shapes were tried. The write has to be registered while the request is still live for the platform to bind it, and the outcome is only known when the stream ends; each attempt to satisfy both lost the usage row entirely. The fix that would work is for a family's meter to report how its stream ended, since that is the one place that knows — a change across all four of them. Left as it was, with the constraint written where the next reader meets it.
The pipeline emitted a run's events and had no sink to emit them to: `run()` recorded into `RunResult.events` and nothing carried them to storage, so a pipelined turn's dump was the edges of its HTTP request and nothing about the run that produced them. `openRunDump` opens one recording per turn. `sink` is what the prologue hands to `run` as `services.dump`; every event it receives is folded into the encoded stream as it happens rather than re-walked at the end, so a stage's exit that hands up what its child handed up costs nothing and a payload the run never rewrote is one object id. `finalize` turns the stream into NDJSON and writes one record through `DumpStore.put` — the same route the edge record takes. Recording is conditional structurally: a key with no retention gets no sink, so `services.dump` is absent and the runner does none of the recording. There is no mode flag and nothing that accumulates into a value it discards.
The seam now records why a truncated stream settles as a success, and names the fix as the family's meter reporting how its stream ended. This family already does that, so the epilogue says so where the next reader meets it, along with what registering the write while the request is still live costs it.
The request detail panel read every record as a pair of edges — request headers, request body, response headers, response body. A pipelined turn is recorded as its whole run instead, so the panel would have read `record.request` off a record that has none. It now dispatches on the record's shape. An edge record draws the four sections it always did; a run record draws its event stream, one block per NDJSON line, labelled with the event's own kind and with whichever stage, object or stream it names. A line that does not parse is shown as it was stored with the failure named, because a record that renders as nothing would read as an empty run. The severity badge reads `meta.status` rather than the response half's copy of it — the metadata is what both shapes have, and the two were always the same number.
…t ended A streaming turn settled as a success whatever happened to it, so a stream that stopped before its terminator wrote a neutral performance row and a dump success — a turn that never finished, reported as one that produced what it said it would. What blocked this was that the write has to be registered while the request is still live for the platform to bind it, and the outcome is only known when the stream ends. The two stop pulling against each other once the family's meter reports both together: it knows the usage and the terminator at the same moment, so it resolves one value carrying each. The audio family arrived at this shape independently and had to write its own epilogue around the seam to use it. Widening the seam is what lets that handler collapse back into it.
The reader dispatch was covered per layer and not end to end, so nothing said that a run record survives the list query, the detail fetch and the wire codec together — which is the path the dashboard actually takes. One list carries a run and a pair of edges, and a detail fetch of each hands back what it is: the run's NDJSON, and the edge's request half. The run fixture encodes the events the way the sink does, object space and folding included, rather than spelling the stored lines out by hand.
Both record shapes fell back to a transport-level read failure when no hook had stamped an error, and each spelled the precedence out for itself — a request-body read failure outranking a response-body one, twice. It belongs beside the assembly it feeds, so it moves there and both call it.
The run's event stream had a writer and no producer: nothing opened one, and a pipelined endpoint still wrote the edge record its handler no longer builds. The shape follows the endpoint. A pipelined turn opens the run recording in its prologue, so one turn is written once and the two shapes never both appear for it. What a turn stamps on either — the requested model, the upstream that answered, what it billed, how it failed — is now a named surface both implement, which is what keeps the metadata common and the stages ignorant of which they are stamping. A frame becomes the stream.frame event the format already specified, which is that event's first producer.
Every non-chat family serves through its pipeline now, so the scaffold whose whole job was to forward a body it had not parsed and hand back one it had not read has no callers. passthrough-serve.ts, passthrough-attempt.ts and their tests go, and PassthroughServeApiName with them — the set of endpoints it named is not a set anything groups by any more. What survives is the one piece that was never about forwarding: reading a JSON body with a model field, which is a request parser and now says so in its name. Audio's respond.ts and usage.ts went the same way — the strategy responder and the usage reader the pipeline replaced, both without a referrer since the family was routed. Audio's handler collapses back into the seam. It wrote its own epilogue because the seam could not carry an upstream's absent media type or a stream's outcome; both now travel, so the two agree again.
# Conflicts: # packages/pipeline/__tests__/boundary_test.ts
Takes the host-independence fix for the ownership tests, which this branch's own suite runs.
The audio test required `Symbol.asyncDispose` to be present on an async generator. That is true on Node 24 and false on Node 22, which CI runs and `apps/platform-node` supports, so the suite passed locally and failed there. What it means to assert is the run's answer, which is the same on every host. Four comments explained the stream wrapper by a mechanism that is gone. They said a bare generator in the record would be adopted as a resource and `return()`ed by the sweep — true when the runner detected ownership structurally, and not since it started claiming it through `own()`. The wrapper still earns its place, for the reason the audio test already stated: it says which value in the record is the resource. That is the upstream's body, and nothing else.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #474. Draft until its base lands.
The shared space every non-chat family's pipeline extends, the stages they all run, the seam
their routes serve through — and three families moved onto it.
What is routed
/v1/embeddings/v2/rerank,/jina/v1/rerank, and the other two mounts/v1/completions/v1/images/{generations,edits}/v1/audio/transcriptionssrtandvttAlpha search keeps its own route: its pipeline composes and settles, but nothing dispatches
through it yet. The passthrough surface itself is gone.
The shared parts
Neither shared stage names a family's own key. Branching is a capability of the
framework and what something branches on is a concept in the domain, so a family hands in
the two domain-shaped things — how to narrow a candidate, and how to read an attempt's
outcome — and the stages stay written against the shared space alone. That is what lets them
drop into a pipeline over any family's larger space with no variance question to lose:
assembly reasons over declarations, and declarations are strings.
Reading the request is separate from opening the run. Whether a request streams is
written in the request, and the body can only be read once — so the run is opened after the
handler knows what was asked for. Guessing is not harmless: the abort controller a streaming
run cancels its upstream with is minted from that flag.
A failure is a value, and so is a dial that never connected. Each ending catches what the
platform raised and hands it up, because a refused connection is an outcome failover has to
see. Attribution is set before the dial, so an attempt that never returned names its own
candidate rather than the one before it.
Settlement sits above the fork, so a run bills once however many candidates it tried. A
stream states its usage in the chunk that ends it, which is after the run has answered — the
pipeline hands that up as a promise and the epilogue writes it, so the row is written once
and where the numbers actually are.
What routing the families found
Wiring is what exercised any of this. Every one of these was invisible while no route ran a
pipeline, and each is fixed here:
retry-afterstopped reaching clients.the very frames the client was waiting for. The terminal
[DONE]was what usually vanished.retention configured a warning or an error reached nothing at all.
finalize is what writes the dump, so a streamed request recorded nothing.
belongs to the family: five spell the endpoint, rerank spells out which part of the request
no candidate could satisfy.
read an escaped JSON document where the message should have been.
/embeddingsanswers without the top-levelmodelthe schema marks required, soparsing refused a body the replaced surface served.
back out unchanged; only a translation has to model it.
Two families that were held back, and what unblocked them
Both were the same shape of blocker — a response the fact space did not model — and both are
routed now.
Images. A
stream: truerequest reached the upstream and the pipeline could not read whatcame back, so the call was made, charged for, and answered 502. The stream is carried as facts
now: the events are metered as they pass, so the reading costs one pass and the client's own
stream is what drives it.
Audio.
srtandvttwere parsed into cues and rendered back, which the protocol statesis not byte-identical to what the upstream sent. The decision went the way the replaced surface
had it: a media type is upstream-owned, so the upstream's own document travels with the
upstream's own label, and the edge names a media type only where the gateway wrote the body
out of nothing the upstream sent.
The deferral question, answered
17-facts.mdsays stages have no fire-and-forget, and20-gateway.mddescribes settlementproviding a
Deferred<T>fact composed with.thenand never awaited. That subsystem wasunimplemented when this branch was written, which left the two documents describing something
no code did.
Deferred<T>is implemented in #474 and the runner's teardown settles what a run deferred.Settlement here still reaches the background scheduler rather than providing a deferred fact,
so the mechanism exists and this one caller has not moved onto it — a smaller gap than the one
originally raised, and one that no longer needs a ruling to close.