Skip to content

feat!: swap provider harness to pi - #3

Open
colinds wants to merge 40 commits into
mainfrom
colin/pi
Open

feat!: swap provider harness to pi#3
colinds wants to merge 40 commits into
mainfrom
colin/pi

Conversation

@colinds

@colinds colinds commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Replaces agentry's hand-written Anthropic and OpenAI adapters (src/providers/, 1,404 lines) with pi (@earendil-works/pi-ai), then reworks its execution mechanics while keeping the JSX API. Agentry keeps its reconciler, engine, handles, conditions and subagents; pi owns the wire.

Net −3,700 lines.

What you get

  • ~35 providers instead of 2, selected by string; OAuth subscription auth; usage.costUSD
  • MCP on every provider — agentry is now the MCP client, so it no longer needs a provider-native connector
  • Thinking signatures survive replay. The old Anthropic adapter dropped them, so multi-turn thinking was silently lossy
  • The model is told when its tools change — previously the tool set could change between turns with no signal

Verification

I captured a live smoke baseline against the old adapters before touching anything, then re-ran the identical scenarios after and diffed: no regressions. Same tool round-trips, same event types, no lost usage keys. The only differences are the three predicted in writing beforehand — stopReason end_turnstop, tool results becoming first-class toolResult messages, and betas disappearing (pi manages beta headers).

Also verified against real API calls: MCP end-to-end against @modelcontextprotocol/server-filesystem, NL conditions, dynamic tools unlocking mid-run, and cross-provider subagents. 151 tests pass; typecheck, lint, format, build clean.

Breaking changes (0.2.0)

  • Single entry point — agentry/anthropic and agentry/openai are gone
  • provider/model are plain strings; run/createAI take an optional pi Models collection, defaulting to pi's full catalog so zero-config works
  • Tool schemas are TypeBox, not Zod (Type is re-exported from agentry; zod peer dep dropped)
  • Removed: <WebSearch>, <CodeExecution>, <Memory>, <Agent websocket>, <System cache>, betas, stopSequences
  • <MCP> now takes type: 'stdio' | 'url'

Why some things were removed rather than ported

pi ships no MCP and no provider-native tools by design. onPayload can inject {type: 'web_search_20250305'} into the request, but pi's Anthropic parser handles four block types with no fallthrough — server_tool_use and friends are dropped and vanish from replayed history. Works for one turn, corrupts the conversation after. So MCP is client-side instead (strictly better: every provider). Agentry now ships no built-in tools at all — search needs a third-party API, code execution is a sandboxing project, and memory is a few lines over storage you already own. All three are better written as ordinary tools.

Post-migration hardening

The migration used a minimal slice of pi — enough not to break anything. These commits close what that left.

Failure modes pi already solved that agentry was re-suffering. isContextOverflow (already in pi, unused) encodes overflow detection for ~18 providers across three modes, so a context blowout now raises a distinct AgentryContextOverflowError instead of an opaque provider error — and triggers a compact-and-retry. retryAssistantCall classifies what is worth retrying (529 retries; 401 fails fast without burning attempts). timeoutMs/headers/samplingParams are now reachable: before this, every run silently inherited the Anthropic SDK's 10-minute default with no way to change it. cacheRetention was declared and read but never set — permanently undefined until now.

Compaction rewritten. It used to replace the entire transcript with one summary, taking the model's recent context away exactly when it was deepest in a task. It now keeps a recent window verbatim, and the cut is forced onto a turn boundary — splitting an assistant message from the tool results answering it produces a transcript providers reject.

Context inspection (concept borrowed from pi-context-view): handle.describeContext() reports what is filling the window — system prompt, per-tool cost sorted largest first, messages. Calibrating against the live API showed a naive estimate runs ~6x under the provider's reported tokens (providers prepend scaffolding a client never sees), so the report carries reportedInputTokens for absolutes and confines section shares to attribution.

A real bug, found by writing the test the plan asked for: when an MCP server left the tree its connection closed but its tools stayed registered, so the model kept being offered tools whose server was gone.

Also: images may now be returned from tools (pi carried them end-to-end; agentry's own ToolResult type was the only thing blocking screenshot/chart tools), thinking levels are clamped to what a model supports, and deferred/pending stop reasons throw rather than silently ending a run with empty output.

Deliberately not built: background/detached subagent runs. Concurrent subagents already work — each handle has its own store, container and session id — and that is now documented. Cross-turn deferral would need a serializable agent identity, but <AgentTool agent={...}> is a closure, which is the point of the JSX API.

Review notes

Commits are split for readability, not bisectability — only HEAD is green, since the message-type change forces src/tests/examples to move together. Start with the core swap and the execution-mechanics commit. The last commit acts on a structural review; its notable finding was that conditions.ts still called pi directly, breaking the "one path to a model" invariant one commit after it was documented.

Test count 178 → 177: ~2,900 lines of provider-wire tests covered deleted code; new suites cover MCP (against a real stdio server), robustness, resource diffing, context usage and the pi facade.

Acting on the code review

A review of the finished branch found four real defects, all now fixed and covered:

  • MCP leaked a server process per message. The handle builds a fresh engine per run(), and the engine owned the connection set — so every sendMessage reconnected each server and orphaned the previous ones, while close() reaped only the last. Measured: three messages, three live processes, two surviving close. Cross-run state now lives on an AgentSession owned by the handle.
  • retry did nothing in the default configuration. Agents default to stream: true, but only the non-streaming path went through pi's retry helper. Streaming now retries failures that arrive before the first token; past that a retry would replay output the caller has already seen, so it stays terminal.
  • <MCP> prop changes were silently dropped — three causes stacked: no reconciler branch for MCP instances, the instance aliasing React's frozen props object (so the write threw inside the commit phase, where it is swallowed), and connections keyed on server name alone.
  • usage reported the final turn while documented as a run total, under-reporting a ten-turn loop tenfold.

Two review findings were checked and dismissed with evidence rather than fixed: the compaction summary request sends tool_use blocks with no tools param, which Anthropic accepts (verified live), and pi's own token estimator is unreachable — dist/utils/* has no entry in its exports map.

Also fixed: MCP tool handlers tested the abort signal from the turn they connected on, dead from turn two onward; close() could throw from a finally and mask the run's result; MCP servers connected serially; and two TurnRequest retry fields no caller ever set.

Introduces `src/pi/` as the single boundary to `@earendil-works/pi-ai`,
ahead of routing the engine through it. Nothing imports this yet.

- `turn.ts` is the seam that will replace `ProviderAdapter.createTurn`. It
  normalizes two pi behaviours the engine depends on: pi reports request,
  model and runtime failures as values (`stopReason: 'error' | 'aborted'`)
  rather than throwing, so those are converted back into throws; and
  non-streaming turns replay the same `AgentStreamEvent` sequence as
  streaming ones.
- `events.ts` maps pi's `AssistantMessageEvent` to `AgentStreamEvent`,
  keyed off `contentIndex` because pi does not guarantee a block's
  start/delta/end run is uninterrupted.
- `models.ts` lazily builds pi's built-in catalog (cached, de-duped) and
  resolves provider/model pairs with errors that list what was available.
- `tools.ts` maps an `InternalTool` to pi's wire shape.

`tests/utils/piMockProvider.ts` replicates the existing `StepMockController`
API so the suite can port mechanically. It is built on pi's `fauxProvider`:
a `FauxResponseFactory` receives the full `(context, options, state, model)`
of every call and may return a promise, which covers both param capture and
turn gating without hand-writing a provider.
Replaces agentry's hand-written Anthropic and OpenAI adapters with
`@earendil-works/pi-ai`. The adapter layer was the framework's least
differentiated code — wire-format plumbing rewritten per provider and
re-tested per SDK bump — and the one place two SDKs' types leaked into
otherwise provider-agnostic code. Agentry keeps its reconciler,
ExecutionEngine, handles, conditions and subagents; pi owns the wire.

Deletes `src/providers/` (1,404 lines) along with the `agentry/anthropic`
and `agentry/openai` entry points, and drops the `@anthropic-ai/sdk`,
`openai` and `zod` peer deps. pi is pinned exactly: it is 0.x and just
moved its previous surface to `/compat`, so the pin plus the single
import site in `src/pi/` keeps that churn to a one-directory blast radius.

Message model
- Agentry's message types are now pi's, re-exported with agentry-flavoured
  aliases rather than converted at the edge. A converter layer would be
  permanent and lossy: thinking signatures, images, per-message usage and
  responseId have nowhere to live in the old types.
- Tool results become first-class `ToolResultMessage`s instead of
  `tool_result` blocks batched into one user turn.
- Stop reasons follow pi (`stop` / `toolUse` / `length`), which removes the
  hard-coded `stop_reason === 'tool_use'` check.
- Side effect: thinking signatures now survive replay. The old Anthropic
  adapter dropped them, so multi-turn thinking was silently lossy.

Conditions
- `conditions.ts` no longer bypasses the adapter to call raw SDKs. NL
  conditions are evaluated in one `models.complete()` call using a
  constrained-sampling tool, forced via `toolChoice` where the API supports
  it and falling back to prompting where it does not.

Tool schemas: Zod -> TypeBox
- A TypeBox schema is plain JSON Schema at runtime, which is exactly what
  pi puts on the wire, so `jsonSchema` and the `z.toJSONSchema()` step both
  collapse. Validation moves to `typebox/value`.
- `<Tool>`/`<AgentTool>` gained overloads: TypeBox's `handler` is a
  function-typed property and so contravariant, which broke inference for
  the inline form and assignability for the `{...someTool}` spread.

Removed capabilities (no pi equivalent; upstream non-goals)
- `<WebSearch>`, `<CodeExecution>`, `<MCP>`: pi models only client-executed
  tools. Injecting native tools via `onPayload` reaches the wire, but pi's
  response parser handles four block types with no fallthrough, so
  `server_tool_use` / `web_search_tool_result` blocks are dropped and
  replayed history is corrupted. Rejected rather than half-working.
- `<Agent websocket>` and the stateful `previous_response_id` chain: pi's
  `openai-responses` is stateless replay; only the Codex OAuth provider
  implements a WebSocket transport.
- `<System cache>`, `betas`, `stopSequences`: pi owns cache retention and
  beta headers, and has no stopSequences option.
- `executeMemoryTool` and `MemoryHandlers` are retained so `<Memory>` can be
  rebuilt as an ordinary client-side tool.

BREAKING CHANGE: single `agentry` entry point; provider/model are plain
strings resolved against pi's catalog; `run`/`createAI` take an optional pi
`Models` collection instead of SDK clients; tool schemas are TypeBox.
The old mocks stubbed the Anthropic and OpenAI SDK client objects, a
boundary that no longer exists, so every suite that used
`createStepMockClient` had to move to `createStepMockModels`. The new mock
deliberately mirrors the old controller API (`nextTurn`, `peekNextCall`,
`resolveNextCall`, ...) so test *logic* is unchanged and the diffs stay
reviewable; assertions now read pi's `context`/`options` instead of
Anthropic wire params, which is a better boundary to assert against.

Deleted (~2,900 lines) because they covered code that no longer exists:
`openai-provider.test.tsx` (Responses wire format, WebSocket continuation,
chain reset), `built-ins.test.tsx`, `synthetic-events.test.ts`, and the two
SDK mock helpers. Provider-agnostic behaviour from those files was folded
into `runtime.test.tsx`.

Rewritten rather than deleted, to assert the new behaviour:
- system parts join into one string (cache breakpoints are pi's job now)
- `strict` requests constrained sampling
- `thinking` passes through as a reasoning level
- thinking signatures survive replay, replacing the old tests asserting
  that `toAnthropicMessage` dropped them
- the OpenAI condition suite collapses to one provider-selection test, as
  NL evaluation is a single provider-agnostic path
- cross-provider tests use a Models collection carrying two faux providers

Two real bugs surfaced and were fixed in the process: `accumulated` on text
events reset per content block instead of accumulating across the message,
in both the streaming and synthetic paths.

128 pass / 0 fail.
Examples drop client construction entirely — pi resolves credentials per
provider from the environment, so `run(<Agent />)` with no options is the
common case. Schemas move to TypeBox, and the `agentry/anthropic` and
`agentry/openai` imports collapse to `agentry`.

Deleted the examples for removed capabilities (`mcp`, `web-search`,
`openai/websocket`, `openai/built-ins`, `anthropic/cache-ephemeral`) and
their npm scripts.

Adds `example:multi-provider`, which is the migration's headline: a parent
agent delegating to a subagent on a different provider, selected by string.
That was possible before but needed two SDK clients wired through
`providers`; it now needs neither.

CLAUDE.md documents the pi seam, the pi-owned message model, the TypeBox
tooling, and — importantly for future work — why MCP and provider-native
tools are absent, so they are not re-attempted via `onPayload`.
The pi swap changes the public API: single entry point, string
provider/model, TypeBox tool schemas, and several removed components.
Breaking changes under 0.x, so this is a minor bump with the migration
notes in CLAUDE.md.
pi ships no MCP support and no provider-native tools, by design. Rather
than leave both capabilities dropped, agentry now provides them itself —
which makes them work on *every* provider pi supports, not just the ones
with a native connector.

MCP
- `src/mcp/` is an MCP client: connect, `tools/list`, and expose each remote
  tool as an ordinary agentry tool whose handler proxies `tools/call`.
- Supports stdio (local subprocess) and streamable HTTP (remote) servers;
  the old component only did Anthropic's URL connector.
- MCP `inputSchema` is already JSON Schema, which is what TypeBox schemas
  are at runtime, so it passes through with no conversion.
- Tools are namespaced `<server>__<tool>` so two servers cannot collide.
- Connections are reconciled per turn: a server that leaves the tree (a
  `<Condition>` deactivating, say) is disconnected, and all connections are
  closed with the handle.
- Server-side tool errors come back as strings so the model can recover.

Memory
- `<Memory handlers={...} />` is now an ordinary tool built on the existing
  `executeMemoryTool` / `MemoryHandlers`, with a hand-written schema for the
  view/create/str_replace/insert/delete/rename union mirroring Anthropic's
  published shape. Storage stays entirely the caller's concern.

Testing: `tests/fixtures/mcp-server.ts` is a real stdio MCP server, so the
suite exercises an actual client/server handshake and tool call rather than
a mock. Verified live end to end against `@modelcontextprotocol/server-
filesystem` and a real model.

Raises the test timeout to 5s to give the MCP subprocess headroom.
The previous reactivity model was more machinery than this problem needs.
This replaces the mechanics while leaving `<Agent>`/`<Tool>`/`<Condition>`
untouched. No user-facing API change.

Turn-boundary rendering
- The tree is rendered exactly once per turn, immediately before the model
  call, via a `renderTurn` hook the handle supplies (the engine knows *when*,
  the handle knows *what*).
- Removes the mid-turn `flushSync(() => {})` + `yieldToSchedulerImmediate()`
  pair that existed to force React to commit tool-handler state, and with it
  `yieldToSchedulerImmediate` itself.
- Behaviour change worth naming: state written during a turn is now visible
  at the *next* turn boundary rather than mid-turn. For tool sets this is
  correct.

Name-keyed resources
- `AgentInstance.tools` is a `Map` keyed by name rather than a positional
  array, which is what makes conditional mounting robust and consecutive
  renders diffable. Insertion order is preserved, so wire order is stable.
- Duplicate names are collected during render but rejected at the turn
  boundary: throwing inside React's commit phase surfaced as an unrelated
  "No agent element found in tree", which is worse than useless.

Narrated resource diff (new capability)
- Each turn snapshots the tool set and diffs it against the last narrated
  one; the delta is announced into the transcript as prose. Previously the
  tool set could change between turns with no signal to the model at all,
  which is exactly the incoherence this fixes.
- Digests use an order-independent stringify so key ordering alone never
  reads as a change.

Resolves the `todo(colin)` on `recollectAll` being experimental: condition
changes still trigger a recollect, but React-driven changes now arrive
through the turn-boundary render instead of ad-hoc synchronization.

150 pass / 0 fail. Verified live: a tool handler unlocking further tools
still works end to end.
… cruft

A structural review of the migration turned up one real architectural
inconsistency and a pile of leftovers. Fixes, most important first.

The seam was already broken
- `conditions.ts` called `models.complete()` directly and hand-duplicated
  `createTurn`'s abort/error-to-throw conversion, one commit after CLAUDE.md
  declared `createTurn` the only path to a model. Forced tool choice is now a
  `forceToolUse` flag on `TurnRequest`, with the per-API spelling
  ("any" vs "required") living behind the seam where it belongs. Condition
  evaluation goes through `createTurn` and inherits its error handling.
- `CONDITION_TOOL`'s schema is a real `Type.Object(...)`, dropping an
  `as unknown as TSchema` cast.
- CLAUDE.md overstated the invariant: 14 files import pi. Every one is
  `import type` — `Models` threaded through signatures and pi's message types
  re-exported by design. Reworded to the invariant that actually holds and
  matters: `src/pi/` is the only place that *calls* pi.

Dead code
- `ExecutionEngine.updateConfig` — no callers.
- `ExecutionEngineConfig.system` and `SystemPrompt` — written by
  `createEngineConfig`, never read, because `makeApiCall` recomputes the
  prompt from the live instance each turn. Turn-boundary rendering made the
  stored copy obsolete; leaving it invites someone to "fix" a prompt by
  editing a field nothing reads.
- `AgentInstance.engine` — always `null`, never assigned or read, and it
  forced every test fixture to carry it.
- `ANTHROPIC_BETAS` — `betas` was removed in the swap.
- The `built_in_tool` JSX intrinsic, importing a type deleted in the swap. It
  survived because `skipLibCheck` means `.d.ts` files are never checked.
- `<System cache="ephemeral">` / `<Context cache>` — pi owns cache retention,
  so the prop was threaded through components, instances, collectors and the
  reconciler to do precisely nothing, silently. Removed end to end.

Duplication
- `PropagatedSettings` was defined twice and had drifted — the reconciler's
  copy still carried `stopSequences`, the last trace of a removed feature.
- `RunAgentOptions` was defined twice with the same fields under two names
  for the same type.

Docs that would actively mislead
- Zod examples in `<Agent>`/`<AgentTool>` docstrings after the TypeBox move.
- `<WebSearch />` in `<Tools>`, `<Condition>` and `run()` examples.
- `createRunAgent({ clients: ... })` — that field no longer exists.
- `<Agent model=...>` without `provider`, which is now a type error.
- "not visible to Claude" in a provider-agnostic framework; a `maxTokens`
  comment describing the `<AgentTool>` inheritance path, not `runAgent`'s.

Also: `AgentryProviderError` is exported from the root (consumers need it for
`instanceof`); test-only model constants moved out of `src/` into `tests/`;
and the once-per-run natural-language condition evaluation is now explained
rather than left as a bare `evaluateNL: isFirstIteration`.

Adds coverage for `getDefaultModels`, which every zero-config `run()` depends
on and nothing exercised.

151 pass / 0 fail. NL conditions re-verified live through the new seam.
@colinds
colinds force-pushed the colin/pi branch 2 times, most recently from 9da3aad to 83055b1 Compare August 16, 2026 03:21
colinds added 20 commits August 16, 2026 03:53
The migration used a minimal slice of pi — enough to not break anything.
That left agentry hitting failure modes pi already solves.

Context overflow now has its own error type
- `isContextOverflow` encodes hand-tuned detection for ~18 providers across
  three modes (error message, silent `usage.input > contextWindow`, and a
  zero-output length stop). Previously a context blowout reached users as an
  opaque `AgentryProviderError`.
- `AgentryContextOverflowError` carries the model's context window, because
  this is the one failure a caller can actually act on — by compacting.

Retry
- Non-streaming turns go through pi's `retryAssistantCall`, which classifies
  what is worth retrying: verified 529 overloaded retries, 401 auth fails
  fast without burning attempts. Retries surface as a new `retry` stream
  event so a UI can show "retrying in 2s".
- Streaming is deliberately not wrapped: it has already emitted events, so a
  retry would replay them. pi's SDK-level `maxRetries` still covers transport
  errors there.

Request options that were never reachable
- `timeoutMs`, `headers`, `samplingParams`, `maxRetries`, `maxRetryDelayMs`
  are now props. Before this every run silently inherited the Anthropic SDK's
  10-minute default with no way to change it; `timeoutMs: 1` now fails in
  ~66ms against the live API.
- `cacheRetention` was declared in `ExecutionEngineConfig` and read in
  `makeApiCall` but never set by `createEngineConfig` — permanently
  undefined. Now wired, so `'long'` retention is reachable.
- `headers` is what lets a user reach a corporate gateway without writing a
  custom provider.

Thinking levels are clamped
- `reasoning` was passed through raw, but `'xhigh'`/`'max'` exist only on some
  model families. `clampThinkingLevel` degrades instead; a model with no
  thinking support clamps to `'off'`, which pi expresses as absence.

Auth preflight
- `resolveModel` listed every catalog model on a miss regardless of whether
  the user had credentials, so a user picked a listed model and failed
  mid-turn. `describeMissingAuth` consults `checkAuth` before the run and
  names the credential type ("No Anthropic API key configured for provider
  ..."), plus the /login hint where the provider supports OAuth.
- pi does not export per-provider env var names (`findEnvKeys` is internal),
  so the message quotes the provider's own credential display name rather
  than guessing at variable names that would rot.

Also
- `cleanupSessionResources(sessionId)` on handle close; providers key
  long-lived resources by session id and nothing released them.
- `ToolResult` widened to `string | Array<TextContent | ImageContent>`. pi
  carries images end-to-end and converts them to native image blocks;
  agentry's own type was the only thing blocking screenshot/chart tools.
- `AgentResult.usage` gains `reasoningTokens` and the per-category `cost`
  breakdown — the cache split is how a user finds out their cache is not
  hitting.

Test note: the mock's models are now `reasoning: true`, since clamping
correctly strips a thinking level from a model that does not support one.

162 pass / 0 fail. Live-verified: timeout fails fast, auth preflight
discriminates configured from unconfigured without false positives.
Compaction replaced the entire transcript with a single summary message, so
the model lost all recent verbatim context at exactly the moment it was
deepest in a task. It also only ever triggered on a token threshold, which
by definition has already failed to fire when the provider actually refuses
a request for being too long.

Extracted to `src/execution/compaction.ts`:
- `findCutIndex` keeps roughly `keepRecentTokens` (default ~16k) of the tail
  verbatim and summarizes only what precedes it.
- The cut must land on a turn boundary. Splitting an assistant message from
  the tool results answering it produces a transcript providers reject — a
  tool call with no matching result — so the scan walks forward past any
  tool-result or unanswered-tool-call message. Tested across several budgets.
- The summary request uses a fresh `sessionId` and `cacheRetention: 'none'`:
  a one-off prompt is never reused, so caching it only evicts entries that
  would have been hit.
- `isFatalCompactionError` keeps the existing classification — compaction is
  best-effort, but aborts, auth failures and programming errors still throw.

Overflow recovery:
- `callWithOverflowRecovery` catches `AgentryContextOverflowError`, forces a
  compaction, and retries the turn once.
- The forced path deliberately does not require a previous turn to measure.
  A pre-loaded transcript that overflows on the very first call is the most
  likely overflow there is, and requiring `lastMessage` made it unrecoverable.

`keepRecentTokens` is exposed on `CompactionControl`, with a note that
changing the transcript invalidates the provider's prompt cache.

166 pass / 0 fail.
Borrowed from `pi-context-view`'s concept — "see what fills your context, for
example, what survives compaction". Overflow detection and compaction tell a
user they ran out of room and then act; neither says *why*. The parts that
are easy to forget are the ones nobody wrote by hand: the assembled system
prompt and the tool JSON schemas, which are re-sent on every single request.

`handle.describeContext()` returns a breakdown: per-section token estimates
(system / tools / messages), per-tool cost sorted largest first, the model's
context window, and free space. As a data API rather than a TUI, since
agentry is a library — the caller renders it.

Reuses `estimateTokens` from the compaction module, so the attribution and
the compaction cut agree with each other by construction.

Estimated vs reported, which matters
- Calibrating against the live API showed the naive estimate is ~6x under the
  provider's reported input tokens: providers prepend their own scaffolding
  (tool-use instructions and the like) that a client never sees. Reporting
  `free: 199,832` when real usage was 674 would have been actively
  misleading.
- So the report carries both. `reportedInputTokens` comes from the most
  recent assistant turn's actual usage and backs `free`; `estimatedUsed` is
  labelled as an estimate, and section `share` is a fraction *of the
  estimate* rather than of the window — so attribution stays meaningful
  without inheriting the absolute error.

Live check on a two-tool agent: window 200,000, reported 747, and tools at
57% of what agentry controls versus system at 8% — which is the insight the
feature exists to surface.

170 pass / 0 fail.
Writing the disconnect test the plan called for surfaced a real bug.

`syncMcpConnections` closed the connection when an `<MCP>` element left the
tree — a `<Condition>` deactivating, say — but the tools it had registered
stayed in the agent's tool map. The reconciler removes the element from
`mcpServers`, but the derived tools were written straight into the map and
nothing took them out. The model kept being offered tools whose server was
gone, and calling one would fail.

Also covers `close()` tearing down a live stdio subprocess.

172 pass / 0 fail.
Researching background subagent runs turned up a latent hole and made the
case for not building the feature.

`stopReason: 'deferred'` (and `'pending'`) fell through `throwIfFailed`.
Such a message carries no content and no tool calls, so the engine's loop
would treat it as a normal completion and end the run with **empty output
and no error**. Unreachable today — agentry never requests deferred
responses, and no real provider implements them in pi 0.84.2 — but a silent
empty result is the worst possible failure, so it now throws.

On background runs: not building them. Concurrent subagents already work —
each `SubagentHandle` has its own store, its own reconciler container and its
own `sessionId`, and `executeTools` already runs a turn's tool calls under
`Promise.all`, so two `<AgentTool>` calls in one turn are already concurrent.
That was simply undocumented, so CLAUDE.md now says so.

What is genuinely missing is cross-turn deferral, and that is where the
feature stops fitting a library: durable runs need a serializable agent
identity, but `<AgentTool agent={...}>` is a closure over lexical scope —
which is the whole point of the JSX API. pi-subagents can do it because its
agents are markdown files addressable by name, and because a CLI owns its own
process. A library does not; a caller who needs durability should queue
`run(<Agent …>)` in their own worker.

173 pass / 0 fail.
`McpConnectionSet` now owns both the sockets and the tools derived from them,
because the two have to move together — a tool whose server has gone is worse
than no tool at all, which is the bug the previous commit fixed. Keeping them
in one object makes that invariant local instead of a rule the engine has to
remember.

The engine keeps a two-line delegation for each of sync and close.
ExecutionEngine drops to ~697 lines.

Behaviour-neutral: 173 pass unmodified, and the MCP example still runs
end-to-end against a real filesystem MCP server and a live model.
`sessionId` was accepted by the handle constructor but no caller could reach
it, so it was always a fresh `crypto.randomUUID()`. That made the prompt
caching agentry already wires — `prompt_cache_key` on OpenAI, session
affinity on Anthropic — unable to survive a single run.

Now exposed on `run`, `createAgent` and `createAI`, so a caller can reuse an
id across runs of the same logical agent. Omitting it keeps the fresh-id
behaviour, which is correct for unrelated work.
The plan called for measuring before writing any upstream pi PR to add a
WebSocket transport to `openai-responses`. Measured; the answer is no.

`wss://api.openai.com/v1/responses` is reachable with a plain API key —
verified — so the transport is available. It just is not worth having.

gpt-4.1-mini, n=12, median:

  SSE       TTFT  447ms   total 1080ms
  WS cold   TTFT  608ms   total 1110ms
  WS warm   TTFT  434ms   total  914ms

Warm WS is 2.8% faster to first token, which is noise. Cold WS is 36%
*slower*, paying for connection setup that the SDK's HTTP keep-alive already
amortises for SSE. Against that: pi's changelog carries nine-plus WS-specific
fixes on the Codex path alone — connection limits, idle timeouts, SSE
fallback, processes kept alive after a response, cached sessions shared
across credentials.

Methodology note worth keeping: an early version timed warm WS from
socket-open and appeared to show a 45% win. That was not a comparison — SSE
was paying for connection establishment and WS was not. The committed script
measures cold and warm separately and says which is which.

Kept as a runnable script under `benchmarks/` (outside tsconfig and lint
scope) so the answer can be re-derived rather than trusted. Longer
conversations, where per-request upload dominates, is the case most likely to
change it.
The turn-boundary walk went the wrong direction, and it defeated the whole
point of the rewrite.

`findCutIndex` walked *forward* past any message that could not open a
conversation — tool results, and assistant turns holding tool calls. When the
transcript tail was a call/result pair, which is what a mid-task conversation
looks like almost by definition, the walk ran off the end. `recent` came back
empty and compaction wiped the entire transcript: exactly the behaviour the
rewrite was meant to remove, now hidden behind a guard that looked correct.

Probed:

  ends with call+result       cut=3/3  kept=0
  assistant-with-calls last   cut=2/2  kept=0
  long, ends mid-tool-use     cut=5/5  kept=0

The premise was wrong. The kept tail is prefixed by the summary, which is
itself a *user* message, so the tail may legally begin with an assistant turn
and its tool calls. The only illegal opener is a tool result whose matching
call was summarized away. So the cut walks backward instead, pulling the
owning assistant turn into the tail — which also keeps strictly more context
rather than less.

  ends with call+result       cut=1/3  kept=2  opens with assistant
  long, ends mid-tool-use     cut=3/5  kept=2  opens with assistant

The existing test passed throughout because it asserted the old invariant —
that the cut never lands on an assistant-with-calls. It now asserts the real
property: the cut never lands on a tool result, and every tool result in the
kept tail still has its call. Plus a direct regression test for the
ends-mid-tool-use case.

175 pass / 0 fail.
Every code sample in the README described the old API. Left alone it would
have been the first thing a new user read, and wrong in every particular.

- Install drops `zod`, `@anthropic-ai/sdk` and `openai`. There is no client to
  construct: credentials come from the environment and pi resolves providers.
- Zod → TypeBox throughout, with `Type` re-exported from `agentry` so no extra
  install is needed.
- The `agentry/anthropic` and `agentry/openai` entry points are gone; the
  Providers section now shows string-selected providers and how to pass a
  custom pi `Models` collection.
- Prompt caching is rewritten: `cache="ephemeral"` no longer exists, replaced
  by `cacheRetention` plus a stable `sessionId`, with a note that changing the
  tool set invalidates the cached prefix.
- Compaction documents `keepRecentTokens` and the overflow-triggered retry.
- `<WebSearch>`/`<CodeExecution>` removed; `<MCP>` rewritten as the
  client-side connector it now is (stdio and HTTP, namespaced tools);
  `<Memory>` takes `handlers`.
- `<Agent>` props drop `websocket`, `betas`, `stopSequences` and gain `retry`,
  `cacheRetention`, `timeoutMs`, `headers`, `samplingParams`; `thinking` is a
  level, not a provider-specific object.
- `ToolContext` carries `models` rather than SDK clients.
- New: a context-inspection section for `handle.describeContext()`, including
  why `reportedInputTokens` is the number to trust and `estimatedUsed` is not.
- Requirements: Node 22.19+, matching pi's engines field.

Every sample was compile-checked against the real API rather than eyeballed.
Tests that did not test anything:
- the thinking-clamp test asserted inside the faux provider callback, where
  a thrown expect() is swallowed into a provider error; it passed with
  clamping deleted. Capture the options and assert after the call, and add
  the positive case.
- the MCP teardown test ended in `expect(true).toBe(true)` and would pass
  with closeMcpConnections() removed. Assert the live connection count
  before and after close(), which also gives McpConnectionSet.size a use.

Both verified by deleting the feature and watching the test fail.

Real defects:
- compaction's 60s guard never applied. The engine always passes a signal,
  so `signal ?? AbortSignal.timeout(60_000)` always took the first branch
  and a hung summary blocked on the SDK's 10-minute default. Compose with
  AbortSignal.any instead.
- estimateTokens counted base64 image payloads at chars/4, swamping the
  estimate and dragging the compaction cut far later than intended. Count
  images at a flat per-image figure.

Cleanup: drop the doc blocks duplicated by the ExecutionEngine extraction,
route its pi import through the barrel, and track benchmarks/README.md
(previously hidden by an unanchored gitignore rule).
An <MCP> element whose url, args, env, or tool_configuration changed mid-run
kept its original settings forever. Three separate causes, all on the same
path:

- applyUpdate had no branch for MCP instances, so commitUpdate diffed the
  props and then discarded the result.
- createMCPServerInstance aliased the props object as `config`. React freezes
  props in development, so writing to it threw inside React's commit phase —
  where the error is swallowed rather than surfaced. Copy at creation; the
  collected `agent.mcpServers` array holds this same reference, so the update
  has to be in place.
- McpConnectionSet keyed connections on server name alone, so even a config it
  could see would not have triggered a reconnect.

Also connect servers in parallel — each is a subprocess spawn or an HTTP
handshake, and doing them in sequence made first-turn latency scale with the
number of servers.

Covered by a test that switches allowed_tools mid-run and asserts the
withdrawn tool is gone; it fails if any of the three fixes is reverted.
The handle builds a fresh ExecutionEngine for every run(), and the engine
owned the McpConnectionSet. So each sendMessage reconnected every <MCP>
server before the first model call and orphaned the previous set, while
close() tore down only the most recent engine's connections. Measured with
the stdio fixture: three messages left three live server processes, two of
which survived close().

Move both pieces of genuinely cross-run state onto an AgentSession owned by
the handle:

- the MCP connection set, so servers connect once per handle;
- lastNarratedResources, which also reset per run, silently re-baselining
  the tool set so a change made during one message was never announced in
  the next.

Also guard cleanupSessionResources: pi throws an AggregateError when a
handler fails, and close() is called from finally blocks where that would
mask the run's own result.

Regression test asserts one connection identity across three runs, so a
reconnect fails it even though the count stays at 1.
… signal

<Agent> defaults to stream: true, but only the non-streaming path went
through pi's retry helper — so the documented `retry` prop did nothing in
the default configuration. Wrap the streaming path too, retrying only
failures that arrive before any content has been emitted; once the consumer
has seen output, a retry would replay it, so the failure stays terminal as
it was before. That covers the common transient case (connection refused,
429/503 on the request itself).

The content gate deliberately ignores lifecycle events and empty text
deltas — a failed turn emits both even when it produced no output.

Separately, MCP tool handlers tested the abort signal captured when their
server connected. Connections outlive that turn, so from the second turn on
the check was against a signal that could never fire and aborting a run
never reached an in-flight tools/call. Read the signal off the ToolContext
instead, and pass it to callTool so the request itself is cancelled. The
connect-time signal had no other use — the MCP SDK's connect() takes no
options — so it is gone rather than left as a no-op argument.
AgentResult.usage read the final assistant message, so a tool loop reported
one turn's tokens and cost while the field is documented as a total — under-
reporting a ten-turn run by roughly ten times. Accumulate per turn instead.

TurnRequest.maxRetries / maxRetryDelayMs and RequestOptions.maxRetries were
never set by any caller. Their absence silently meant "SDK default" while the
surrounding comment implied deliberate coverage. `retry` is the supported
knob; remove the phantom ones rather than grow the surface.
Agentry ships no built-in tools. Web search needs a third-party search API and
code execution needs a sandbox; memory was a client-side tool pretending to be
a provider built-in. All three are better written as ordinary user-defined
tools, and pi does not model provider-native tools anyway.

Removes <Memory>, defineMemoryTool, memoryTool and MemoryHandlers along with
their tests, and every remaining mention of native tools in the README, the
examples and CLAUDE.md.

The chatbot example previously told the model to use a web_search tool that
did not exist, over a subagent with an empty <Tools>. It now has a real
fetch_url tool, which is what the example was demonstrating all along.

Also fixes docs rot the same sweep turned up: a README link to the deleted
web-search example, mcp.tsx listed twice, a tools.ts comment referring to
provider adapters that no longer exist, and `EXAMPLE_PROVIDER === 'openai'
? createAI() : createAI()` in three examples.

Dead workspace catalog entries (@anthropic-ai/sdk, openai, zod) are gone;
nothing has referenced them since the migration.
The SSE/WebSocket comparison was investigation scaffolding for a decision
already made; it does not belong in the published package. Removed along
with its note in CLAUDE.md.

Comments added during the review pass had drifted into multi-paragraph
incident reports. Trimmed to the 1-3 line why-focused form used elsewhere,
and test comments to the one-liners the suite already uses.
`children` is in RESERVED_PROPS, so diffProps never reports it — but for these
two elements the children ARE the content. applyUpdate's System/Context
branches read `payload.children`, which was therefore always undefined, so a
state-driven system prompt stayed frozen at whatever it rendered on the first
turn. `<System>Mode: {mode}</System>` kept sending the initial mode forever.

commitUpdate now compares the flattened text itself and passes it through.
Comparing the flattened string rather than the raw ReactNode matters: JSX
children are a fresh array every render, so an identity check would report a
change on every turn and rebuild the system prompt needlessly.

Found by a branch code review; the test fails without the fix.
@colinds colinds changed the title feat!: swap provider harness to pi; borrow Flue's execution mechanics feat!: swap provider harness to pi Aug 17, 2026
colinds added 12 commits August 17, 2026 02:34
rebuildSystemPrompt walked only agent.children, while collectChild recurses
into <Tools> and active <Condition>s. The two were not mirror images, so any
rebuild dropped every nested part.

This became reachable in 3893356, which enabled the text-update path for
<System>/<Context> — so an updating <System> erased the conditional blocks
around it, including its own if it sat inside a <Condition>.

The traversal now lives next to collectChild in collectors.ts, so the two
sets of rules cannot drift apart again. Order matches collect order, since a
mismatch would silently reorder the prompt between an update-driven rebuild
and a condition-driven recollect.
Partial sync failure orphaned every server that opened. sync() ran all
connects under Promise.all and only stored them after the join, so one
failing server discarded the successful siblings before they were tracked —
unreachable from closeAll(), and for stdio that is a live child process.
This was a regression from making connects parallel; the sequential version
stored each one before attempting the next. Now allSettled: register what
opened, publish its tools, then raise the failure.

Duplicate <MCP name> spawned two servers and tracked one. `missing` is
computed before anything is stored, so both connected and the second
overwrote the first with no close. Rejected at sync time — a collision is an
authoring error, and the tool names would collide anyway.

A failing tools/list leaked the client and its subprocess. Only connect()
was inside the try; by listTools() the transport is live, and the MCP SDK
only tears down from a transport close, so a timeout or error response left
the child running forever.

Tests count real fixture processes rather than trusting the connection map,
since the whole failure mode is state the map cannot see.
The AbortController was a per-turn local, and only the Streaming state kept a
copy — WaitingForTools and ExecutingTools carry no controller field. So from
the moment the model asked for a tool there was no reachable reference, and
abort() gated on `status === Streaming` did nothing: the tool's
ToolContext.signal never fired, the turn ran to completion, and the loop fell
through to buildResult() and resolved successfully with stopReason 'toolUse'.

The controller now lives on the engine, abort() fires it unconditionally, and
the run rejects with AbortError instead of reporting success.

Aborted tools still produce a result. The assistant toolUse message is pushed
before tools run, so rejecting out of executeTools would leave a tool call
with no answer — and the store is reused by sendMessage, so the *next* call
would send an invalid conversation. Each cancelled tool yields an error
result, the transcript is completed, and only then does the run reject.

abort() no longer emits 'error' or transitions to Error itself: run()'s catch
already does both, and doing it twice made consumers see 'error' followed by
'complete' for a single run.
CONDITION_DEFAULT_MODELS lists anthropic and openai only, and an unlisted
provider threw — despite the comment directly above it promising a fallback
to the agent's own model. The engine catches that throw, and NL conditions
are only evaluated on the first iteration, so the three-strikes abort could
never fire: the run continued with every NL condition permanently false.
Silently wrong gating rather than a visible failure.

Implemented the documented fallback.

Also in the same seam:

- forcedToolChoice covered 6 of pi's 10 APIs. The four missing
  (bedrock-converse-stream, mistral-conversations, pi-messages,
  openai-codex-responses) all support toolChoice, so this was four absent
  switch cases, not a capability gap. Unforced requests are why condition
  evaluation was flaky on those providers.
- reasoning was silently dropped under forceToolUse: it lives on pi's
  SimpleStreamOptions, but the forced path uses `complete`, which takes raw
  per-API options and ignores unknown keys. Nothing requests both today, so
  this now throws rather than quietly sending a request without the thinking
  configuration it asked for.
…i seam leak

isFatalCompactionError checked `error.status`, but pi surfaces failures as
values on the assistant message and classifies them by message text, so
nothing reaching this function carries a status — the branch was unreachable
and an auth failure during compaction cost one wasted request per iteration.
Match the message wording instead.

Nine types the README documents as public were not exported from the package
root, which is the only entry point: AgentState, AgentStoreState,
CompactionControl, RunAgentOptions, ToolResult, RunOptions and
CreateAgentOptions. AgentState is the worst of them — documented as both the
useExecutionState() return type and the type of handle.state, so a
TypeScript user could annotate neither.

cleanupSessionResources was the one runtime pi call outside src/pi/, out of
24 import sites. Wrapped as releaseSessionResources so the invariant that
bounds pi's 0.x churn holds again.

createEngineConfig defaulted stream to false while createInstance had
already applied `?? true`, so the branch never fired and only served to
imply the opposite of the real default. Aligned, and documented that
<AgentTool> subagents are the genuine exception.
getEffectiveAgent returned null for a <Condition> parent, so anything
appended directly under a condition mid-run was never collected — and never
uncollected when removed. Meanwhile findParentAgent, used for the <Tools>
case, walked parent links without ever checking isActive, so a tool
appearing into a <Tools> nested in an *inactive* condition was collected and
offered to the model.

The two are the same defect pulling in opposite directions: fixing either in
isolation creates the other. Collection now walks up through <Tools> and
active <Condition>s in one place, and stops at an inactive one.

findParentAgent stays condition-blind: prop updates must still find their
agent under an inactive condition, and only collection routing gains the
gate.

Both directions are covered, and each test fails against the old
implementation — one missed the tool that should appear, the other was
offered the tool that should not.
applyUpdate had branches for Agent, System, Context, Tool, MCP and Condition
but none for AgentTool — isAgentToolInstance was not even imported. diffProps
did report a change (the agentTool object is rebuilt every render, and
deepEqual returns false for two distinct closures), so applyUpdate ran and
silently did nothing.

The result was a description, schema and agent closure frozen at first
render, with the registered synthetic tool snapshotting them at collect
time. A subagent whose prompt closes over parent state kept serving the
state it saw on turn one.

name is not affected: <agent_tool> is keyed by it, so a rename remounts.

Guarded on the tool already being collected, matching the Tool branch —
otherwise updating props would resurrect a tool an inactive <Condition> is
deliberately withholding.
…ate tracking

Conditional <Message> duplication. recollectAll skipped messages at the top
level, but a <Condition> child recursed through collectChild, which pushes.
Broader than it looks: recollectAll runs whenever ANY condition in the tree
flips, and re-pushes messages for EVERY currently-active condition —
including ones active since turn one — while deactivation never removes
them, so copies accumulated monotonically. Three copies after two flips.

The skip is now threaded through collectChild rather than applied only at
the top, so the recursive paths honour it too.

duplicateToolNames was a one-way Set. A tool relocated between containers
registers in its new position before the old one is uncollected, so the name
was marked duplicate and never unmarked — aborting every later turn, and
every later sendMessage, over a duplicate that no longer existed. Worse, the
following uncollect deleted the newly registered tool, so it vanished from
the model's list entirely.

Now a count map: incremented on collision, decremented on uncollect, and the
uncollect leaves the surviving registration in place.
…ored

createSubagentInstance always produced a maxTokens — 4096 when nothing
supplied one — and SubagentHandle then wrote every defined subagent prop
over the rendered instance. So <AgentTool agent={() => <Agent
maxTokens={7777}>}> silently ran at 4096, and the value the author wrote was
never reachable. The instance layer no longer manufactures a default;
createEngineConfig already applies one, so it is applied in exactly one
place and the element's own value survives.

runAgent's documented provider/model overrides were discarded whenever the
element declared its own: createAgentInstance reads the element's props
first, and SubagentHandle only fills gaps with `??=`. maxTokens and
temperature already honoured the override, which is what gave it away.

Applied to the element itself rather than to SubagentHandle's assignment,
because on the <AgentTool> path `sub.provider` carries the parent's provider
— making that assignment unconditional would have broken cross-provider
subagents, which are a supported feature.
Most of conditions.test.tsx asserted result.content, which is the scripted
mock reply and identical whether or not conditions activate. Deleting
condition activation outright left 18 of 20 tests passing; it now fails 5,
including the boolean activate/deactivate cases that are the feature's core.

'should collect routes into router.children array' had no assertions at all
— it ended on 'passes if it completes without errors', which held with
condition activation deleted, with abort() broken, and with Condition
instances missing from children entirely. It now asserts which routes
reached the system prompt.
Three features could be deleted outright with the suite staying green.

describeMissingAuth: the test probed the environment for a provider that
happened to be unconfigured and returned early when it found none — so
gutting the function to `return undefined` made it find none and assert
nothing. Now builds an unconfigured provider deterministically.

<Agent retry>: every retry test called createTurn directly, so
`retry: agent.props.retry` could be dropped from createEngineConfig and a
529 would kill runs with nothing failing. Now exercised end to end through
run().

Context overflow: only the error-shaped overflow was covered, but two of the
three shapes pi recognises arrive as successful turns — which is the whole
reason detection was hoisted out of the error branch. Now covers the silent
case.

Each verified by reapplying the exact breakage and watching the new test
fail.
…ation

isFatalCompactionError could return false unconditionally with the suite
staying green — so an abort mid-compaction was swallowed and the run carried
on. Now covers all four classes: abort and programming errors are fatal,
auth failures are matched from the message (pi never sets error.status,
which is why the old numeric check was dead), and a 529 stays non-fatal so
best-effort compaction does not kill a healthy run.

'subagent has isolated message context' asserted toBeLessThan(5) against an
actual value of 2 — pointing useMessages() at the parent store would also
produce 2, so the bound held whether or not the context was isolated.
Asserts the exact count now, plus that the parent's user message never
appears in what the subagent sees.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant