Skip to content

fix(runtime): gate provider reasoning replay by source model - #4286

Open
Astro-Han wants to merge 12 commits into
apache:mainfrom
Astro-Han:fix/reasoning-replay-provenance
Open

fix(runtime): gate provider reasoning replay by source model#4286
Astro-Han wants to merge 12 commits into
apache:mainfrom
Astro-Han:fix/reasoning-replay-provenance

Conversation

@Astro-Han

@Astro-Han Astro-Han commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Make provider-owned reasoning replay use one fail-closed provider-state contract across normal turns, continuations, compaction, overflow recovery, and durable reload.

  • Persist and replay Anthropic signed and redacted_thinking blocks through ModelAdapter → thinking RuntimeEvent → the existing AI SDK request converter.
  • Reuse the existing openai-chat-plaintext contract for GitHub Copilot openai-chat, preserving observed reasoning_content request-field behavior.
  • Freeze one opaque providerStateIdentity per provider Run from the canonical Host connection snapshot: immutable connection ID, provider type, effective endpoint, connection-credential version, and request-header credential version.
  • Replay provider-owned reasoning only when source providerStateIdentity + modelId exactly matches the current target. Missing legacy provenance fails closed while text, tool calls, and tool results remain.
  • Bind safe-boundary continuation admission to that same target identity and ordered replay projection; bump the durable provider replay projection to v2.
  • Keep persisted v1 claims readable and migration-safe, but explicitly reject them for v2 replay instead of interpreting their old digest under new semantics.

Root cause

The original contract treated connectionId + modelId as the exact provider route. That is not true in this repository: a connection keeps its ID when its endpoint, API key/OAuth material, or custom request-header credential is replaced. A same-ID next turn could therefore replay signed, redacted, or encrypted reasoning created by another relay or account.

A second versioning defect came from changing continuation admission semantics without changing PROVIDER_REPLAY_PROJECTION_VERSION. The new digest included target route state while persisted v1 claims had been produced without it. Rejection was safe, but the durable protocol version no longer described the bytes it authenticated.

Both findings have one owner-level correction: freeze the provider-state identity before provider dispatch, persist it once on the existing Run route-provenance record, and use that same identity in replay projection and durable continuation admission. RuntimeEvent remains the canonical transcript/execution ledger; event.runId joins to the Run-opening provenance needed to decide whether opaque provider state may cross the current boundary.

Lifecycle and replay exits

flowchart TD
  P["Host RuntimePolicy authority<br/>connection + endpoint + credential versions"] --> I["Resolve providerStateIdentity<br/>opaque SHA-256, no secrets"]
  I --> R["Create AgentRunHeader before dispatch<br/>freeze identity + model"]
  R --> B["Build backend in the same activation gate"]
  P --> B2["Resolve latest provider target again"]
  B2 --> B
  B --> C{"Run identity equals<br/>resolved backend identity?"}
  C -- no --> X["Reject before provider dispatch"]
  C -- yes --> D["Provider stream"]
  D --> A["ModelAdapter<br/>ordered reasoning part boundaries"]
  A --> E["RuntimeEvent ledger<br/>thinking, text, tools, results"]

  E --> J["Replay projection"]
  R --> J
  T["Current target<br/>providerStateIdentity + modelId"] --> J
  J --> G{"Source Run identity<br/>exactly matches target?"}
  G -- yes --> H["Admit provider-owned reasoning"]
  G -- no or missing --> O["Omit provider-owned reasoning only"]
  J --> K["Always preserve portable text and tool evidence"]

  H --> N["Normal next turn"]
  O --> N
  K --> N
  H --> Q["Continuation admission v2<br/>digest target identity + ordered items"]
  O --> Q
  K --> Q
  H --> M["Compaction / overflow / durable reload"]
  O --> M
  K --> M
  Q --> V{"Execution rebuilds same v2 digest?"}
  V -- no --> X
  V -- yes --> W["Durable claim + continuation-start"]
  N --> S["AiSdkBackend materializer"]
  M --> S
  W --> S
  S --> Z["ModelAdapter + existing AI SDK request converter"]
  Z --> Y["Provider request"]

  L["Persisted projection v1"] --> L2["Readable after schema migration"]
  L2 --> L3["Explicit unsupported-version rejection"]
Loading

The Host computes identity; Runtime owns replay admission; ModelAdapter remains the only AI SDK boundary; RuntimeKernel and ToolRuntime continue to own execution, permissions, concurrency, and recovery. No provider-specific serializer, RuntimeEvent provenance copy, providerOptions side channel, SDK prepareStep/stopWhen loop, or second provider-state authority was added.

The broader Run-header retirement/migration remains tracked in #4311 and #4283. This PR extends the current Run-opening provenance seam by one opaque identity because that is the smallest durable fact that can authenticate provider-owned replay today.

Compatibility and risk

  • Legacy Run headers without providerStateIdentity continue to decode; only provider-owned reasoning is omitted.
  • SQLite runtime schema v15 preserves existing v1 continuation rows and admits v2 rows. Readers accept v1/v2, while current replay requires v2 and rejects unknown versions.
  • Endpoint, primary credential, and request-header credential replacement all change the identity even when connection ID and model remain unchanged.
  • A policy change after Run/continuation admission is caught again during real Host backend construction before dispatch.
  • Existing OpenAI Codex V3 checkpoint connection/model semantics remain separate and unchanged; this PR does not conflate that durable checkpoint contract with RuntimeEvent reasoning admission.

Commit structure

The original provider-contract and recovery series remains independently reviewable. The two review corrections added here are:

  1. feat(runtime): read provider replay projection v2
  2. fix(runtime): bind reasoning replay to provider state

The first is compatibility-only: read v1/v2, migrate storage, reject unknown versions. The second switches current admission to v2 and introduces the single Host-owned identity.

Verification

  • Observed RED then GREEN for same connectionId + modelId but different provider-state identities: Anthropic reasoning/signature is removed while portable text remains.
  • Verified real RuntimePolicy mutations: endpoint move, API-key rotation, and request-header credential addition each change providerStateIdentity.
  • Verified Host backend creation rejects a Run identity that no longer matches the resolved provider target.
  • Verified real SessionManager.resumeSafeBoundaryContinuation() composition carries the same identity through source Run headers, v2 plan/digest, target claim/Run, backend input, and the following normal Run.
  • Verified persisted v1 continuation rows survive SQLite schema migration; v2 rows are accepted; unknown versions are rejected; v1 replay is explicitly unsupported under the v2 projection.
  • Exact-head focused suites: 332 tests passed across AiSdkBackend/provider wire composition, continuation planning/lineage/resume, SessionManager continuation dispatch, SQLite runtime authority, Host connection effects, and Host backend creation.
  • Typecheck passed for @maka/core, @maka/runtime, @maka/storage, and @maka/runtime-host.
  • Biome check passed on all 27 affected files; git diff --check passed.
  • package-lock.json is unchanged.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Codex implemented and verified the runtime contract using debug, TDD, adversarial review-feedback, and simplification-audit workflows. Material commits include a Generated-by: Codex trailer.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

@github-actions github-actions Bot added the effort/L Under 1000 readable lines label Aug 30, 2026
@Astro-Han
Astro-Han force-pushed the fix/reasoning-replay-provenance branch 4 times, most recently from a8d4bcb to 1feb2ef Compare August 31, 2026 01:54
@github-actions github-actions Bot added effort/XL Over 1000 readable lines and removed effort/L Under 1000 readable lines labels Aug 31, 2026
@Astro-Han
Astro-Han marked this pull request as ready for review August 31, 2026 09:15
Requires the source-route replay gate established by the preceding commit.

Generated-by: Codex
Requires the source-route replay gate established by the first commit in this series.

Generated-by: Codex

@jackwener jackwener left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the latest head 1715b59c2. No P0 or P1 in the logic — approving the content. No inline comments. Two gate caveats below, and the second one matters for how much this approval is worth.

The admission model is correctly fail-closed, which is the property that counts here. compatibleProviderReasoningReplayEventIds starts from an empty set (plus the current run) and only adds runs that match on both llmConnectionId and modelId. Reasoning from a different connection or a different model is therefore never admitted. Crucially, when targetConnectionId or runHeaders is missing the loop is skipped entirely, so absent inputs produce a smaller allowlist rather than a wider one — missing data cannot silently widen replay. That is the right direction, and it is the opposite of the "optional guard silently inert" pattern I flagged on #4300.

The restriction is also correctly scoped. admitProviderReasoningReplayItems filters only kind === 'thinking', so portable transcript content and tool evidence pass through untouched. It gates provider-owned reasoning specifically rather than trimming history generally.

A note on reading this one commit-by-commit. Intermediate commit 6583c6ebd carries providerReasoningReplayEventIds?: ReadonlySet<string> with a !== undefined escape, which would have been worth flagging. That does not survive to the head: model-history.ts:110 declares it required, and the filter has no undefined branch. I reviewed the head. This is the second of your PRs where an intermediate commit shows a weaker form than the final tree, so it is worth knowing that reviewing by commit here produces findings that no longer exist.

Gate caveat 1: label was still queued; test and windows_recovery are green.

Gate caveat 2, and the more important one: GitHub reports this as CONFLICTING. I confirmed it — merging into current main conflicts, and the single conflicted file is packages/runtime/src/ai-sdk-backend.ts. That is the file carrying the replay-admission wiring I just reviewed: the compatibleProviderReasoningReplayEventIds call site and the threading of the resulting set through the replay and compaction paths. So this approval is bound to this tree, not to whatever comes out of the conflict resolution. Resolving it involves choosing between two versions of security-relevant wiring, which is a judgement call for you rather than something I should do to your branch unasked — but I am happy to rebase it if you want that. Either way the admission wiring is worth re-confirming on the rebased head, and I will re-review on request.

Automated review notice: This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.

@ARE404 ARE404 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving fix(runtime): gate provider reasoning replay by source model at head 1715b59con the code; noting merge is currently blocked on the branch's DIRTY/CONFLICTING state (GitHub reports mergeable: CONFLICTING) so it must be rebased onto current main before it can merge.

I reviewed the gating logic (compatibleProviderReasoningReplayEventIds / admitProviderReasoningReplayItems) and the digest binding:

  • Reasoning-replay is now admitted only when the source run's llmConnectionId+modelId match the target route (or the reasoning is from the current run), which correctly prevents a provider's reasoning being replayed/trusted under a different model connection. The providerReplayDigest binding the admission route prevents a plan built for one model being silently replayed for another.
  • test is green on this exact head (run 33374629392, 24m23s); 0 unresolved review threads.
  • One [P2] note is inline (continuation-replay.ts:129) about PROVIDER_REPLAY_PROJECTION_VERSION staying at 1 despite the changed plan/digest semantics — verify the replay plan isn't durably persisted across upgrades, or bump the version with the change.

No P0–P1.

简体中文

批准 fix(runtime): gate provider reasoning replay by source model,head 1715b59c——基于代码批准;同时注明合并目前被 DIRTY/CONFLICTING 状态阻塞(GitHub 报 mergeable: CONFLICTING),必须先 rebase 到当前 main 才能合。
审查了 gating 逻辑(compatibleProviderReasoningReplayEventIds/admitProviderReasoningReplayItems)与摘要绑定:reasoning-replay 现在仅当源 run 的 llmConnectionId+modelId 与目标路由一致(或 reasoning 来自当前 run)时才被采纳,正确防止某 provider 的 reasoning 在另一模型连接下被重放/信任;providerReplayDigest 绑定 admission 路由,防止为模型 A 生成的 plan 被静默用于模型 B。test 在 exact head 绿(run 33374629392,24m23s);0 未解决线程。行内一条 [P2](continuation-replay.ts:129):plan/digest 语义变了但 PROVIDER_REPLAY_PROJECTION_VERSION 仍是 1——确认 replay plan 不会跨升级持久化,否则应随语义变更 bump 版本。无 P0–P1。

providerProjectionVersion: input.providerProjectionVersion,
boundary: createRuntimeBoundaryCursor(boundaries),
providerReplayDigest: digestProviderReplay(input.providerProjectionVersion, providerItems),
providerReplayDigest: digestProviderReplayAdmission({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] The providerReplayDigest (and the plan protocol shape) now bind the admission route (targetConnectionId, targetModelId), i.e. replay-plan/digest semantics changed, but PROVIDER_REPLAY_PROJECTION_VERSION stays 1 (model-history.ts:76). If a v1 replay plan/digest is ever persisted across an upgrade (e.g. in durable resume/continuation state), the old digest was computed without the route, so a persisted v1 plan will not match the new digestProviderReplayAdmission — either rejected (safe) or misread. Please confirm whether the replay plan is durable and, if so, bump the projection/replay-version alongside this semantics change so a stored plan can't be mis-decoded. Non-blocking for merge once the conflict/rebase is handled, but worth settling.

@Astro-Han
Astro-Han force-pushed the fix/reasoning-replay-provenance branch from 1715b59 to 89a0646 Compare August 31, 2026 09:57

@zhiiw zhiiw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at exact head 1715b59c (verified unchanged at review time; test 24m23s and windows_recovery completed/success on this head).

Not approving yet: the PR is CONFLICTING against current main. git merge-tree shows the conflict is confined to packages/runtime/src/ai-sdk-backend.ts — exactly the file where this PR and the now-merged #4287 both rewired the tool-result path. That rebase is semantic, not textual, so the resulting head needs fresh verification; approve then and this becomes a fast review, because the content below is already verified.

What I verified on this head:

  • The reasoning-replay gate is fail-closed and route-exact: compatibleProviderReasoningReplayEventIds admits historical thinking events only when the source run's header matches the target on immutable llmConnectionId + modelId (current-run events are same-route by construction); with no target connection id, nothing historical qualifies. admitProviderReasoningReplayItems filters only thinking items, so text, tool calls, and tool results survive a rejection.
  • The continuation digest now authenticates the route, not just the items: digestProviderReplayAdmission binds provider_replay_admission_v1 to the target connection/model plus the ordered admitted items — closing the planned-for-B/executed-on-C hole where two incompatible targets produced the same item digest.
  • Codex checkpoints bind immutable identity: connectionSlug is gone from openai_codex_remote_v2 in favor of connectionId, and the validator requires it — legacy slug-bound checkpoints fail validation and fall back to the text-summary/raw-history path rather than replaying opaque provider state.
  • Producer side is per-part now: ModelAdapter lowers every SDK reasoning start to one provider-neutral part boundary, Anthropic redactedData rides providerOptions, and the Copilot route throws unless the plaintext contract is selected.

Executed on a real Windows machine at this head: clean forced rebuild, then 568/568 across model-adapter, history-compact-checkpoint, continuation-replay, runtime-resume, runtime-continuation, ai-sdk-backend, session-manager, overflow-reactive-recovery, and computer-use-provider-protocol.


Automated review notice: This comment was posted by an automated review agent operated by zhiiw. It is not an independent human review and does not replace one.

简体中文

暂不批:PR 对当前 main 是 CONFLICTING——merge-tree 显示冲突集中在 ai-sdk-backend.ts,正是本 PR 与已合并的 #4287 同时改写的工具结果路径。这个 rebase 是语义性的,结果 head 需要重新验证;内容本身我已验完(见下),rebase 后重审会很快。机制核实:推理重放门禁 fail-closed 且按不可变 connectionId+modelId 精确匹配(无目标连接则历史一律不放行),只过滤 thinking 项;continuation digest 现在认证目标路由+有序准入项(堵住 B 计划 C 执行的洞);Codex checkpoint 从可变 slug 改绑不可变 connectionId,旧 slug 形校验失败回落文本恢复路径;生产侧 ModelAdapter 把每个 SDK reasoning start 降为中立 part 边界,Anthropic redactedData 走 providerOptions,Copilot 非 plaintext 契约直接抛。本机真 Windows 干净重建后 568/568。

@hqhq1025 hqhq1025 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 89a06464e898002325db8668325d0dca04a7237a. One P1 blocks approval: the new provider-reasoning admission key does not change when the connection's endpoint or authentication material is replaced in place. The rebase onto current main is otherwise clean, and the reasoning/compaction integration with the durable Tool Result projection passed local validation.

Validation: clean npm ci, npm run build:test, full workspace typecheck, full Runtime suite (3090 passed / 13 skipped), focused Host compaction tests, changed-file Biome, ASF headers, and git diff --check. One unrelated managed-sandbox Host test failed in this Linux container; the changed portion of that file is confined to the Codex compaction test, which passed. Hosted test was queued and windows_recovery was still running at publication time.

Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.

Comment thread packages/runtime/src/model-history.ts Outdated
const compatibleRunIds = new Set(currentRunId ? [currentRunId] : []);
if (targetConnectionId && runHeaders) {
for (const run of runHeaders) {
if (run.llmConnectionId === targetConnectionId && run.modelId === targetModelId) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Include the connection generation in the replay boundary. connectionId + modelId is not the exact provider route in this repository: ConnectionCatalogDocumentOwner.update preserves previous.connectionId while replacing baseUrl, and the credential vault can rotate the API key, OAuth token, or request headers under that same ID. The next backend resolves the latest endpoint and credentials, but prior AgentRunHeader records contain only this unchanged ID/model, so this branch admits provider-owned signed/redacted/encrypted reasoning produced by the old relay or account and sends it to the new one. I reproduced the admission directly on this head, and the production connection-effect tests confirm endpoint-plus-secret replacement in place while retaining the ID. Persist/freeze a provider-state identity that changes with endpoint/auth ownership (or rotate the connection identity on those edits), and add a regression covering: prior run on relay/account A, in-place connection edit to B, then a next turn that must omit A's provider reasoning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/XL Over 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants