feat: add observable commentary and collapsible work logs - #4270
feat: add observable commentary and collapsible work logs#4270hqhq1025 wants to merge 10 commits into
Conversation
Normalize model-authored progress and final-answer text across provider, persistence, Runtime Host, CLI, TUI, and UI boundaries. Infer phases for providers without native support and preserve explicit OpenAI Responses phases. Generated-by: OpenAI Codex
|
Closing this draft for now while local end-to-end validation is completed. The branch remains available and the proposal continues in Discussion #4268. |
Astro-Han
left a comment
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks for building this out end to end. I reviewed e39eeb3, and the implementation helped make the protocol differences and edge cases concrete.
After discussing the product boundary, I think we can solve the user problem with a much smaller architecture:
flowchart LR
A[Responses / Chat Completions / Anthropic] -->|ordinary assistant text| B[Existing text stream and timeline]
B --> C{Runtime Turn}
C -->|active| D[Progress visible and interruptible]
C -->|terminal| E[Last settled text is the final reply]
F[Responses native phase] -->|provider metadata only| G[Adapter and durable replay]
What users need is straightforward: while the Turn is active, the model should occasionally explain its progress; when the Turn finishes, it should give a final reply. All three protocols can already do that through ordinary assistant text, so the common fix can be the progress-update prompt plus the existing text pipeline.
Only Responses has a native phase. Keeping it as provider-owned replay metadata avoids inventing an equivalent fact for Chat Completions and Anthropic. It also avoids having a normalized top-level phase and the original provider phase disagree, or having to propagate the new field through storage, Runtime Host, CLI, and UI before there is a concrete presentation requirement.
My suggestion is therefore to keep:
- the progress-update prompt and cross-provider behavior tests;
- lossless Responses phase/item-boundary preservation at the adapter and replay boundary.
And defer the provider-neutral stored phase, phase inference, Runtime Host epoch change, phase-aware consumers, and commentary-only continuation.
I left one P1 and two P2 inline for the concrete risks in the current implementation. The broader work was still useful—it showed us exactly where the complexity would spread. I am very open to pushback if there is a current product consumer or provider constraint that needs a stable cross-provider phase.
AI-assisted review using OpenAI Codex; I verified the exact head, provider paths, persistence/replay boundary, and continuation behavior.
中文架构建议
建议先用更小的架构解决当前问题：
- 三种协议都通过普通 assistant text 输出工作进展；
- Runtime Turn 的进行中和终态负责区分“进展”和“最终回复”；
- Responses 原生
phase只作为 provider metadata 无损保存和回放； - 暂不为其他协议推断 phase，也不扩展 Runtime Host、CLI 和 UI 协议。
这样已经能让用户看到进展并及时打断，同时避免出现两个 phase 权威不一致的问题。以后有独立样式或 answer-only export 等具体需求时，再增加 Maka 自有语义也不迟。
| ts: number; | ||
| text: string; | ||
| /** User-visible role of this model-authored text. */ | ||
| phase?: AssistantTextPhase; |
There was a problem hiding this comment.
P1 — Category ② (reasonable provider irregularity): keep one durable authority for assistant phase
This adds a second persisted semantic for text that already retains its native Responses phase in providerOptions.
On a reachable Responses path where the provider labels text as final_answer but the same step later returns a client tool call, normalizedAssistantTextPhase() stores message.phase = "commentary" while providerOptions.openai.phase remains "final_answer". The UI, CLI, copy behavior, and child summaries consume the new top-level field, while provider replay consumes providerOptions. The same durable assistant row therefore has two contradictory identities, and reconstruction depends on which consumer reads it.
Could we avoid persisting a second top-level phase authority? My suggested boundary is:
- keep the original Responses phase losslessly in provider metadata;
- preserve native text-item boundaries where replay requires them;
- let Runtime's active/terminal Turn lifecycle and chronological text position own the product distinction between progress and the settled reply.
That would also remove the need to propagate inferred phase through AssistantMessage, RuntimeEvent, Runtime Host continuity, and the public wire protocol.
There was a problem hiding this comment.
Thanks, this distinction is now explicit in the implementation and PR description. AssistantMessage.phase is the sole Maka product authority consumed by UI, CLI, copy, summaries, and continuity. Provider options remain opaque replay evidence and are not consulted for product classification after normalization. A malformed native Responses step can therefore replay its original provider payload while Maka consistently treats text followed by tool work as commentary. The adapter-capability gate and the new reconnect/read-model tests make that ownership boundary explicit.
| // Responses supplies phase directly. Chat Completions and Anthropic Messages | ||
| // do not, so their response topology is the portable signal: text before a | ||
| // client tool call is progress, and completed text-only output is terminal. | ||
| if (input.hasClientToolCall) return 'commentary'; |
There was a problem hiding this comment.
P2 — Category ① (supported provider-tool path): phase-less text around server-executed tools is classified as final
This inference only considers client-executed tool calls. Maka also supports provider-executed tools, including Anthropic and OpenAI native web search; those events follow the provider-tool-input / provider-tool-result path and never set stepHasClientToolCall.
Consequently, phase-less assistant text emitted before a provider-executed tool reaches flushStep(completedStep = true) with hasClientToolCall = false and is labeled final_answer, even though tool activity followed and the work was still in progress. That value can then drive copy behavior, CLI output, memory extraction, or child summaries as though it were the settled answer.
If normalized phase remains, the topology would need to account for every supported form of tool activity and preserve the relevant text/tool boundaries. I think the smaller fix is to avoid cross-provider phase inference entirely and let the Turn lifecycle plus chronology own this distinction.
There was a problem hiding this comment.
Fixed in b1b8b1a2c. Pending assistant content is now flushed as commentary when provider tool input or a provider-executed call begins, then the assistant message identity rotates so post-tool text can settle independently as the final answer. The regression test covers commentary -> provider WebSearch -> final_answer, and the existing OpenAI/Anthropic native web-search tests still pass.
| currentStepMessageId = this.newId(); | ||
| continue agentLoop; | ||
| } | ||
| throw { |
There was a problem hiding this comment.
P2 — Category ② (reasonable model noncompliance): do not add a paid retry and hard failure without demonstrated need
When a provider completes a response with explicit commentary and no tool call, this changes the existing terminal behavior in two ways: Maka silently sends another provider request, and a repeated commentary response becomes a non-retryable Turn failure.
That changes request count, cost, latency, and the final Turn status for a recoverable model-output problem. The evidence in Discussion #4268 demonstrates that models currently fail to produce progress updates; it does not demonstrate that commentary-only completion is occurring in production or that converting it into a failed Turn is the desired product policy.
Could we remove the bounded continuation and throw from this slice? The prompt can instruct the model not to stop after a progress update. If real telemetry later shows premature commentary-only completion, we can design recovery from that concrete failure mode without coupling it to the basic observability feature.
There was a problem hiding this comment.
I kept the bounded continuation, but made its scope and cost explicit. It applies only when the provider explicitly labels the terminal text as commentary, because final-only CLI/copy/child-summary consumers cannot safely treat that semantic as a successful answer. Maka performs at most one additional request; a repeated violation becomes a visible non-retryable failure instead of silently exporting commentary as the answer. Phase-less inferred commentary does not trigger this terminal guard.
Add a provider-neutral ProgressUpdate transport for phase-less model APIs, hide its implementation-detail activity, and present commentary, reasoning, and tool work as a Codex-style log that folds when the final answer begins. Preserve native Responses phases and keep CLI final-output selection phase-aware. Generated-by: OpenAI Codex
…y-phase # Conflicts: # packages/runtime-host/src/protocol/index.ts
hqhq1025
changed the title
UI comparisonThe original baseline comparison remains at the same persisted Turn, application state, content, zoom, and Flat baseline
Completed work collapsed
Current revisionThe current
|
hqhq1025
left a comment
hqhq1025
left a comment
There was a problem hiding this comment.
Reviewed exact head b1b8b1a2cd0154b82770c0c4fdb9b2c7901f0dcf. This remains NO-GO due to the two inline P1 findings: the compatibility epoch collides with a different current-main wire change, and the new raw disclosure buttons introduce an Astryx blocker that already fails the hosted test job. The provider-executed tool split and the focused phase, replay, Runtime Host, UI, and CLI paths otherwise passed local review.
Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.
| // Increment when the same protocol version no longer guarantees safe Client-Host | ||
| // interoperability. Mismatches are rejected before domain commands are admitted. | ||
| export const RUNTIME_HOST_COMPATIBILITY_EPOCH = 78 as const; | ||
| export const RUNTIME_HOST_COMPATIBILITY_EPOCH = 79 as const; |
There was a problem hiding this comment.
P1 — allocate a fresh compatibility epoch after rebasing
Current main is already at epoch 80 and assigns epoch 79 to the queued Skill-outcome wire change. This branch assigns 79 to assistant text phases. A conflict resolution that keeps this value would allow peers with different closed wire shapes to pass the same epoch handshake. Preserve main's 79/80 history and assign this change the next epoch (currently 81), with the protocol test updated accordingly.
| data-collapsible={props.collapsed ? 'true' : undefined} | ||
| > | ||
| {props.collapsed && ( | ||
| <button |
There was a problem hiding this comment.
P1 — use the Astryx disclosure/button primitive for both new toggles
This file introduces raw <button> controls here and again in ProcessingBlock (line 1435). The repository's generated Astryx inventory classifies that as a blocker: npm run astryx:surface-inventory fails on this head, and regenerating changes this file from aligned to raw <button (API Use-the-System) | blocker. Replace both controls with the available Astryx Button/Collapsible boundary and regenerate the inventory rather than committing the blocker state.
|
Left detailed feedback on #4268 (discussion) rather than here, since the questions are about the shape rather than the code: Not blocking with a formal request for changes yet — I'd rather settle the shape on the discussion first. The collapse hierarchy and the Codex import fix look good to me independently of that. |
|
Updated the branch to The implementation now follows the smaller architecture proposed in review and Discussion #4268:
The previous CI failure was in Desktop E2E. The stable regression was the accessibility scenario still trying to reach a tool result through the retired flat layout. It now follows the real keyboard path through the collapsed outer work log and inner processing summary before opening the tool result. The second reported transcript-width failure passed after syncing current Validation on this revision:
A single long local Desktop E2E run passed 90 tests before an Electron teardown timeout caused cascading closed-page/fixture-start failures. Every earliest and representative failure from that cascade passed in isolation, including all tests related to this PR. Hosted CI has been retriggered by the push. |
|
Hosted CI is now green on exact head The first attempt reached Storybook smoke after Desktop E2E had passed, then intermittently missed the post-navigation focus assertion in the unchanged Attempt 2 passed the complete workflow, including:
Review has been re-requested from @Astro-Han and @M4n5ter. The PR still requires an independent human approval before it is merge-ready. |
2 participants
Summary
thinking/tools/textorder for provider-executed tool steps so live and replayed transcripts agreeRefs #4268
Architecture
All supported protocols can emit ordinary assistant text around tool calls. Maka therefore uses the existing text pipeline for progress updates and final replies:
flowchart LR A[Responses / Chat Completions / Anthropic] -->|ordinary assistant text| B[Existing text stream and timeline] B --> C{Runtime Turn} C -->|active| D[Progress remains visible and interruptible] C -->|terminal| E[Chronology selects the settled final reply] F[Responses native item metadata] -->|adapter and replay only| G[Provider options and item boundaries] H[Runtime tool events] --> I[Derived activity summaries]This revision deliberately does not add:
AssistantTextPhaseProgressUpdatecompatibility toolOpenAI Responses metadata remains provider-owned replay evidence. Product presentation uses the existing Turn terminal state and ordered timeline.
User experience
Final-reply rule
For the latest assistant segment of a completed Turn:
The same selector controls the visible final answer, copied text, and whether Copy is enabled.
Verification
npm run typechecknpm run lintnpm run format:checknpm run check:asf-headersnpm run astryx:surface-inventorynpm run check:renderer-architecture -- --base origin/mainorigin/main(98fc50508) and re-ran the checks aboveThe first post-merge typecheck read stale
@maka/core/distdeclarations from before the new main-branch attachment and/copyAPIs. Rebuilding@maka/corerefreshed the workspace declarations; the complete typecheck then passed.AI use
Select exactly one:
Tool(s) and scope: OpenAI Codex investigated the provider/runtime/UI contracts, implemented and tested the progress and work-log behavior, incorporated reviewer feedback, performed adversarial self-review, and prepared this revision. The human contributor remains responsible for review and submission.
Checklist
mainDoes this PR entail a change in behavior?