Skip to content

rendering.md

Latest commit

 

History

History
919 lines (717 loc) · 49.2 KB

rendering.md

File metadata and controls

919 lines (717 loc) · 49.2 KB
Raw

Rendering Pipeline — LLM Response → Terminal Display

Status: production
Last Updated: 2026-10-10

3D terminal rendering note: Three.js WebGPU via @opentui/three has known issues on this platform (see §14). The working 3D pipeline uses GPU compute shaders + Sixel output via Python wgpu — see experiments_history/2026-07-28_rotating-cube-3d/test_cube.py.

1. Overview

The rendering pipeline has two distinct paths:

Path Target Framework Key Libraries
TUI (OpenTUI) CLI terminal SolidJS + OpenTUI custom renderables @opentui/core, @opentui/solid, @opentui/three
Web/Desktop HTML DOM SolidJS/React + marked + DOMPurify marked, DOMPurify, remend

The TUI path is the primary rendering pipeline for the interactive CLI. This document traces the end-to-end flow from LLM response → terminal display.

2. End-to-End Data Flow

Text paint graph — verified against the code 2026-09-23. Every node names the owner of the step, so a change can be traced from the symptom back to one file. The two annotated numbers are MEASURED, not estimated (isolated replay of a real 843-delta SSE stream; see plans/to_be_confirmed/2026-09-22_reasoning-stream-render-stability.md).

flowchart TD
  SSE["SSE chat.completion.chunk<br/>reasoning_content · content"]

  subgraph server["server · session"]
    PROC["processor.ts:816-827<br/>a delta APPENDS to part.text"]
    BUS["bus publish — typed PubSub"]
  end

  subgraph ingest["TUI · ingest"]
    SDKS["sdk.tsx SSE stream → batch queue"]
    SYNC["context/sync.tsx<br/>deltaBuffer → setStore<br/>DELTA_DEBOUNCE_MS = 25"]
  end

  subgraph route["TUI · routes/session/index.tsx"]
    MEMO["createMemo → store.part[messageID]"]
    AM["AssistantMessage → PART_MAPPING<br/>text · reasoning · tool · file"]
    TP["TextPart"]
    RP["ReasoningPart<br/>THINKING_DISPLAY_MAX = 6_000"]
    RT["RichText — the ONE prose entry<br/>splitTextSegments → markdown · mermaid"]
  end

  subgraph core["@opentui/core · MarkdownRenderable"]
    MD["markdown element<br/>internalBlockMode = coalesced (default)<br/>5.3 ms/frame — top-level costs 8.6"]
    PARSE["markdown-parser.ts:parseMarkdownIncremental<br/>tokens matched from OFFSET 0"]
    BLK["block dispatch"]
    TXT["TextRenderable"]
    TAB["TextTableRenderable"]
    COD["CodeRenderable — fenced code"]
  end

  subgraph hl["@opentui/core · highlight"]
    HL["TreeSitterClient.highlightOnce"]
    LAST["_lastHighlights — the stored parse"]
    PREV["updateStreamingPreview<br/>paints FROM _lastHighlights,<br/>else from the caller's text"]
    QW["quietHighlightMs — per call,<br/>default 0 = synchronous"]
    SNAP["_highlightSnapshotId<br/>a late result is discarded"]
  end

  subgraph paint["layout & paint"]
    YOGA["yoga layout — Renderable.ts:1830"]
    LOOP["renderer.ts:4740 frame loop<br/>frame admission + diff"]
    BUF["OptimizedBuffer → ANSI / DEC 2026"]
  end

  subgraph rwin["reasoning window"]
    RV["text-segments.ts:reasoningView<br/>window quantised to 1024, counter ABOVE the markdown"]
  end

  SSE --> PROC --> BUS --> SDKS --> SYNC --> MEMO --> AM
  AM --> TP --> RT
  AM --> RP --> RV --> RT
  RT -->|"markdown segment"| MD
  RT -->|"mermaid segment"| MER["MediaMermaid"]
  MD --> PARSE --> BLK
  BLK --> TXT
  BLK --> TAB
  BLK --> COD
  COD --> HL --> LAST --> PREV
  QW -.->|"defers the parse"| HL
  SNAP -.->|"discards a stale result"| PREV
  PREV -->|"styled chunks"| COD
  TXT --> YOGA
  TAB --> YOGA
  COD --> YOGA
  MER --> YOGA
  YOGA --> LOOP --> BUF
Loading

Two properties this graph exists to make obvious, because both cost a defect:

  • The counter lives OUTSIDE the markdown. reasoningView keeps the markdown body append-only between window rotations; with the counter as the first token, parseMarkdownIncremental's offset-0 prefix match failed on EVERY delta and the whole 6 000-character window was re-lexed.
  • coalesced is the default block mode, and the prose path returns to it (2026-09-23). top-level renders each block as its own renderable — better structure, +63 % cost per frame on the replay — and that cost multiplies by the number of blocks a session holds.

The ASCII sketch below was checked against the code on 2026-09-23 and is STALE in three places: TextPart no longer renders through <code filetype="markdown">, ReasoningPart is not a <code> element at all, and the highlight / block-mode half of the path is missing entirely. The graph above is the accurate one; this sketch is kept only until someone deletes it.

LLM Provider Response
  │
  ▼
Session Processor (processor.ts)
  └─ text-delta event → session.updatePartDelta()
       │
       ▼
Bus (bus/index.ts)
  └─ publish via typed PubSub + GlobalBus.emit()
       │
       ▼
SDK SSE Stream (sdk.tsx)
  └─ event → handleEvent → 16ms batch queue → emitter.emit("event", event)
       │
       ▼
Sync Store (sync.tsx)
  └─ event.subscribe → switch(event.type)
       ├─ "message.part.delta" (sync.tsx):
       │     deltaBuffer → Binary.search → produce → setStore
       ├─ "message.part.updated" (sync.tsx):
       │     Binary.search → reconcile → setStore
       └─ "message.part.removed" (sync.tsx)
            │
            ▼
SolidJS Reactivity
  └─ createMemo → store.part[messageID]
       │
       ▼
Session Route (index.tsx)
  └─ <For each={messagesList()}>
       ├─ UserMessage (index.tsx): plain text + file badges
       └─ AssistantMessage (index.tsx)
            └─ <For each={props.parts}>
                 ├─ TextPart (index.tsx):
                 │     splitTextSegments → <code filetype="markdown"> or <markdown>
                 │     └─ tree-sitter syntax highlighting + streaming fallback
                 ├─ ReasoningPart (index.tsx):
                 │     <code filetype="markdown"> dimmed, [REDACTED] stripped
                 └─ ToolPart (index.tsx):
                       ├─ InlineTool (index.tsx): compact icon+text
                       └─ BlockTool (index.tsx): bordered, expandable
                            └─ <diff> for edits, <code> for writes, nested media
                              │
                              ▼
OpenTUI Core (@opentui/core)
  └─ CliRenderer → targetFps:30 → render loop
       └─ ScrollBoxRenderable for scrollable content (stickyScroll="bottom")
            └─ MediaImage → OpenTUI <image> (PixelBuffer)
                 └─ Kitty / Sixel / half-block escape codes
                      │
                      ▼
Terminal Output

3. Stage-by-Stage Details

Stage A: Provider Streaming → Session Processor

File: packages/opencode/src/session/processor.ts

Event Type Handler What it does
text-start case "text-start" Creates ctx.currentText part, calls session.updatePart() → publishes message.part.updated
text-delta case "text-delta" Appends text to StringBuilder, calls session.updatePartDelta() → publishes message.part.delta via bus
text-end case "text-end" Finalizes text part with ctx.textBuilder.toString(), triggers experimental.text.complete plugin hook
reasoning-start case "reasoning-start" Creates reasoning part in ctx.reasoningMap
reasoning-delta case "reasoning-delta" Appends to StringBuilder, calls updatePartDelta()
reasoning-end case "reasoning-end" Finalizes reasoning part

Stage B: Session → Event Bus

File: packages/opencode/src/session/session.ts

Method What it publishes Event Type
updatePart SyncEvent.run(MessageV2.Event.PartUpdated, ...) — persists to DB + bus message.part.updated
updatePartDelta bus.publish(MessageV2.Event.PartDelta, input) — live streaming only (no DB write) message.part.delta

File: packages/opencode/src/bus/index.ts

  • publish(): Publishes event to typed PubSub AND wildcard PubSub AND GlobalBus.emit()
  • subscribe() / subscribeAll(): Stream-based subscriptions

Stage C: Event Bus → SDK Client → TUI

Files:

  • packages/opencode/src/cli/cmd/tui/context/sdk.tsx — SSE connection (startSSE())
  • packages/opencode/src/cli/cmd/tui/context/event.ts — Event subscription

Key behavior:

  1. sdk.tsx: startSSE() initiates SSE stream via sdk.global.event()
  2. sdk.tsx: Each event → handleEvent(event)
  3. sdk.tsx: Events queued with 16ms batch window, flushed in SolidJS batch() call
  4. event.ts: useEvent().subscribe() subscribes via sdk.event.on("event", ...), filters by workspace/directory

Stage D: TUI Sync Store → SolidJS Reactivity

File: packages/opencode/src/cli/cmd/tui/context/sync.tsx

The sync store receives events and updates the SolidJS store:

Event Type Store Mutation Notes
message.part.updated setStore("part", messageID, index, reconcile(...)) Flushes delta buffer for this messageID
message.part.delta setStore("part", messageID, produce(...)) Guarded by DELTA_SAFE_FIELDS: ["text", "output"]
message.updated Updates store.message[sessionID][index], evicts oldest if > 100 Eviction only if no active parts
message.removed Removes from store.message —
session.diff Stores store.session_diff —
session.deleted Cleans up all orphaned session-keyed stores Prevents RSS leak (was causing 1.18 GB Bun segfault on Windows)

Delta buffering (sync.tsx): If a delta event arrives before the part is created, it's buffered in deltaBuffer (Map<messageID, Map<partID, accumulatedText>). When the part arrives via part.updated, flushDeltaBuffer() is called to apply buffered content.

4. TUI Rendering Components

4a. App Entry Point

File: packages/opencode/src/cli/cmd/tui/app.tsx

Anchor What it does
targetFps: 30 rendering rate
createCliRenderer(cfg) + renderer.waitForThemeMode(400) initialize the OpenTUI renderer, resolve light/dark

Provider tree:

ErrorBoundary → Args → Exit → KV → Toast → Route → TuiConfig
  → SDK → Project → Sync → Theme → Local → Keybind → PromptStash
    → Dialog → Command → Frecency → PromptHistory → PromptRef
      → EditorContext → App

4b. Session Route — Main Display View

File: packages/opencode/src/cli/cmd/tui/routes/session/index.tsx Size: ~3.2k lines — the largest source file in the TUI tree

Anchor Component Role
addDefaultParsers(resolvedParsers) parser registration Registers the tree-sitter parsers used for syntax highlighting
root <box> layout Root flex container: messages + sidebar
<scrollbox> message container Scrollable, stickyScroll="bottom", vertical scrollbar on by default (session_chat_scrollbar)
jump-to-live chip scroll affordance When not at bottom: ↓ Live · N% · rows above (click / End / command palette) — see util/scroll-position.ts
<For each={messagesList()}> message loop Iterates over messages
UserMessage message renderer User messages with file badges, compaction markers
AssistantMessage message renderer Assistant messages with parts loop
PART_MAPPING part dispatch Maps part types to renderer components

4c. Part Rendering Components

RichText / TextPart (routes/session/index.tsx) — Primary text rendering

TextPart wraps RichText, which is the streaming text renderer:

  1. createMemo(() => splitTextSegments(props.content())) — splits on ```mermaid fences into TextSegment[], then <Index each={segments()}> keeps one renderable per segment, so a closed fence renders while the model keeps writing after it.
  2. Markdown segment → the OpenTUI <markdown> renderable with streaming, syntaxStyle (subtle variant when props.subtle), conceal, and theme fg/bg.
  3. Mermaid segment → <MediaMermaid source={…}>.

Incremental parsing and healing are no longer owned by this route. They moved into OpenTUI's markdown renderable: parseMarkdownIncremental in packages/opentui/packages/core/src/renderables/markdown-parser.ts reuses the stable token prefix (reuseCount) and always re-parses the last trailingUnstable = 2 tokens, which is what keeps mid-stream markdown from flickering. The web side has its own healer, packages/ui/src/components/markdown-stream.ts (marked + remend).

Historical: route-level markdown healing

Earlier versions closed unclosed formatting in the route before tree-sitter processing:

  • Closes **bold**, *italic*, ~~strikethrough~~, `inline code`
  • Operates on text outside code blocks (strips fenced/inline code before counting)
  • Drastically reduces zero-highlights events during streaming

ReasoningPart (routes/session/index.tsx)

  • Strips [REDACTED] from OpenRouter reasoning (encrypted reasoning payloads)
  • Renders as <code filetype="markdown"> with streaming=true, muted colors, dimmed border
  • Controlled by ctx.showThinking() toggle

ToolPart (routes/session/index.tsx)

Uses PART_MAPPING with tool-specific renderers:

Tool Handler (line) Display
bash 2108 Full output with expand/collapse (10 line threshold)
write 2175 Code content with line numbers + diagnostics
edit 2476 Diff view (<diff> renderable) — split/unified auto-detect
read 2217 File paths + inline attachment previews
grep 2271 Pattern + match count
glob 2206 Pattern + match count
webfetch 2282 URL + attachment previews
task 2406 Sub-agent progress, toolcalls, duration
question 2624 Questions + answers display
todowrite 2603 Todo list with status checkboxes

Inline vs Block tools:

  • InlineTool: Compact single-line display with icon, spinner, hover
  • BlockTool: Full-width bordered display with expand/collapse

5. MarkdownRenderable Internals (@opentui/core)

File: node_modules/@opentui/core/renderables/Markdown.d.ts (types)
Implementation: packages/opentui/packages/core/src/renderables/Markdown.ts (workspace source)
Status: the former npm patch (@opentui/core@0.4.3) is gone — the fixes live in the vendored fork

The TUI has two rendering paths for markdown, controlled by Flag.OPENCODE_MARKDOWN (default true):

Path Component Tokenizer Inline Formatting
New (default) <markdown> → MarkdownRenderable marked@17 (gfm: true) renderInlineContent() on marked tokens
Legacy <code filetype="markdown"> → CodeRenderable tree-sitter markdown grammar tree-sitter highlights.scm queries

5a. MarkdownRenderable Architecture

Assistant text content
  │
  ├── splitTextSegments(): mermaid fences only
  │   └── non-mermaid → { type: "markdown", text }
  │
  └── MarkdownRenderable.renderSelf()
        │
        ├── parseMarkdownIncremental(newContent, prevState, trailingUnstable)
        │     └── marked Lexer.lex(newContent, { gfm: true })
        │           → MarkedToken[] (heading, list, paragraph, table, code, etc.)
        │
        └── internalBlockMode?
              │
              ├── "coalesced" (default) ───────────────────────────────────┐
              │   └── buildRenderableTokens(tokens)                        │
              │         │                                                  │
              │         ├── shouldRenderSeparately(token)?                 │
              │         │     ├── code / table / blockquote / hr           │
              │         │     │   └── flush raw buffer → render separately │
              │         │     ├── heading / list (PATCHED)                 │
              │         │     │   └── flush raw buffer → render separately │
              │         │     └── paragraph / space / others               │
              │         │           └── accumulate into markdownRaw        │
              │         │                                                  │
              │         └── remaining raw → createMarkdownBlockToken()     │
              │               → type:"paragraph", tokens:[] (FLAT)         │
              │                                                  │
              └── "top-level" ──────────────────────────────────┤
                  └── buildTopLevelRenderBlocks(tokens)         │
                        └── each block keeps its token identity │
                                                                 │
                    ┌────────────────────────────────────────────┘
                    ▼
          createDefaultRenderable(token, index, nextToken)
                │
                ├── code     → createCodeRenderable()
                ├── blockquote → createBlockquoteRenderable()
                ├── list     → createListRenderable()
                │               └── per item → createListItemRenderable()
                │                     └── per child → createListChildRenderable()
                │                           → createMarkdownCodeRenderable(...)
                ├── hr       → createHorizontalRuleRenderable()
                ├── table    → createTableBlock() → TextTableRenderable
                └── heading / paragraph → createMarkdownCodeRenderable(raw, ..., initialStyledText)
                                              │
                                              └── CodeRenderable
                                                    │
                                                    ├── streaming: mirrors Markdown.streaming
                                                    ├── drawUnstyledText: true while streaming (progressive paint)
                                                    ├── initialStyledText? → rich first paint when available
                                                    └── tree-sitter async highlight (debounced 150ms)
                                                          └── paints unstyled immediately; styles catch up
                                                          └── failure → plain text stays visible

5b. Token Dispatch Table

shouldRenderSeparately(token) controls which block types get individual renderables vs being coalesced into raw text:

Token Type shouldRenderSeparately Render Path Inline Tokens Preserved?
code ✅ true CodeRenderable (syntax-highlighted) N/A (code block)
table ✅ true TextTableRenderable via marked token tree ✅ Yes — renderInlineContent per cell
blockquote ✅ true BoxRenderable with left border → CodeRenderable ❌ No — uses token.raw → tree-sitter
hr ✅ true BoxRenderable with top border N/A
heading ✅ true (patched) createMarkdownCodeRenderable(raw, ..., initialStyledText) ✅ Yes — marked inline tokens → createInitialStyledText
list ✅ true (patched) createListRenderable() → per-item createListChildRenderable() ✅ Yes — marked inline tokens in each paragraph/text child
paragraph ❌ false (coalesced) Accumulated into flat raw → single CodeRenderable ⚠️ Partial — via x.lexInline() fallback when streaming guard removed

5c. Coalescing Behavior and Impact

In buildRenderableTokens() (default coalesced mode), non-separate tokens are concatenated into a single raw string via markdownRaw += token.raw. A synthetic paragraph token with tokens: [] is created from the combined raw text. This:

  • Destroys all inline token structure — strong, em, codespan, link tokens are lost
  • Lists lose their hierarchical structure — markers, nesting, item boundaries are flattened
  • Headings lose their depth and inline tokens — the ### prefix and inline bold/italic become raw text

The coalesced raw string is passed to createMarkdownCodeRenderable() which creates a CodeRenderable with filetype: "markdown". This CodeRenderable depends on tree-sitter's markdown grammar to recover the formatting from the raw text.

5d. Streaming Guard (createInitialStyledText)

createInitialStyledText(token) generates pre-computed styled text from marked's inline tokenizer:

createInitialStyledText(token) {
    // streaming guard removed (PATCHED) — now runs for all content
    const chunks = [];
    if ("tokens" in token && Array.isArray(token.tokens)) {
        this.renderInlineContent(token.tokens, chunks);  // from marked tokens
    }
    if (chunks.length === 0 && "text" in token && typeof token.text === "string") {
        this.renderInlineContent(x.lexInline(token.text), chunks);  // inline re-parse
    }
    return chunks.length > 0 ? new StyledText(chunks) : undefined;
}

When initialStyledText is provided to createMarkdownCodeRenderable():

  • drawUnstyledText: true — CodeRenderable immediately renders the pre-computed styled text
  • Content is visible immediately without waiting for async tree-sitter highlighting
  • Inline formatting (**bold**, *italic*, `code`) is rendered correctly

When initialStyledText is undefined:

  • drawUnstyledText: false — CodeRenderable's _shouldRenderTextBuffer is set to false
  • Content is invisible until tree-sitter highlighting completes asynchronously
  • If tree-sitter highlighting fails, falls back to plain text

Pre-patch: The streaming guard (if (!this._streaming) return;) prevented initialStyledText from being generated for static (non-streaming) content, leaving all non-separately-rendered blocks dependent on tree-sitter for inline formatting.

5h. Tree-Sitter Overwrite Bug (Race Condition)

STALE in two places — measured 2026-09-23 (plans/to_be_confirmed/2026-09-22_reasoning-stream-render-stability.md T11, instrument packages/opentui/packages/core/src/renderables/__tests__/stream-replay-sources.test.ts). (1) The guard shown below is NOT in Code.ts: startHighlight overwrites the buffer unconditionally. (2) «tree-sitter never highlights inline formatting» is refuted: the tree ships a markdown_inline injection (lib/tree-sitter/default-parsers.ts:39-58). What actually alternated on the real stream was the list-marker colour and the conceal of backticks, because the content setter repainted the caller's styling over the stored parse on every delta. The fix is Code.ts:paintFromStoredHighlights, shared by the setter and updateStreamingPreview: one style source per frame. The text below is kept as history.

Even with initialStyledText correctly populated, a second issue occurs: tree-sitter's async startHighlight() unconditionally overwrites the styled text buffer when it completes, even when its highlights are less rich than initialStyledText.

Timeline:
  1. createInitialStyledText → chunks with markup.strong, markup.italic ✅
  2. CodeRenderable renders initialStyledText → user sees bold/italic ✅
  3. [async tree-sitter starts...]
  4. tree-sitter completes → highlights: [heading.1, list.marker] only
     → treeSitterToTextChunks → chunks WITHOUT markup.strong/markup.italic
     → textBuffer.setStyledText(chunks)  ← OVERWRITES rich text ❌
  5. User sees flat text — "bw" (black and white) ❌

Root cause: tree-sitter's markdown grammar highlights.scm query captures structural nodes (heading markers, list bullets, code fences) but not inline formatting (strong, em, codespan, del, link). When treeSitterToTextChunks() converts these structural highlights to styled text chunks, inline formatting like **bold** becomes plain text instead of markup.strong.

Fix: In CodeRenderable.startHighlight(), skip the textBuffer.setStyledText() overwrite when initialStyledText is available and filetype === "markdown":

// Before (index-xt9f071j.js:3385-3386):
const styledText = new StyledText(chunks);
this.textBuffer.setStyledText(styledText);

// After (PATCHED):
if (!(this._initialStyledText && filetype === "markdown")) {
    const styledText = new StyledText(chunks);
    this.textBuffer.setStyledText(styledText);
}

This preserves the richer inline formatting from marked's x.lexInline() tokenizer, which correctly handles strong, em, codespan, del, and link tokens.

5e. renderInlineContent — Inline Token Rendering

Maps marked inline token types to OpenTUI styled text chunks:

Marked Inline Token Styled Text Chunk(s) Concealed?
text default chunk —
strong markup.strong wrapper + child tokens ** markers concealed when conceal: true
em markup.italic wrapper + child tokens * markers concealed
codespan markup.raw chunk ` markers concealed
del markup.strikethrough wrapper ~~ markers concealed
link markup.link + markup.link.label + markup.link.url URL hidden when conceal: true

5f. Known Issues and Edge Cases

Issue Context Workaround
Inline formatting in coalesced paragraphs Coalesced paragraph tokens have tokens: [], so renderInlineContent falls through to x.lexInline(raw) x.lexInline fully handles inline formatting from raw text
Consecutive paragraphs without separation Multiple paragraphs without intervening separate blocks accumulate into one raw block Add an explicit flush on double-newline within buildRenderableTokens
List item paragraph margin List items use marginTop: /\n[ \t]*\n$/.test(item.raw) ? 1 : 0 — double-newline inside item adds paragraph break Controlled by marked's list item tokenization
tree-sitter highlight failure If markdown highlight query file is missing or parser fails to load ensureVisibleTextBeforeHighlight sets textBuffer.setText(content) as plain text
Zero-highlights during streaming Incomplete markdown mid-stream produces zero tree-sitter highlights onHighlight callback in index.tsx preserves last known good highlights
Black empty lines during stream CodeRenderable.renderSelf returned early during highlight debounce without painting; content setter also skipped buffer sync when drawUnstyledText=false Fixed: always sync buffer + paint unstyled progressive text; only defer tree-sitter; Markdown children use streaming: this._streaming

5g. Incremental Parsing for Streaming

parseMarkdownIncremental() reuses previous parse state by matching token.raw at character offsets:

  1. Compares newContent.startsWith(token.raw, offset) for each previous token
  2. Reuses matching tokens up to reuseCount
  3. Subtracts trailingUnstable (default 2 when streaming: true) from reuseCount
  4. Re-parses only the remaining newContent.slice(offset) with x.lex()

The trailingUnstable parameter:

  • streaming: true → trailingUnstable = 2 — last 2 tokens always re-parsed
  • streaming: false → trailingUnstable = 0 — all matching tokens are stable

6. Mermaid Diagram Rendering

Text size contract (2026-10-10)

The native path measures the terminal cell width and sets Mermaid's font size BEFORE layout: fontSize = cellWidth * unitsPerEm / advanceWidth(M). Fontkit reads the same Consolas face that WASM registers and resvg uses; embedded Cascadia Mono remains the fallback. Resetting WASM also resets font registration and metrics. The cache includes cell width.

SVG is rasterized at its natural size. ImageRenderable.fit="none" preserves source pixels even when the reserved cell box rounds up. Narrow diagrams keep their width; wide diagrams use a horizontal scrollbox whose content width/minWidth/maxWidth equal the image's reserved columns. Its full height belongs to the surrounding session scrollbox. Attachments retain their previous fit behavior.

The PNG/symbols fallback remains available without terminal graphics; physical font equality is only defined for the native pixel path. Fixed-width enlargement and raster budget inputs were removed.

Regression predicates: glyph advance at 12/24px; unchanged label size across short/long diagrams; font reset; pixel-exact 23x19 source in a rounded 24x20 cell box; horizontal wheel reaches the last column at 40/80 terminal columns; tall diagrams scroll with the parent; nonempty symbol fallback. Tests live in test/util/mermaid-text-size.test.ts, test/tui/mermaid-text-size.test.tsx, and OpenTUI src/tests/image-natural-size.test.ts.

Pipeline

Mermaid source
  → splitTextSegments() (text-segments.ts regex)
  → MediaMermaid → renderMermaidToRgba (mermaid.ts)
    → getRenderer() lazy loader — first call dynamically imports WASM
    → withTimeout() — 10s timeout guard
    → renderMermaidToSvg (mermaid.ts) — mermaid-wasm-renderer v0.3.1 (Rust → WASM)
    → font metrics → SVG layout → resvg at natural size → RGBA
  → MediaImage diagram → horizontal viewport → OpenTUI image fit=none → Kitty/Sixel
    → PNG → half-block symbols when the terminal has no graphics protocol

Files:

  • packages/opencode/src/util/mermaid.ts — WASM SVG → PNG pipeline, lazy loader, MERMAID_RENDER_TIMEOUT guard
  • packages/opencode/src/cli/cmd/tui/routes/session/text-segments.ts — splitTextSegments, splitter for ```mermaid fences
  • packages/opencode/src/cli/cmd/tui/component/media-image.tsx — decode, contain-fit, graphics-protocol path, zoom/pan viewport
  • packages/opencode/src/cli/cmd/tui/routes/session/index.tsx — RichText text-part render integration

6a. Lazy WASM Loader

The mermaid-wasm-renderer package ("type": "commonjs") reads a 2.8MB .wasm file synchronously at module load via require('fs').readFileSync(__dirname + '/...bg.wasm'). To avoid blocking startup:

  1. Top-level import replaced with lazy import() — triggered on first renderMermaidToSvg() call, not at module load
  2. Concurrent call deduplication — parallel first calls share one Promise
  3. Cache reset — resetRendererCache() clears the cached module for testing/recovery
  4. Error isolation — WASM load failures log warn("bug:...") and return null, don't crash the caller

6b. Timeout Guard

A 10-second timeout wraps every WASM render call via Promise.race:

Promise.race([
    mod.renderSvg(source),     // Rust WASM render
    timeout(10_000),           // Abort after 10s
])

On timeout: logs warn("bug: Mermaid render timed out after 10000ms"), returns null. This prevents pathological diagrams (Rust recursion, infinite loop) from hanging the TUI.

6c. Error Logging

Location Before After Rationale
mermaid.ts — WASM failure log.debug(...) log.warn("bug: ...") Rendering failure is a real problem, not ignorable
mermaid.ts — resvg failure log.debug(...) log.warn("bug: ...") PNG conversion failure should be observable
index.tsx — TextPart failure Log.Default.debug(...) Log.Default.warn("bug: ...") Silent blank image is a user-visible bug
media-mermaid.tsx log.debug(...) log.warn("bug: ...") Same — visible failure

6d. TexturePlaneRenderable Improvements

Improvement Detail
Responsive width Defaults to 70% of terminal columns (capped at 80) instead of hardcoded 70
Aspect-ratio height Height derived from actual image aspect ratio, not a fixed 0.5 fallback
Cleanup hardening Temp file deletion is always try/catch in finally
Error visibility Failures log warn("bug: ...") instead of silent failures

6e. Identified Limitations

Issue Impact Status
No progressive preview during streaming Users see raw source code until closing ``` fence Unresolved — needs mermaid partial-render support in WASM
__dirname in bundled binary bun build --compile fails if __dirname is undefined at runtime Mitigated by lazy import() — at least error is caught gracefully
MediaMermaid component (media-mermaid.tsx) dead code CORRECTED 2026-09-26 — it is live: routes/session/index.tsx:69,2295 renders <MediaMermaid> Resolved
No font registered with WASM renderer RESOLVED 2026-09-24 — ensureMermaidFont registers Consolas (terminal face) with embedded Cascadia Mono fallback (mermaid.ts:224-268) Resolved
Strict regex for fence detection (case-sensitive, 3 backticks) Non-standard mermaid blocks don't match Low priority — all LLM outputs use standard fencing
Streaming gap — incomplete blocks show as raw code Visual source-code flash on every mermaid block Low priority — needs partial SVG rendering during stream

6f. Why every image looked wrong on 2026-09-23 — the raster was fine, the box was ignored

Owner: «до миграции было четкое форматирование mermaid по размеру шрифта — он был читаемый, сейчас он не читаемый», and of the isolated raster: «в тесте норм». Both statements are correct, and together they localise the defect: the raster is good, the display is not.

Measured, in this order:

  1. The raster pipeline is CORRECT. Rendering a five-node diagram through renderMermaidToSvg + renderSvgToPngDataUrl produces a crisp diagram whose node frames fit their text exactly (experiments/2026-09-23_mermaid-font/baseline.png).
  2. registerMermaidFont is DEFINED AND NEVER CALLED (src/util/mermaid.ts:157 — its only other mention is the WASM binding). A real gap, but NOT this defect: step 1 renders correctly without it, so the «no font» row in §6e describes a risk, not the cause.
  3. The cause is one IGNORED ARGUMENT. The caller passes the box it can actually show — maxWidth from the column budget, maxHeight from the row budget, cellHeight from the measured cell (media-image.tsx:452-455) — and resvgOptionsForSvg answered width-only by design («Height is never set here»). A five-node diagram therefore left the rasteriser ~735×1600 px, taller than any box that would show it, and was cut down again by NATIVE_IMAGE_PIXELS/NATIVE_IMAGE_MAX_PIXELS = 512 (media-image.tsx:218). Text that the font anchor had just made exactly one terminal row tall arrived at a fraction of a row — or the diagram was simply CUT at the frame edge, which is what the owner's screenshot showed.
  4. The font anchor itself is right. Measured on a live capture, one terminal row on this host is 23 px (consecutive text ink bands 87–104 and 110–127), and labelCells: 1 means one line of diagram text per row — the «по размеру шрифта» behaviour to restore.

FIX (2026-09-23): resvgOptionsForSvg now re-fits by HEIGHT when the font-anchored width would overflow the caller's row budget. Verified differentially, same source and same code: with no height budget the raster is 52 426 bytes (width-only behaviour), with the 38-row / 874 px budget it is 34 003 bytes — every node still fits its frame, and one line of text is one row. The 512 ceiling stays where its justification lives: an image whose natural size we did NOT choose (a pasted screenshot). It is the right rule there and the wrong one for a raster generated to fit a box.

CORRECTION (2026-09-26): the height re-fit was itself the regression. On tall diagrams it divided the font-anchored scale — text that was exactly one terminal row arrived smaller than a row, the «нечитаемо» it had meant to fix. Owner ruling, verbatim: «клампить ширину, высоту отпускать и вставлять как есть». resvgOptionsForSvg no longer re-fits by height; maxHeight is accepted for compatibility and has NO effect. Only the width clamp inside fitFontAnchoredSize may reduce the scale, and only when the diagram genuinely does not fit horizontally. A raster taller than the row budget is inserted as-is (the MediaImage stays interactive: wheel zoom · drag pan). Line anchors above are from the 09-21 snapshot; current ones: media-image.tsx:453-457 passes the box, mermaid.ts:78-141 builds the options.

Progressive Rendering (2026-07-12 improvement)

Before: Mermaid diagrams only rendered after part.time?.end — entire LLM response had to finish. Users saw raw ```mermaid code as plain text fallback for the entire streaming duration.

After: Renders each mermaid block as soon as its code fence closes, even if the LLM continues writing after it. Re-render avoidance is structural rather than a cache: the segment list is a createMemo and <Index each={segments()}> keys renderables by segment position, so an unchanged segment keeps its renderable across streaming ticks. (The original implementation used a renderedSources Map keyed by index + source hash; that map is gone.)

7. Image & Media Rendering

7a. MediaImage — inline image display

File: packages/opencode/src/cli/cmd/tui/component/media-image.tsx

The Three.js <image-plane> / TexturePlaneRenderable path is gone; images go through OpenTUI's own <image> renderable.

  • Primary path: decode → RGBA → OpenTUI <image> (PixelBuffer → Kitty or Sixel)
  • Fallback: half-block symbols (util/image-to-ansi.ts) when no graphics protocol
  • Sizing: natural width/height contain-fit into the terminal box (util/fit-image.ts), capped at 80 cols × 40 rows; never a forced fixed width
  • Capability detection can land after first paint — the path stays unlocked for 1s
  • interactive (mermaid): keeps a high-res source buffer, wheel = zoom, drag = pan, middle-click = reset (util/image-viewport.ts)

7b. Terminal Graphics Protocol Detection

File: packages/opencode/src/util/terminal-graphics.ts

  • detectGraphicsProtocol() detects Kitty, iTerm2, Sixel, or Unicode symbols fallback
  • Detection based on TERM, TERM_PROGRAM, KITTY_WINDOW_ID, WT_SESSION env vars
  • WezTerm: prefers Kitty, falls back through chain
  • VS Code terminal: always "symbols" fallback

7c. Image Components

File Component Rendering
media-image.tsx <MediaImage> OpenTUI <image> → Kitty / Sixel, half-block fallback
media-video.tsx <MediaVideo> Video playback widget
media-audio.tsx <MediaAudio> Audio playback widget
media-mermaid.tsx <MediaMermaid> Mermaid → SVG → PNG → <MediaImage interactive>

7d. ANSI / Kitty Renderers

File Purpose Key Details
kitty-render.ts Kitty protocol escape sequences Chunked for images > 4096 bytes
image-to-ansi.ts Pure ANSI TrueColor fallback Half-block characters (▀), Jimp-based

8. Markdown Rendering (Web/Desktop)

File: packages/ui/src/components/markdown.tsx

  • Uses marked for markdown → HTML conversion
  • DOMPurify for XSS sanitization
  • 200-entry LRU cache (Map<string, Entry>)
  • Code blocks get copy buttons

File: packages/ui/src/components/markdown-stream.ts

  • heal(text) — closes unclosed formatting via remend library
  • stream(text, live) — splits markdown into stable content vs incomplete trailing code fence
  • Used by web UI for streaming markdown display

Document Conversion (Binary → Markdown)

File: packages/opencode/src/util/markdownify.ts

  • Uses external binary opencode-markdownify
  • Supports PDF, docx, xlsx, pptx, CSV, HTML, media files
  • Resolves from Global.Path.bin, config, executable dir, source checkout

9. Syntax Highlighting (Tree-sitter)

Configuration

File: packages/opencode/parsers-config.ts

Defines 28 language parsers (Python, Rust, Go, TypeScript, etc.) with WASM files and tree-sitter query URLs.

Parser Loading

File: packages/opencode/src/util/parser-wasm.ts

  • Line 56-60: getGrammarWasm(filetype) — loads from local WASM cache or CDN fallback
  • Line 66-71: preloadGrammars() — preloads all grammars during initialization
  • Registered in session route at index.tsx: addDefaultParsers(resolvedParsers)

10. Attachment Rendering

File: packages/opencode/src/attachment/registry.ts

Each attachment type has a render() method returning TuiRenderResult:

Handler File Type render() result
handlers/image.ts Images TuiRenderResult → <MediaImage>
handlers/video.ts Video → <MediaVideo>
handlers/audio.ts Audio → <MediaAudio>
handlers/document.ts Documents → "describe" (converted to markdown)
handlers/archive.ts Archives → Directory listing
handlers/data.ts Data files → Structured display
handlers/spatial.ts GIS → Map/geo display
handlers/sensor.ts Sensor data → Gauge/chart display

11. TUI Configuration

File: packages/opencode/src/cli/cmd/tui/config/tui-schema.ts

Setting Values Default Line
diff_style "auto", "stacked" — 26
image_protocol "auto", "kitty", "sixel", "iterm2", "symbols" — 31

Themes: 30+ theme JSON files in tui/themes/ with full color/syntax definitions.

12. Test Infrastructure

Test Fixtures

File: packages/opencode/test/fixture/fixture.ts

  • tmpdir(options?) — creates temporary directories with optional git/config/init
  • tmpdirScoped(options?) — Effect-scoped temp directory
  • provideTmpdirInstance((dir) => effect, options?) — convenience helper

Known Test Considerations

SolidJS SSR Evaluation: SolidJS evaluates renderable components during module import (when .tsx files are loaded). If a component requires context providers (Toast, Dialog, Theme) that don't exist at module-load time, it throws. Mitigations:

  • DialogProvider (ui/dialog.tsx) gracefully handles missing ToastProvider — useToast() is wrapped in try/catch, copy-on-select guards with if (!toast) return
  • Tests that don't depend on SolidJS should be placed outside test/cli/tui/ to avoid the SSR cascade (e.g., test/cli/editor-context.test.ts)

Log Spy Target: The plugin loader (runtime.ts) routes warnings through Log.Default.warn() → JSONL file I/O, not console.warn. Tests must spy on the log module, not console.warn.

13. Key File Index

Anchors are symbol names, not line numbers — line ranges rot on every edit.

File Anchor Purpose
packages/opencode/src/session/processor.ts "reasoning-delta" / "text-delta" / "tool-input-delta" cases Delta event generation
packages/opencode/src/session/session.ts bus.publish(MessageV2.Event.PartDelta, …), bus.publish(Event.Updated, …) Event bus publishing
packages/opencode/src/cli/cmd/tui/context/sync.tsx deltaBuffer + MAX_DELTA_BUFFER_SIZE / DELTA_BUFFER_TTL_MS Delta store mutations, buffered pre-store deltas
packages/opencode/src/cli/cmd/tui/routes/session/index.tsx RichText, TextPart Text rendering (markdown + mermaid segments)
packages/opentui/packages/core/src/renderables/markdown-parser.ts parseMarkdownIncremental, reuseCount, trailingUnstable Incremental markdown parsing / stream stability
packages/ui/src/components/markdown-stream.ts marked + remend Web markdown stream healing
packages/opencode/src/cli/cmd/tui/routes/session/text-segments.ts splitTextSegments Mermaid text splitter
packages/opencode/src/util/mermaid.ts renderMermaidToPngDataUrl Mermaid SVG→PNG pipeline
packages/opencode/src/cli/cmd/tui/component/media-image.tsx MediaImage, mediaImageCellBounds Inline image renderable (Kitty / Sixel / half-block)
packages/opencode/src/util/terminal-graphics.ts detectGraphicsProtocol, detectBestProtocol, GRAPHICS_PROTOCOLS Protocol detection
packages/opencode/src/util/kitty-render.ts kittyImage Kitty escape sequences
packages/opencode/src/util/image-to-ansi.ts imageToChunks, imageToAnsi ANSI TrueColor fallback
packages/opencode/src/util/markdownify.ts convertDocument, isSupportedDocumentFormat Document→Markdown conversion
packages/ui/src/components/markdown.tsx Markdown Web markdown→HTML rendering
packages/opencode/src/cli/cmd/tui/ui/dialog.tsx DialogProvider Dialog provider (tolerates a missing toast provider)
packages/opencode/src/cli/cmd/tui/ui/toast.tsx Toast, ToastProvider, useToast Toast notifications
packages/opencode/parsers-config.ts parser table Tree-sitter parser config
packages/opencode/src/util/parser-wasm.ts getGrammarWasm, preloadGrammars, availableLanguages Parser WASM loading
packages/opencode/src/attachment/registry.ts registry, Service Attachment type registry
packages/opencode/src/cli/cmd/tui/config/tui-schema.ts TUI config schema TUI config schema
packages/opencode/test/cli/editor-context.test.ts whole file ZED editor tests (moved from TUI dir)
packages/opencode/test/cli/tui/plugin-loader-entrypoint.test.ts whole file Plugin entrypoint tests

OpenTUI is consumed from the workspace, not from npm. Root package.json pins "@opentui/core": "workspace:*" → packages/opentui/packages/core. Read the TypeScript source there; the old node_modules/@opentui/core/index.js bundle offsets this document used to cite are gone, and so is the @opentui/core@0.4.3 patch (patchedDependencies now holds only @npmcli/agent and @standard-community/standard-openapi).

File Anchor Purpose
packages/opentui/packages/core/src/renderables/Markdown.ts MarkdownRenderable, shouldRenderSeparately, buildRenderableTokens, createDefaultRenderable, createListRenderable, createInitialStyledText Markdown renderable and token→renderable routing
packages/opentui/packages/core/src/renderables/markdown-parser.ts parseMarkdownIncremental, reuseCount, trailingUnstable Incremental parse, stable-prefix reuse
packages/opentui/packages/core/src/renderables/Code.ts CodeRenderable, startHighlight, ensureVisibleTextBeforeHighlight, _shouldRenderTextBuffer Tree-sitter highlight lifecycle

14. 3D Terminal Rendering — Current Status

Working Pipeline (Python wgpu + Sixel)

WGSL compute shader (GPU)
  → map_sync readback → numpy → pixel array
    → Sixel escape codes → terminal (WezTerm/WT)

The working 3D rendering approach uses Python's wgpu package for WebGPU bindings and outputs directly via the Sixel protocol. Key advantages:

Aspect How it works
Rendering WGSL compute shader does rotation, projection, line rasterization on GPU
Readback map_sync() — synchronous, no double-map hazard
Output Raw pixel bits (uint32 ON/OFF) → Sixel → terminal
Deps python-wgpu, numpy
Files experiments_history/2026-07-28_rotating-cube-3d/test_cube.py
Run python test_cube.py (with DXC DLLs in PATH)

The resolve_and_inject_dependencies() function in test_cube.py handles DXC DLL discovery automatically using os.add_dll_directory().

Blocked Pipeline (@opentui/three + bun-webgpu)

Three.js WebGPURenderer
  → CLICanvas.readPixelsIntoBuffer()
    → setCellWithAlphaBlending("\u2588", pixelColor, ...)
      → FFI boundary → U+FFFD (replacement char)

The @opentui/three package (CLICanvas.readPixelsIntoBuffer) passes the character \u2588 (█) through bun's FFI to the native bufferSetCellWithAlphaBlending function. At the FFI boundary the character is corrupted to U+FFFD in all supersampling modes (none, CPU, GPU). This is an upstream bug in @opentui/three's canvas readback pipeline.

Issue Status Fix location
DXC DLL not found ✅ Fixed Copy x64 DLLs from Windows SDK or install DXC
getMappedRange + getMappedRangePtr overlap ✅ Fixed Fixed in the vendored packages/opentui/packages/three source
setCellWithAlphaBlending char → U+FFFD ❌ Upstream @opentui/three CLICanvas.readPixelsIntoBuffer()

Quick Test Commands

# Working: Python WGSL + Sixel (WebGPU, needs DXC DLLs in PATH)
python experiments_history/2026-07-28_rotating-cube-3d/test_cube.py

# Working: Python software raster + Sixel (no GPU needed)
python experiments_history/2026-07-28_rotating-cube-3d/term_rend.py

# Working: Three.js WebGL in browser
# Open experiments_history/2026-07-28_rotating-cube-3d/web.html

# Not working: Three.js WebGPU in terminal (U+FFFD bug)
bun run experiments_history/2026-07-28_rotating-cube-3d/smoke.ts