TypeAgent Studio: Trace Viewer - an A/B resolution-pipeline debugger for utterance regressions by TalZaccai · Pull Request #2689 · microsoft/TypeAgent

TalZaccai · 2026-07-17T05:37:53Z

What & why

When the Impact Report flags an utterance that resolved differently between two
versions (A = base, B = compare), this adds a Trace Viewer that answers
where resolution diverged and what change caused it.

For one utterance it renders resolution top-to-bottom as a pipeline —
construction cache → grammar match → wildcard validation → result — for both
versions. Agreeing stages collapse to one line; the first diverging stage expands
into a side-by-side A/B card, names the responsible grammar/schema file, and
offers an inline A↔B file diff plus jump-to-source. Opened in one click from an
Impact Report row (right-click, detail-pane button, or arrow-key nav).

Design decisions worth a look

Parity is decided by comparing the two sides' final actions, not per-layer
outcomes. Both sides can "hit" yet produce different actions, and a live
cache hit vs a git-ref grammar match are different branches, not the same step.
Cause attribution walks layers in order and blames the first that explains
the difference (cache short-circuit → different grammar rule → wildcard
rejection → action payload). Grammar blame uses the compiled-grammar hash
(resolver-independent); when the captured parse can't be trusted, attribution
falls back to the action level and the conclusion is marked low-confidence.
Tracing is side-effect-free on the hot path — resolveWithTrace() shares
resolveSide() with the normal resolve(), so uninstrumented replay pays
nothing.
Heavy traces stay out of the webview payload — the report posts only a
lightweight tracedUtteranceIds list (MB → KB); full traces are persisted
separately and loaded on open. Per-run capture is changed-first and capped.

Where the changes live

typeagent-core — produces/captures/persists/recomputes traces: trace
DTOs, resolver instrumentation (grammarResolver,
constructionCacheResolver.matchEntry), per-row capture + single-utterance
recompute (studioRuntimeCore).
typeagent-studio — the viewer: A/B pipeline client + host (owns all
file/diff opening), the pure divergence view model, source resolver,
workspace-state trace store (retention/eviction), and the Impact Report entry
points.
grammarTools — gt-trace-timeline gains a display-only, CSP-safe mode so
a persisted match can render inside a webview.
studio-service — one RPC handler bridging the recompute call.

Testing

Divergence view model, source resolver, trace store, protocol, engine
capture/recompute, and the timeline display-only path are all covered; full
typeagent-studio and typeagent-core suites green.

Add serializable per-row resolution traces to the replay resolver so the Impact Report can drill into why a red row lost or changed its match. - resolutionTrace.ts: DTOs for the four-layer resolution trace (cache-consult, grammar-match, wildcard-validation, action) plus construction-cache entry and run-descriptor shapes, with GrammarDebugInfo serialize/deserialize. Type-only grammar-tools-core imports keep it safe in the browser barrel. - resolutionTraceCapture.ts: pure captureGrammarMatchTrace() that loads a grammar from text, runs traceMatch, computes ranking parity against the chosen action, and anchors a headline .agr source span. Value imports keep it out of the barrel; reachable via the replayResolver entry. - grammarResolver.ts: retain grammar text/filename on the built grammar; add resolveWithTrace() and captureResolutionTrace() built on a shared resolveSide() so the normal resolve() path pays no added cost. Live-only layers report not-applicable on a git ref. - constructionCacheResolver.ts: matchEntry() to expose the hit entry for traces. - Tests: capture unit tests and resolveWithTrace/captureResolutionTrace tests; full core replay/runtime suite green. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

Pure, JSON-safe builders that assemble the run-level descriptor persisted with a replay run so a stored trace can later be reproduced (or flagged as drifted). - buildTraceVersionPin() derives workingTree from the spec and enforces the side invariants: a commit SHA identifies a git-ref side (dropped for the working tree), and only the working tree can drift so content hashes are kept for it alone. SHA/hash resolution stays in the runtime; pins are passed in. - buildReplayRunDescriptor() assembles the descriptor and stamps runAt from an injectable clock (explicit runAt wins). - 5 unit tests covering both invariants, optional-field omission, JSON round trip, and clock precedence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

Add a display-only mode to gt-trace-timeline so a persisted match trace can be rendered inside the Studio webview, where the strict Content-Security-Policy forbids inline style attributes. - Add displayTrace/displayDebugInfo properties; when set, the component renders the supplied trace directly instead of recomputing via the backend, and hides the match-options panel, input row, and Trace button. - Route trace and debug-info reads through _activeTrace/_activeDebugInfo getters. - Replace all inline style attributes with CSS classes: per-event-kind icon colors, depth indentation, and status padding. - Add part-span source jumps alongside the existing rule-name jumps. - Cover the display-only path with component tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

Pure, browser-neutral view-model that turns a captured ReplayResolutionTrace into the conclusion the Trace Viewer leads with. - Resolution parity is decided by comparing the two sides' canonical final actions, not per-layer outcomes. - When actions differ, an ordered causal walk attributes the difference to the first explaining layer: a cache short-circuit, a different grammar rule, a wildcard-validation rejection, or an action-parameter difference. - Capability asymmetry (a git-ref side's not-applicable layers) is reported as a fidelity-path note, never as the headline divergence. - A diverged grammar ranking-parity demotes attribution to the action level and marks the conclusion low-confidence, since the captured parse may not be the resolver-selected one. - Emits compact per-node summaries (execution/outcome pills, grammar rule and ranking-parity, inline cache entry, action name/schema) for the node cards. Covered by unit tests over representative traces. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

Add a workspace-state store for the red-row resolution traces and the run descriptor so the Trace Viewer can reopen the exact side-by-side resolution that produced a row, and recompute a fresh trace from the same pinned inputs. Grammar debug info is deduped to one blob per grammar hash per run (stripped on save, rehydrated on read). Retention mirrors the Impact Report: a new run for an agent evicts that agent's previous run, and a global cap bounds how many agents' runs are kept. A rotated-out run reads back as evicted so the viewer can say its traces are no longer available rather than erroring. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

After a grammar replay streams its rows, re-resolve each changed (red) row with tracing on the still-live resolver and return the per-row resolution traces on the run result, capped so a large changed set stays bounded. The identity baseline captures nothing. The Impact Report run handler builds the run descriptor from the pinned version provenance and persists it alongside the captured traces, so the Trace Viewer can reopen the exact resolution behind a row and later recompute a fresh one from the same inputs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

…ion trace Add replayResolutionTrace to the Studio runtime and service channel: given a stored run descriptor and an utterance id, rebuild the same grammar resolution the run used and capture a fresh trace for that one row. Returns entry-missing when the utterance is no longer in the corpus and unavailable when the descriptor's versions don't reconstruct into a grammar run. Threaded through the runtime interface/impl, the wire protocol, the service handler, the client proxy, the facade, and the test stub, with runtime tests covering the recomputed, entry-missing, and unavailable outcomes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

…vidence view) Adds the per-red-row Trace Viewer webview — its own protocol union, host panel (keyed runId::utteranceId, opened beside the Impact Report), and a browser client that renders the divergence conclusion, a row-centric A|B fidelity grid, and a Recorded/Fresh replay toggle. Source-jump and the inline timeline expand follow as separate steps; the Impact Report entry point is wired in trace-entrypoint2. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

Wire the Trace Viewer's per-node jump-to-source: clicking a winning grammar rule or a produced action opens the exact file the row resolved against. - traceSourceResolver.ts (pure): sourceTargetFor(trace, side, node) extracts a grammar-rule span (resolving fileId via the node's debug-info file table, with a display-path fallback) or an action's schema file from the stored trace. - traceViewerView.ts: open-source handler opens the working-tree file live with best-effort selection, or a pinned git ref through a read-only virtual document backed by git show <sha>:<relpath>, with repo-relative path guarding. - traceDivergenceViewModel.ts: expose hasSource on the grammar summary so the client only offers a jump when the trace recorded a span. - client/traceViewer.ts + traceViewer.css: clickable jump affordances on the chosen rule and action name, a monotonic source-request id, and source-result handling that clears or surfaces the host's note. Tests: traceSourceResolver unit (6); studio suite 234. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

Complete the J4 find-a-regression journey: a red row's drill-in can now open the full side-by-side resolution trace beside the report. - protocol.ts: new openTrace message (runId + utteranceId), host-validated in parseWebviewMessage. - impactReportView.ts: openTrace handler calls openTraceViewer beside the report. The traceViewerView back-import stays type-only so the two modules don't form a runtime cycle. - client/impactReport.ts: the drill-in detail header gains an Open-trace button, shown only for rows that captured a trace (payload.resolutionTraces, which survive both in-session recovery and persisted restore since only rows are capped). - impactReport.css: group the trace button with close on the header's right. Tests: openTrace parse accept/reject (2); studio suite 236. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

The detail-header Open-trace button uses the list-tree codicon, but the webview stylesheet defines codicon glyphs one rule at a time and had no .codicon-list-tree rule, so the button rendered with an empty glyph and was effectively invisible. Add the missing rule with the list-tree codepoint; the bundled codicon font already contains it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

The replay result now also returns a lightweight tracedUtteranceIds list, which is all the webview needs to offer the Open-trace affordance. The host strips the heavy per-row resolutionTraces before posting to the webview and before writing the persisted run to workspaceState; the full traces are still persisted separately by the Trace Viewer store. The client builds its traced-id set from tracedUtteranceIds, falling back to the traces when absent. This drops a large changed set's webview payload from megabytes to kilobytes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b770d9f-e456-4b3f-aef2-3d84e8900a4a

The Trace Viewer now renders an utterance's resolution top-to-bottom as a pipeline (construction cache, grammar match, wildcard validation, result) for both the A and B versions, and attributes an A/B divergence to its most likely cause, surfacing the responsible file with an inline diff. File and diff links open in a stable side group that never hides the viewer. Impact Report rows are navigable with Up/Down arrows and open in the Trace Viewer from a right-click menu, with a jump-to-viewer control in the detail pane. Arrow navigation is handled at the document level and reclaims keyboard focus on hover, so it keeps working after focus leaves the table. Core replay captures a resolution trace for every compared row rather than only divergent ones, and records the matched construction-cache entry so the viewer can show which cache construction resolved an utterance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2c29bc06-0d89-4e92-bc27-e5cac77c3331

Copilot

Pull request overview

This PR adds a Trace Viewer to TypeAgent Studio to debug utterance regressions by rendering an A/B, stage-by-stage resolution pipeline (construction cache → grammar match → wildcard validation → action), with host-driven source opening and cross-version diffs backed by persisted traces.

Changes:

Capture and persist per-utterance resolution traces (bounded, changed-first) and enable single-utterance trace recomputation via a new runtime/RPC surface.
Add a new Trace Viewer webview (protocol, view model, store, source resolver) and integrate Impact Report entry points (open/focus, context menu, keyboard navigation).
Extend grammar tools timeline UI with a CSP-safe “display-only” mode for rendering stored traces without a backend.

Show a summary per file

File	Description
ts/packages/typeagent-studio/src/webviewKit/traceProtocol.ts	Defines Trace Viewer-specific webview message contract + host-side parsing.
ts/packages/typeagent-studio/src/webviewKit/replayViewModel.ts	Generalizes `toActionDiff` to accept `{actionA, actionB}` pairs.
ts/packages/typeagent-studio/src/webviewKit/protocol.ts	Adds `openTrace` / `focusTrace` messages to Impact Report protocol parsing.
ts/packages/typeagent-studio/src/webviewKit/host.ts	Adds configurable `viewColumn` support when creating webview panels.
ts/packages/typeagent-studio/src/webviewKit/client/impactReport.ts	Adds trace affordances, row context menu, and arrow-key row navigation behavior.
ts/packages/typeagent-studio/src/traceViewerView.ts	Implements the Trace Viewer panel, source open/diff behavior, and virtual ref-backed documents.
ts/packages/typeagent-studio/src/traceStore.ts	Persists run descriptors + per-row traces into workspace state with retention/eviction.
ts/packages/typeagent-studio/src/traceSourceResolver.ts	Pure extraction of source jump targets from a captured trace DTO.
ts/packages/typeagent-studio/src/test/webviewProtocol.spec.ts	Adds tests for new `openTrace` / `focusTrace` protocol parsing.
ts/packages/typeagent-studio/src/test/webviewBundle.spec.ts	Ensures the new Trace Viewer bundle remains browser-safe (no node/vscode/ws).
ts/packages/typeagent-studio/src/test/traceStore.spec.ts	Covers trace persistence, dedupe, and eviction behavior for the trace store.
ts/packages/typeagent-studio/src/test/traceSourceResolver.spec.ts	Covers grammar/action source target extraction from traces.
ts/packages/typeagent-studio/src/test/traceProtocol.spec.ts	Covers Trace Viewer message parsing hardening.
ts/packages/typeagent-studio/src/test/traceDivergenceViewModel.spec.ts	Adds broad unit coverage of the divergence attribution and pipeline VM.
ts/packages/typeagent-studio/src/test/studioRuntimeCore.spec.ts	Adds tests for single-utterance trace recomputation path.
ts/packages/typeagent-studio/src/test/stubInvokeHandlers.ts	Extends test stubs with `replayResolutionTrace` RPC shape.
ts/packages/typeagent-studio/src/studioServiceClient.ts	Adds client RPC method to request recomputed traces.
ts/packages/typeagent-studio/src/serviceRuntimeFacade.ts	Plumbs `replayResolutionTrace` through the runtime facade.
ts/packages/typeagent-studio/src/impactReportView.ts	Integrates Trace Viewer open/focus, slims webview payloads, persists traces separately.
ts/packages/typeagent-studio/src/impactReportStore.ts	Stops persisting heavy traces inside the Impact Report stored payload.
ts/packages/typeagent-studio/media/traceViewer.css	Adds Trace Viewer styling and codicon font embedding rules.
ts/packages/typeagent-studio/media/impactReport.css	Adds context-menu styling and aligns A/B visual language with Trace Viewer.
ts/packages/typeagent-studio/esbuild.mjs	Adds an esbuild config for the Trace Viewer webview bundle.
ts/packages/typeagent-core/test/resolutionTraceCapture.spec.ts	Tests grammar-match trace capture, parity, and path recording.
ts/packages/typeagent-core/test/resolutionTrace.spec.ts	Tests version pin and run descriptor builders for JSON-safe behavior.
ts/packages/typeagent-core/test/grammarReplayResolver.spec.ts	Tests `resolveWithTrace` and trace capture semantics across workingTree/git ref.
ts/packages/typeagent-core/test/constructionCacheResolver.spec.ts	Tests new `matchEntry` for construction cache identity DTO.
ts/packages/typeagent-core/test/captureRowTraces.spec.ts	Tests capped, changed-first per-row trace capture behavior.
ts/packages/typeagent-core/src/runtime/studioRuntimeCore.ts	Captures traces post-run and adds `replayResolutionTrace` API to recompute one trace.
ts/packages/typeagent-core/src/runtime/studioProtocol.ts	Adds studio service invoke signature for `replayResolutionTrace`.
ts/packages/typeagent-core/src/replay/resolutionTraceCapture.ts	Implements grammar-tools-based grammar match tracing and source anchoring.
ts/packages/typeagent-core/src/replay/resolutionTrace.ts	Introduces serializable trace DTOs, pins, and descriptor builders.
ts/packages/typeagent-core/src/replay/index.ts	Re-exports trace types/builders from `@typeagent/core/replay`.
ts/packages/typeagent-core/src/replay/grammarResolver.ts	Adds `resolveWithTrace`, trace assembly, and `captureResolutionTrace`.
ts/packages/typeagent-core/src/replay/constructionCacheResolver.ts	Adds `matchEntry` to surface inspectable construction identity alongside the action.
ts/packages/studio-service/src/studioRpcHandlers.ts	Adds RPC handler to bridge `replayResolutionTrace` to the runtime.
ts/packages/grammarTools/ui/test/components/gt-trace-timeline.test.ts	Adds tests for trace timeline display-only mode and CSP-safe rendering.
ts/packages/grammarTools/ui/src/gt-trace-timeline.ts	Adds display-only inputs and removes inline color styles (class-based).

Review details

Files reviewed: 40/40 changed files
Comments generated: 4
Review effort level: Low

The Trace Viewer's pre-action stages already show a compact one-liner when the two versions agree, but the Result stage always drew the side-by-side A/B card, so an unchanged Result looked heavier than an unchanged grammar match. A converged Result now renders that same compact summary and drops its card border; a diverged Result still shows the bordered side-by-side card with its output diff and schema chip. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2c29bc06-0d89-4e92-bc27-e5cac77c3331

Break the single attributeCause walk into cacheCause, grammarCause, wildcardCause, actionPayloadCause and unattributedCause helpers chained with ??, sharing one AttributionContext computed up front. Behavior is unchanged (the view-model spec guards it); this drops the function's cyclomatic complexity under the repo cap so the changed-file complexity ratchet passes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2c29bc06-0d89-4e92-bc27-e5cac77c3331

- grammarResolver: only attach the schema source file when a side actually produced an action, so a miss offers no bogus schema jump or A/B compare. Adds a regression test covering the miss and hit cases. - traceViewerView: reject only a real ".." path segment when validating a git-relative path, so an in-repo file whose name starts with two dots is no longer treated as escaping the repo root. - impactReport: refresh the tracedIds and renderDetail comments to describe the current capture-all behavior and remove a duplicated JSDoc block. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2c29bc06-0d89-4e92-bc27-e5cac77c3331

Note the part7 branch, add a Trace Viewer capability-matrix row, and move Gate D (Debug a trace) from not-started to partial with the remaining cache-entry inspect and replay-this-trace follow-ups. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2c29bc06-0d89-4e92-bc27-e5cac77c3331

robgruen · 2026-07-17T20:41:09Z

        }
    }

+    /** True when the component shows a supplied stored trace instead of running


Is there a way we can stop copilot from embedding HTML and instead use a template/replacement pattern?

robgruen · 2026-07-17T20:44:11Z

 } from "../health/index.js";
 import type { CorpusEntry } from "../corpus/types.js";
-import type { VersionSpec } from "./types.js";
+import type { VersionSpec, ReplayCacheState } from "./types.js";


As this evolves consider if some/part/all of this can be folded into the grammar toolset as a first class citizen.

robgruen · 2026-07-17T20:47:37Z

+
+vscode.postMessage({ type: "ready" });
+
+// --- Rendering ------------------------------------------------------------


this seems silly..if there's a logical break in the logic of the file shouldn't we just split the file and name them accordingly?

robgruen · 2026-07-17T20:48:13Z

+    return id;
+}
+
+// --- State screens & bits -------------------------------------------------


Yeah, let's refactor this into it's logical parts.

robgruen · 2026-07-17T20:49:28Z

+
+/** Keep the last run for at most this many agents so the persisted payload stays
+ *  bounded even if the user cycles through many agents. */
+const MAX_RETAINED_RUNS = 8;


configuration setting?

TalZaccai and others added 14 commits July 13, 2026 13:24

Merge branch 'main' into dev/talzacc/typeagent_studio_part7

fbf54a8

TalZaccai requested a review from Copilot July 17, 2026 05:38

Copilot started reviewing on behalf of TalZaccai July 17, 2026 05:38 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

typeagent-bot Bot and others added 6 commits July 17, 2026 05:45

docs: regenerate README.AUTOGEN.md for changed packages

bc8a333

style: apply prettier formatting and policy fixes

d7d1b43

TalZaccai marked this pull request as ready for review July 17, 2026 20:25

docs: regenerate README.AUTOGEN.md for changed packages

18f0206

robgruen approved these changes Jul 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

TypeAgent Studio: Trace Viewer - an A/B resolution-pipeline debugger for utterance regressions#2689

TypeAgent Studio: Trace Viewer - an A/B resolution-pipeline debugger for utterance regressions#2689
TalZaccai wants to merge 21 commits into
mainfrom
dev/talzacc/typeagent_studio_part7

TalZaccai commented Jul 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robgruen Jul 17, 2026

Uh oh!

robgruen Jul 17, 2026

Uh oh!

robgruen Jul 17, 2026

Uh oh!

robgruen Jul 17, 2026

Uh oh!

robgruen Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		vscode.postMessage({ type: "ready" });

		// --- Rendering ------------------------------------------------------------

Uh oh!

Conversation

TalZaccai commented Jul 17, 2026

What & why

Design decisions worth a look

Where the changes live

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robgruen Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

robgruen Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

robgruen Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

robgruen Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

robgruen Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants