fix(vscode-lm): sanitize surrogates, recover leaked tool calls, and window-safe tool_result truncation

.roo/skills/probe-vscode-lm-api/SKILL.md

@@ -0,0 +1,99 @@
+---
+name: probe-vscode-lm-api
+description: How to empirically probe the VS Code Language Model API (`vscode.lm`) with a scratch extension against a real extension host, and the measured findings about Copilot Claude models leaking tool-call markup into text. Use when asked to "test the vscode.lm API", "probe Copilot model behavior", "capture a raw LM transcript", "does the model leak tool-call markup", "verify LanguageModelToolCallPart behavior", or when reasoning about `extractLeakedToolCalls()` in the vscode-lm provider.
+---
+
+# Probing the VS Code LM API Empirically
+
+## When to Use This Skill
+
+- A claim is being made about what a Copilot-backed model _actually_ emits over `vscode.lm` (tool-call parts vs. text), and it needs evidence rather than inference.
+- Changing [`extractLeakedToolCalls()`](../../../src/api/providers/vscode-lm.ts) or its guards, and you need a false-positive corpus.
+- Any question that can only be answered by real `model.sendRequest()` traffic — the mocked unit tests cannot answer it.
+
+## When NOT to Use This Skill
+
+- Ordinary provider work covered by [`src/api/providers/__tests__/vscode-lm.spec.ts`](../../../src/api/providers/__tests__/vscode-lm.spec.ts). Live probing is slow and burns Copilot quota.
+- Anything about non-`vscode-lm` providers. Anthropic-API behavior does not transfer.
+
+## Running the Probe
+
+Scripts live in [`scripts/probe-vscode-lm-api/`](../../../scripts/probe-vscode-lm-api/) at the repo root.
+
+1. Copy `scripts/probe-vscode-lm-api/package.json` and `scripts/probe-vscode-lm-api/extension.js` into a scratch directory, e.g. `<repo>\.tmp\lmprobe\`. No build, no `npm install` — it is plain CommonJS against the `vscode` module.
+2. Adjust `OUT_DIR` at the top of the copied `extension.js` (or set `LM_PROBE_OUT_DIR`) to the transcript output directory.
+3. Launch a **new** extension host window:
+
+```
+code --extensionDevelopmentPath=<repo>\.tmp\lmprobe --new-window <repo>
+```
+
+4. In that new window: `Ctrl+Shift+P` -> **LM Probe: Run** (or click "Run probe" on the toast).
+5. Wait for the completion notification. Transcripts and `summary.json` land in `OUT_DIR`.
+
+Each run writes a `.json` of every stream part and a `.txt` of the exact concatenated text, named `<modelId>__<scenario>__run<N>`.
+
+### Delegate the UI driving
+
+Steps 3-5 involve a live window. Delegate them to **`ui-operator`** mode rather than doing them inline; screenshots and control trees consume large amounts of context.
+
+## Gotchas
+
+### The consent gate needs a real user gesture
+
+**Do not call `model.sendRequest()` from `activate()`.** Every request fails with:
+
+```
+Language model '<model-id>' cannot be used by '<publisher>.<ext-id>'
+```
+
+This is not a quota, auth, or manifest problem — `vscode.lm` grants consent only in response to a genuine user gesture. The probe must therefore be triggered from the Command Palette (or a notification button click). This is the single most expensive trap here; it silently fails 100% of requests and looks like an entitlement bug.
+
+### Never `Stop-Process` filtered on window title
+
+**WARNING:** During this experiment, killing processes matched by window title destroyed the user's unrelated VS Code windows and their unsaved work.
+
+Safe alternative: only ever _launch_ new windows with `--new-window`, and close the probe window by hand. Never bulk-terminate `Code.exe` by title, `MainWindowTitle`, or any other fuzzy match.
+
+### Run tests with pnpm, not npx
+
+```
+pnpm --dir src exec vitest run <path>
+```
+
+Never `npx vitest` — it resolves a wrong hoisted 3.2.4 instead of the pinned 4.1.9 and produces phantom failures.
+
+## Measured Findings
+
+Sample: 210 live requests, 7 Copilot Claude models x 6 scenarios x 5 repeats, 0 errors. The raw transcripts were not retained; the counts below are the retained record of that run, and re-running the probe is the way to regenerate the underlying evidence.
+
+| Scenario | Setup                                     | Runs | `<invoke` in text |
+| -------- | ----------------------------------------- | ---- | ----------------- |
+| A        | tools declared + agent system prompt      | 35   | 0                 |
+| B        | tools declared, no system prompt          | 35   | 0                 |
+| C        | tools declared + ~300KB filler context    | 35   | 0                 |
+| D        | no tools, model asked to emit the markup  | 35   | 14                |
+| E        | asked to quote the markup in prose        | 35   | 23                |
+| F        | asked to quote the markup in a code fence | 35   | 21                |
+
+Observations:
+
+- **The leak did not reproduce.** 105/105 tool-declared runs (A+B+C) emitted a proper `LanguageModelToolCallPart` and leaked nothing into text parts. This bounds the leak rate at a low value; it is **not** proof of absence. 105 runs across 7 models cannot exclude a rare or prompt-specific trigger.
+- **Wrapped vs. bare inverts the intuition.** All 14 genuine emitted invocations (D) were wrapped in `<function_calls>`; 0 were bare. All 44 quoted-in-prose cases (E+F) were bare; 0 were wrapped. In this sample, _bare correlates with quoting and wrapped with genuine invocation_ — so requiring a `<function_calls>` wrapper would not have been the discriminator it appears to be.
+- **No `antml:` prefix appeared** in any of the 210 runs.
+- **Zero false positives.** Replaying `extractLeakedToolCalls()` over all 58 transcripts containing `<invoke` with `validToolNames = {read_file}`: 9 recovered (all genuine wrapped invocations, arguments parsed correctly), 49 passed through as text, including all 44 bare quoted cases. The fenced/quoted guard is what does the work here, not the wrapper requirement.
+
+### Caveats
+
+- Copilot's `vscode.lm` endpoint sits behind its own prompt assembly; results describe that surface, not the raw Anthropic API.
+- The real-world shape of the leak that motivated the recovery code is **inferred** from third-party Anthropic-API reports (anthropics/claude-code#66153, #73808), not captured from `vscode-lm`. No `vscode-lm` transcript of the failure exists.
+
+## Reproducing the False-Positive Replay
+
+[`scripts/probe-vscode-lm-api/probe-false-positives.spec.ts`](../../../scripts/probe-vscode-lm-api/probe-false-positives.spec.ts) replays [`extractLeakedToolCalls()`](../../../src/api/providers/vscode-lm.ts) over a transcript directory and writes a `RECOVERED`/`passthrough` report. It has no committed inputs — run the probe first to produce them. Its `../vscode-lm` import and `TRANSCRIPTS` default are written for the copy destination, not for where it is committed. Drop it into `src/api/providers/__tests__/`, point `TRANSCRIPTS` (or `LM_PROBE_TRANSCRIPTS`) at the probe's `OUT_DIR`, then:
+
+```
+pnpm --dir src exec vitest run api/providers/__tests__/probe-false-positives.spec.ts
+```
+
+It is a scratch harness, not a committed test — remove it afterwards.

scripts/probe-vscode-lm-api/extension.js

@@ -0,0 +1,208 @@
+const vscode = require("vscode")
+const fs = require("fs")
+const path = require("path")
+
+// Absolute: the extension host's cwd is not the repo, so a relative path would scatter output.
+const OUT_DIR = process.env.LM_PROBE_OUT_DIR || "c:\\git\\<repo>\\.tmp\\transcripts"
+
+function write(name, data) {
+	fs.mkdirSync(OUT_DIR, { recursive: true })
+	fs.writeFileSync(path.join(OUT_DIR, name), typeof data === "string" ? data : JSON.stringify(data, null, 2))
+}
+
+const READ_TOOL = {
+	name: "read_file",
+	description: "Read the contents of a file at the given path.",
+	inputSchema: {
+		type: "object",
+		properties: { path: { type: "string", description: "File path to read" } },
+		required: ["path"],
+	},
+}
+
+const TOOL_SYSTEM_PROMPT = [
+	"You are Zoo Code, an autonomous coding agent.",
+	"You accomplish tasks by calling the tools provided to you.",
+	"You MUST call exactly one tool per message. Never ask the user a question.",
+	"Do not answer from memory; always read the file first using the read_file tool.",
+].join("\n")
+
+async function runOnce(model, scenario) {
+	const record = {
+		scenario: scenario.name,
+		modelId: model.id,
+		modelFamily: model.family,
+		modelVendor: model.vendor,
+		modelVersion: model.version,
+		maxInputTokens: model.maxInputTokens,
+		parts: [],
+		concatenatedText: "",
+		toolCallParts: [],
+		error: null,
+	}
+	const messages = []
+	if (scenario.system) {
+		messages.push(vscode.LanguageModelChatMessage.Assistant(scenario.system))
+	}
+	for (const userText of scenario.userMessages) {
+		messages.push(vscode.LanguageModelChatMessage.User(userText))
+	}
+	const options = { justification: "Empirical probe of leaked tool-call formatting." }
+	if (scenario.tools) {
+		options.tools = [READ_TOOL]
+	}
+	const source = new vscode.CancellationTokenSource()
+	try {
+		const response = await model.sendRequest(messages, options, source.token)
+		for await (const chunk of response.stream) {
+			const typeName = chunk && chunk.constructor ? chunk.constructor.name : typeof chunk
+			if (chunk instanceof vscode.LanguageModelTextPart) {
+				record.parts.push({ type: typeName, value: chunk.value })
+				record.concatenatedText += chunk.value
+			} else if (chunk instanceof vscode.LanguageModelToolCallPart) {
+				const call = { type: typeName, name: chunk.name, callId: chunk.callId, input: chunk.input }
+				record.parts.push(call)
+				record.toolCallParts.push(call)
+			} else {
+				record.parts.push({ type: typeName, raw: String(chunk) })
+			}
+		}
+	} catch (error) {
+		record.error = { name: error && error.name, message: error && error.message, stack: error && error.stack }
+	} finally {
+		source.dispose()
+	}
+
+	const text = record.concatenatedText
+	record.markers = {
+		hasInvoke: /<(?:antml:)?invoke\b/i.test(text),
+		hasFunctionCalls: /<(?:antml:)?function_calls\b/i.test(text),
+		hasAntmlNamespace: /antml:/i.test(text),
+		invokeWrappedInFunctionCalls: /<(?:antml:)?function_calls\s*>[\s\S]*?<(?:antml:)?invoke\b/i.test(text),
+		bareInvokeWithoutWrapper: /<(?:antml:)?invoke\b/i.test(text) && !/<(?:antml:)?function_calls\b/i.test(text),
+		insideFencedCodeBlock: /```[\s\S]*?<(?:antml:)?invoke\b/i.test(text),
+	}
+	return record
+}
+
+function buildScenarios() {
+	const longFiller = "This is filler context line used to grow the prompt toward the context window. ".repeat(4000)
+	return [
+		{
+			name: "A_tools_declared_compelling_prompt",
+			tools: true,
+			system: TOOL_SYSTEM_PROMPT,
+			userMessages: ["Read the file c:/git/Zoo-Code/package.json and tell me the version field."],
+		},
+		{
+			name: "B_tools_declared_no_system_prompt",
+			tools: true,
+			userMessages: ["Read the file c:/git/Zoo-Code/package.json and tell me the version field."],
+		},
+		{
+			name: "C_tools_declared_long_context",
+			tools: true,
+			system: TOOL_SYSTEM_PROMPT,
+			userMessages: [longFiller, "Now read the file c:/git/Zoo-Code/package.json and tell me the version field."],
+		},
+		{
+			name: "D_no_tools_asked_to_emit_markup",
+			tools: false,
+			system: TOOL_SYSTEM_PROMPT,
+			userMessages: [
+				"You have a tool named read_file that takes a path. Emit the tool invocation using Anthropic's internal function-call XML format, exactly as you would internally, to read c:/git/Zoo-Code/package.json. Output only the markup.",
+			],
+		},
+		{
+			name: "E_quoted_markup_in_prose_false_positive_check",
+			tools: true,
+			system: TOOL_SYSTEM_PROMPT,
+			userMessages: [
+				'Do NOT call any tool. Instead, explain in prose what an invoke block looks like, and include a literal example using the tag name "invoke" with a name attribute of read_file and a parameter named path, written as plain text in your answer.',
+			],
+		},
+		{
+			name: "F_quoted_markup_in_fenced_code_block",
+			tools: true,
+			system: TOOL_SYSTEM_PROMPT,
+			userMessages: [
+				"Do NOT call any tool. Show me, inside a fenced markdown code block, an example of an invoke block naming the tool read_file with a parameter named path set to /etc/passwd. Only output the fenced code block.",
+			],
+		},
+	]
+}
+
+async function run() {
+	const summary = { startedAt: new Date().toISOString(), vscodeVersion: vscode.version, models: [], runs: [] }
+	let models = []
+	let selectError = null
+	try {
+		models = (await vscode.lm.selectChatModels({ vendor: "copilot" })) || []
+	} catch (error) {
+		selectError = { name: error && error.name, message: error && error.message }
+	}
+	summary.selectError = selectError
+	summary.allModels = models.map((model) => ({
+		id: model.id,
+		family: model.family,
+		vendor: model.vendor,
+		version: model.version,
+		maxInputTokens: model.maxInputTokens,
+	}))
+
+	const claudeModels = models.filter((model) => /claude/i.test(model.id) || /claude/i.test(model.family))
+	summary.claudeModelIds = claudeModels.map((model) => model.id)
+
+	if (claudeModels.length === 0) {
+		write("summary.json", summary)
+		vscode.window.showErrorMessage(
+			`LM Probe: no Claude models. selectChatModels returned ${models.length}. ${selectError ? selectError.message : ""}`,
+		)
+		return
+	}
+
+	const scenarios = buildScenarios()
+	const REPEATS = 5
+	for (const model of claudeModels) {
+		for (const scenario of scenarios) {
+			for (let iter = 1; iter <= REPEATS; iter++) {
+				const record = await runOnce(model, scenario)
+				record.iteration = iter
+				summary.runs.push({
+					scenario: scenario.name,
+					modelId: model.id,
+					iteration: iter,
+					markers: record.markers,
+					toolCallPartCount: record.toolCallParts.length,
+					textLength: record.concatenatedText.length,
+					error: record.error ? record.error.message : null,
+				})
+				write(`${model.id}__${scenario.name}__run${iter}.json`, record)
+				write(`${model.id}__${scenario.name}__run${iter}.txt`, record.concatenatedText)
+			}
+		}
+	}
+
+	summary.finishedAt = new Date().toISOString()
+	write("summary.json", summary)
+	vscode.window.showInformationMessage(`LM Probe complete: ${summary.runs.length} runs written to ${OUT_DIR}`)
+}
+
+function activate(context) {
+	// MUST be user-initiated: vscode.lm consent is only granted from a real user gesture, so an
+	// activation-time sendRequest is auto-denied with "cannot be used by 'scratch.lmprobe'".
+	context.subscriptions.push(
+		vscode.commands.registerCommand("lmprobe.run", () =>
+			run().catch((error) => {
+				write("fatal.json", { message: String(error && error.message), stack: String(error && error.stack) })
+			}),
+		),
+	)
+	vscode.window.showInformationMessage("LM Probe ready", "Run probe").then((choice) => {
+		if (choice === "Run probe") {
+			vscode.commands.executeCommand("lmprobe.run")
+		}
+	})
+}
+
+module.exports = { activate, deactivate() {} }

scripts/probe-vscode-lm-api/package.json

@@ -0,0 +1,21 @@
+{
+	"name": "lmprobe",
+	"displayName": "LM Leak Probe",
+	"version": "0.0.1",
+	"publisher": "scratch",
+	"engines": {
+		"vscode": "^1.95.0"
+	},
+	"activationEvents": [
+		"onStartupFinished"
+	],
+	"main": "./extension.js",
+	"contributes": {
+		"commands": [
+			{
+				"command": "lmprobe.run",
+				"title": "LM Probe: Run"
+			}
+		]
+	}
+}

scripts/probe-vscode-lm-api/probe-false-positives.spec.ts

@@ -0,0 +1,26 @@
+import fs from "fs"
+import path from "path"
+
+// Paths here are written for the copy destination `src/api/providers/__tests__/`, not for this
+// file's committed location — it is a template to be copied there, never run in place.
+import { extractLeakedToolCalls } from "../vscode-lm"
+
+// Point this at the probe's OUT_DIR. Scratch harness: not a committed test.
+const TRANSCRIPTS = process.env.LM_PROBE_TRANSCRIPTS ?? path.resolve(__dirname, "../../../../.tmp/transcripts")
+
+describe("probe: false-positive check against real transcripts", () => {
+	it("reports which transcripts extractLeakedToolCalls would treat as real calls", () => {
+		const names = new Set(["read_file"])
+		const report: string[] = []
+		for (const file of fs.readdirSync(TRANSCRIPTS).filter((name) => name.endsWith(".txt"))) {
+			const text = fs.readFileSync(path.join(TRANSCRIPTS, file), "utf8")
+			if (!/<(?:antml:)?invoke/i.test(text)) {
+				continue
+			}
+			const { calls } = extractLeakedToolCalls(text, names)
+			report.push(`${calls.length > 0 ? "RECOVERED" : "passthrough"}\t${file}\t${JSON.stringify(calls)}`)
+		}
+		fs.writeFileSync(path.join(TRANSCRIPTS, "..", "false-positive-report.txt"), report.join("\n"))
+		console.log(report.join("\n"))
+	})
+})