Tracking: expand first-class built-in (provider-executed) tools

[cpsievert]

chatlas currently ships two first-class built-in (provider-executed) tools: tool_web_search() and tool_web_fetch() (#248), plus the generic ToolBuiltIn escape hatch (#134). This is a tracking issue for expanding that lineup, based on a July 2026 survey of the provider-executed tools offered by Anthropic (Messages API), OpenAI (Responses API), and Google (Gemini API / Vertex AI).

What the survey found

The three first-class providers have converged on a similar set of server-executed tools — the kind that fit the existing ToolBuiltIn model, where the provider runs the tool and chatlas only needs to send a config and parse result content:

Tool	Anthropic	OpenAI (Responses)	Google Gemini
Web search	✅ GA (shipped)	✅ GA (shipped)	✅ GA (shipped)
Web fetch / URL context	✅ GA (shipped)	❌	✅ GA (shipped)
Code execution	✅ GA	✅ GA (code_interpreter)	✅ GA
Remote MCP connector	✅ beta	✅ GA (mcp)	⚠️ experimental (not Vertex)
File search (hosted RAG)	❌	✅ GA	✅ GA (not Vertex)
Tool search / deferred tools	✅ GA	✅ GA	❌
Image generation	❌	✅ GA	❌ (separate models)
Maps grounding	❌	❌	✅ GA

A second family is provider-defined but client-executed (Anthropic: memory / bash / text editor / computer use; OpenAI: computer use / apply_patch / local shell; Google: computer use). These have official schemas but the application must execute them, so they don't fit ToolBuiltIn — supporting them means shipping an execution harness, a much larger design.

Sub-issues

Roughly by priority:

• feat: tool_code_execution() — provider-executed code execution #364 — tool_code_execution(): the only remaining server tool that is GA across all three providers
• feat: server-side MCP connector as a built-in tool #365 — server-side MCP connector: complements the existing client-side register_mcp_tools_* support
• Built-in tools should fail loudly on providers/platforms that don't support them #367 — make built-in tools fail loudly on providers/platforms that don't support them (pre-existing bug; grows in importance with each new tool)
• Scope decision: file search / provider-hosted RAG as a built-in tool? #366 — (decision) file search / provider-hosted RAG: in tension with the "no RAG primitives" non-goal from Tracking: competitive review workstreams (July 2026) #360; may close as "no"

Explicitly not planned (for now)

• Computer use, bash, text editor, apply-patch, shell — client-executed; chatlas would have to ship a screenshot/action or command-execution harness. Out of scope for a chat client.
• Image generation (OpenAI-only) and Maps grounding (Gemini-only) — single-provider; the ToolBuiltIn escape hatch already handles them (image generation is the worked example in Using built-in tools #134). A docs recipe may be worth more than first-class wrappers.
• Anthropic tool search / advisor tool — too new/niche; revisit if users accumulate large deferred-tool catalogs.
• Anthropic memory tool — client-executed but cheap to back with a filesystem; interesting, but a different pattern from ToolBuiltIn. Revisit separately if there's demand.

Cross-provider caveats that affect all of the above

• Amazon Bedrock supports essentially none of Anthropic's server tools; Vertex AI supports only basic Anthropic web search. Built-in tools should fail loudly on providers/platforms that don't support them #367 covers surfacing this early.
• Gemini models before Gemini 3 can't mix built-in tools with custom function declarations in one request.
• ellmer currently has no code execution or structured citation model, so this is an area where chatlas leads; worth coordinating naming/design with the ellmer team as these land.