Multimodal input: analyze_file, MCP binary capture, and a fixture server (#513)

[rockfordlhotka]

Steps (A), (B) and (C) of the multimodal plan in #513: binary content now reaches the shared volume instead of the model's context, and a model that can actually see it can be asked about it.

The problem

RockBot has been text-only end to end. Every model it talks to can see, but nothing in the framework could put a non-text byte in front of one. #513 found this from the outside: an agent asked an MCP server for an image, got 167K characters of textual representation back, chunked it into working memory, and never showed the model an image.

Five separate chokepoints, all verified in the tree and inventoried in the design doc. The one that shapes this PR:

Images cannot ride in a tool result. RegistryToolFunction already maps MCP image blocks to DataContent, which looks like a working path. It isn't. The text-based tool-calling path reduces the result with result?.ToString(), so a List<AIContent> becomes a bare type name. And on the FICC path, where it survives, every provider is reached through OpenAIClient(...).AsIChatClient() — and the OpenAI Chat Completions wire format accepts only text in tool-role messages. A DataContent there is JSON-serialised into a data-URI string: full base64 token cost, no image.

On OpenAI-compatible APIs, bytes can only enter as content parts on a user message.

What this adds

analyze_file(path, prompt, tier) — a side call rather than a richer tool result, which sidesteps the constraint entirely:

analyze_file({ path: "attachments/architecture.png",
               prompt: "Describe the components and how they connect.",
               tier: "high" })
    -> containment check, MIME allowlist, size limit
    -> ILlmClient.GetResponseAsync([User: TextContent(prompt), DataContent(bytes, mime)], tier)
    -> "Three services arranged left to right. ..."

A path goes in, prose comes out. Bytes never touch the agent's context, the message bus, or the context budget — which is why the byte-blind EstimateMessageChars gap can wait for the inbound-attachment work.

LlmTierConfig.SupportsImageInput is the second half. It is opt-in per tier because a blind model rejects image content with an opaque provider error deep in the stack; making the operator say "this model can see" is one config line and removes the whole class of failure.

Notable decisions

• Not registered unless a tier declares vision. Offering the tool otherwise teaches the model a capability the deployment does not have, which it then spends turns trying to use. The file-tools skill guide documents the tool under the same predicate — shared via VisionTiers.From so the two cannot disagree whichever hosted service starts first.
• A requested tier that cannot see is substituted, not attempted. ILlmClient retries a failed Low/High call on Balanced. Sending a vision request to a blind tier would fail twice and surface the less informative second error.
• LlmTierOptions is now registered in DI by the agent, after the backward-compat fixups, so anything reading it sees the same resolved tiers the chat clients were built from. Consumers that do not register it get an analyze_file that never registers — the dependency is optional and its absence reads the same as "no tier declares vision".
• Lives in RockBot.Tools.FileSystem, which already references RockBot.Host (so ILlmClient is in scope), already owns SafeResolvePath, and already defaults its base path to /rockbot/shared — so the MCP attachment gateway's output directory is already inside its reachable scope.

Enabling it

{ "LLM": { "High": { "ModelId": "openai/gpt-5.5", "SupportsImageInput": true } } }

Or LLM__High__SupportsImageInput=true. Nothing changes for a deployment that sets neither.

(C) Binary capture in the MCP bridge

analyze_file only helps once bytes are a file on the shared volume, and the attachment gateway only gets them there for servers that implement RockBot's convention. Most don't — including the one in the issue. BinaryResponseCapture runs on every MCP response, for every server, manifest or not:

• Typed image/audio content blocks are captured with no configuration. MCP has already labelled them, so nothing is guessed. Bytes go to the attachments directory, the block becomes {path, name, size, mime, note}, other blocks are untouched.
• Base64 inside a JSON response needs a declared rule. Sniffing for "a field that looks like base64" is the fragile heuristic the attachment design explicitly rejected, so this half stays declarative — and Gitea's shape needs no server change, only a description of the response it already sends:

{ "attachments": { "capture": { "rules": [
  { "tools": ["get_file_contents"], "contentField": "content",
    "nameField": "name", "encodingField": "encoding" }
] } } }

The load-bearing detail is deciding what is actually binary. A repository server returns a README and a PNG through the same tool and the same field, so capturing on the presence of base64 alone would take away text the model could simply have read. Name or MIME decides when there is one — SVG deliberately excluded, an image by MIME and text by nature — and otherwise the bytes do, via a NUL byte or a strict UTF-8 decode failure.

Capture never fails a tool call: a bad rule, an unwritable volume, or a payload that isn't what the rule claimed logs and passes the server's original response through. That is the opposite of the outbound gateway's stance, deliberately — outbound, a failure means the model's file never got attached and silence would be a lie; inbound, capture is an optimisation on a response that is already complete.

The response object survives the rewrite. Only the content field is removed, so sha, url, and whatever else the server sent stay alongside the added path descriptor.

Content-block payloads were being read wrong

Found by pointing a live agent at the fixture server below, and it affects any MCP server that returns typed image or audio blocks — which is to say it was broken before this PR too, just invisibly, because nothing in the reference deployment returns that shape.

ImageContentBlock.Data is typed ReadOnlyMemory<byte>, which reads like "the file's bytes". In SDK 1.4.0 it is not: it carries the wire field verbatim, and the wire field is base64 text. So capture wrote Data.ToArray() straight to disk and a 783-byte PNG landed as 1044 bytes of iVBORw0KGgo… under a .png name, which the vision model then rejected as invalid_image_format. The same field was being double-encoded in McpToolExecutor.MapContentBlocks (Convert.ToBase64String over already-base64 bytes) — pre-existing, fixed alongside.

McpBinaryPayload reads either convention, so an SDK version that starts storing decoded bytes needs no change.

The fixture server

McpServer.BinaryFixture returns each binary shape on demand — typed image and audio blocks, a repository server's base64-in-JSON for both an image and a text file, binary mangled into text, and plain text as the control. It exists because none of those shapes occur in a real deployment, which is how both bugs in this PR stayed hidden behind unit tests that had encoded assumptions rather than observations.

Payloads are generated in code: no binary blobs in the repo, and — the actual point — the image's content is documented, so a vision model's description is checked against a known answer instead of an impression of a photo. The fixtures are also served over plain HTTP (/fixtures/chart.png) for when a live result and a model's account of it disagree.

It sits outside the Helm chart deliberately (deploy/k8s/mcp-binary-fixture.yaml, applied for a test and deleted afterwards) and is IsPackable=false. See src/McpServer.BinaryFixture/README.md.

Not in this PR

design/multimodal-input.md sequences what remains, now filed:

• Blazor and CLI cannot send files to the agent #565 — Blazor and CLI cannot send files to the agent. The other direction from attach_image: UserMessage.Attachments, an upload control, --attach, and the loop injecting image parts onto the user message. Also carries the one-word ClientCapabilityPresets.Blazor fix (it omits ImageAttachment, so the agent is never told it may attach images to Blazor replies even though the client renders them).
• Context-size estimation is blind to non-text content parts #564 — context-size estimation counts a DataContent as 50 characters, so images are effectively invisible to the watermark trim. Unreachable today, because nothing puts image parts into the agent's own message list; Blazor and CLI cannot send files to the agent #565 is what makes it reachable, so it should land first.

Two claims in the design doc were corrected while filing those: EstimateMessageChars scores non-text parts at a flat 50 rather than zero, and inbound attachments need no schema migration — adding an optional property to ConversationTurn is additive, and the conversation store is not enrolled in schema migrations at all. Both had been written from the shape of the code rather than its detail.

Version: 0.15.0, not a patch

Two things here are behavioural changes for an existing consumer, and a patch release should not carry them:

• Binary capture is on by default for every MCP server, with no opt-in. A consumer whose server returns typed image or audio blocks now receives a {path, name, size, mime} descriptor where it previously received the block.
• McpToolExecutor.MapContentBlocks stops double-encoding those payloads, so ToolContentBlock.Data changes shape for the same consumers.

Plus new public surface across three packages (LlmTierConfig.SupportsImageInput, FileSystemOptions.AnalyzeFileMaxBytes/AnalyzeFileMimeTypes, AttachmentManifest.Capture and its config types, BinaryResponseCapture), a new chart value, and a capability the framework did not have at all.

The 0.14.43–0.14.47 trail on the branch was an artefact of needing a fresh image tag per round of live testing; it is collapsed into this one number.

Testing

48 new tests. analyze_file (15): containment rejection, missing file, unrecognised extension, allowlist enforcement in both directions, the size limit, tier honouring and substitution, multimodal content shape (prompt text and bytes both arrive intact), empty response, and provider exceptions. Capture (19): image and audio blocks, mixed blocks, text-only responses left alone, the disable switch, error results, the Gitea shape end to end, a README through the same field left in the response, sniffing fallback in both directions, wrong encoding, malformed base64, unmatched tools, no rules, MIME override, storage failure, and binary a server mangled into text (dropped with an explanation rather than flooding context). Payload decoding (7): base64-text and raw-byte payloads both read correctly, no double-encoding, round trip. Storage containment (7): traversal names, absolute paths, subdirectories, empty-name fallback, collisions, and read-side rejection — pinned because capture now names files from a remote server's response. Full suite green: 3198 passed, 0 failed. Capture was also smoke-tested against a live MCP server, and against McpServer.BinaryFixture (added here) which returns every shape on demand — see the comments. Those two runs are what turned up the mangled-binary case and the base64-payload bug respectively.

Closes #513

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf8a2Z1v9dY3YjjM42zJEj

[rockfordlhotka]

Verified live on the reference deployment (agent 0.14.43 built from this branch, all three tiers flagged SupportsImageInput=true via agent.extraEnv — no chart change needed).

Vision support, measured rather than assumed. A throwaway in-cluster probe pod (envFrom the same config/secret sources the agent uses, so keys never left the cluster) POSTed a 32×32 solid-red PNG as an image_url content part to each configured tier's real endpoint:

Tier	Model	Result
Balanced	gpt-5.4	HTTP 200 → "red"
High	gpt-5.5	HTTP 200 → "Red"
Low	gpt-5.4-mini	HTTP 200 → "red"

All three on the Azure Foundry openai/v1 endpoint. The Balanced fallback chain (gpt-5.4 → gpt-5.4-mini → google/gemini-3-flash-preview → anthropic/claude-haiku-4.5) is vision-capable the whole way down, so declaring Balanced as seeing does not create a blind fallback.

Registration gate works:

Registered file tool: analyze_file (vision tiers: Low, Balanced, High)

End to end. A 240×120 PNG was written to the shared volume: three bars, left to right red (h=60), green (h=95), blue (h=40). Asked through the CLI, the agent chose the tool and the High tier on its own:

Executing tool analyze_file(path=vision-test-chart.png, prompt=Describe exactly what this image shows…, tier=high)
analyze_file: vision-test-chart.png (image/png, 494 bytes) on High tier
LLM call: tier=High model=gpt-5.5

Final reply: "It shows 3 bars. Left to right, the colours are red, green, blue. By height, blue is shortest, red is in the middle, and green is tallest."

Correct on count, on left-to-right colour order, and on the height ordering — which is only recoverable by actually looking at the image. Test file removed from the shared volume afterwards.

Includes a version bump to 0.14.43.

[rockfordlhotka]

Added secrets.llm.<tier>.supportsImageInput as a first-class chart value (8a963e9), replacing the agent.extraEnv escape hatch used for the earlier live test.

It follows the reasoningEffort pattern in secret.yaml and is emitted only when true, so a release that doesn't set it renders byte-identically to before:

secrets:
  llm:
    high:
      modelId: gpt-5.5
      supportsImageInput: true

balancedModels entries deliberately take no flag of their own — the app reads SupportsImageInput per tier, not per fallback model, so one there would be silently inert. values.yaml documents that along with its consequence: if balanced.supportsImageInput is true, every model in the fallback chain must read images, or a fallback gets handed one and fails.

Verified on the reference deployment: the three keys now render into the Secret and no longer into the ConfigMap, the unrelated Dream__* extraEnv entries survived the migration untouched, and after a restart the agent picks the values up from the new source:

LLM__Balanced__SupportsImageInput=true
LLM__High__SupportsImageInput=true
LLM__Low__SupportsImageInput=true
Registered file tool: analyze_file (vision tiers: Low, Balanced, High)

Worth noting for whoever changes this later: the chart has no checksum annotation on the Secret, so the Helm upgrade alone left the running pod on its old env — LLM config changes still need an explicit rollout restart. That's pre-existing behaviour, not something this PR changes.

[rockfordlhotka]

Smoke-tested capture against a live MCP server, and it found a failure mode the design didn't anticipate — now fixed (fc76780).

Baseline

github/get_file_contents for docs/assets/images/rockbot.png (345 KB), on 0.14.43:

← MCP github/get_file_contents OK in 1973ms (1366356 chars)
Tool result for 'mcp_invoke_tool' is large (1,366,356 chars) and has been split into 22 chunk(s)
Truncating embedding input for '…chunk0' from 64089 to 12000 chars
… ×22

1.37 million characters, 22 working-memory chunks, 22 embedding calls. The issue reported 167K; this is eight times worse.

What the smoke test found

Capture didn't fire, and was right not to. Reading the raw payload out of working memory rather than trusting the agent's summary:

'"content": "\uFFFDPNG\r\n\u001A\n\u0000\u0000\u0000\rIHDR…'

That is not base64. This server decodes file bytes as UTF-8 before returning them, so the PNG's leading 0x89 arrives as U+FFFD and the bytes are destroyed at the source. Convert.FromBase64String fails, the rule declines, and the response passes through untouched — the correct answer to "can you save these bytes?", but the wrong outcome overall: the flood still happened, and an agent reading mojibake tends to retry the same call.

Worth noting the issue's Gitea case is genuinely base64 (167K chars matches base64 of a ~125 KB image), so this is a second, distinct shape — one only a live test was going to surface.

The fix

Under a declared rule, a content field that fails base64 decoding is now checked for the signature of lossy binary-to-text decoding: eight or more U+FFFD in at least a kilobyte, thresholds set so a document with a couple of encoding glitches is still a document. On a match the field is dropped and replaced with a note; name, sha, size and html_url survive, which is usually enough to fetch the file another way.

Nothing is lost — the bytes were already unrecoverable when they arrived.

After (0.14.45, same call)

Binary capture: github/get_file_contents returned binary as text in 'content'
  (327935 chars, unrecoverable); dropping the field rather than letting it into context
← MCP github/get_file_contents OK in 1361ms (588 chars)

1,366,356 chars and 22 chunks → 588 chars and none. The agent reported the note back accurately and did not retry.

Regression check

Same tool, same capture rule, CODE_OF_CONDUCT.md: no capture fired, and the agent quoted # Code of Conduct from the actual file content. Text through a rule-bearing tool still arrives as text — the property that matters most, since a repository server returns prose and images through the same field.

What this did not exercise

No server in that fleet returns base64 file content or typed image/audio content blocks, so the positive capture paths — bytes actually written to the shared volume — remain covered by unit tests (19 of them) rather than live traffic. Happy to stand up a purpose-built MCP server if that's worth proving end to end.

[rockfordlhotka]

Added McpServer.BinaryFixture — an MCP server that returns binary in every shape the bridge has to cope with — and pointing a live agent at it immediately found two real bugs that the unit tests had encoded as assumptions rather than observations.

Bug 1: content-block payloads are base64 text, not bytes

ImageContentBlock.Data is typed ReadOnlyMemory<byte>, which reads like "the file's bytes". In SDK 1.4.0 it is not — it carries the wire field verbatim, and the wire field is base64 text.

Capture was writing Data.ToArray() straight to disk, so a 783-byte PNG landed as 1044 bytes of iVBORw0KGgo… under a .png name. The agent got a plausible-looking path and analyze_file came back with:

Error: Analysis failed: HTTP 400 (invalid_request_error: invalid_image_format)

1044 is exactly the base64 length of 783 bytes, which is what gave it away. McpBinaryPayload now reads either convention, so a future SDK that starts storing decoded bytes needs no change. The same field was being double-encoded in McpToolExecutor.MapContentBlocks (Convert.ToBase64String over already-base64 bytes) — pre-existing, fixed alongside.

Bug 2: the same thing in reverse, in the fixture

Handing raw PNG bytes to Data serialises them as mojibake and the receiving client rejects the block outright:

Error: The JSON value could not be converted to ModelContextProtocol.Protocol.ContentBlock. Path: $.content[0]

The fixture now writes base64-as-bytes, with the reason documented where it's used. Verified against the wire JSON rather than inferred.

The fixture server

Tool	Shape	Live result
get_image	Typed image block	Captured → valid 783-byte PNG (89 50 4e 47 magic)
get_audio	Typed audio block	Captured → 4044-byte WAV, exact size
get_image_with_text	Text + image blocks	Text preserved verbatim, only the image rewritten
get_file_base64 kind=image	Repository-style base64 JSON	Captured, content stripped, sha/size kept
get_file_base64 kind=text	Same shape, markdown	Not captured — content stayed readable
get_file_mangled	Binary decoded as UTF-8	218,751 chars → 482-char explanation
describe_fixtures	The image's known description	Used to check the vision answer

End to end, unprompted by me — the agent chose the tools itself:

Binary capture: binary-fixture/get_image image/png block (783 bytes) → /rockbot/shared/attachments/get-image-4e8a1d98.png
analyze_file: attachments/get-image-4e8a1d98.png (image/png, 783 bytes) on Balanced tier

analyze_file: 3 bars. Colours left to right: red, green, blue. Tallest: green. Shortest: blue.
describe_fixtures: Three vertical bars on a light background, left to right: red (medium height), green (tallest), blue (shortest), sitting on a dark horizontal baseline.

The payloads are generated in code, so the repo carries no binary blobs — but the real reason is that the image's content is documented, so a vision model's answer is checked against a known one rather than against an impression of a photo. Fixtures are also served over plain HTTP (/fixtures/chart.png) for when a live result and a model's account of it disagree.

One case needed the fixture itself fixed rather than the code: an 783-byte flat-colour PNG mangles to ~700 characters, below the guard's 1 KB floor, so the mangled fixture now uses an incompressible noise image at a realistic size.

Cluster left clean

Fixture deployment deleted, binary-fixture deregistered from mcp.json, captured test files removed. The github capture rule stays — it's a genuine improvement. The fixture lives outside the Helm chart deliberately: applied for a test, deleted afterwards, per src/McpServer.BinaryFixture/README.md.