Skip to content

Let people send files to the agent - #567

Merged
rockfordlhotka merged 2 commits into
mainfrom
issue-565/blazor-cli-send-files-to-agent
Sep 7, 2026
Merged

Let people send files to the agent#567
rockfordlhotka merged 2 commits into
mainfrom
issue-565/blazor-cli-send-files-to-agent

Conversation

@rockfordlhotka

Copy link
Copy Markdown
Member

Summary

Attachments were outbound only. The agent could attach an image to a reply (attach_imageAgentAttachment → Blazor's /attachments endpoint), but someone with a screenshot, a scanned invoice, or a photo of a whiteboard had nowhere to put it. This is the missing direction.

#563 gave the agent analyze_file; #564 made the context estimate honest about DataContent, which is what made inline images safe to land.

The upload goes through the agent, not the file share

The issue sketched a Blazor upload writing into the shared directory. That would have broken the rule the chart states plainly — "Read-only: Blazor only serves attachment bytes; the agent owns writes" — so it isn't what shipped.

Instead: AttachmentUploadRequest carries the bytes to the agent, which writes them through its existing IAttachmentStorage and returns an AgentAttachment path reference. The message the user then sends carries only that reference.

client writes the share upload through the agent
Blazor shared mount must become read-write stays readOnly: true
Containment, sanitise, allowlist, size cap once per client once, agent-side
CLI from a laptop impossible (no mount) works
Chart change required, plus fsGroup risk none

This is the one place bytes cross the bus, and it is deliberately its own request/reply rather than a field on UserMessage — so the conversation message, the persisted turn, and every history replay stay byte-free. (AgentAttachment's "bytes never ride the bus" is about the outbound direction, where the frontend already mounts the volume and shipping bytes would be waste; inbound is the opposite case.)

Validation is authoritative agent-side: image/* plus PDF, an 8 MB cap matching AnalyzeFileMaxBytes, and a check that the declared type agrees with the extension — so a file cannot be stored under a name the next reader resolves differently. Clients pre-check the same rules only to spare a pointless round trip.

What the model sees

InboundAttachmentInjector appends the current turn's attachments to the last user message — bytes can only enter a conversation as content parts on a user message on OpenAI-compatible APIs:

  • image and the tier declares SupportsImageInputDataContent, the real thing
  • otherwise (blind tier, a PDF, a file that has gone missing) → a line naming the path and pointing at analyze_file

A file someone deliberately attached is never silently dropped. That is the behaviour with the most tests behind it.

Replayed history turns keep only the marker line. Re-materialising every image still in the context window would cost a disk read per image per request and leave several thousand image tokens standing in context; the model can still reach an older image deliberately through analyze_file on the path the marker names.

Also here

  • UserMessage.Attachments and ConversationHistoryTurn.Attachments reuse AgentAttachment outright rather than adding a parallel type, so AttachmentList.razor and AttachmentPlaceholder render user attachments with no change.
  • ConversationTurn.Attachments is additive — per design/schema-migrations.md no migration, and the conversation store is not enrolled in them anyway.
  • Blazor: a 📎 picker, staged chips with remove, upload-in-progress and error states, send enabled by an attachment alone. OpenReadStream is passed an explicit maxAllowedSize (the default is 512 KB and throws past it).
  • CLI: --attach <PATH>, repeatable.
  • The bundled one-word fixClientCapabilityPresets.Blazor now declares ImageAttachment. The test asserting it should not had already documented its own expiry: "until AgentReply.Attachments lands", which it has.
  • design/multimodal-input.md — concern (D) marked landed, with the gateway decision and the current-turn-only rule recorded.

Deliberate limits

  • --attach requires --message (one-shot). Attaching per-message in the REPL needs an affordance it does not have; a clear error beats quietly attaching to whichever message happened to go first.
  • PDFs never ride inline — only image/* becomes DataContent; a PDF is announced with its path.
  • A picked-then-abandoned file stays on the volume. The upload happens on pick, and nothing reaps the shared directory today (agent-produced attachments accumulate the same way). Worth a follow-up on retention rather than solving here.

Test plan

  • AttachmentUploadHandlerTests (8) — valid PNG writes and returns a relative reference; a second file of the same name does not clobber the first; disallowed type rejected with a message naming what is accepted; declared type disagreeing with the extension rejected; both JPEG extensions accepted; over-cap and empty rejected before touching disk; a traversal in the supplied filename stays inside the base directory
  • InboundAttachmentInjectorTests (9) — image on a seeing tier becomes DataContent and no other message is touched; blind tier gets the marker, no bytes, and no disk read; a PDF is announced even on a seeing tier; an unreadable file degrades to a marker that says so rather than failing the turn; several attachments all land; no attachments leaves the context byte-identical; no user message does not throw
  • ChatCommandAttachTests (4) — --attach without --message fails before a host is built; the option binds as string[] so it is genuinely repeatable; MIME guessing covers the accepted types and defers to the agent for anything else
  • ChatStateServiceLoadHistoryTests (+3) — attachments survive a history reload, absent ones stay null, and a sent message shows its attachment immediately
  • ClientCapabilitiesTests — the Blazor preset assertion flipped
  • Full suite: dotnet test RockBot.slnx — 20 assemblies, 0 failures

Not yet exercised end-to-end against a running stack; the compose path (ROCKBOT_SHARED_PATH as a local directory) is the cheap way to do that.

Closes #565

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf8a2Z1v9dY3YjjM42zJEj

rockfordlhotka and others added 2 commits September 7, 2026 01:09
Attachments were outbound only. The agent could attach an image to a reply,
but someone with a screenshot, a scanned invoice, or a photo of a whiteboard
had nowhere to put it.

The upload goes through the agent, not the file share. Frontends co-mount the
shared volume read-only ("the agent owns writes") and the CLI does not mount
it at all, so a client cannot stage a file itself. AttachmentUploadRequest
carries the bytes to the agent, which writes them through its existing
IAttachmentStorage and returns an AgentAttachment path reference; the message
the user then sends carries only that reference.

Letting each client write directly would have meant flipping the Blazor mount
to read-write, duplicating the containment check, filename sanitisation, MIME
allowlist and size cap per client, and still leaving the CLI unable to attach
anything. Validation is authoritative agent-side: image/* plus PDF, an 8 MB
cap matching AnalyzeFileMaxBytes, and a check that the declared type agrees
with the extension so a file cannot be stored under a name the next reader
resolves differently.

This is the one place bytes cross the bus, and it is its own request/reply
rather than a field on UserMessage, so the conversation message, the persisted
turn and every history replay stay byte-free.

InboundAttachmentInjector appends the current turn's attachments to the last
user message — a DataContent when the attachment is an image and the tier
declares SupportsImageInput, otherwise a line naming the path and pointing at
analyze_file. A file someone deliberately attached is never silently dropped.
Replayed history turns keep only the marker: re-materialising every image in
the window would cost a disk read per image per request and leave thousands of
image tokens standing in context.

ConversationTurn.Attachments is additive, so per schema-migrations.md it needs
no migration and the conversation store is not enrolled in them anyway.

Also flips ClientCapabilityPresets.Blazor to declare ImageAttachment. The test
asserting it should not had already documented its own expiry — "until
AgentReply.Attachments lands", which it has.

Closes #565

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uf8a2Z1v9dY3YjjM42zJEj
…turn

Two changes, both found by pushing on the first cut.

An API alternative to the bus. Routing every upload through RabbitMQ meant
multi-megabyte bodies sitting in broker memory until they were acked, to move a
file onto a volume the agent already mounts. AttachmentUploadEndpoint is a small
Kestrel listener in the agent process: POST /attachments writes it directly.
Clients use it when AttachmentUploadUrl is configured and fall back to the bus
when it is not or the attempt fails, so a CLI on a laptop keeps working. A
rejection is not a fallback trigger — a file the agent refused over HTTP would be
refused over the bus, and retrying would only make the user wait twice.

The validation rules moved into InboundAttachmentService so both transports share
them. An allowlist enforced on one path and not the other is no allowlist at all.

The chart gains a ClusterIP Service for the agent (it had none) and points Blazor
at it. The endpoint has no auth of its own — same posture as the introspection
sidecar in that Pod — so the Service is deliberately ClusterIP and documented as
not safe to expose without auth in front.

An attachment now survives its own turn. On a seeing tier the image was injected
as DataContent and the path never entered the persisted turn text, so a follow-up
had nothing to reach. A live Blazor session found it: asked to look again, the
agent answered "I don't have the image in this chat to re-check" and called
file_list hunting for it. design/multimodal-input.md claimed replayed turns keep
a marker; they did not. DescribeAttachments now appends the path to a replayed
turn, and the same session answers correctly from the file afterwards.

Every unit test passed through that defect because they all exercise a single
turn.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uf8a2Z1v9dY3YjjM42zJEj
@rockfordlhotka
rockfordlhotka merged commit 4f04133 into main Sep 7, 2026
2 checks passed
@rockfordlhotka
rockfordlhotka deleted the issue-565/blazor-cli-send-files-to-agent branch September 7, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Blazor and CLI cannot send files to the agent

1 participant