Skip to content

docs(#3996): describe portable generated media and Gemini output limits #4030

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aheritier wants to merge 3 commits into from
Open

docs(#3996): describe portable generated media and Gemini output limits #4030

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0df48c0
docs: explain the workspace artifact lifecycle
aheritier Jul 27, 2026
4c0d297
docs: describe generated-image requests and workspace-only saves
aheritier Jul 27, 2026
4b04a89
docs: clarify uncapped generated-media storage and resolution
aheritier Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 28 additions & 22 deletions docs/configuration/models/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ models:
pdf: boolean # Optional: whether the model accepts PDF attachments
audio: boolean # Optional: whether the model accepts audio attachments
video: boolean # Optional: whether the model accepts video attachments
output_capabilities: # Optional: owner-declared generative output capabilities (never inferred)
image: boolean # Optional: whether the model is declared able to generate image output
output_capabilities: # Optional: override generative output capabilities (otherwise detected from models.dev)
image: boolean # Optional: whether the model can generate image output
cost: # Optional: explicit token pricing (USD per 1M tokens)
input: float # Optional: price per 1M input tokens
output: float # Optional: price per 1M output tokens
Expand Down Expand Up @@ -76,7 +76,7 @@ models:
| `track_usage` | boolean | ✗ | Track and report token usage for this model |
| `routing` | array | ✗ | Rule-based routing to different models. See [Model Routing](../routing/index.md). |
| `capabilities` | object | ✗ | Override attachment (input) capabilities for this model. See [Attachment Capability Overrides](#attachment-capability-overrides). |
| `output_capabilities` | object | ✗ | Owner-declared generative output capabilities for this model, e.g. image generation. Never inferred. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). |
| `output_capabilities` | object | ✗ | Override generative output capabilities for this model, e.g. image generation. Omitted flags are detected from models.dev; explicit values take precedence. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). |
| `cost` | object | ✗ | Explicit token pricing in USD per 1M tokens, overriding the built-in catalogue. See [Custom Token Pricing](#custom-token-pricing). |
| `provider_opts` | object | ✗ | Provider-specific options (see provider pages) |
| `title_model` | string | ✗ | Model used for session-title generation. Can be a named model from the `models:` section or an inline `provider/model` string. When omitted, the agent's primary model generates titles. Cannot be combined with `first_available`. |
Expand Down Expand Up @@ -151,14 +151,15 @@ See [`examples/capability-overrides.yaml`](https://github.com/docker/docker-agen
[`examples/strip-unsupported-media.yaml`](https://github.com/docker/docker-agent/blob/main/examples/strip-unsupported-media.yaml) for a fixture demonstrating the
stripping behaviour with and without an override.

## Output Capabilities
### Output capabilities

`output_capabilities` declares what a model can generate, as opposed to
`capabilities`, which declares what it accepts as input. There is no
automatic detection for output capabilities: no catalogue of
output-capable models exists, and matching on the model name string is
deliberately avoided as unreliable. A model's output capabilities are
therefore always unknown/off unless the owner declares them.
`output_capabilities` overrides what a model can generate, as opposed to
`capabilities`, which overrides what it accepts as input. Resolution follows
one precedence chain: explicit `false`, explicit `true`, then an exact
models.dev record whose `Modalities.Output` contains `image`. An omitted image
flag (including `output_capabilities: {}`) therefore uses catalogue metadata;
an unknown model or unavailable catalogue leaves image output disabled. Docker
Agent never infers this capability from the model name.

```yaml
models:
Expand All @@ -173,18 +174,23 @@ models:
| --------------------------- | ------- | -------------------------------------------------------------|
| `output_capabilities.image` | boolean | Whether the model is declared able to generate image output |

Omitting `output_capabilities`, or leaving `image` unset or `false`, always
preserves existing behavior. Setting it to `true` only opts the model into
behavior that specifically keys off a declared image-output capability (for
example, a provider-specific request-shape guard); it does not by itself
change what Docker Agent sends to or renders from the model.

One side effect of declaring `image: true`: the model is skipped as a
session-title candidate, because titles are generated by a plain text-only
completion that image-output routes can reject. Title generation uses the
first non-image-output candidate (dedicated `title_model`, then the agent's
model, then its fallbacks); when every candidate declares image output, the
automatic title is skipped and the session keeps its default title.
Omitting `output_capabilities`, using an empty block, or omitting `image` uses
models.dev metadata for that exact model when available. Setting `image`
explicitly overrides the catalogue; an explicit `false` has highest precedence
and disables image response modalities even when the catalogue lists image
output. Enabling image output only opts the model into behavior that keys off
that capability (for example, a provider-specific image-output request
contract); it does not guarantee that a provider will return an image.

Session-title and compaction requests omit image response modalities and
bypass the guard even for image-output-capable models; they do not explicitly
force TEXT-only output. On supported Google surfaces, the guard runs only when
image output resolves as enabled and an ordinary request includes custom tools
or structured output; matching requests are rejected locally. Google
server-side built-ins remain available. For a custom-tool conflict,
models.dev's `tool_call` capability makes the error say
whether the model lacks tool calls entirely or only cannot combine them with
image output; unavailable metadata keeps a conservative generic message.

> [!WARNING]
> **Constraint**
Expand Down
31 changes: 31 additions & 0 deletions docs/features/sessions/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,37 @@ Override the location with `-s`/`--session-db`, or by overriding the data direct
$ docker agent run agent.yaml --session-db ./sessions.db
```

## Generated Media Files

Some models (e.g. Gemini image-output models) can generate an image as part
of a reply. Docker Agent saves each generated image into the session's
workspace as an ordinary, untracked file. After the file and its manifest
entry are saved, it stores a portable copy in `session.db`. Reopening or moving
the session can therefore render the original generated bytes even when the
workspace file was edited, moved, or deleted. Ordinary follow-up turns replace
generated-media parts with metadata placeholders rather than resending the
bytes. User attachments are unaffected; explicitly attaching the file or
reading it through a tool can send its contents to a model. The workspace file
remains yours to edit, commit, or delete.

Existing session databases are upgraded in place when a newer Docker Agent
opens them. The database now carries portable generated-media bytes as well
as session records; an older binary may not understand the upgraded schema.
Do not assume a database opened by a newer version remains readable by an
older version. Sessions created before portable copies were introduced
continue to render through their manifest-gated workspace files. Historical
manifest entries for files outside the workspace are rejected. An unrecorded
path is never read. Every resolution rechecks the current manifest, so deleting
a session revokes its references even if workspace provenance is cached. A
legacy workspace file that was deleted, replaced by
a symlink, made invalid, or cannot be read appears as a short "unavailable"
note. Manifest and containment checks do not verify the integrity of
ordinary-file contents, which may have changed. Generated-media storage and
resolution have no byte cap. Deleting a session also deletes its stored
generated-media blobs. See
[Generated Media](../tui/index.md#generated-media) for naming, collision
handling, and rendering details.

## Resuming a Session

Pass `--session <id>` to continue a previous conversation instead of starting a new one:
Expand Down
111 changes: 106 additions & 5 deletions docs/features/tui/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,11 +294,112 @@ Attached files are also recorded on the session so sub-agents spawned by task tr

## Generated Media

Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of their reply. When that happens, docker-agent writes the generated bytes into the session's workspace (the directory the session was started in) as an ordinary, visible file, and the assistant message keeps only a relative reference to that file plus its MIME type, display name, and size — never the raw bytes.

This keeps session JSON/database rows lightweight regardless of how many images a conversation accumulates, and the generated file is a regular workspace deliverable — visible to every tool, and yours to edit, commit, move, or delete — the same way generated code or text lands there.

Generated media is **not** automatically resent to the model on later turns: only the surrounding text is replayed in the outgoing history, the same way a large tool result would be summarized rather than repeated. This avoids silently ballooning the context window with image bytes on every follow-up message. A future step will add TUI rendering for these files (e.g. displaying the generated image inline); today this slice covers the domain, persistence, and safety mechanics only.
Some models (e.g. Gemini image-output models like `gemini-2.5-flash-image`)
are designed to generate an image directly as part of their reply, not just
describe one. When models.dev reports that a model can generate images, or
[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities)
explicitly enables it, Docker Agent asks it for text *and* image output on the
models gateway, direct Gemini API, and Vertex AI. An explicit `false` disables
this behavior. See
[Google Gemini: Generated Images](../../providers/google/index.md#generated-images)
for exact configuration and limitations — ordinary image-output requests with
custom tools or structured output are rejected locally before any request is
sent. Session-title and compaction requests omit image response modalities and
skip this guard; they do not explicitly force TEXT-only output. Google
Search, Maps, and code-execution built-ins remain available. For custom tools,
models.dev metadata lets the error distinguish
a model that cannot call tools from an image-output request shape that cannot
combine both capabilities; unknown metadata keeps a conservative generic
message.

At a text-only stop, Docker Agent checks the last user prompt for phrases
such as "generate an image" or "draw a picture". A match preserves the reply
and adds this nonfatal warning: `The model returned text but no image for this
image-generation request. Try rephrasing the request.` Prompts without a
matching phrase do not trigger it. This is phrase matching, not semantic
intent detection: negated or quoted phrases can match, other wording can be
missed, and the check does not require an image-output-capable model. It does
not track a whole submission across tool calls, steering, stop hooks, or
handoffs. A terminal provider error skips this check, as does structured
output on the current agent model; the check does not inspect every override
or parse the reply to determine whether it is structured.

**Where images land.** Docker Agent attempts to save each generated image
as an ordinary workspace file and record it in the session manifest. After
both steps succeed, it stores a complete portable copy in the session database.
A failed portable-copy write does not remove the saved workspace file and
produces a per-item warning.
Database upgrades are in-place and older binaries may not understand the
upgraded schema — see [Sessions](../sessions/index.md#generated-media-files).
Generated files are untracked workspace files, yours to edit, commit, move,
or delete. A remote runtime writes to its own workspace; the local TUI does
not receive a remote binary-rendering path from this feature.

**Naming.** The model is instructed to honor an explicit prompt filename
such as `assets/red-panda.jpg` in a private naming marker. This is a request,
not a guarantee: emitted markers take precedence. When exactly one image is
returned without a marker, a single explicit filename found by a conservative
prompt parser is used. An unmarked item among several returned images does
not qualify. Otherwise names come from the provider, then `generated-1`,
`generated-2`, and so on. Parent directories are created when the validated
save succeeds. Two rules always apply:

- **The extension matches the data.** The image format is decided by the
provider (typically PNG) — asking for `sunshine.gif` or `diagram.svg`
does not transcode anything. If the model returns PNG data, the file is
saved as `sunshine.png` and a notice tells you so.
- **Existing files are never overwritten.** A name collision gets a dash
suffix instead: a second `red-panda.jpg` is saved as `red-panda-1.jpg`.
Publication requires hard-link support; filesystems without it fail safely
with a save warning instead of using a replacing rename.

**Paths stay in the workspace.** A prompt-directed target that is absolute,
`~`-rooted, or climbs above the workspace with `..` is not written outside the
owning session's workspace. Docker Agent discards the directory portion,
sanitizes the basename, saves it at the workspace root, and adds a bounded
warning to the turn. This does not prompt or wait for confirmation, including
over ACP and other interfaces without an elicitation consumer. General MCP and
tool elicitation is unaffected. If the owning session has no workspace root,
the save fails instead of falling back to the data directory. Relative
subdirectories remain supported after containment and symlink checks. An unusable
basename falls back to `generated-N`; a redirected save can still fail and warn
without discarding successful siblings or assistant text.

**Rendering.** Successfully resolved images can appear inline in the same
assistant turn, using Kitty-graphics support and `render_images` as described
under [Markdown Images](#markdown-images). Graphics-disabled or unsupported
terminals show a filename/path fallback. Resolution checks the owning session's
manifest before preferring its portable database copy; saved bytes can survive
workspace edits, deletion, or missing provenance. Without workspace provenance,
the label uses the recorded relative path rather than a verified absolute file. It falls back to the manifest-gated workspace file only when the
session store has no blob interface or the blob is not found. Other blob errors
fail closed. Historical manifest entries that identify an external root are
rejected. Stores without blob support and sessions created before portable
blobs were introduced continue to use legacy workspace files.
Generated-media resolution has no byte cap.

Ordinary outgoing history replaces generated-media parts with metadata
placeholders, so follow-up turns do not resend stored bytes. Explicitly
attaching a generated file or asking a tool to read it can send its contents
to a model. Legacy files still require manifest authorization, containment,
regular-file and symlink checks. These are not content-integrity checks: an
ordinary file's bytes may have changed. Failed resolution shows an unavailable
label rather than reading an unauthorized fallback.
This generated-media behavior is separate from the existing input bound for
ordinary Markdown images rendered from assistant text.

If a save fails (unwritable directory, full disk, …), only that image is
dropped, with a concise warning — the reply text and any sibling images in
the same turn are kept. If all saves in a media-only reply fail, an empty
assistant record may remain alongside the warnings. Disk-full, quota, and
unclassified failures currently use generic retry/debug advice.
Note also that an image-capable model can answer
with text only and generate no image at all; that is provider behavior, so
reword or repeat the prompt.

Inline image rendering in the TUI also covers a tool/MCP result that
returns an image, or a Markdown image reference to a file a tool actually
writes to disk — see [Markdown Images](#markdown-images) above.

### Team Context Budgets and Targeted Compaction

Expand Down
Loading
Loading