diff --git a/docs/configuration/models/index.md b/docs/configuration/models/index.md index 3549f3507a..7e7656c1e8 100644 --- a/docs/configuration/models/index.md +++ b/docs/configuration/models/index.md @@ -40,8 +40,8 @@ models: pdf: boolean # Optional: whether the model accepts PDF attachments audio: boolean # Optional: whether the model accepts audio attachments video: boolean # Optional: whether the model accepts video attachments - output_capabilities: # Optional: owner-declared generative output capabilities (never inferred) - image: boolean # Optional: whether the model is declared able to generate image output + output_capabilities: # Optional: override generative output capabilities (otherwise detected from models.dev) + image: boolean # Optional: whether the model can generate image output cost: # Optional: explicit token pricing (USD per 1M tokens) input: float # Optional: price per 1M input tokens output: float # Optional: price per 1M output tokens @@ -76,7 +76,7 @@ models: | `track_usage` | boolean | ✗ | Track and report token usage for this model | | `routing` | array | ✗ | Rule-based routing to different models. See [Model Routing](../routing/index.md). | | `capabilities` | object | ✗ | Override attachment (input) capabilities for this model. See [Attachment Capability Overrides](#attachment-capability-overrides). | -| `output_capabilities` | object | ✗ | Owner-declared generative output capabilities for this model, e.g. image generation. Never inferred. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). | +| `output_capabilities` | object | ✗ | Override generative output capabilities for this model, e.g. image generation. Omitted flags are detected from models.dev; explicit values take precedence. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). | | `cost` | object | ✗ | Explicit token pricing in USD per 1M tokens, overriding the built-in catalogue. See [Custom Token Pricing](#custom-token-pricing). | | `provider_opts` | object | ✗ | Provider-specific options (see provider pages) | | `title_model` | string | ✗ | Model used for session-title generation. Can be a named model from the `models:` section or an inline `provider/model` string. When omitted, the agent's primary model generates titles. Cannot be combined with `first_available`. | @@ -151,14 +151,15 @@ See [`examples/capability-overrides.yaml`](https://github.com/docker/docker-agen [`examples/strip-unsupported-media.yaml`](https://github.com/docker/docker-agent/blob/main/examples/strip-unsupported-media.yaml) for a fixture demonstrating the stripping behaviour with and without an override. -## Output Capabilities +### Output capabilities -`output_capabilities` declares what a model can generate, as opposed to -`capabilities`, which declares what it accepts as input. There is no -automatic detection for output capabilities: no catalogue of -output-capable models exists, and matching on the model name string is -deliberately avoided as unreliable. A model's output capabilities are -therefore always unknown/off unless the owner declares them. +`output_capabilities` overrides what a model can generate, as opposed to +`capabilities`, which overrides what it accepts as input. Resolution follows +one precedence chain: explicit `false`, explicit `true`, then an exact +models.dev record whose `Modalities.Output` contains `image`. An omitted image +flag (including `output_capabilities: {}`) therefore uses catalogue metadata; +an unknown model or unavailable catalogue leaves image output disabled. Docker +Agent never infers this capability from the model name. ```yaml models: @@ -173,18 +174,23 @@ models: | --------------------------- | ------- | -------------------------------------------------------------| | `output_capabilities.image` | boolean | Whether the model is declared able to generate image output | -Omitting `output_capabilities`, or leaving `image` unset or `false`, always -preserves existing behavior. Setting it to `true` only opts the model into -behavior that specifically keys off a declared image-output capability (for -example, a provider-specific request-shape guard); it does not by itself -change what Docker Agent sends to or renders from the model. - -One side effect of declaring `image: true`: the model is skipped as a -session-title candidate, because titles are generated by a plain text-only -completion that image-output routes can reject. Title generation uses the -first non-image-output candidate (dedicated `title_model`, then the agent's -model, then its fallbacks); when every candidate declares image output, the -automatic title is skipped and the session keeps its default title. +Omitting `output_capabilities`, using an empty block, or omitting `image` uses +models.dev metadata for that exact model when available. Setting `image` +explicitly overrides the catalogue; an explicit `false` has highest precedence +and disables image response modalities even when the catalogue lists image +output. Enabling image output only opts the model into behavior that keys off +that capability (for example, a provider-specific image-output request +contract); it does not guarantee that a provider will return an image. + +Session-title and compaction requests omit image response modalities and +bypass the guard even for image-output-capable models; they do not explicitly +force TEXT-only output. On supported Google surfaces, the guard runs only when +image output resolves as enabled and an ordinary request includes custom tools +or structured output; matching requests are rejected locally. Google +server-side built-ins remain available. For a custom-tool conflict, +models.dev's `tool_call` capability makes the error say +whether the model lacks tool calls entirely or only cannot combine them with +image output; unavailable metadata keeps a conservative generic message. > [!WARNING] > **Constraint** diff --git a/docs/features/sessions/index.md b/docs/features/sessions/index.md index dd264814b4..39f2c0c19b 100644 --- a/docs/features/sessions/index.md +++ b/docs/features/sessions/index.md @@ -29,6 +29,37 @@ Override the location with `-s`/`--session-db`, or by overriding the data direct $ docker agent run agent.yaml --session-db ./sessions.db ``` +## Generated Media Files + +Some models (e.g. Gemini image-output models) can generate an image as part +of a reply. Docker Agent saves each generated image into the session's +workspace as an ordinary, untracked file. After the file and its manifest +entry are saved, it stores a portable copy in `session.db`. Reopening or moving +the session can therefore render the original generated bytes even when the +workspace file was edited, moved, or deleted. Ordinary follow-up turns replace +generated-media parts with metadata placeholders rather than resending the +bytes. User attachments are unaffected; explicitly attaching the file or +reading it through a tool can send its contents to a model. The workspace file +remains yours to edit, commit, or delete. + +Existing session databases are upgraded in place when a newer Docker Agent +opens them. The database now carries portable generated-media bytes as well +as session records; an older binary may not understand the upgraded schema. +Do not assume a database opened by a newer version remains readable by an +older version. Sessions created before portable copies were introduced +continue to render through their manifest-gated workspace files. Historical +manifest entries for files outside the workspace are rejected. An unrecorded +path is never read. Every resolution rechecks the current manifest, so deleting +a session revokes its references even if workspace provenance is cached. A +legacy workspace file that was deleted, replaced by +a symlink, made invalid, or cannot be read appears as a short "unavailable" +note. Manifest and containment checks do not verify the integrity of +ordinary-file contents, which may have changed. Generated-media storage and +resolution have no byte cap. Deleting a session also deletes its stored +generated-media blobs. See +[Generated Media](../tui/index.md#generated-media) for naming, collision +handling, and rendering details. + ## Resuming a Session Pass `--session ` to continue a previous conversation instead of starting a new one: diff --git a/docs/features/tui/index.md b/docs/features/tui/index.md index 288f47233c..13795c29b3 100644 --- a/docs/features/tui/index.md +++ b/docs/features/tui/index.md @@ -294,11 +294,112 @@ Attached files are also recorded on the session so sub-agents spawned by task tr ## Generated Media -Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of their reply. When that happens, docker-agent writes the generated bytes into the session's workspace (the directory the session was started in) as an ordinary, visible file, and the assistant message keeps only a relative reference to that file plus its MIME type, display name, and size — never the raw bytes. - -This keeps session JSON/database rows lightweight regardless of how many images a conversation accumulates, and the generated file is a regular workspace deliverable — visible to every tool, and yours to edit, commit, move, or delete — the same way generated code or text lands there. - -Generated media is **not** automatically resent to the model on later turns: only the surrounding text is replayed in the outgoing history, the same way a large tool result would be summarized rather than repeated. This avoids silently ballooning the context window with image bytes on every follow-up message. A future step will add TUI rendering for these files (e.g. displaying the generated image inline); today this slice covers the domain, persistence, and safety mechanics only. +Some models (e.g. Gemini image-output models like `gemini-2.5-flash-image`) +are designed to generate an image directly as part of their reply, not just +describe one. When models.dev reports that a model can generate images, or +[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities) +explicitly enables it, Docker Agent asks it for text *and* image output on the +models gateway, direct Gemini API, and Vertex AI. An explicit `false` disables +this behavior. See +[Google Gemini: Generated Images](../../providers/google/index.md#generated-images) +for exact configuration and limitations — ordinary image-output requests with +custom tools or structured output are rejected locally before any request is +sent. Session-title and compaction requests omit image response modalities and +skip this guard; they do not explicitly force TEXT-only output. Google +Search, Maps, and code-execution built-ins remain available. For custom tools, +models.dev metadata lets the error distinguish +a model that cannot call tools from an image-output request shape that cannot +combine both capabilities; unknown metadata keeps a conservative generic +message. + +At a text-only stop, Docker Agent checks the last user prompt for phrases +such as "generate an image" or "draw a picture". A match preserves the reply +and adds this nonfatal warning: `The model returned text but no image for this +image-generation request. Try rephrasing the request.` Prompts without a +matching phrase do not trigger it. This is phrase matching, not semantic +intent detection: negated or quoted phrases can match, other wording can be +missed, and the check does not require an image-output-capable model. It does +not track a whole submission across tool calls, steering, stop hooks, or +handoffs. A terminal provider error skips this check, as does structured +output on the current agent model; the check does not inspect every override +or parse the reply to determine whether it is structured. + +**Where images land.** Docker Agent attempts to save each generated image +as an ordinary workspace file and record it in the session manifest. After +both steps succeed, it stores a complete portable copy in the session database. +A failed portable-copy write does not remove the saved workspace file and +produces a per-item warning. +Database upgrades are in-place and older binaries may not understand the +upgraded schema — see [Sessions](../sessions/index.md#generated-media-files). +Generated files are untracked workspace files, yours to edit, commit, move, +or delete. A remote runtime writes to its own workspace; the local TUI does +not receive a remote binary-rendering path from this feature. + +**Naming.** The model is instructed to honor an explicit prompt filename +such as `assets/red-panda.jpg` in a private naming marker. This is a request, +not a guarantee: emitted markers take precedence. When exactly one image is +returned without a marker, a single explicit filename found by a conservative +prompt parser is used. An unmarked item among several returned images does +not qualify. Otherwise names come from the provider, then `generated-1`, +`generated-2`, and so on. Parent directories are created when the validated +save succeeds. Two rules always apply: + +- **The extension matches the data.** The image format is decided by the + provider (typically PNG) — asking for `sunshine.gif` or `diagram.svg` + does not transcode anything. If the model returns PNG data, the file is + saved as `sunshine.png` and a notice tells you so. +- **Existing files are never overwritten.** A name collision gets a dash + suffix instead: a second `red-panda.jpg` is saved as `red-panda-1.jpg`. + Publication requires hard-link support; filesystems without it fail safely + with a save warning instead of using a replacing rename. + +**Paths stay in the workspace.** A prompt-directed target that is absolute, +`~`-rooted, or climbs above the workspace with `..` is not written outside the +owning session's workspace. Docker Agent discards the directory portion, +sanitizes the basename, saves it at the workspace root, and adds a bounded +warning to the turn. This does not prompt or wait for confirmation, including +over ACP and other interfaces without an elicitation consumer. General MCP and +tool elicitation is unaffected. If the owning session has no workspace root, +the save fails instead of falling back to the data directory. Relative +subdirectories remain supported after containment and symlink checks. An unusable +basename falls back to `generated-N`; a redirected save can still fail and warn +without discarding successful siblings or assistant text. + +**Rendering.** Successfully resolved images can appear inline in the same +assistant turn, using Kitty-graphics support and `render_images` as described +under [Markdown Images](#markdown-images). Graphics-disabled or unsupported +terminals show a filename/path fallback. Resolution checks the owning session's +manifest before preferring its portable database copy; saved bytes can survive +workspace edits, deletion, or missing provenance. Without workspace provenance, +the label uses the recorded relative path rather than a verified absolute file. It falls back to the manifest-gated workspace file only when the +session store has no blob interface or the blob is not found. Other blob errors +fail closed. Historical manifest entries that identify an external root are +rejected. Stores without blob support and sessions created before portable +blobs were introduced continue to use legacy workspace files. +Generated-media resolution has no byte cap. + +Ordinary outgoing history replaces generated-media parts with metadata +placeholders, so follow-up turns do not resend stored bytes. Explicitly +attaching a generated file or asking a tool to read it can send its contents +to a model. Legacy files still require manifest authorization, containment, +regular-file and symlink checks. These are not content-integrity checks: an +ordinary file's bytes may have changed. Failed resolution shows an unavailable +label rather than reading an unauthorized fallback. +This generated-media behavior is separate from the existing input bound for +ordinary Markdown images rendered from assistant text. + +If a save fails (unwritable directory, full disk, …), only that image is +dropped, with a concise warning — the reply text and any sibling images in +the same turn are kept. If all saves in a media-only reply fail, an empty +assistant record may remain alongside the warnings. Disk-full, quota, and +unclassified failures currently use generic retry/debug advice. +Note also that an image-capable model can answer +with text only and generate no image at all; that is provider behavior, so +reword or repeat the prompt. + +Inline image rendering in the TUI also covers a tool/MCP result that +returns an image, or a Markdown image reference to a file a tool actually +writes to disk — see [Markdown Images](#markdown-images) above. ### Team Context Budgets and Targeted Compaction diff --git a/docs/providers/google/index.md b/docs/providers/google/index.md index ff5958e19c..d0436667ae 100644 --- a/docs/providers/google/index.md +++ b/docs/providers/google/index.md @@ -64,29 +64,70 @@ models: Some Gemini models (e.g. `gemini-2.5-flash-image`) are designed to generate an image directly as part of their reply, not just describe one. Docker -Agent's Gemini request path doesn't yet ask for that image output — that -support is still being completed — so today a request like this gets a -text-only reply. See -[Generated Media](../../features/tui/index.md#generated-media) for the -current, verified state. +Agent requests that image output on supported Google surfaces — the models +gateway, direct Gemini API, and Vertex AI — according to one binding policy: +explicit `false`, explicit `true`, then an exact models.dev record whose +`Modalities.Output` contains `image`. An omitted image flag, including +`output_capabilities: {}`, uses that catalogue default. Unknown models or +unavailable catalogue data leave image response modalities disabled; model +names are never used to guess capability. +Each eligible ordinary chat request asks for text *and* image output. +Vertex AI has deterministic guard/predicate coverage; live image-generation +validation is deferred. + +Docker Agent attempts to save each returned image and display it in the TUI — see [Generated Media](../../features/tui/index.md#generated-media) +for file naming, collision handling, and rendering details. + +The workspace file is the visible deliverable. After that file and its manifest +entry are saved, a portable copy is also stored in the session database and +preferred when the session is reopened. It can render the original generated +bytes even if the workspace file was edited, moved, or deleted. +Generated-media storage and resolution do not impose a size cap. If portable +persistence fails, the workspace file is still kept and the turn includes a +warning. +Sessions created before portable copies were introduced continue to use their +manifest-gated workspace files. ```yaml -agents: - root: - model: google/gemini-2.5-flash-image +models: + gemini-image: + provider: google + model: gemini-2.5-flash-image ``` -When the model is accessed through a Docker AI Gateway and explicitly -declared image-output-capable with -[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities), -Docker Agent has verified that request combined with custom function tools, -a built-in tool (e.g. `google_search`), or structured output gets rejected -by the gateway with an opaque, empty-body HTTP 400. To avoid that, Docker -Agent rejects such a combination itself, before any request is sent, with a -clear error naming which feature is incompatible. Plain text requests to -that model (no tools, no structured output) are unaffected, as is every -other route: direct Gemini API/Vertex AI calls, and gateway calls to a model -without the declaration. +When `output_capabilities.image` is omitted, including in an empty block, +Docker Agent uses models.dev output modalities for the exact known model. Set +it explicitly for custom models or to override incorrect catalogue data; +unknown or unavailable metadata remains disabled and capability is never +guessed from the model name. + +Session-title and compaction requests omit image response modalities and +bypass the guard even for image-output-capable models. They do not explicitly +force TEXT-only output. Ordinary image-output requests with custom function tools or +structured output are rejected locally before any request is sent. Google +Search, Maps, and code-execution built-ins remain available. When custom +tools conflict, Docker Agent uses models.dev's `tool_call` capability to clarify +whether the model cannot call tools at all or supports tools only outside an +image-output request. Unknown catalogue data keeps the conservative generic +message. + +A few provider-side behaviors to know: + +- **The provider decides the image format** (typically PNG). Asking for a + `.gif` or `.svg` filename does not transcode anything — the saved file's + extension is corrected to match the data actually returned. +- **An image is not guaranteed.** Even a correctly configured image model + can answer with text only and generate no image. At a text-only stop, + phrases such as "generate an image" or "draw a picture" in the last user + prompt trigger a nonfatal warning while preserving the reply: + `The model returned text but no image for this image-generation request. Try rephrasing the request.` + This phrase-based check is not semantic intent detection and does not + check output capability: negated or quoted phrases can match and other + wording can be missed. It does not track a whole submission across tool + calls, steering, stop hooks, or handoffs. A terminal provider error skips + this check, as does structured output configured on the current agent + model; per-call overrides and reply content are not independently + classified. ## Thinking Budget diff --git a/examples/README.md b/examples/README.md index 759c7f603d..2d8b7d811b 100644 --- a/examples/README.md +++ b/examples/README.md @@ -216,7 +216,7 @@ remote MCP endpoints. | [`rule_based_routing.yaml`](rule_based_routing.yaml) | Cheap router model dispatches the user message to fast or capable models. | | [`structured-output.yaml`](structured-output.yaml) | Forces the model to return JSON matching a schema. | | [`google_search_grounding.yaml`](google_search_grounding.yaml) | Enables Google Search grounding on Gemini models. | -| [`gemini_image_output.yaml`](gemini_image_output.yaml) | Gemini image-output model (generated images are saved into the workspace, not inlined as base64). | +| [`gemini_image_output.yaml`](gemini_image_output.yaml) | Gemini image-output model: generated images are saved into the workspace, stored portably with the session, and rendered inline in the TUI. | | [`sampling-opts.yaml`](sampling-opts.yaml) | Provider-specific sampling parameters (`top_k`, `repetition_penalty`, …). | | [`thinking_budget.yaml`](thinking_budget.yaml) | Reasoning/thinking budgets across OpenAI, Anthropic and Google. | | [`task_budget.yaml`](task_budget.yaml) | Anthropic `task_budget`: cap total tokens spent across a multi-step agentic task. | diff --git a/examples/gemini_image_output.yaml b/examples/gemini_image_output.yaml index e8490a4f62..1531b974a1 100644 --- a/examples/gemini_image_output.yaml +++ b/examples/gemini_image_output.yaml @@ -1,25 +1,56 @@ # Gemini image-output model: the model can generate an image directly as # part of its reply, instead of only describing one. # -# The `gemini-image` model below declares `output_capabilities.image: true` — -# an explicit, owner-provided statement that this model can generate image -# output. It is never inferred from the model name or any catalogue; omit -# it, or leave it false, and behavior is unchanged. +# The `gemini-image` model below relies on the exact models.dev record's +# `Modalities.Output=image`. An explicit `output_capabilities.image` value can +# override catalogue data; unknown or unavailable catalogue metadata stays off. +# Capability is never guessed from the model name. # -# Native generated images aren't currently presented inline in the -# terminal UI, so the model's text reply is what you'll see today. +# On the models gateway, direct Gemini API, and Vertex AI, Docker Agent +# requests text AND image output for this image-capable model. It attempts +# to save each image in the workspace, record portable bytes, and display it +# in the TUI. Failures warn per item; unsupported graphics use path labels. # +# Notes: +# - Session-title/compaction omit image modalities and bypass the guard; +# they do not explicitly force TEXT-only output. Ordinary requests +# with custom tools or structured output are rejected locally before any +# request is sent; Google Search, Maps, and code-execution built-ins remain +# available. For custom tools, models.dev metadata lets the error say +# whether the model lacks tool calls or only cannot combine them with +# image output; unknown metadata keeps a conservative generic message. +# - Naming: the model is instructed to honor an explicit prompt filename +# in a naming marker; otherwise it chooses a meaningful name. A missing +# marker falls back to a single parsed prompt filename only for a single +# returned image, then to the provider name or a numbered generic name. +# Multi-image replies use markers, provider names, or numbered names. +# Extensions are corrected to match the returned data (the provider +# decides the format, typically PNG), and existing files are never +# overwritten — collisions get a dash suffix (logo-1.png). +# - A text-only stop can trigger a nonfatal warning for phrases such as +# "generate an image" in the last prompt. This is phrase matching, not +# semantic intent detection: quoted/negated phrases can match and other +# wording can be missed. It does not check output capability or track +# a whole submission across tools, steering, stop hooks, or handoffs. +# Terminal provider errors and current-agent structured-output config +# skip the check; per-call overrides/reply content are not classified. +# - Ordinary follow-ups send metadata placeholders, not generated bytes. +# Attaching a saved image or reading it through a tool can send it again. +# +# Optional live trials require credentials, may be billed, and should use +# a disposable workspace and session store, not an active session database. +# Live Vertex image-generation validation is deferred. # Try it out: # docker agent run examples/gemini_image_output.yaml \ # "Generate an image of a red panda working at a terminal" # docker agent run examples/gemini_image_output.yaml \ -# "Generate an image of a lighthouse at sunset, and describe the color palette you used" +# "Generate an image of a lighthouse at sunset as assets/lighthouse.png" +# docker agent run examples/gemini_image_output.yaml \ +# "Generate an image of a whale and save it as whale.jpg" models: gemini-image: provider: google model: gemini-2.5-flash-image - output_capabilities: - image: true agents: root: