From 0df48c0b85490c0b3bb430e3bdff187d6b75acf5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Arnaud=20He=CC=81ritier?= Date: Mon, 27 Jul 2026 22:13:34 +0200 Subject: [PATCH 1/3] docs: explain the workspace artifact lifecycle Describe owning-workspace saves, sanitized redirects, manifest-gated references, portable session copies, no-resend history and inline fallbacks. Keep optional validation isolated from active session databases and distinguish deterministic coverage from deferred live Vertex validation. --- docs/features/sessions/index.md | 22 +++++++++++++++++++ docs/features/tui/index.md | 38 ++++++++++++++++++++++----------- 2 files changed, 47 insertions(+), 13 deletions(-) diff --git a/docs/features/sessions/index.md b/docs/features/sessions/index.md index dd264814b4..8c575c5881 100644 --- a/docs/features/sessions/index.md +++ b/docs/features/sessions/index.md @@ -29,6 +29,28 @@ Override the location with `-s`/`--session-db`, or by overriding the data direct $ docker agent run agent.yaml --session-db ./sessions.db ``` +## Generated-Media Artifacts + +Generated images are ordinary files in the owning session's workspace. Absolute, +parent-traversing and home-relative requests redirect to a sanitized workspace +basename with a warning; generated media never writes externally or asks for a +path decision. Relative directories, MIME-derived extensions and collision +suffixes preserve existing files. + +Session messages store owner-qualified references and sanitized metadata, not +inline generated bytes. Branching and forking retain the original owner. The +manifest authorizes each reference before resolution; unknown owners, roots or +paths fail closed. The viewing working directory never determines ownership. + +After a workspace save and manifest record succeed, the session store keeps a +portable copy. A portability failure warns without removing the workspace file. +Deleting a session removes stored manifest and blob rows, not workspace files. +Session databases migrate in place when opened; compatibility testing belongs +in a disposable store, not an active session database. + +Ordinary follow-ups send placeholders, not the stored generated bytes. Explicit +file attachment or tool reads are separate operations. + ## Resuming a Session Pass `--session ` to continue a previous conversation instead of starting a new one: diff --git a/docs/features/tui/index.md b/docs/features/tui/index.md index 288f47233c..44b0cc9ccd 100644 --- a/docs/features/tui/index.md +++ b/docs/features/tui/index.md @@ -294,19 +294,31 @@ Attached files are also recorded on the session so sub-agents spawned by task tr ## Generated Media -Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of their reply. When that happens, docker-agent writes the generated bytes into the session's workspace (the directory the session was started in) as an ordinary, visible file, and the assistant message keeps only a relative reference to that file plus its MIME type, display name, and size — never the raw bytes. - -This keeps session JSON/database rows lightweight regardless of how many images a conversation accumulates, and the generated file is a regular workspace deliverable — visible to every tool, and yours to edit, commit, move, or delete — the same way generated code or text lands there. - -Generated media is **not** automatically resent to the model on later turns: only the surrounding text is replayed in the outgoing history, the same way a large tool result would be summarized rather than repeated. This avoids silently ballooning the context window with image bytes on every follow-up message. A future step will add TUI rendering for these files (e.g. displaying the generated image inline); today this slice covers the domain, persistence, and safety mechanics only. - -### Team Context Budgets and Targeted Compaction - -The `/context` dialog also shows a **Live sessions** section: the current session plus every currently running sub-agent session (foreground children spawned by task transfer and long-running `run_background_agent` tasks). Each row shows the agent name, a short session ID (so two concurrent runs of the same agent stay distinguishable), and that session's context budget: used tokens, context limit, and percentage, or an explicit "limit unknown" reading when the model's window cannot be resolved. Live-sessions rows do not repeat the compaction-cap wording themselves — the dialog's header line is the sole authority on which model, if any, caps the effective limit. - -When a compaction has occurred, the dialog displays the verbatim text of the most recent compaction summary below the file inventory, under a "Latest compaction summary" section. This shows exactly what was summarized, preserving hard newlines and soft-wrapping long lines to the dialog width. - -Select a live session with ↑/↓ and press Enter to explicitly compact it. Cross-agent compaction happens only on this explicit request: no idle-triggered automatic compaction is added, and the existing automatic threshold and overflow-recovery compaction of sub-agent sessions is unchanged. The request is queued onto the target session's own run loop and executes at the next safe point between model turns, so it cannot corrupt an in-flight turn. The dialog closes and a notification confirms the request; a second notification reports the outcome (compacted, skipped, or failed) with the agent's name. Selecting the main row runs the same compaction as `/compact`. `/compact` itself keeps compacting the current root session. Remote runtimes do not expose live-session tracking, so the section is omitted there. +Returned media is saved in the owning session's workspace with safe names, +MIME-derived extensions and collision suffixes. Absolute, traversal, +home-relative and symlink-escaping targets redirect to a sanitized workspace +basename and warn without an external write or confirmation dialog. An unusable +basename uses `generated-N`. Failed saves retain assistant text and successful +siblings; a missing owning workspace never falls back to a data directory. + +The session message holds owner-qualified references and sanitized metadata. +Manifest authorization gates resolution before portable bytes or workspace files +are read. Ordinary follow-up history uses placeholders rather than resending the +stored bytes. Branches and restored messages keep the original artifact owner. + +Supported terminals render generated images inline; others show a safe path +label. Unavailable or forged references keep an unavailable placeholder. File +fallback requires workspace containment and regular-file identity checks, not +content-integrity verification. See [Generated-Media Artifacts](../sessions/index.md#generated-media-artifacts) +for storage and lifecycle details. + +### Optional Validation + +Use a disposable workspace and session store for optional image-generation +trials. Live model calls require credentials, may be billed and can return text +only; Vertex live image-generation validation is deferred. Deterministic tests +cover naming, redirection, persistence, no-resend history and restoration without +provider credentials. ## Runtime Model Switching From 4c0d297f5df9410a1f3222264d790c4814400f17 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Arnaud=20He=CC=81ritier?= Date: Tue, 28 Jul 2026 00:35:48 +0200 Subject: [PATCH 2/3] docs: describe generated-image requests and workspace-only saves Document catalog defaults, explicit capability overrides, TEXT+IMAGE requests, workspace-only redirection, provider-controlled formats, normal ACP completion and inline fallback labels. Vertex live image validation remains deferred. --- docs/configuration/models/index.md | 50 ++++++----- docs/features/sessions/index.md | 50 ++++++----- docs/features/tui/index.md | 138 +++++++++++++++++++++++------ docs/providers/google/index.md | 79 +++++++++++++---- examples/README.md | 2 +- examples/gemini_image_output.yaml | 49 ++++++++-- 6 files changed, 271 insertions(+), 97 deletions(-) diff --git a/docs/configuration/models/index.md b/docs/configuration/models/index.md index 3549f3507a..7e7656c1e8 100644 --- a/docs/configuration/models/index.md +++ b/docs/configuration/models/index.md @@ -40,8 +40,8 @@ models: pdf: boolean # Optional: whether the model accepts PDF attachments audio: boolean # Optional: whether the model accepts audio attachments video: boolean # Optional: whether the model accepts video attachments - output_capabilities: # Optional: owner-declared generative output capabilities (never inferred) - image: boolean # Optional: whether the model is declared able to generate image output + output_capabilities: # Optional: override generative output capabilities (otherwise detected from models.dev) + image: boolean # Optional: whether the model can generate image output cost: # Optional: explicit token pricing (USD per 1M tokens) input: float # Optional: price per 1M input tokens output: float # Optional: price per 1M output tokens @@ -76,7 +76,7 @@ models: | `track_usage` | boolean | ✗ | Track and report token usage for this model | | `routing` | array | ✗ | Rule-based routing to different models. See [Model Routing](../routing/index.md). | | `capabilities` | object | ✗ | Override attachment (input) capabilities for this model. See [Attachment Capability Overrides](#attachment-capability-overrides). | -| `output_capabilities` | object | ✗ | Owner-declared generative output capabilities for this model, e.g. image generation. Never inferred. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). | +| `output_capabilities` | object | ✗ | Override generative output capabilities for this model, e.g. image generation. Omitted flags are detected from models.dev; explicit values take precedence. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). | | `cost` | object | ✗ | Explicit token pricing in USD per 1M tokens, overriding the built-in catalogue. See [Custom Token Pricing](#custom-token-pricing). | | `provider_opts` | object | ✗ | Provider-specific options (see provider pages) | | `title_model` | string | ✗ | Model used for session-title generation. Can be a named model from the `models:` section or an inline `provider/model` string. When omitted, the agent's primary model generates titles. Cannot be combined with `first_available`. | @@ -151,14 +151,15 @@ See [`examples/capability-overrides.yaml`](https://github.com/docker/docker-agen [`examples/strip-unsupported-media.yaml`](https://github.com/docker/docker-agent/blob/main/examples/strip-unsupported-media.yaml) for a fixture demonstrating the stripping behaviour with and without an override. -## Output Capabilities +### Output capabilities -`output_capabilities` declares what a model can generate, as opposed to -`capabilities`, which declares what it accepts as input. There is no -automatic detection for output capabilities: no catalogue of -output-capable models exists, and matching on the model name string is -deliberately avoided as unreliable. A model's output capabilities are -therefore always unknown/off unless the owner declares them. +`output_capabilities` overrides what a model can generate, as opposed to +`capabilities`, which overrides what it accepts as input. Resolution follows +one precedence chain: explicit `false`, explicit `true`, then an exact +models.dev record whose `Modalities.Output` contains `image`. An omitted image +flag (including `output_capabilities: {}`) therefore uses catalogue metadata; +an unknown model or unavailable catalogue leaves image output disabled. Docker +Agent never infers this capability from the model name. ```yaml models: @@ -173,18 +174,23 @@ models: | --------------------------- | ------- | -------------------------------------------------------------| | `output_capabilities.image` | boolean | Whether the model is declared able to generate image output | -Omitting `output_capabilities`, or leaving `image` unset or `false`, always -preserves existing behavior. Setting it to `true` only opts the model into -behavior that specifically keys off a declared image-output capability (for -example, a provider-specific request-shape guard); it does not by itself -change what Docker Agent sends to or renders from the model. - -One side effect of declaring `image: true`: the model is skipped as a -session-title candidate, because titles are generated by a plain text-only -completion that image-output routes can reject. Title generation uses the -first non-image-output candidate (dedicated `title_model`, then the agent's -model, then its fallbacks); when every candidate declares image output, the -automatic title is skipped and the session keeps its default title. +Omitting `output_capabilities`, using an empty block, or omitting `image` uses +models.dev metadata for that exact model when available. Setting `image` +explicitly overrides the catalogue; an explicit `false` has highest precedence +and disables image response modalities even when the catalogue lists image +output. Enabling image output only opts the model into behavior that keys off +that capability (for example, a provider-specific image-output request +contract); it does not guarantee that a provider will return an image. + +Session-title and compaction requests omit image response modalities and +bypass the guard even for image-output-capable models; they do not explicitly +force TEXT-only output. On supported Google surfaces, the guard runs only when +image output resolves as enabled and an ordinary request includes custom tools +or structured output; matching requests are rejected locally. Google +server-side built-ins remain available. For a custom-tool conflict, +models.dev's `tool_call` capability makes the error say +whether the model lacks tool calls entirely or only cannot combine them with +image output; unavailable metadata keeps a conservative generic message. > [!WARNING] > **Constraint** diff --git a/docs/features/sessions/index.md b/docs/features/sessions/index.md index 8c575c5881..4ff72abddb 100644 --- a/docs/features/sessions/index.md +++ b/docs/features/sessions/index.md @@ -29,27 +29,35 @@ Override the location with `-s`/`--session-db`, or by overriding the data direct $ docker agent run agent.yaml --session-db ./sessions.db ``` -## Generated-Media Artifacts - -Generated images are ordinary files in the owning session's workspace. Absolute, -parent-traversing and home-relative requests redirect to a sanitized workspace -basename with a warning; generated media never writes externally or asks for a -path decision. Relative directories, MIME-derived extensions and collision -suffixes preserve existing files. - -Session messages store owner-qualified references and sanitized metadata, not -inline generated bytes. Branching and forking retain the original owner. The -manifest authorizes each reference before resolution; unknown owners, roots or -paths fail closed. The viewing working directory never determines ownership. - -After a workspace save and manifest record succeed, the session store keeps a -portable copy. A portability failure warns without removing the workspace file. -Deleting a session removes stored manifest and blob rows, not workspace files. -Session databases migrate in place when opened; compatibility testing belongs -in a disposable store, not an active session database. - -Ordinary follow-ups send placeholders, not the stored generated bytes. Explicit -file attachment or tool reads are separate operations. +## Generated Media Files + +Some models (e.g. Gemini image-output models) can generate an image as part +of a reply. Docker Agent saves each generated image into the session's +workspace as an ordinary, untracked file. After the file and its manifest +entry are saved, it stores a portable copy in `session.db`. Reopening or moving +the session can therefore render the original generated bytes even when the +workspace file was edited, moved, or deleted. Ordinary follow-up turns replace +generated-media parts with metadata placeholders rather than resending the +bytes. User attachments are unaffected; explicitly attaching the file or +reading it through a tool can send its contents to a model. The workspace file +remains yours to edit, commit, or delete. + +Existing session databases are upgraded in place when a newer Docker Agent +opens them. The database now carries portable generated-media bytes as well +as session records; an older binary may not understand the upgraded schema. +Do not assume a database opened by a newer version remains readable by an +older version. Sessions created before portable copies were introduced +continue to render through their manifest-gated workspace files. Historical +manifest entries for files outside the workspace are rejected. An unrecorded +path is never read. Every resolution rechecks the current manifest, so deleting +a session revokes its references even if workspace provenance is cached. A +legacy workspace file that was deleted, replaced by +a symlink, made invalid, or cannot be read appears as a short "unavailable" +note. Manifest and containment checks do not verify the integrity of +ordinary-file contents, which may have changed. Deleting a session also deletes its stored +generated-media blobs. See +[Generated Media](../tui/index.md#generated-media) for naming, collision +handling, and rendering details. ## Resuming a Session diff --git a/docs/features/tui/index.md b/docs/features/tui/index.md index 44b0cc9ccd..2636f97ed5 100644 --- a/docs/features/tui/index.md +++ b/docs/features/tui/index.md @@ -294,31 +294,119 @@ Attached files are also recorded on the session so sub-agents spawned by task tr ## Generated Media -Returned media is saved in the owning session's workspace with safe names, -MIME-derived extensions and collision suffixes. Absolute, traversal, -home-relative and symlink-escaping targets redirect to a sanitized workspace -basename and warn without an external write or confirmation dialog. An unusable -basename uses `generated-N`. Failed saves retain assistant text and successful -siblings; a missing owning workspace never falls back to a data directory. - -The session message holds owner-qualified references and sanitized metadata. -Manifest authorization gates resolution before portable bytes or workspace files -are read. Ordinary follow-up history uses placeholders rather than resending the -stored bytes. Branches and restored messages keep the original artifact owner. - -Supported terminals render generated images inline; others show a safe path -label. Unavailable or forged references keep an unavailable placeholder. File -fallback requires workspace containment and regular-file identity checks, not -content-integrity verification. See [Generated-Media Artifacts](../sessions/index.md#generated-media-artifacts) -for storage and lifecycle details. - -### Optional Validation - -Use a disposable workspace and session store for optional image-generation -trials. Live model calls require credentials, may be billed and can return text -only; Vertex live image-generation validation is deferred. Deterministic tests -cover naming, redirection, persistence, no-resend history and restoration without -provider credentials. +Some models (e.g. Gemini image-output models like `gemini-2.5-flash-image`) +are designed to generate an image directly as part of their reply, not just +describe one. When models.dev reports that a model can generate images, or +[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities) +explicitly enables it, Docker Agent asks it for text *and* image output on the +models gateway, direct Gemini API, and Vertex AI. An explicit `false` disables +this behavior. See +[Google Gemini: Generated Images](../../providers/google/index.md#generated-images) +for exact configuration and limitations — ordinary image-output requests with +custom tools or structured output are rejected locally before any request is +sent. Session-title and compaction requests omit image response modalities and +skip this guard; they do not explicitly force TEXT-only output. Google +Search, Maps, and code-execution built-ins remain available. For custom tools, +models.dev metadata lets the error distinguish +a model that cannot call tools from an image-output request shape that cannot +combine both capabilities; unknown metadata keeps a conservative generic +message. + +At a text-only stop, Docker Agent checks the last user prompt for phrases +such as "generate an image" or "draw a picture". A match preserves the reply +and adds this nonfatal warning: `The model returned text but no image for this +image-generation request. Try rephrasing the request.` Prompts without a +matching phrase do not trigger it. This is phrase matching, not semantic +intent detection: negated or quoted phrases can match, other wording can be +missed, and the check does not require an image-output-capable model. It does +not track a whole submission across tool calls, steering, stop hooks, or +handoffs. A terminal provider error skips this check, as does structured +output on the current agent model; the check does not inspect every override +or parse the reply to determine whether it is structured. + +**Where images land.** Docker Agent attempts to save each generated image +as an ordinary workspace file and record it in the session manifest. After +both steps succeed, it stores a complete portable copy in the session database. +A failed portable-copy write does not remove the saved workspace file and +produces a per-item warning. +Database upgrades are in-place and older binaries may not understand the +upgraded schema — see [Sessions](../sessions/index.md#generated-media-files). +Generated files are untracked workspace files, yours to edit, commit, move, +or delete. A remote runtime writes to its own workspace; the local TUI does +not receive a remote binary-rendering path from this feature. + +**Naming.** The model is instructed to honor an explicit prompt filename +such as `assets/red-panda.jpg` in a private naming marker. This is a request, +not a guarantee: emitted markers take precedence. When exactly one image is +returned without a marker, a single explicit filename found by a conservative +prompt parser is used. An unmarked item among several returned images does +not qualify. Otherwise names come from the provider, then `generated-1`, +`generated-2`, and so on. Parent directories are created when the validated +save succeeds. Two rules always apply: + +- **The extension matches the data.** The image format is decided by the + provider (typically PNG) — asking for `sunshine.gif` or `diagram.svg` + does not transcode anything. If the model returns PNG data, the file is + saved as `sunshine.png` and a notice tells you so. +- **Existing files are never overwritten.** A name collision gets a dash + suffix instead: a second `red-panda.jpg` is saved as `red-panda-1.jpg`. + Publication requires hard-link support; filesystems without it fail safely + with a save warning instead of using a replacing rename. + +**Paths stay in the workspace.** A prompt-directed target that is absolute, +`~`-rooted, or climbs above the workspace with `..` is not written outside the +owning session's workspace. Docker Agent discards the directory portion, +sanitizes the basename, saves it at the workspace root, and adds a bounded +warning to the turn. This does not prompt or wait for confirmation, including +over ACP and other interfaces without an elicitation consumer. General MCP and +tool elicitation is unaffected. If the owning session has no workspace root, +the save fails instead of falling back to the data directory. Relative +subdirectories remain supported after containment and symlink checks. An unusable +basename falls back to `generated-N`; a redirected save can still fail and warn +without discarding successful siblings or assistant text. + +**Rendering.** Successfully resolved images can appear inline in the same +assistant turn, using Kitty-graphics support and `render_images` as described +under [Markdown Images](#markdown-images). Graphics-disabled or unsupported +terminals show a filename/path fallback. Resolution checks the owning session's +manifest before preferring its portable database copy; saved bytes can survive +workspace edits, deletion, or missing provenance. Without workspace provenance, +the label uses the recorded relative path rather than a verified absolute file. It falls back to the manifest-gated workspace file only when the +session store has no blob interface or the blob is not found. Other blob errors +fail closed. Historical manifest entries that identify an external root are +rejected. Stores without blob support and sessions created before portable +blobs were introduced continue to use legacy workspace files. + +Ordinary outgoing history replaces generated-media parts with metadata +placeholders, so follow-up turns do not resend stored bytes. Explicitly +attaching a generated file or asking a tool to read it can send its contents +to a model. Legacy files still require manifest authorization, containment, +regular-file and symlink checks. These are not content-integrity checks: an +ordinary file's bytes may have changed. Failed resolution shows an unavailable +label rather than reading an unauthorized fallback. +This generated-media behavior is separate from the existing input bound for +ordinary Markdown images rendered from assistant text. + +If a save fails (unwritable directory, full disk, …), only that image is +dropped, with a concise warning — the reply text and any sibling images in +the same turn are kept. If all saves in a media-only reply fail, an empty +assistant record may remain alongside the warnings. Disk-full, quota, and +unclassified failures currently use generic retry/debug advice. +Note also that an image-capable model can answer +with text only and generate no image at all; that is provider behavior, so +reword or repeat the prompt. + +Inline image rendering in the TUI also covers a tool/MCP result that +returns an image, or a Markdown image reference to a file a tool actually +writes to disk — see [Markdown Images](#markdown-images) above. + +### Team Context Budgets and Targeted Compaction + +The `/context` dialog also shows a **Live sessions** section: the current session plus every currently running sub-agent session (foreground children spawned by task transfer and long-running `run_background_agent` tasks). Each row shows the agent name, a short session ID (so two concurrent runs of the same agent stay distinguishable), and that session's context budget: used tokens, context limit, and percentage, or an explicit "limit unknown" reading when the model's window cannot be resolved. Live-sessions rows do not repeat the compaction-cap wording themselves — the dialog's header line is the sole authority on which model, if any, caps the effective limit. + +When a compaction has occurred, the dialog displays the verbatim text of the most recent compaction summary below the file inventory, under a "Latest compaction summary" section. This shows exactly what was summarized, preserving hard newlines and soft-wrapping long lines to the dialog width. + +Select a live session with ↑/↓ and press Enter to explicitly compact it. Cross-agent compaction happens only on this explicit request: no idle-triggered automatic compaction is added, and the existing automatic threshold and overflow-recovery compaction of sub-agent sessions is unchanged. The request is queued onto the target session's own run loop and executes at the next safe point between model turns, so it cannot corrupt an in-flight turn. The dialog closes and a notification confirms the request; a second notification reports the outcome (compacted, skipped, or failed) with the agent's name. Selecting the main row runs the same compaction as `/compact`. `/compact` itself keeps compacting the current root session. Remote runtimes do not expose live-session tracking, so the section is omitted there. ## Runtime Model Switching diff --git a/docs/providers/google/index.md b/docs/providers/google/index.md index ff5958e19c..a5919aa9aa 100644 --- a/docs/providers/google/index.md +++ b/docs/providers/google/index.md @@ -64,29 +64,70 @@ models: Some Gemini models (e.g. `gemini-2.5-flash-image`) are designed to generate an image directly as part of their reply, not just describe one. Docker -Agent's Gemini request path doesn't yet ask for that image output — that -support is still being completed — so today a request like this gets a -text-only reply. See -[Generated Media](../../features/tui/index.md#generated-media) for the -current, verified state. +Agent requests that image output on supported Google surfaces — the models +gateway, direct Gemini API, and Vertex AI — according to one binding policy: +explicit `false`, explicit `true`, then an exact models.dev record whose +`Modalities.Output` contains `image`. An omitted image flag, including +`output_capabilities: {}`, uses that catalogue default. Unknown models or +unavailable catalogue data leave image response modalities disabled; model +names are never used to guess capability. +Each eligible ordinary chat request asks for text *and* image output. +Vertex AI has deterministic guard/predicate coverage; live image-generation +validation is deferred. + +Docker Agent attempts to save each returned image and display it in the TUI — see [Generated Media](../../features/tui/index.md#generated-media) +for file naming, collision handling, and rendering details. + +The workspace file is the visible deliverable. After that file and its manifest +entry are saved, a portable copy is also stored in the session database and +preferred when the session is reopened. It can render the original generated +bytes even if the workspace file was edited, moved, or deleted. +If portable +persistence fails, the workspace file is still kept and the turn includes a +warning. +Sessions created before portable copies were introduced continue to use their +manifest-gated workspace files. ```yaml -agents: - root: - model: google/gemini-2.5-flash-image +models: + gemini-image: + provider: google + model: gemini-2.5-flash-image ``` -When the model is accessed through a Docker AI Gateway and explicitly -declared image-output-capable with -[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities), -Docker Agent has verified that request combined with custom function tools, -a built-in tool (e.g. `google_search`), or structured output gets rejected -by the gateway with an opaque, empty-body HTTP 400. To avoid that, Docker -Agent rejects such a combination itself, before any request is sent, with a -clear error naming which feature is incompatible. Plain text requests to -that model (no tools, no structured output) are unaffected, as is every -other route: direct Gemini API/Vertex AI calls, and gateway calls to a model -without the declaration. +When `output_capabilities.image` is omitted, including in an empty block, +Docker Agent uses models.dev output modalities for the exact known model. Set +it explicitly for custom models or to override incorrect catalogue data; +unknown or unavailable metadata remains disabled and capability is never +guessed from the model name. + +Session-title and compaction requests omit image response modalities and +bypass the guard even for image-output-capable models. They do not explicitly +force TEXT-only output. Ordinary image-output requests with custom function tools or +structured output are rejected locally before any request is sent. Google +Search, Maps, and code-execution built-ins remain available. When custom +tools conflict, Docker Agent uses models.dev's `tool_call` capability to clarify +whether the model cannot call tools at all or supports tools only outside an +image-output request. Unknown catalogue data keeps the conservative generic +message. + +A few provider-side behaviors to know: + +- **The provider decides the image format** (typically PNG). Asking for a + `.gif` or `.svg` filename does not transcode anything — the saved file's + extension is corrected to match the data actually returned. +- **An image is not guaranteed.** Even a correctly configured image model + can answer with text only and generate no image. At a text-only stop, + phrases such as "generate an image" or "draw a picture" in the last user + prompt trigger a nonfatal warning while preserving the reply: + `The model returned text but no image for this image-generation request. Try rephrasing the request.` + This phrase-based check is not semantic intent detection and does not + check output capability: negated or quoted phrases can match and other + wording can be missed. It does not track a whole submission across tool + calls, steering, stop hooks, or handoffs. A terminal provider error skips + this check, as does structured output configured on the current agent + model; per-call overrides and reply content are not independently + classified. ## Thinking Budget diff --git a/examples/README.md b/examples/README.md index 759c7f603d..2d8b7d811b 100644 --- a/examples/README.md +++ b/examples/README.md @@ -216,7 +216,7 @@ remote MCP endpoints. | [`rule_based_routing.yaml`](rule_based_routing.yaml) | Cheap router model dispatches the user message to fast or capable models. | | [`structured-output.yaml`](structured-output.yaml) | Forces the model to return JSON matching a schema. | | [`google_search_grounding.yaml`](google_search_grounding.yaml) | Enables Google Search grounding on Gemini models. | -| [`gemini_image_output.yaml`](gemini_image_output.yaml) | Gemini image-output model (generated images are saved into the workspace, not inlined as base64). | +| [`gemini_image_output.yaml`](gemini_image_output.yaml) | Gemini image-output model: generated images are saved into the workspace, stored portably with the session, and rendered inline in the TUI. | | [`sampling-opts.yaml`](sampling-opts.yaml) | Provider-specific sampling parameters (`top_k`, `repetition_penalty`, …). | | [`thinking_budget.yaml`](thinking_budget.yaml) | Reasoning/thinking budgets across OpenAI, Anthropic and Google. | | [`task_budget.yaml`](task_budget.yaml) | Anthropic `task_budget`: cap total tokens spent across a multi-step agentic task. | diff --git a/examples/gemini_image_output.yaml b/examples/gemini_image_output.yaml index e8490a4f62..1531b974a1 100644 --- a/examples/gemini_image_output.yaml +++ b/examples/gemini_image_output.yaml @@ -1,25 +1,56 @@ # Gemini image-output model: the model can generate an image directly as # part of its reply, instead of only describing one. # -# The `gemini-image` model below declares `output_capabilities.image: true` — -# an explicit, owner-provided statement that this model can generate image -# output. It is never inferred from the model name or any catalogue; omit -# it, or leave it false, and behavior is unchanged. +# The `gemini-image` model below relies on the exact models.dev record's +# `Modalities.Output=image`. An explicit `output_capabilities.image` value can +# override catalogue data; unknown or unavailable catalogue metadata stays off. +# Capability is never guessed from the model name. # -# Native generated images aren't currently presented inline in the -# terminal UI, so the model's text reply is what you'll see today. +# On the models gateway, direct Gemini API, and Vertex AI, Docker Agent +# requests text AND image output for this image-capable model. It attempts +# to save each image in the workspace, record portable bytes, and display it +# in the TUI. Failures warn per item; unsupported graphics use path labels. # +# Notes: +# - Session-title/compaction omit image modalities and bypass the guard; +# they do not explicitly force TEXT-only output. Ordinary requests +# with custom tools or structured output are rejected locally before any +# request is sent; Google Search, Maps, and code-execution built-ins remain +# available. For custom tools, models.dev metadata lets the error say +# whether the model lacks tool calls or only cannot combine them with +# image output; unknown metadata keeps a conservative generic message. +# - Naming: the model is instructed to honor an explicit prompt filename +# in a naming marker; otherwise it chooses a meaningful name. A missing +# marker falls back to a single parsed prompt filename only for a single +# returned image, then to the provider name or a numbered generic name. +# Multi-image replies use markers, provider names, or numbered names. +# Extensions are corrected to match the returned data (the provider +# decides the format, typically PNG), and existing files are never +# overwritten — collisions get a dash suffix (logo-1.png). +# - A text-only stop can trigger a nonfatal warning for phrases such as +# "generate an image" in the last prompt. This is phrase matching, not +# semantic intent detection: quoted/negated phrases can match and other +# wording can be missed. It does not check output capability or track +# a whole submission across tools, steering, stop hooks, or handoffs. +# Terminal provider errors and current-agent structured-output config +# skip the check; per-call overrides/reply content are not classified. +# - Ordinary follow-ups send metadata placeholders, not generated bytes. +# Attaching a saved image or reading it through a tool can send it again. +# +# Optional live trials require credentials, may be billed, and should use +# a disposable workspace and session store, not an active session database. +# Live Vertex image-generation validation is deferred. # Try it out: # docker agent run examples/gemini_image_output.yaml \ # "Generate an image of a red panda working at a terminal" # docker agent run examples/gemini_image_output.yaml \ -# "Generate an image of a lighthouse at sunset, and describe the color palette you used" +# "Generate an image of a lighthouse at sunset as assets/lighthouse.png" +# docker agent run examples/gemini_image_output.yaml \ +# "Generate an image of a whale and save it as whale.jpg" models: gemini-image: provider: google model: gemini-2.5-flash-image - output_capabilities: - image: true agents: root: From 4b04a89084ac6ea9d35630b3e1111ee73ceeb18a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Arnaud=20He=CC=81ritier?= Date: Fri, 4 Sep 2026 20:09:14 +0200 Subject: [PATCH 3/3] docs: clarify uncapped generated-media storage and resolution Generated media has no application-level byte cap in portable session storage or authorized file resolution. Ordinary attachment limits are unchanged. --- docs/features/sessions/index.md | 3 ++- docs/features/tui/index.md | 1 + docs/providers/google/index.md | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/features/sessions/index.md b/docs/features/sessions/index.md index 4ff72abddb..39f2c0c19b 100644 --- a/docs/features/sessions/index.md +++ b/docs/features/sessions/index.md @@ -54,7 +54,8 @@ a session revokes its references even if workspace provenance is cached. A legacy workspace file that was deleted, replaced by a symlink, made invalid, or cannot be read appears as a short "unavailable" note. Manifest and containment checks do not verify the integrity of -ordinary-file contents, which may have changed. Deleting a session also deletes its stored +ordinary-file contents, which may have changed. Generated-media storage and +resolution have no byte cap. Deleting a session also deletes its stored generated-media blobs. See [Generated Media](../tui/index.md#generated-media) for naming, collision handling, and rendering details. diff --git a/docs/features/tui/index.md b/docs/features/tui/index.md index 2636f97ed5..13795c29b3 100644 --- a/docs/features/tui/index.md +++ b/docs/features/tui/index.md @@ -376,6 +376,7 @@ session store has no blob interface or the blob is not found. Other blob errors fail closed. Historical manifest entries that identify an external root are rejected. Stores without blob support and sessions created before portable blobs were introduced continue to use legacy workspace files. +Generated-media resolution has no byte cap. Ordinary outgoing history replaces generated-media parts with metadata placeholders, so follow-up turns do not resend stored bytes. Explicitly diff --git a/docs/providers/google/index.md b/docs/providers/google/index.md index a5919aa9aa..d0436667ae 100644 --- a/docs/providers/google/index.md +++ b/docs/providers/google/index.md @@ -82,7 +82,7 @@ The workspace file is the visible deliverable. After that file and its manifest entry are saved, a portable copy is also stored in the session database and preferred when the session is reopened. It can render the original generated bytes even if the workspace file was edited, moved, or deleted. -If portable +Generated-media storage and resolution do not impose a size cap. If portable persistence fails, the workspace file is still kept and the turn includes a warning. Sessions created before portable copies were introduced continue to use their