feat(#3996): resolve and filter input media per request model - #4016
Open
aheritier wants to merge 3 commits into
Open
feat(#3996): resolve and filter input media per request model#4016aheritier wants to merge 3 commits into
aheritier wants to merge 3 commits into
Conversation
aheritier
added
area/config
aheritier
force-pushed
the
media-input-modalities
branch
2 times, most recently
from
August 25, 2026 20:24
2000915 to
d2b8ce4
Compare
aheritier
force-pushed
the
media-input-modalities
branch
3 times, most recently
from
September 1, 2026 16:29
d2b8ce4 to
26adf11
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
aheritier
force-pushed
the
media-input-modalities
branch
7 times, most recently
from
September 8, 2026 17:30
1898a54 to
0180c77
Compare
Extend ModelCapabilities, CapsWith, CapsOverride and catalogue parsing with audio/video input support. Unknown models remain conservative. Update provider and attachment callers for the expanded capability shape without adding audio/video conversion support or changing the config schema. Retain the existing cancellable catalogue-load timeout and conservative fallback; waiting for the shared store lock remains outside that timeout. Cover modality and MIME matching, overrides, and fallback behavior.
Extend latest.CapabilitiesConfig with Audio and Video and wire all four input flags through base.Config.CapsOverride. This lets custom or uncatalogued models override incomplete models.dev input metadata. Update agent-schema.json, model-reference docs, capability-overrides.yaml, and schema/config/provider tests. Numbered config packages stay frozen.
Generalize strip_unsupported_modalities across image, audio, and video parts using the capabilities of the model selected for each attempt. Prepare history again for each fallback provider rather than reusing the primary model's filtered copy, so a capable fallback retains user media. Keep stored history unchanged, preserve text and document ordering, and register the builtin by default for agents that bypass the YAML loader. Expose ModelAudioCapable and ModelVideoCapable to transform hooks. Unknown catalogue entries remain conservative for audio/video; retain DMR provider flags and the existing Anthropic/Claude image/PDF fallback. Update the configuration reference and example, and test MIME mapping, runtime-only agents, per-tool models, and primary/fallback capability differences.
aheritier
force-pushed
the
media-input-modalities
branch
from
September 8, 2026 20:32
0180c77 to
309a956
Compare
aheritier
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Resolve audio/video input capability independently and support latest-only configuration overrides. Filter unsupported image/audio/video parts using the model selected for each request, preserve provider-specific image fallbacks, and prepare history again for fallback models. Stored history and supported text/document ordering remain intact. Input audio/video capability invariants are retained; this does not add universal upload or conversion support.
Part of #3996. Review this PR against its immediate parent,
main, rather than the aggregate stack againstmain.Commit inventory
Head:
309a95662d205a795722eae67a1b3c9d906f99bc; parent SHA:316a5d2fe3c1b3a97fe0e6a867f6715d173cbd70.d8ca5eea7173a77a34021b57d357ffba77d4ea8b— feat(Epic: Gemini image output with workspace deliverables and portable session copies #3996): detect audio/video input modalities98639c4860fe8f42193f377033a1866c6c1757e4— feat(Epic: Gemini image output with workspace deliverables and portable session copies #3996): add audio/video capability overrides309a95662d205a795722eae67a1b3c9d906f99bc— feat(Epic: Gemini image output with workspace deliverables and portable session copies #3996): strip unsupported audio/video media partsValidation
Build, test compilation, owning-package tests and the named fixture passed at this PR head.
Exact deterministic fixture command:
Matched top-level tests:
pkg/config/latest: 2;pkg/hooks/builtins: 1;pkg/modelinfo: 2;pkg/runtime: 3.Deterministic scope: the named local fixture exercises this PR boundary with disposable configuration/stores and fake or loopback providers as applicable. Every listed package ran nonzero matching top-level tests.
Deferred/live scope: Live provider, remote CI, and platform execution are not claimed by this deterministic receipt. The final stack head passed build, lint, full tests, an uncached full suite, focused race tests and documentation checks in disposable environments. Remote CI is tracked by the checks below; no new paid-provider or active-database validation was run.