feat: Add content_video_file() and content_video_youtube() for Gemini video input - #347
Open
cpsievert wants to merge 4 commits into
Open
feat: Add content_video_file() and content_video_youtube() for Gemini video input#347cpsievert wants to merge 4 commits into
content_video_file() and content_video_youtube() for Gemini video input#347cpsievert wants to merge 4 commits into
Conversation
Introduces ContentVideoInline and ContentVideoUrl (plus the shared ContentVideo base and VideoContentTypes literal) to represent Gemini video input: small inline clips, and public YouTube URLs referenced with no upload and no MIME type. content_video_file() and content_video_youtube() are the public constructors, exported from chatlas and chatlas.types alongside the existing image/PDF helpers.
ChatGoogle()/ChatVertex() send ContentVideoInline as inline Blob data and ContentVideoUrl as a bare Part(file_data=FileData(...)) -- deliberately not Part.from_uri(), which falls back to mimetypes.guess_type() and raises for a YouTube watch URL that has no file extension to guess from. ChatOpenAI(), ChatAnthropic(), and the OpenAI-compatible Chat Completions providers (Groq, Mistral, Ollama, etc.) now raise a clear NotImplementedError for video content instead of falling through to a generic "unknown content type" error, consistent with how remote images are already rejected on Google. Also updates docs/chat.qmd and the quartodoc reference with a video input section, and adds a CHANGELOG entry.
# Conflicts: # chatlas/_provider_openai.py # chatlas/_provider_openai_completions.py # docs/_quarto.yml # tests/test_provider_anthropic.py # tests/test_provider_openai.py
content_video_file() and content_video_youtube() for Gemini video input
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You can now point a chat at a public YouTube video and ask about it — no upload, no download, no local copy:
Gemini fetches the video server-side, accepts up to 10 per request on 2.5+ models, and currently charges nothing for it.
A small local clip works the same way, sent inline:
Previously the only route to video was the Files API:
That path is genuinely right for large files — it already waits for Gemini to finish processing before returning — but it's a two-step round trip for a ten-second clip, and it cannot express a YouTube URL at all: there's no file to upload. This mirrors how images already get both an inline and a Files-API path, and the docs now say which to reach for.
What each provider accepts
Gemini is the only major provider that accepts video, so the interesting column is the first one. The rest raise a clear
NotImplementedErrornaming Gemini rather than falling through to a generic "unknown content type".ChatGoogle()/ChatVertex()ChatOpenAI()ChatOpenAICompletions()ChatAnthropic().mp4,.mpeg,.mov,.avi,.flv,.mpg,.webm,.wmv,.3gppchat.files.upload()Inline video is capped by the ~100 MB request ceiling; past that,
chat.files.upload()remains the answer and the docstring points there.One finding worth flagging
Gemini's documented approach for YouTube URLs doesn't work through the SDK constructor.
Part.from_uri()falls back tomimetypes.guess_type()whenmime_typeis omitted, and that can't infer anything from ayoutube.com/watch?v=...URL, so it raisesValueError. Gemini genuinely wants no MIME type here — it determines the format itself. The provider buildsPart(file_data=FileData(file_uri=url))directly instead, with the reasoning in a comment at the call site so it doesn't get "simplified" back later. Verified against the installedgoogle-genai, not inferred from docs.Notes for review
YouTube URLs are modeled as their own
ContentVideoUrl, notContentUploadedwith a synthetic MIME type. Nobody uploaded a YouTube video: it doesn't expire in 48 hours, it isn't listable or deletable throughchat.files, and inventing a MIME type for it would put a falsehood into serialized turn history. The class shape stays generic ({url: str}) in case Gemini widens this to other URIs.No generic "remote video URL" helper, because that isn't a verified Gemini capability — only YouTube is special-cased in their API.
No VCR cassette for a live YouTube round trip, since recording one needs a real
GOOGLE_API_KEY. The dispatch-level tests cover the serialization logic, including a regression guard assertingfile_data.mime_type is Noneso thePart.from_uri()trap can't come back.Verified after merging
main:pyrightclean,ruffclean, 198 tests passing across the content and provider-dispatch suites, of which 29 are video-specific.ellmer parity
New capability, not a port — ellmer has no video content of any kind today:
content_image_*andcontent_pdf_*are the only file inputs; theContentclasses are Text, Image, ToolRequest, ToolResult, Json, Uploaded, Thinking, PDF.google_upload()has a video MIME table (R/provider-google-upload.R:210-216— mp4, avi, mkv, mov, wmv, webm). That's the analogue ofchat.files.upload(), not inline video.If ellmer wants parity:
ContentVideoparent with inline and URL variants, pluscontent_video_file()/content_video_youtube(). The Geminias_json()methods are short:inlineDatawith base64 + MIME type for the inline case, andfileDatawith a barefile_urifor the YouTube case. R sidesteps the trap that bit here — there's noPart.from_uri()equivalent doing helpful MIME guessing, so hand-building the list is the only option anyway.as_json()dispatch already errors on unhandled content, but the default message won't tell the user that Gemini is the one provider that would have worked.google_upload()'s MIME table while you're there: it's missing3gpp,flv,mpg/mpeg, and includesmkv, which Gemini's inline-video docs don't list.No ellmer issue tracks any of this (searched video, youtube), so it would all be new.
Landing order
Part of a three-PR set (documents, audio, video). All three touch
ContentTypeEnum,ContentUnion, andcreate_content(). #345 has since landed and this branch is merged up to currentmain, so the only remaining overlap is with #346 (audio) — whichever of the two goes second needs a trivial re-merge of those three lists.