feat: allow images as urls for supported backends#1260
Conversation
Signed-off-by: AngeloDanducci <angelo.danducci.ii@ibm.com>
Signed-off-by: AngeloDanducci <angelo.danducci.ii@ibm.com>
planetf1
left a comment
There was a problem hiding this comment.
One test regression and a docstring gap to fix before merge, plus a couple of things worth flagging.
Test assertion needs updating (test/backends/test_vision_openai.py:163–164): The ollama version (test_vision_ollama.py:148–150) was updated to pull out first_image and compare .value, but the OpenAI test still compares last_action.images[0] directly against image_block.value. Since images now returns block objects, this is comparing an ImageBlock to a str and will fail when the e2e test runs:
# first image in image list should be the same as the image block
first_image = last_action.images[0] # type: ignore
assert isinstance(first_image, ImageBlock)
assert first_image.value == ImageBlock.from_pil_image(pil_image).valueRaises: entry missing (mellea/backends/ollama.py:366–367): generate_from_chat_context now raises ValueError for ImageUrlBlock, but the docstring only lists RuntimeError. Per AGENTS.md (Section 10, item 6) every raise in a public function wants a matching entry — the quality gate won’t catch this one since RuntimeError is already listed:
Raises:
RuntimeError: If not called from a thread with a running event loop.
ValueError: If a message contains an ``ImageUrlBlock``; Ollama requires
base64-encoded images — convert to an ``ImageBlock`` first.Breaking change: Message.images now returns list[ImageBlock | ImageUrlBlock] rather than list[str]. Right call, but worth a line in the PR description and a changelog entry so people know to expect it on upgrade.
Missing tests: Three new paths that’d be worth covering before this merges, since they’re the heart of the feature:
get_image_blocks()with anhttps://URL returning anImageUrlBlock— the http/https branch atcli/serve/models.py:89–90isn’t exercised yet (the existing pydantic-parse test only checks the model deserialises)- Ollama rejecting an
ImageUrlBlockwithValueError— the guard atollama.py:401–402has no test message_to_openai_messagepassing a URL straight through as{"image_url": {"url": ...}}instead of wrapping it in a data URI (openai_compatible_helpers.py:200–206) — nothing hits that branch currently
| if TYPE_CHECKING: | ||
| from ..core import Formatter, ModelToolCall | ||
| from ..core.base import AbstractMelleaTool, ModelOutputThunk | ||
| from ..core.base import AbstractMelleaTool, ImageUrlBlock, ModelOutputThunk |
There was a problem hiding this comment.
Nit: ImageUrlBlock is already imported at runtime on line 11, so the copy in the TYPE_CHECKING block is dead. Ruff doesn’t flag it across the guard, but it reads as if it’s type-only.
| from ..core.base import AbstractMelleaTool, ImageUrlBlock, ModelOutputThunk | |
| from ..core.base import AbstractMelleaTool, ModelOutputThunk |
Pull Request
Issue
Fixes #127
Description
allows images as urls for supported backends
Testing
Attribution
Adding a new component, requirement, sampling strategy, or tool?
If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.
NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.