Skip to content

feat: allow images as urls for supported backends#1260

Open
AngeloDanducci wants to merge 2 commits into
generative-computing:mainfrom
AngeloDanducci:ad-127
Open

feat: allow images as urls for supported backends#1260
AngeloDanducci wants to merge 2 commits into
generative-computing:mainfrom
AngeloDanducci:ad-127

Conversation

@AngeloDanducci

Copy link
Copy Markdown
Contributor

Pull Request

Issue

Fixes #127

Description

allows images as urls for supported backends

Testing

  • Tests added to the respective file if code was changed
  • New code has 100% coverage if code was added
  • Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

  • AI coding assistants used

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

  • Component
  • Requirement
  • Sampling Strategy
  • Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

Signed-off-by: AngeloDanducci <angelo.danducci.ii@ibm.com>
@AngeloDanducci AngeloDanducci requested review from a team, jakelorocco and nrfulton as code owners June 12, 2026 06:12
@AngeloDanducci AngeloDanducci requested a review from planetf1 June 12, 2026 06:12
@github-actions github-actions Bot added the enhancement New feature or request label Jun 12, 2026
@AngeloDanducci AngeloDanducci enabled auto-merge June 12, 2026 06:15
Signed-off-by: AngeloDanducci <angelo.danducci.ii@ibm.com>

@planetf1 planetf1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One test regression and a docstring gap to fix before merge, plus a couple of things worth flagging.

Test assertion needs updating (test/backends/test_vision_openai.py:163–164): The ollama version (test_vision_ollama.py:148–150) was updated to pull out first_image and compare .value, but the OpenAI test still compares last_action.images[0] directly against image_block.value. Since images now returns block objects, this is comparing an ImageBlock to a str and will fail when the e2e test runs:

    # first image in image list should be the same as the image block
    first_image = last_action.images[0]  # type: ignore
    assert isinstance(first_image, ImageBlock)
    assert first_image.value == ImageBlock.from_pil_image(pil_image).value

Raises: entry missing (mellea/backends/ollama.py:366–367): generate_from_chat_context now raises ValueError for ImageUrlBlock, but the docstring only lists RuntimeError. Per AGENTS.md (Section 10, item 6) every raise in a public function wants a matching entry — the quality gate won’t catch this one since RuntimeError is already listed:

        Raises:
            RuntimeError: If not called from a thread with a running event loop.
            ValueError: If a message contains an ``ImageUrlBlock``; Ollama requires
                base64-encoded imagesconvert to an ``ImageBlock`` first.

Breaking change: Message.images now returns list[ImageBlock | ImageUrlBlock] rather than list[str]. Right call, but worth a line in the PR description and a changelog entry so people know to expect it on upgrade.

Missing tests: Three new paths that’d be worth covering before this merges, since they’re the heart of the feature:

  • get_image_blocks() with an https:// URL returning an ImageUrlBlock — the http/https branch at cli/serve/models.py:89–90 isn’t exercised yet (the existing pydantic-parse test only checks the model deserialises)
  • Ollama rejecting an ImageUrlBlock with ValueError — the guard at ollama.py:401–402 has no test
  • message_to_openai_message passing a URL straight through as {"image_url": {"url": ...}} instead of wrapping it in a data URI (openai_compatible_helpers.py:200–206) — nothing hits that branch currently

if TYPE_CHECKING:
from ..core import Formatter, ModelToolCall
from ..core.base import AbstractMelleaTool, ModelOutputThunk
from ..core.base import AbstractMelleaTool, ImageUrlBlock, ModelOutputThunk

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: ImageUrlBlock is already imported at runtime on line 11, so the copy in the TYPE_CHECKING block is dead. Ruff doesn’t flag it across the guard, but it reads as if it’s type-only.

Suggested change
from ..core.base import AbstractMelleaTool, ImageUrlBlock, ModelOutputThunk
from ..core.base import AbstractMelleaTool, ModelOutputThunk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

allow images to be urls

2 participants