Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion .github/workflows/quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ jobs:
# projector Ollama needs for image input). Drop the hf.co prefix if the
# model is ever published to the Ollama library directly.
run: |
for model in granite4.2:3b granite4:micro-h hf.co/ibm-granite/granite-vision-4.1-4b-GGUF:Q4_K_M; do
for model in granite4.2:3b granite4:micro-h granite4.1:3b hf.co/ibm-granite/granite-vision-4.1-4b-GGUF:Q4_K_M; do
pulled=false
for i in 1 2 3 4 5; do
ollama pull "$model" && { pulled=true; break; }
Expand All @@ -153,6 +153,25 @@ jobs:
# The names below are what the tests match against; print them so a
# tag-name mismatch is diagnosable from the log alone.
ollama list
- name: Constrain the granite4.1:3b context for the CI runner
# granite4.1:3b ships the same oversized num_ctx default as
# granite4.2:3b below. The uncertainty aLoRA Ollama model is built
# `FROM` this tag, so it must be constrained *before* that build step
# or the bundled adapter model inherits the unconstrained context.
run: |
ollama cp granite4.1:3b granite4.1:3b-128k
printf 'FROM granite4.1:3b-128k\nPARAMETER num_ctx 8192\n' \
> /tmp/MODELFILE-granite41-ci
ollama create granite4.1:3b -f /tmp/MODELFILE-granite41-ci
ollama show granite4.1:3b --modelfile | grep -qx 'PARAMETER num_ctx 8192'
- name: Build the official uncertainty aLoRA Ollama model
Comment thread
planetf1 marked this conversation as resolved.
env:
# read-only public-repo HF token; used only to make the pinned build
# reliable under Hub rate limits.
HF_TOKEN: ${{ secrets.HF_TOKEN_READ_PUBLIC_ONLY }} # zizmor: ignore[secrets-outside-env]
run: |
model=$(./test/scripts/build_ollama_uncertainty_adapter.sh)
echo "MELLEA_OLLAMA_UNCERTAINTY_MODEL=$model" >> "$GITHUB_ENV"
- name: Constrain the granite4.2:3b context for the CI runner
# The published tag ships num_ctx=131072, a ~6 GB KV cache at load —
# too much for the 16 GB runner and a cause of the CI stalls. Re-point
Expand Down
8 changes: 5 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,15 +226,17 @@ For lower-level control (custom adapters, model options), use `mfuncs.act()` wit

### Weights binding shapes

`Adapter.weights` normalizes each deployment's activation mechanism behind one of
two shapes — a `WeightsBinding` lifecycle for weights you stage yourself, or
`EmbeddedBinding.apply_activation` for weights already in the served model. The
`Adapter.weights` normalizes each deployment's activation mechanism behind three
shapes — a `WeightsBinding` lifecycle for weights you stage yourself,
`EmbeddedBinding.apply_activation` for weights already in the served model, or
`ServerMediatedBinding` for a model tag selected by the provider. The
post-activation shape each produces:

| Binding | Reality | Lifecycle verbs | Caller invokes | Normalized post-activation state |
|---------|---------|------------------|-----------------|-----------------------------------|
| `LocalFileBinding` | LocalFile/PEFT | `prepare` / `activate` / `deactivate` / `release` | `activate()` / `deactivate()`, via `adapter_scope` | Backend-internal PEFT adapter state toggled; the outgoing request is untouched |
| `EmbeddedBinding` | Embedded/Granite Switch | none — weights are already in the served model | `apply_activation(request, identity)` | `request.extra_body["chat_template_kwargs"]["adapter_name"]` set; `request.api_params["model"]` removed if present |
| `ServerMediatedBinding` | Ollama bundled adapter model | none for the current Ollama path | select the configured model tag during intrinsic generation | Ollama request's `model` is the bundled adapter tag; full lifecycle telemetry remains follow-up work |

### Project Resources

Expand Down
79 changes: 73 additions & 6 deletions docs/docs/advanced/intrinsics.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,12 @@ checkpoints. Both local paths require a GPU or Apple Silicon Mac. An
OpenAIBackend using a Granite Switch model served via vLLM uses
`uv sync --extra switch` when it downloads embedded adapter metadata.

Adapter functions are adapter-accelerated operations for RAG quality checks. They use
LoRA/aLoRA adapters loaded directly into the Hugging Face backend — faster and more
reliable than prompting a general-purpose model for these specialized micro-tasks.
Adapter functions are adapter-accelerated operations for RAG quality checks. Their
LoRA/aLoRA weights are loaded locally, selected from an embedded checkpoint, or
bundled into an Ollama model — faster and more reliable than prompting a
general-purpose model for these specialised micro-tasks.

> **Backend note:** Adapter functions work with two backends:
> **Backend note:** Adapter functions work with three backends:
>
> - **LocalHFBackend** — loads LoRA/aLoRA adapters from the catalog at runtime.
> A local Granite Switch checkpoint can instead use
Expand All @@ -25,8 +26,12 @@ reliable than prompting a general-purpose model for these specialized micro-task
> `load_embedded_adapters=True`. Only adapter functions embedded in the model are
> available — check the model's `adapter_index.json` for the list.
> See `docs/docs/examples/granite-switch/README.md`
> - **OllamaModelBackend** — uses an Ollama model that bundles the adapter.
> Ollama bundles one adapter per model, so pass
> `adapter_models={"uncertainty": "<tag>", ...}` to route each adapter function
> to its model. Install `mellea[switch]` to download the adapter's `io.yaml`.
>
> Adapter functions do not work with Ollama or other remote backends.
> Adapter functions do not work with other remote backends.

Set up the backend once and reuse it across adapter function calls:

Expand All @@ -38,6 +43,66 @@ from mellea.backends.huggingface import LocalHFBackend
backend = LocalHFBackend(model_id="ibm-granite/granite-4.1-3b")
```

## Use an adapter bundled in an Ollama model

Ollama serves adapter weights as part of a model tag; Mellea does not load the
weights separately. For local development, build a bundled uncertainty model
from the pinned official Granite base and adapter artefacts:

```bash
MELLEA_OLLAMA_UNCERTAINTY_MODEL="$(
./test/scripts/build_ollama_uncertainty_adapter.sh
)"
export MELLEA_OLLAMA_UNCERTAINTY_MODEL
```

Install the lightweight Hugging Face Hub dependency that retrieves the
catalogued `io.yaml`:

```bash
uv sync --extra switch
```

For one adapter function, use the bundled tag for both ordinary chat and the
adapter route. Before the invocation tokens appear, the aLoRA model behaves as
the base model; keeping one model identity lets Ollama reuse its own prefix
cache where available.

```python
import os

from mellea.backends import ModelOption
from mellea.backends.ollama import OllamaModelBackend
from mellea.stdlib.components import Message
from mellea.stdlib.components.intrinsic import core
from mellea.stdlib.context import ChatContext

backend = OllamaModelBackend(
model_id=os.environ["MELLEA_OLLAMA_UNCERTAINTY_MODEL"],
adapter_base_model_name="granite-4.1-3b",
model_options={ModelOption.CONTEXT_WINDOW: 4096},
adapter_models={
"uncertainty": os.environ["MELLEA_OLLAMA_UNCERTAINTY_MODEL"],
},
)
context = (
ChatContext()
.add(Message("user", "What is the square root of 4?"))
.add(Message("assistant", "The square root of 4 is 2."))
)

print(core.check_certainty(context, backend))
```

When an application needs several adapter functions, keep the base model as
`model_id` and map each function to its bundled model tag. Current Ollama
model packaging exposes one adapter per tag, so calls across those tags do not
share a KV cache. Use a Granite Switch checkpoint when multi-adapter,
single-model serving is important.

See `docs/examples/intrinsics/uncertainty_ollama.py` for the complete
executable example.

## Use a local Granite Switch checkpoint

Granite Switch checkpoints contain their adapter functions already. Pass the
Expand Down Expand Up @@ -342,8 +407,10 @@ Weights-binding support by backend today:
| --- | --- | --- | --- |
| `LocalHFBackend` | ✅ shipping — `add_adapter` accepts a composed `Adapter` or a bare `LocalFileBinding` directly | ✅ shipping — `load_embedded_adapters=True`, or `add_adapter(adapter, config=...)`/`register_embedded_adapter_model` with a composed `Adapter` | — |
| `OpenAIBackend` | — | ✅ shipping — `load_embedded_adapters=True`, or `add_adapter(adapter, config=...)`/`register_embedded_adapter_model` with a composed `Adapter` | — |
| `OllamaModelBackend` | — | — | ✅ model selection through `adapter_models` for catalogued adapter functions; lifecycle telemetry is tracked separately |

`ServerMediatedBinding` has no backend implementation yet — see discussion #1486.
`ServerMediatedBinding` currently supports Ollama's bundled-model path. A full
server-mediated lifecycle and telemetry contract is tracked separately.
Discovering *multiple* embedded adapters from a Granite Switch checkpoint or
Hub repo (rather than one already-known name) still goes through
`register_embedded_adapter_model`, which builds the composed `Adapter`
Expand Down
9 changes: 7 additions & 2 deletions docs/docs/advanced/lora-and-alora-adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,13 @@ and use it as a requirement validator in any Mellea program.
Apple Silicon Mac with sufficient VRAM for the chosen base model. Uploading requires a
Hugging Face account.

> **Backend note:** Custom-trained adapters can only be loaded into `LocalHFBackend`.
> They do not work with Ollama, OpenAI, or other remote backends.
> **Backend note:** Custom-trained adapters can only be loaded directly into
> `LocalHFBackend`. Ollama can use a custom adapter bundled into a model with a
> Modelfile `ADAPTER` line, but Mellea does not discover custom Ollama adapter
> functions from `adapter_models` alone. Register a composed adapter with its
> `io.yaml` explicitly, then map its name to the bundled model tag. See
> [Adapter functions](./intrinsics.md) for the supported catalogue-adapter
> workflow.
>
> Granite Switch models ship with pre-trained adapter functions embedded in the
> model weights. Use them through `OpenAIBackend` with a served checkpoint, or
Expand Down
12 changes: 7 additions & 5 deletions docs/docs/tutorials/04-making-agents-reliable.md
Original file line number Diff line number Diff line change
Expand Up @@ -365,8 +365,8 @@ response = m.instruct(

output_text = str(response)

# Guardian adapter functions require a LocalHFBackend — they load LoRA adapters
# that are not supported by OllamaModelBackend.
# This tutorial uses LocalHFBackend. Ollama requires a compatible bundled model
# for each Guardian adapter function; see Adapter functions for that setup.
guardian_backend = LocalHFBackend(model_id="ibm-granite/granite-4.1-3b")

# Build a context containing the exchange to check.
Expand Down Expand Up @@ -397,9 +397,11 @@ and dynamic applications with ease. The word "Mellea" consists of
6 characters.
```

> **Note:** Guardian adapter functions load LoRA adapters and require `LocalHFBackend`.
> They cannot run against `OllamaModelBackend`. The main agent and the Guardian
> checks can use different backends — only the Guardian calls need `LocalHFBackend`.
> **Note:** This tutorial uses `LocalHFBackend` because it loads Guardian LoRA
> adapters directly. `OllamaModelBackend` can run a Guardian adapter function
> only when you provide an Ollama model tag that bundles that adapter through
> `adapter_models`. The main agent and the Guardian checks can use different
> backends.

Scores are floats between 0.0 (safe) and 1.0 (risk detected); 0.5 is the
threshold. The available criteria are: `"harm"`, `"jailbreak"`, `"social_bias"`,
Expand Down
54 changes: 54 additions & 0 deletions docs/examples/intrinsics/uncertainty_ollama.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# pytest: e2e, ollama

"""Example usage of the uncertainty/certainty intrinsic with Ollama.

Evaluates how certain the model is about its response to a user question.
The context should contain a user question followed by an assistant answer.

Ollama bundles one adapter per model. For this single-adapter example, the
uncertainty aLoRA tag is used for normal chat and the certainty helper, so
Ollama can retain one model identity.

Requires `mellea[switch]` to download the adapter's `io.yaml`.

To run this script from the root of the Mellea source tree, use the command:
```
uv run python docs/examples/intrinsics/uncertainty_ollama.py
```
"""

import os
import subprocess
from pathlib import Path

from mellea import start_backend
from mellea.stdlib import functional as mfuncs
from mellea.stdlib.components.intrinsic import core

adapter_model = os.environ.get("MELLEA_OLLAMA_UNCERTAINTY_MODEL")
if adapter_model is None:
builder = (
Path(__file__).parents[3] / "test/scripts/build_ollama_uncertainty_adapter.sh"
)
adapter_model = subprocess.run(
[builder], check=True, stdout=subprocess.PIPE, text=True
).stdout.strip()

ctx, backend = start_backend(
"ollama",
# Before its invocation tokens, this bundled aLoRA behaves as the base model.
model_id=adapter_model,
# The bundled tag cannot identify the Hugging Face adapter directory itself.
adapter_base_model_name="granite-4.1-3b",
context_type="chat",
# The certainty helper uses the same model identity.
adapter_models={"uncertainty": adapter_model},
)

# Add the exchange whose answer the adapter will score.
response, ctx = mfuncs.chat("What is 2 + 2?", ctx, backend) # type: ignore
print(f"Response: {response.content}")

# This call uses the mapped bundled adapter model, not the normal chat model.
result = core.check_certainty(ctx, backend) # type: ignore
print(f"Certainty score: {result}")
22 changes: 12 additions & 10 deletions mellea/backends/adapters/adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -565,24 +565,26 @@ def add_adapter(self, adapter: AdapterInput, *, config: dict | None = None) -> N
still be rejected at runtime.

`config` is the raw io.yaml mapping for a composed `Adapter`/
`_AdapterCore` whose `weights` is an `EmbeddedBinding` — that reality's
config cannot be cheaply re-derived later, so it must be supplied here
(see `_discover_embedded_adapters`/`resolve_adapter`), rather than
being fetched lazily the way a `LocalFileBinding`'s io.yaml is.
`_AdapterCore` whose `weights` is an `EmbeddedBinding` or
`ServerMediatedBinding`. Those realities do not retain the raw
configuration required by the legacy rewriter, so it must be supplied
at registration rather than being fetched lazily like a
`LocalFileBinding`'s io.yaml.

Args:
adapter (AdapterInput): The adapter to register with this backend.
config (dict | None): Raw io.yaml config for a composed
`EmbeddedBinding` adapter. Ignored (and rejected) for every
other adapter reality.
`EmbeddedBinding` or `ServerMediatedBinding` adapter. Ignored
(and rejected) for every other adapter reality.

Raises:
TypeError: If `adapter` belongs to a reality this backend does not
support, or `config` is given for a reality other than a
composed `EmbeddedBinding` adapter.
ValueError: If `adapter.weights` is an `EmbeddedBinding` and
`config` is not given — registering it without a config would
make it discoverable but permanently unable to generate.
composed `EmbeddedBinding` or `ServerMediatedBinding` adapter.
ValueError: If `adapter.weights` is an `EmbeddedBinding` or
`ServerMediatedBinding` and `config` is not given —
registering it without a config would make it discoverable
but permanently unable to generate.
"""

@abc.abstractmethod
Expand Down
16 changes: 14 additions & 2 deletions mellea/backends/adapters/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,11 +162,15 @@ def effective_capability(self) -> str:
############################################
# Core adapter functions
############################################
# context-attribution, citations, and hallucination_detection publish only
# a `lora/` subdirectory on the Hub for granite-4.1-3b (no `alora/`), unlike
# every other entry below.
IntrinsicsCatalogEntry(
name="context-attribution",
capability="context_attribution",
repo_id=_CORE_R1_REPO,
revision=_CORE_R1_SHA,
adapter_types=(AdapterType.LORA,),
),
IntrinsicsCatalogEntry(
name="requirement-check",
Expand All @@ -181,9 +185,17 @@ def effective_capability(self) -> str:
# RAG adapter functions
############################################
IntrinsicsCatalogEntry(name="answerability", repo_id=_RAG_REPO, revision=_RAG_SHA),
IntrinsicsCatalogEntry(name="citations", repo_id=_RAG_REPO, revision=_RAG_SHA),
IntrinsicsCatalogEntry(
name="hallucination_detection", repo_id=_RAG_REPO, revision=_RAG_SHA
name="citations",
repo_id=_RAG_REPO,
revision=_RAG_SHA,
adapter_types=(AdapterType.LORA,),
),
IntrinsicsCatalogEntry(
name="hallucination_detection",
repo_id=_RAG_REPO,
revision=_RAG_SHA,
adapter_types=(AdapterType.LORA,),
),
IntrinsicsCatalogEntry(
name="query_clarification", repo_id=_RAG_REPO, revision=_RAG_SHA
Expand Down
Loading
Loading