From 54a5da7429f1820cb3142bd951ddf220d7cf4023 Mon Sep 17 00:00:00 2001 From: Nigel Jones Date: Thu, 10 Sep 2026 14:43:21 +0100 Subject: [PATCH 1/5] docs: document ModelOption.THINKING behaviour for Granite and add example Adds a per-backend reasoning/thinking-mode reference table (native Ollama, OpenAI/LiteLLM, HF) to configure-model-options.md, including Granite 4.2's "low"-only effort distinction and its default-on behaviour, and cross-links it from openai.md's existing Qwen3/vLLM extra_body section as the preferred portable entry point. Adds docs/examples/thinking_mode.py demonstrating THINKING=True/False/"low" against a local Ollama Granite model, verified live against granite4.2:3b. Fixes #1638 Assisted-by: Claude Code Signed-off-by: Nigel Jones --- docs/docs/how-to/configure-model-options.md | 50 ++++++++++++++++++++- docs/docs/integrations/openai.md | 7 +++ docs/examples/thinking_mode.py | 36 +++++++++++++++ 3 files changed, 92 insertions(+), 1 deletion(-) create mode 100644 docs/examples/thinking_mode.py diff --git a/docs/docs/how-to/configure-model-options.md b/docs/docs/how-to/configure-model-options.md index b460a7b42..cfb6affa8 100644 --- a/docs/docs/how-to/configure-model-options.md +++ b/docs/docs/how-to/configure-model-options.md @@ -120,7 +120,7 @@ non-deterministic session. | `ModelOption.STREAM` | `bool` | `False` | Enable streaming output. | | `ModelOption.STREAM_TIMEOUT` | `float \| None` | `120.0` | Timeout in seconds applied to every chunk, including time-to-first-token. Only applies to streaming responses; non-streaming calls are unaffected. If no chunk arrives within this window the stream aborts with a `TimeoutError`. Set to `None` to disable. Increase for slow local inference. | | `ModelOption.STOP_SEQUENCES` | `list[str]` | `None` | Strings that halt generation when produced by the model. | -| `ModelOption.THINKING` | varies | `None` | Enable or configure reasoning/thinking mode (model-dependent). | +| `ModelOption.THINKING` | `bool \| str` | `None` | Enable or configure reasoning/thinking mode. See [Reasoning and thinking mode](#reasoning-and-thinking-mode) below. | | `ModelOption.CONTEXT_WINDOW` | `int` | backend default | Context window size override. | | `ModelOption.TOOLS` | `list[MelleaTool]` | `None` | Tools exposed to the model for tool calling. | | `ModelOption.TOOL_CHOICE` | `str` | `"auto"` | Tool selection strategy (`"none"`, `"auto"`, or a specific tool name). | @@ -176,6 +176,54 @@ mot = await m.ainstruct( ) ``` +## Reasoning and thinking mode + +`ModelOption.THINKING` enables or configures a model's reasoning/thinking mode. +Accepted values and their effect are backend-dependent: + +| Backend | `True` | `False` | `"low"` / `"medium"` / `"high"` | +| ------- | ------ | ------- | -------------------------------- | +| Native `OllamaModelBackend` | Enables thinking (Ollama `think=True`) | Disables thinking (`think=False`) | Passed through to Ollama's `think=` param, which handles string effort levels itself | +| `OpenAIBackend` / LiteLLM (OpenAI-compatible) | Enables thinking (`reasoning_effort="medium"`, plus `chat_template_kwargs.enable_thinking=True` for vLLM-served templates) | Disables thinking | Sent as `reasoning_effort` verbatim | +| `LocalHFBackend` | Forwards to whichever chat-template variable is declared (`think`, `thinking`, or `enable_thinking`) | Same | **Not supported** — silently dropped, model falls back to full-length thinking. Tracked in [#1636](https://github.com/generative-computing/mellea/issues/1636). | + +For Granite 4.2 specifically: the chat template only distinguishes `"low"` +effort from everything else — `reasoning_effort == "low"` triggers genuine +low-effort (short) reasoning, while `"medium"`/`"high"` are accepted but +behave the same as `True`/omitted (full-length reasoning). Granite defaults +to thinking **on** when `ModelOption.THINKING` is not set at all. + +```python +from mellea.backends import ModelOption, model_ids +from mellea.backends.ollama import OllamaModelBackend +import mellea + +m = mellea.MelleaSession( + backend=OllamaModelBackend(model_id=model_ids.IBM_GRANITE_4_2_3B) +) + +# Full reasoning (the default for Granite 4.2) +answer = m.instruct("What is 17 * 24?", model_options={ModelOption.THINKING: True}) +print(answer.thinking) # reasoning trace +print(answer.value) # final answer + +# Short, low-effort reasoning — use when you need an answer within a small +# token budget rather than an exhaustive trace +answer = m.instruct("What is 17 * 24?", model_options={ModelOption.THINKING: "low"}) + +# No reasoning at all +answer = m.instruct("What is 17 * 24?", model_options={ModelOption.THINKING: False}) +``` + +If you're serving Granite 4.2 via vLLM, make sure your vLLM install picks up +the model's latest reasoning-parser update (shipped in the model's Hugging +Face files) — an older cached parser produces stale thinking behaviour. + +For non-Granite thinking models served through an OpenAI-compatible endpoint +(e.g. Qwen3 on vLLM), see +[Empty `value` from a thinking-mode model](../integrations/openai.md#empty-value-from-a-thinking-mode-model) +in the OpenAI integration guide. + ## System prompts Set a system prompt with `ModelOption.SYSTEM_PROMPT`. At session level it applies to all diff --git a/docs/docs/integrations/openai.md b/docs/docs/integrations/openai.md index a127a7e9c..24227b1f2 100644 --- a/docs/docs/integrations/openai.md +++ b/docs/docs/integrations/openai.md @@ -395,6 +395,13 @@ final answer. The OpenAI backend reports the response faithfully — the model genuinely returned `content=None` — but the reasoning content is preserved separately on the underlying `ModelOutputThunk`. +> **Preferred entry point:** for backends that honour it (this one included), +> `ModelOption.THINKING` is the portable way to enable/disable/level thinking — +> see [Reasoning and thinking mode](../how-to/configure-model-options.md#reasoning-and-thinking-mode). +> The `extra_body`/`enable_thinking` pattern below remains useful as a fallback +> for runtime-specific params `ModelOption.THINKING` doesn't cover, and for the +> post-#1617 `default_extra_body` merge semantics described here. + Diagnose with: ```python diff --git a/docs/examples/thinking_mode.py b/docs/examples/thinking_mode.py new file mode 100644 index 000000000..da439f05a --- /dev/null +++ b/docs/examples/thinking_mode.py @@ -0,0 +1,36 @@ +# pytest: ollama, e2e, qualitative + +"""Demonstrates ModelOption.THINKING against a local Ollama Granite model. + +Requires `ollama serve` running with `granite4.2:3b` pulled +(`ollama pull granite4.2:3b`). +""" + +import mellea +from mellea.backends import ModelOption, model_ids +from mellea.backends.ollama import OllamaModelBackend + +m = mellea.MelleaSession( + backend=OllamaModelBackend(model_id=model_ids.IBM_GRANITE_4_2_3B) +) + +question = "What is 17 * 24?" + +# Full reasoning — Granite 4.2 thinks by default even without THINKING set, +# but setting it explicitly makes the intent visible in the code. +full = m.instruct(question, model_options={ModelOption.THINKING: True}) +print("=== THINKING=True ===") +print("thinking:", full.thinking) +print("answer:", full.value) + +# Low-effort reasoning — a short trace, useful when the token budget is tight. +low = m.instruct(question, model_options={ModelOption.THINKING: "low"}) +print("=== THINKING='low' ===") +print("thinking:", low.thinking) +print("answer:", low.value) + +# No reasoning at all. +off = m.instruct(question, model_options={ModelOption.THINKING: False}) +print("=== THINKING=False ===") +print("thinking:", off.thinking) +print("answer:", off.value) From 4ed20031bbe87492e600cb804fa22a141686f200 Mon Sep 17 00:00:00 2001 From: Nigel Jones Date: Thu, 10 Sep 2026 15:21:15 +0100 Subject: [PATCH 2/5] docs: describe THINKING doc as post-#1639/#1616 target state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents ModelOption.THINKING string levels and result.thinking as supported on LocalHFBackend too, matching the state after #1639 (HF string forwarding fix) and #1616 (HF tag parsing) merge. Both are still open — this docs PR should merge after them, or the LocalHFBackend row will describe behaviour that isn't live yet. Assisted-by: Claude Code Signed-off-by: Nigel Jones --- docs/docs/how-to/configure-model-options.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/docs/how-to/configure-model-options.md b/docs/docs/how-to/configure-model-options.md index cfb6affa8..3b4a55092 100644 --- a/docs/docs/how-to/configure-model-options.md +++ b/docs/docs/how-to/configure-model-options.md @@ -185,13 +185,19 @@ Accepted values and their effect are backend-dependent: | ------- | ------ | ------- | -------------------------------- | | Native `OllamaModelBackend` | Enables thinking (Ollama `think=True`) | Disables thinking (`think=False`) | Passed through to Ollama's `think=` param, which handles string effort levels itself | | `OpenAIBackend` / LiteLLM (OpenAI-compatible) | Enables thinking (`reasoning_effort="medium"`, plus `chat_template_kwargs.enable_thinking=True` for vLLM-served templates) | Disables thinking | Sent as `reasoning_effort` verbatim | -| `LocalHFBackend` | Forwards to whichever chat-template variable is declared (`think`, `thinking`, or `enable_thinking`) | Same | **Not supported** — silently dropped, model falls back to full-length thinking. Tracked in [#1636](https://github.com/generative-computing/mellea/issues/1636). | +| `LocalHFBackend` | Forwards to whichever chat-template variable is declared (`think`, `thinking`, or `enable_thinking`) | Same | Forwarded verbatim as `reasoning_effort` when the tokenizer's chat template declares that variable — same mechanism as the OpenAI backend | For Granite 4.2 specifically: the chat template only distinguishes `"low"` effort from everything else — `reasoning_effort == "low"` triggers genuine low-effort (short) reasoning, while `"medium"`/`"high"` are accepted but -behave the same as `True`/omitted (full-length reasoning). Granite defaults -to thinking **on** when `ModelOption.THINKING` is not set at all. +behave the same as `True`/omitted (full-length reasoning). This holds across +all three backends. Granite defaults to thinking **on** when +`ModelOption.THINKING` is not set at all. + +`LocalHFBackend` also parses Granite's `...` block out of the +response, so `result.thinking` and `result.value` are populated separately — +matching the other backends — rather than leaving the reasoning trace +embedded raw in `result.value`. ```python from mellea.backends import ModelOption, model_ids From bf8bc09d4780b113090343f9726eb171feb64a43 Mon Sep 17 00:00:00 2001 From: Nigel Jones Date: Thu, 10 Sep 2026 15:59:45 +0100 Subject: [PATCH 3/5] docs: address code-review findings on THINKING docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Qualify the OpenAIBackend/LiteLLM `False` cell: real OpenAI reasoning models and non-Ollama LiteLLM targets never receive `reasoning_effort= "none"` (openai.py's server-type guard, litellm.py's ollama-prefix guard), so `False` does not actually disable thinking there. - Add an inline callout marking the LocalHFBackend string-forwarding and result.thinking claims as contingent on #1639/#1616 merging, instead of only noting it in the PR description/comment. - Correct "same mechanism as the OpenAI backend" — HF forwards through a gated chat-template variable, OpenAI sends an ungated top-level param. - Note the runtime-forwarding dependency for the cross-backend Granite claim, add a `> Full example:` link, fix the unresolvable bare "#1617" reference, add a non-determinism note to the new code block, drop the unnecessary `qualitative` marker, and fix US-English spelling in new content per CONTRIBUTING_DOCS.md. - Add cheap assertions to docs/examples/thinking_mode.py pinning the documented per-arm behaviour; reran live against granite4.2:3b, all pass. Assisted-by: Claude Code Signed-off-by: Nigel Jones --- docs/docs/how-to/configure-model-options.md | 18 ++++++++++++------ docs/docs/integrations/openai.md | 2 +- docs/examples/thinking_mode.py | 5 ++++- 3 files changed, 17 insertions(+), 8 deletions(-) diff --git a/docs/docs/how-to/configure-model-options.md b/docs/docs/how-to/configure-model-options.md index 3b4a55092..5eecdb1ee 100644 --- a/docs/docs/how-to/configure-model-options.md +++ b/docs/docs/how-to/configure-model-options.md @@ -184,15 +184,18 @@ Accepted values and their effect are backend-dependent: | Backend | `True` | `False` | `"low"` / `"medium"` / `"high"` | | ------- | ------ | ------- | -------------------------------- | | Native `OllamaModelBackend` | Enables thinking (Ollama `think=True`) | Disables thinking (`think=False`) | Passed through to Ollama's `think=` param, which handles string effort levels itself | -| `OpenAIBackend` / LiteLLM (OpenAI-compatible) | Enables thinking (`reasoning_effort="medium"`, plus `chat_template_kwargs.enable_thinking=True` for vLLM-served templates) | Disables thinking | Sent as `reasoning_effort` verbatim | -| `LocalHFBackend` | Forwards to whichever chat-template variable is declared (`think`, `thinking`, or `enable_thinking`) | Same | Forwarded verbatim as `reasoning_effort` when the tokenizer's chat template declares that variable — same mechanism as the OpenAI backend | +| `OpenAIBackend` / LiteLLM (OpenAI-compatible) | Enables thinking (`reasoning_effort="medium"`, plus `chat_template_kwargs.enable_thinking=True` for vLLM-served templates) | Disables thinking on vLLM-served/OpenAI-compatible servers that honour `chat_template_kwargs`. **Real OpenAI reasoning models, and LiteLLM targets that aren't Ollama, deliberately never receive `reasoning_effort="none"`** (real OpenAI rejects that value) — there is no supported way to fully disable reasoning on them via `ModelOption.THINKING` | Sent as `reasoning_effort` verbatim — a top-level request parameter, independent of any chat template | +| `LocalHFBackend` | Forwards to whichever chat-template variable is declared (`think`, `thinking`, or `enable_thinking`) | Same, `False` value | Forwarded verbatim as the chat template's own `reasoning_effort` variable when the template declares one. This is a different transport than the OpenAI backend's top-level parameter — it only takes effect if the served model's template exposes that variable | For Granite 4.2 specifically: the chat template only distinguishes `"low"` effort from everything else — `reasoning_effort == "low"` triggers genuine low-effort (short) reasoning, while `"medium"`/`"high"` are accepted but behave the same as `True`/omitted (full-length reasoning). This holds across -all three backends. Granite defaults to thinking **on** when -`ModelOption.THINKING` is not set at all. +all three backends, provided the serving runtime forwards the effort level +into the chat template (Ollama and vLLM do). Granite defaults to thinking +**on** when `ModelOption.THINKING` is not set at all. + +> **Depends on [#1639](https://github.com/generative-computing/mellea/issues/1639) and [#1616](https://github.com/generative-computing/mellea/pull/1616):** the `LocalHFBackend` string-forwarding behaviour above, and the paragraph below, describe the state once those two open PRs merge. On current `main`, `LocalHFBackend` only forwards boolean `THINKING` values and does not populate `result.thinking`. `LocalHFBackend` also parses Granite's `...` block out of the response, so `result.thinking` and `result.value` are populated separately — @@ -200,9 +203,9 @@ matching the other backends — rather than leaving the reasoning trace embedded raw in `result.value`. ```python +import mellea from mellea.backends import ModelOption, model_ids from mellea.backends.ollama import OllamaModelBackend -import mellea m = mellea.MelleaSession( backend=OllamaModelBackend(model_id=model_ids.IBM_GRANITE_4_2_3B) @@ -212,6 +215,7 @@ m = mellea.MelleaSession( answer = m.instruct("What is 17 * 24?", model_options={ModelOption.THINKING: True}) print(answer.thinking) # reasoning trace print(answer.value) # final answer +# Output will vary — reasoning traces are non-deterministic. # Short, low-effort reasoning — use when you need an answer within a small # token budget rather than an exhaustive trace @@ -221,9 +225,11 @@ answer = m.instruct("What is 17 * 24?", model_options={ModelOption.THINKING: "lo answer = m.instruct("What is 17 * 24?", model_options={ModelOption.THINKING: False}) ``` +> **Full example:** [`docs/examples/thinking_mode.py`](https://github.com/generative-computing/mellea/blob/main/docs/examples/thinking_mode.py) + If you're serving Granite 4.2 via vLLM, make sure your vLLM install picks up the model's latest reasoning-parser update (shipped in the model's Hugging -Face files) — an older cached parser produces stale thinking behaviour. +Face files) — an older cached parser produces stale thinking behavior. For non-Granite thinking models served through an OpenAI-compatible endpoint (e.g. Qwen3 on vLLM), see diff --git a/docs/docs/integrations/openai.md b/docs/docs/integrations/openai.md index 24227b1f2..cc5942140 100644 --- a/docs/docs/integrations/openai.md +++ b/docs/docs/integrations/openai.md @@ -400,7 +400,7 @@ separately on the underlying `ModelOutputThunk`. > see [Reasoning and thinking mode](../how-to/configure-model-options.md#reasoning-and-thinking-mode). > The `extra_body`/`enable_thinking` pattern below remains useful as a fallback > for runtime-specific params `ModelOption.THINKING` doesn't cover, and for the -> post-#1617 `default_extra_body` merge semantics described here. +> `default_extra_body` merge semantics described below. Diagnose with: diff --git a/docs/examples/thinking_mode.py b/docs/examples/thinking_mode.py index da439f05a..540fc68a2 100644 --- a/docs/examples/thinking_mode.py +++ b/docs/examples/thinking_mode.py @@ -1,4 +1,4 @@ -# pytest: ollama, e2e, qualitative +# pytest: ollama, e2e """Demonstrates ModelOption.THINKING against a local Ollama Granite model. @@ -22,15 +22,18 @@ print("=== THINKING=True ===") print("thinking:", full.thinking) print("answer:", full.value) +assert full.thinking # Low-effort reasoning — a short trace, useful when the token budget is tight. low = m.instruct(question, model_options={ModelOption.THINKING: "low"}) print("=== THINKING='low' ===") print("thinking:", low.thinking) print("answer:", low.value) +assert low.thinking # No reasoning at all. off = m.instruct(question, model_options={ModelOption.THINKING: False}) print("=== THINKING=False ===") print("thinking:", off.thinking) print("answer:", off.value) +assert not off.thinking From b53d7a3938007bea7482ffcadc382f82194c7e37 Mon Sep 17 00:00:00 2001 From: Nigel Jones Date: Tue, 15 Sep 2026 16:36:17 +0000 Subject: [PATCH 4/5] docs: fix stale THINKING dependency note now that #1639 merged #1639 (HF string-forwarding) merged 2026-09-10; only #1616 (result.thinking parsing) remains open. Also note that #1616's -tag split is skipped for streaming (m serve) calls, per its own PR description. Assisted-by: Claude Code Signed-off-by: Nigel Jones --- docs/docs/how-to/configure-model-options.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/docs/how-to/configure-model-options.md b/docs/docs/how-to/configure-model-options.md index 5eecdb1ee..d595a0020 100644 --- a/docs/docs/how-to/configure-model-options.md +++ b/docs/docs/how-to/configure-model-options.md @@ -195,12 +195,21 @@ all three backends, provided the serving runtime forwards the effort level into the chat template (Ollama and vLLM do). Granite defaults to thinking **on** when `ModelOption.THINKING` is not set at all. -> **Depends on [#1639](https://github.com/generative-computing/mellea/issues/1639) and [#1616](https://github.com/generative-computing/mellea/pull/1616):** the `LocalHFBackend` string-forwarding behaviour above, and the paragraph below, describe the state once those two open PRs merge. On current `main`, `LocalHFBackend` only forwards boolean `THINKING` values and does not populate `result.thinking`. +The `LocalHFBackend` string-forwarding behaviour above is live on `main` +(landed via [#1639](https://github.com/generative-computing/mellea/pull/1639)). + +> **Depends on [#1616](https://github.com/generative-computing/mellea/pull/1616):** +> the paragraph below describes the state once that open PR merges. On current +> `main`, `LocalHFBackend` does not populate `result.thinking` — the reasoning +> trace stays embedded raw in `result.value`. `LocalHFBackend` also parses Granite's `...` block out of the response, so `result.thinking` and `result.value` are populated separately — matching the other backends — rather than leaving the reasoning trace -embedded raw in `result.value`. +embedded raw in `result.value`. This split is skipped for streaming (`m serve`) +calls, where the reasoning trace still arrives inline in `result.value`; +incremental splitting for streaming is tracked separately in +[#1604](https://github.com/generative-computing/mellea/issues/1604). ```python import mellea From f1b37fe1f5eb7dd191a4355708c4d992a1b20476 Mon Sep 17 00:00:00 2001 From: Nigel Jones Date: Wed, 16 Sep 2026 13:04:44 +0000 Subject: [PATCH 5/5] docs: drop merged-PR references from THINKING doc #1616 and #1639 have both merged, so LocalHFBackend's result.thinking split is now current-main behaviour, not a future dependency. Assisted-by: Claude Code Co-Authored-By: Claude Sonnet 5 Signed-off-by: Nigel Jones --- docs/docs/how-to/configure-model-options.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/docs/how-to/configure-model-options.md b/docs/docs/how-to/configure-model-options.md index d595a0020..6c468b257 100644 --- a/docs/docs/how-to/configure-model-options.md +++ b/docs/docs/how-to/configure-model-options.md @@ -195,14 +195,6 @@ all three backends, provided the serving runtime forwards the effort level into the chat template (Ollama and vLLM do). Granite defaults to thinking **on** when `ModelOption.THINKING` is not set at all. -The `LocalHFBackend` string-forwarding behaviour above is live on `main` -(landed via [#1639](https://github.com/generative-computing/mellea/pull/1639)). - -> **Depends on [#1616](https://github.com/generative-computing/mellea/pull/1616):** -> the paragraph below describes the state once that open PR merges. On current -> `main`, `LocalHFBackend` does not populate `result.thinking` — the reasoning -> trace stays embedded raw in `result.value`. - `LocalHFBackend` also parses Granite's `...` block out of the response, so `result.thinking` and `result.value` are populated separately — matching the other backends — rather than leaving the reasoning trace