diff --git a/docs/docs/how-to/configure-model-options.md b/docs/docs/how-to/configure-model-options.md index b460a7b42..6c468b257 100644 --- a/docs/docs/how-to/configure-model-options.md +++ b/docs/docs/how-to/configure-model-options.md @@ -120,7 +120,7 @@ non-deterministic session. | `ModelOption.STREAM` | `bool` | `False` | Enable streaming output. | | `ModelOption.STREAM_TIMEOUT` | `float \| None` | `120.0` | Timeout in seconds applied to every chunk, including time-to-first-token. Only applies to streaming responses; non-streaming calls are unaffected. If no chunk arrives within this window the stream aborts with a `TimeoutError`. Set to `None` to disable. Increase for slow local inference. | | `ModelOption.STOP_SEQUENCES` | `list[str]` | `None` | Strings that halt generation when produced by the model. | -| `ModelOption.THINKING` | varies | `None` | Enable or configure reasoning/thinking mode (model-dependent). | +| `ModelOption.THINKING` | `bool \| str` | `None` | Enable or configure reasoning/thinking mode. See [Reasoning and thinking mode](#reasoning-and-thinking-mode) below. | | `ModelOption.CONTEXT_WINDOW` | `int` | backend default | Context window size override. | | `ModelOption.TOOLS` | `list[MelleaTool]` | `None` | Tools exposed to the model for tool calling. | | `ModelOption.TOOL_CHOICE` | `str` | `"auto"` | Tool selection strategy (`"none"`, `"auto"`, or a specific tool name). | @@ -176,6 +176,67 @@ mot = await m.ainstruct( ) ``` +## Reasoning and thinking mode + +`ModelOption.THINKING` enables or configures a model's reasoning/thinking mode. +Accepted values and their effect are backend-dependent: + +| Backend | `True` | `False` | `"low"` / `"medium"` / `"high"` | +| ------- | ------ | ------- | -------------------------------- | +| Native `OllamaModelBackend` | Enables thinking (Ollama `think=True`) | Disables thinking (`think=False`) | Passed through to Ollama's `think=` param, which handles string effort levels itself | +| `OpenAIBackend` / LiteLLM (OpenAI-compatible) | Enables thinking (`reasoning_effort="medium"`, plus `chat_template_kwargs.enable_thinking=True` for vLLM-served templates) | Disables thinking on vLLM-served/OpenAI-compatible servers that honour `chat_template_kwargs`. **Real OpenAI reasoning models, and LiteLLM targets that aren't Ollama, deliberately never receive `reasoning_effort="none"`** (real OpenAI rejects that value) — there is no supported way to fully disable reasoning on them via `ModelOption.THINKING` | Sent as `reasoning_effort` verbatim — a top-level request parameter, independent of any chat template | +| `LocalHFBackend` | Forwards to whichever chat-template variable is declared (`think`, `thinking`, or `enable_thinking`) | Same, `False` value | Forwarded verbatim as the chat template's own `reasoning_effort` variable when the template declares one. This is a different transport than the OpenAI backend's top-level parameter — it only takes effect if the served model's template exposes that variable | + +For Granite 4.2 specifically: the chat template only distinguishes `"low"` +effort from everything else — `reasoning_effort == "low"` triggers genuine +low-effort (short) reasoning, while `"medium"`/`"high"` are accepted but +behave the same as `True`/omitted (full-length reasoning). This holds across +all three backends, provided the serving runtime forwards the effort level +into the chat template (Ollama and vLLM do). Granite defaults to thinking +**on** when `ModelOption.THINKING` is not set at all. + +`LocalHFBackend` also parses Granite's `...` block out of the +response, so `result.thinking` and `result.value` are populated separately — +matching the other backends — rather than leaving the reasoning trace +embedded raw in `result.value`. This split is skipped for streaming (`m serve`) +calls, where the reasoning trace still arrives inline in `result.value`; +incremental splitting for streaming is tracked separately in +[#1604](https://github.com/generative-computing/mellea/issues/1604). + +```python +import mellea +from mellea.backends import ModelOption, model_ids +from mellea.backends.ollama import OllamaModelBackend + +m = mellea.MelleaSession( + backend=OllamaModelBackend(model_id=model_ids.IBM_GRANITE_4_2_3B) +) + +# Full reasoning (the default for Granite 4.2) +answer = m.instruct("What is 17 * 24?", model_options={ModelOption.THINKING: True}) +print(answer.thinking) # reasoning trace +print(answer.value) # final answer +# Output will vary — reasoning traces are non-deterministic. + +# Short, low-effort reasoning — use when you need an answer within a small +# token budget rather than an exhaustive trace +answer = m.instruct("What is 17 * 24?", model_options={ModelOption.THINKING: "low"}) + +# No reasoning at all +answer = m.instruct("What is 17 * 24?", model_options={ModelOption.THINKING: False}) +``` + +> **Full example:** [`docs/examples/thinking_mode.py`](https://github.com/generative-computing/mellea/blob/main/docs/examples/thinking_mode.py) + +If you're serving Granite 4.2 via vLLM, make sure your vLLM install picks up +the model's latest reasoning-parser update (shipped in the model's Hugging +Face files) — an older cached parser produces stale thinking behavior. + +For non-Granite thinking models served through an OpenAI-compatible endpoint +(e.g. Qwen3 on vLLM), see +[Empty `value` from a thinking-mode model](../integrations/openai.md#empty-value-from-a-thinking-mode-model) +in the OpenAI integration guide. + ## System prompts Set a system prompt with `ModelOption.SYSTEM_PROMPT`. At session level it applies to all diff --git a/docs/docs/integrations/openai.md b/docs/docs/integrations/openai.md index a127a7e9c..cc5942140 100644 --- a/docs/docs/integrations/openai.md +++ b/docs/docs/integrations/openai.md @@ -395,6 +395,13 @@ final answer. The OpenAI backend reports the response faithfully — the model genuinely returned `content=None` — but the reasoning content is preserved separately on the underlying `ModelOutputThunk`. +> **Preferred entry point:** for backends that honour it (this one included), +> `ModelOption.THINKING` is the portable way to enable/disable/level thinking — +> see [Reasoning and thinking mode](../how-to/configure-model-options.md#reasoning-and-thinking-mode). +> The `extra_body`/`enable_thinking` pattern below remains useful as a fallback +> for runtime-specific params `ModelOption.THINKING` doesn't cover, and for the +> `default_extra_body` merge semantics described below. + Diagnose with: ```python diff --git a/docs/examples/thinking_mode.py b/docs/examples/thinking_mode.py new file mode 100644 index 000000000..540fc68a2 --- /dev/null +++ b/docs/examples/thinking_mode.py @@ -0,0 +1,39 @@ +# pytest: ollama, e2e + +"""Demonstrates ModelOption.THINKING against a local Ollama Granite model. + +Requires `ollama serve` running with `granite4.2:3b` pulled +(`ollama pull granite4.2:3b`). +""" + +import mellea +from mellea.backends import ModelOption, model_ids +from mellea.backends.ollama import OllamaModelBackend + +m = mellea.MelleaSession( + backend=OllamaModelBackend(model_id=model_ids.IBM_GRANITE_4_2_3B) +) + +question = "What is 17 * 24?" + +# Full reasoning — Granite 4.2 thinks by default even without THINKING set, +# but setting it explicitly makes the intent visible in the code. +full = m.instruct(question, model_options={ModelOption.THINKING: True}) +print("=== THINKING=True ===") +print("thinking:", full.thinking) +print("answer:", full.value) +assert full.thinking + +# Low-effort reasoning — a short trace, useful when the token budget is tight. +low = m.instruct(question, model_options={ModelOption.THINKING: "low"}) +print("=== THINKING='low' ===") +print("thinking:", low.thinking) +print("answer:", low.value) +assert low.thinking + +# No reasoning at all. +off = m.instruct(question, model_options={ModelOption.THINKING: False}) +print("=== THINKING=False ===") +print("thinking:", off.thinking) +print("answer:", off.value) +assert not off.thinking