fix(hf): parse <think> tags into mot.thinking on LocalHFBackend

[planetf1]

Fixes #1610.

What changed

LocalHFBackend never populated ModelOutputThunk.thinking. Other backends split reasoning from the answer at the SDK level; HF decoded one flat string, leaving Granite's <think>...</think> block unparsed inside mot.value.

Adds _split_think_tags() in post_processing():

• Splits on the closing </think> tag (the opening tag lives in the prompt, never in the model's output).
• Only splits when the template declares a thinking variable and ModelOption.THINKING for this call isn't explicitly False.
• Skipped for streaming (m serve) — incremental splitting is follow-up work (feat: implement better hugging face output parsing #1604).
• A documented fallback for transformers' parse_response(); no tokenizer today declares a response_schema.

Behaviour change: reasoning replay on multi-turn conversations

Splitting reasoning out of mot.value breaks history replay: Granite's template used to see reasoning inline on every turn by accident (no opening tag, so its truncation gate never fired). Once mot.value is clean, the template drops it.

to_chat() now forwards captured reasoning as reasoning_content — the key Granite/Qwen3 templates already consume. Measured against the real template:

Turn shape	Reasoning replayed?
Assistant turn with a tool call	✅
Plain assistant turn, followed by another user turn	❌ (template's truncate_history_thinking strips it)

Tool-call turns match the cross-backend convention (#1201: replay on tool-call turns only). Plain turns are left as-is by design — matching that convention, not diverging from it just because Granite's template allows it. Pinned by test_rendered_prompt_preserves_reasoning_on_tool_call_turn / test_rendered_prompt_drops_reasoning_on_plain_multi_turn.

Other fixes

• Cache key (KV-cache LRU) computed after the split, not before — the old key was an object identity that changed a few lines later.
• GenerateLog.extra includes captured reasoning.
• reasoning_content forward is assistant-only; a provider_fields collision on that key resolves in Mellea's favour (tested).
• Intrinsic/adapter functions on HF now get reasoning-free input on thinking turns — a correctness fix, but a real input change for anyone tracking intrinsic eval numbers.

Design decisions

• Plain-turn replay — not extended, stays with Better handling of reasoning output from thinking models #1201.
• response_schema — no tokenizer offers one; nothing to do until one does.
• First vs. last </think> — kept first, deliberately opposite to Granite's own template. The template is discarding old reasoning, where dropping too much is safe; we're extracting a clean answer, where an answer mentioning </think> must stay intact.

Test plan

• test_huggingface_thinking.py — splitter units, post_processing() gating/streaming/cache-key tests, rendered-prompt tests against the real template.
• test_huggingface_thinking_e2e.py — real granite-4.2-3b generation, GPU-gated, skipped in CI. Run locally on Apple Silicon MPS: both pass, mot.thinking holds a genuine trace, mot.value a clean answer.
• test_utils.py — to_chat() reasoning tests (attach, omit, role guard, collision).
• pytest, ruff, mypy clean.

[jakelorocco]

I think I did not realize how many aspects of the hf backend this would impact when I created the issue. I think there's actually a fair bit of design work that might be required to address these concerns.

[jakelorocco]

I think we may need to actually look at the tokens for this to work properly? The model may produce </think> in text form, which we wouldn't want to parse as the end of a thinking section.

[planetf1]

Looked into token-level checks — won't work here: Granite's </think> token is itself non-special, so it's indistinguishable from literal text even at the token level. transformers' own parser has the same limit. Documented the splitter as an explicit fallback and gated it on the resolved thinking value, which cuts the practical risk.

[jakelorocco]

I did test this and I do see that <think> and </think> are specific tokens. When I asked the model what it's thinking token was, it output <, think, > or something like that. So there might be more to this.

[planetf1]

The string check is the safe superset here: it fires on every case a token check would, plus any model whose delimiter isn't a single vocab token — a token check would regress those, missing real splits entirely. Reason enough to keep it as the default mechanism on its own.

Separately, verified against ibm-granite/granite-4.2-3b's tokenizer (one model, four test strings): encoding </think> — standalone, mid-sentence, glued to other text — always yields the same single vocab token, id 100275, non-special. Generation doesn't re-encode a target string though — it samples token-by-token — so the model could in principle emit <, /think, > as separate steps decoding to the same string, which a check against the raw generated ids (hf_output.sequences[0], which includes prompt tokens and would need slicing) could tell apart from the atomic-token case, unlike a string match on decoded text.

My expectation is this doesn't help the case we actually care about — a model naturally writing the tag inline is more likely to hit the same atomic token as a genuine close, since training data containing that literal substring would itself have been encoded into the atomic token. Haven't run the sampling experiment to confirm that — it's a hypothesis, not something verified like the tokenizer facts above.

The superset argument doesn't depend on that hypothesis either way — suggest keeping the string-fallback as primary. Happy to add a token check as a supplementary non-streaming-only check if you want it, but it's real complexity (guarding models without a single-token delimiter, slicing prompt tokens, a second code path, more tests) for a benefit that's currently a guess, not a fact.

[jakelorocco]

Wouldn't this still trigger if one of the thinking vars is set to false?

[planetf1]

Fixed — gate now checks the actual ModelOption.THINKING value for the call, not just whether the template mentions the variable. Tests cover both explicit False and unset.

[jakelorocco]

I think for this PR; it might be good enough to just always attempt to split on </think>? I just worry that if someone asks the model to output that token in an answer, it will be problematic.

[jakelorocco]

I may have been wrong about my earlier comment; since some models default to thinking on, we can't go off of model options alone here (and their truthy value).

Maybe we should just always split on unless it's structured output? And then we can eventually do more sophisticated stuff like hf does with looking at the chat template, etc...

[planetf1]

reverting to draft. will revisit next week

[planetf1]

Message.thinking is forwarded as reasoning_content for every assistant turn that has one (mellea/backends/utils.py) — HF has no should_replay_reasoning() gate the way litellm/watsonx/openai/ollama do. What decides whether it survives is Granite's own template: a turn keeps reasoning if its index is at or after the most recent user message, and loses it otherwise — turn-recency, not turn-type. (truncate_history_thinking defaults True and mellea never overrides it, so this always applies today.) A tool-call turn usually has no user message before its continuation, so it tends to survive; a plain turn from an earlier exchange tends not to — a side effect of recency, not a rule Mellea enforces.

So a false split — real answer text landing in mot.thinking — gets attached as reasoning_content and is silently dropped from every subsequent prompt once a newer user turn exists. Not destroyed everywhere (it survives in mot.thinking/the ChatContext for that turn, and Guardian's safety check reads mot.thinking directly — guardian.py:407), but once conversation moves on, it's gone from what the model sees, with no trace anything was dropped.

A missed split — raw tags staying in mot.value — replays through ordinary content instead. Nothing re-parses it; it just gets read and judged as part of the answer. So "recoverable" overstates it too — the real difference is visibility: a missed split stays visible in the output, a false split disappears silently once history moves past it.

Suggest keeping the current gate on that basis — an invisible failure is worse than a visible one, even if neither is strictly safe. This reasoning is Granite-template-specific though, and the gate is generic across LocalHFBackend, so a different model's template semantics could shift the calculus. Documented both failure modes at the gate (huggingface.py:1871-1894), with a pointer to #1604 for always-thinking models without a declared var.