Skip to content

sampling: a reasoning budget of 0 bounds the think, on the integration line (#95 merged to the stale master) - #96

Merged
joelteply merged 4 commits into
889580fc/seq-state-across-stream-countsfrom
c5fb32b7/budget-zero-integration
Nov 7, 2026
Merged

joelteply merged 4 commits into
889580fc/seq-state-across-stream-countsfrom
c5fb32b7/budget-zero-integration

Conversation

@joelteply

Copy link
Copy Markdown

The reasoning-budget-0 fix (#95, reviewed by Fable) on the integration line continuum pins. #95 merged into fork master, which is 99 commits behind continuum's pin 0d7dda6; every fork PR since #84 targets this branch. Same three commits, cherry-picked clean onto a8c33da:

  • sampling: in FORCING, force_pos advances only on the forced token, and a natural end tag ends the forcing. A prefilled <think>\n can no longer consume the forced end, which had let the model think to its cap at budget 0.
  • gguf.cpp: #include <cerrno>.
  • Missing libc includes: the includes upstream ggml-org has and this line lacks (the WASM WebGPU build stopped providing them transitively).

The fork's default branch is now this line, so new PRs target what continuum ships.

Measured

Integration build 45ae996 (this head), IntelMac CPU, Qwen3.5-0.8B Q4_K_M, --jinja --reasoning-format deepseek -t 2, 2026-11-07 ~01:20Z. The sampler and its test are byte-identical to master's, but server-context.cpp differs by ~880 lines, so the probe was re-run on this build.

  • test-reasoning-budget: OK (13 tests). Mutation (on master's build): reverting the fix aborts the suite at activate immediately budget=0.
request deployed 0d7dda6 this head
schema, budget 0 thinks to the cap, content empty reasoning empty, 78-token schema JSON, stop
no schema, budget 0 thinks to the cap reasoning empty, answers from token 1
schema, budget 1 / 40 / none bounded / forced end + JSON / thinks to the cap unchanged
budget 0 + enable_thinking=false (continuum ggml-org#5808's pair) reasoning empty (the template closes it) reasoning empty, 95 tokens, stop

🤖 Generated with Claude Code

https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

joelteply and others added 3 commits November 6, 2026 18:46
… cannot consume the forced end

With budget 0, the generation prompt's prefilled `<think>` moves the budget sampler from
IDLE to FORCING, then the prefilled newline was accepted IN FORCING and advanced force_pos,
so the sampler reached DONE without ever sampling the end tag and the model thought to its
cap. Measured on 0d7dda6 (Qwen3.5-0.8B Q4_K_M, 2026-11-06): reasoning_budget_tokens 0
thought 400 of 400 tokens with empty content, with or without a json_schema; 1 was bounded.

FORCING now advances only on the forced token, and a natural end tag accepted while forcing
(a prefilled `<think></think>`) ends it, so a closed think is never forced a second time.

Tests: the realistic budget-0 case (start, then the forced end) and the budget-exhausted case
now feed the forced tokens as sampled; a new test covers budget 0 with the prefilled start and
newline (end tag alone; message then end tag), a prefill that already closed the think, and
budgets 1 and 3. Reverting the fix aborts the suite at 'activate immediately budget=0'.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
gguf.cpp:996 logs strerror(errno) but never included <cerrno>; it compiled only while another
header pulled it in transitively. The WASM WebGPU build (emscripten) no longer does: ubuntu-webgpu
failed on #95 with "use of undeclared identifier 'errno'" (it passed on #94, 2026-11-04).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
…ger provides them transitively

ubuntu-webgpu (emscripten) failed on #95 at gguf.cpp (errno), then at llama-graph.h:103
(getenv/atoi undeclared): the same build passed on #94 (2026-11-04), so the toolchain stopped
pulling these headers in through others. Upstream ggml-org/llama.cpp master already includes
them; this adds exactly the ones upstream has and the fork lacked (diffed per file over src/,
common/, ggml/src, ggml/include): llama-graph.h, llama-mmap.cpp, llama-vocab.cpp <cstdlib>;
llama-adapter.cpp <cerrno> <cstring>; common/jinja/value.cpp <cstdio> <cstdlib> <cstring>;
common/common.h and common/build-info.h <cstdio>.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply

Copy link
Copy Markdown
Author

PASS at 45ae996.

Verified: I diffed #95's squashed patch (25c5ec17b~1..25c5ec17b) against this PR's total patch (889580fc/seq-state-across-stream-counts..45ae99681). Every added and removed line is identical. The only differences are blob index hashes and one unchanged context line (#include <array> vs <memory> near an insertion), because the integration line's files differ around the edits.

Logic: the same FORCING-accept fix I passed on #95. It is re-measured on a server built from this head, because server-context.cpp differs by ~880 lines from master: unit 13/13; budget 0 gives empty reasoning and 78-token schema JSON with finish=stop; budgets 1, 40 and none are unchanged; and ggml-org#5808's two-lever pair also gives no think. Merge on green CI. Then bump the pin (f0a909bd), and set the fork's default branch to the integration line.

…ed runner exists

Its runner is [self-hosted, Linux], which exists only upstream. In this fork it sat queued for
24 h on every PR and was cancelled, and a merge gate reads a cancelled check as a failure (#94,
#95, #96). The fork's four self-hosted workflows (CI, Server, UI, UI Build) are disabled in
the repo settings for the same reason; this job is called from the CPU workflow, so it gets
a job condition instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@github-actions github-actions Bot added the devops label Nov 7, 2026
@joelteply

Copy link
Copy Markdown
Author

PASS carries to 9bd4d63. The delta from 45ae996 is .github/workflows/build-cpu.yml +3 lines: a comment, plus if: github.repository == 'ggml-org/llama.cpp' on build-cmake-pkg, whose runner is self-hosted and exists only upstream. It fixes the root cause: every fork PR has waited 24 h on runner-less jobs that were then cancelled. Disabling the four self-hosted-only workflows in repo settings (reversible) covers the rest. No logic change.

@joelteply
joelteply merged commit 444f386 into 889580fc/seq-state-across-stream-counts Nov 7, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant