[joelteply]

The crash

Windows node, 2026-11-07 07:08Z, on exactly 444f386: bad allocation on slot 0, then on slot 1, right after a checkpoint restore; each slot released; then tools/server/server-context.cpp:3442: GGML_ASSERT(batch.slot_batched || batch.size() == 0) failed, and the process aborted.

Mechanism (line numbers at 444f386)

• iterate() (:3331) catches any std::exception from a slot callback, sends the error and calls slot.release().
• In the prompt fill (iterate(prompt_slots, ...), :3726) a slot calls batch.add() for its tokens (:4197) and slot.prompt.tokens.push_back() (:4202). After that, create_checkpoint() (:4299) puts an empty checkpoint in the slot's list (:2516) and then allocates the state copy (update_tgt, :2525, a std::vector::resize). That allocation is where std::bad_alloc was thrown. slot_batched = &slot comes only after it (:4305). The generate path sets slot_batched before it adds anything (:3593).
• So a fill that throws leaves three things behind:
  1. Tokens in the batch with no owner. If every filling slot fails, the assert at :3442 fires. That is the abort above.
  2. Prompt tokens whose memory was never written. release() keeps the slot's prompt for reuse, so the next request with that prefix trusts cells that do not exist. Cormac traced this. The test below exercises it: with the batch rollback alone, the retry hits pos_min == -1, but n_past > 0. On a slot whose memory already held an earlier prefix, it would reuse the wrong cache silently.
  3. A half-built checkpoint (pos/n_tokens set, no data) in the slot's checkpoint list.
• The speculative checkpoint in the generate path allocates the same kind of copy, with no guard: spec_ckpt.update_dft when a draft starts (:3625) and update_tgt/update_dft before a draft is verified (:3683, :3695). A bad_alloc there releases a generating slot mid-reply. The slot is still in generating, so handle_last_sampled_token() then aborts on GGML_ASSERT(task) in need_embd() (:418; :446 in the measured build). This is measured below.

Fix

• Batch rollback. server_batch::get_mark() / rollback() (one owner, beside clear()) records the batch's tokens, embd, slot_batched and alora fields before a slot adds anything, and restores them.
• Prompt fill. The per-slot prompt fill is now one lambda, fill_prompt, with its body unchanged. Its caller in iterate(prompt_slots, ...) catches any throw, rolls the batch back to the mark, cuts slot.prompt.tokens back to n_written, then rethrows.
  • n_written is the prompt's length when the text-token loop starts. That is after keep_first/seq_rm and after any mtmd chunks (which decode), so it counts only tokens whose memory is written.
  • iterate() still reports the error and releases the slot.
  • prefill_order.served() runs only on success, so a failed fill uses up no service turn.
• One owner for every checkpoint copy. copy_checkpoint(slot, ckpt, kind, parts) is now the only place a checkpoint's state copy is allocated, for prompt and speculative checkpoints alike. A copy that cannot be allocated (std::bad_alloc, a typed catch, no string matching) is refused: the checkpoint is cleared, the refusal is logged, false is returned, and the caller goes on without it.
  • Prompt (create_checkpoint): the copy is built in a local and added to the list only when complete. A refusal means a restore from there is cold, which is how a missing checkpoint sidecar is already handled.
  • Speculative, when a draft starts: no draft this step.
  • Speculative, before verification: the draft is dropped, and the step decodes the sampled token alone, exactly as when the drafter proposes nothing. The reply is unchanged. Dropped drafts are not counted in n_draft_tokens.
• The assert at :3442 is untouched.

Test

tools/server/server-test-faults.h holds the test-only faults. They compile in only with cmake -DLLAMA_SERVER_TEST_FAULTS=ON, a CMake option that is off by default and is set only by the server CI builds (server.yml, server-sanitize.yml, server-self-hosted.yml) and the tests README. In any other build, server_test_faults::enabled is false, every use sits under if constexpr (server_test_faults::enabled) and is discarded, and from_env() reads nothing, so a stray environment variable can never fail a real slot. test_failed_fill.py skips, saying why, when the server it runs has no fault code. In the test build the faults are read from the environment once at load and are zero unless set:

• LLAMA_SERVER_TEST_FAIL_CHECKPOINTS=N: the next N prompt checkpoint copies throw std::bad_alloc, at the point where the copy allocates (inside copy_checkpoint).
• LLAMA_SERVER_TEST_FAIL_SPEC_CHECKPOINTS=N: the same for the next N speculative checkpoint copies.
• LLAMA_SERVER_TEST_FAIL_PROMPT_FILLS=N: the next N prompt fills throw after they have added their tokens.

tools/server/tests/unit/test_failed_fill.py:

• test_a_refused_checkpoint_copy_is_not_a_slot_failure (tinygemma3, sliding window, so its prompts checkpoint; 2 faults; two concurrent completions pinned to slots 0 and 1, --cache-ram 0): both complete with equal content, and /health returns 200.
• test_a_failed_fill_leaves_the_batch_and_its_prompt_clean (tinyllama2, 1 fault, same two-slot setup): exactly one request errors and the other completes. Retrying on the faulted slot returns cache_n == 0 and the same content as the good slot.
• test_a_refused_speculative_checkpoint_drops_the_draft_not_the_reply:
  • Model: mamba-130m, which is recurrent, so its memory cannot drop a rejected draft's tail and every verified draft is checkpointed (speculative decoding will use checkpoints). Speculation is --spec-type ngram-simple.
  • Prompt: a repeated phrase whose period is longer than the 12-token n-gram, so it drafts (draft_n 44 on the clean run).
  • Checks: the run with 1 faulted copy completes, returns exactly the clean run's tokens, logs speculative checkpoint refused, and /health returns 200.

The draft-start copy only runs when there is a draft context whose memory cannot drop a tail. The test covers the verification copy; both go through the same copy_checkpoint.

Measured (macOS M5, CPU-only build -DGGML_METAL=OFF, 2026-11-07)

Gating (5f3b59f). These checks scan every binary and library under build/bin plus libserver-context.a with strings -a (for LLAMA_SERVER_TEST_FAIL, test faults: and test fault: the prompt fill) and with nm -C (for server_test_faults):

• Default build (LLAMA_SERVER_TEST_FAULTS:BOOL=OFF): 0 fault strings and 0 fault symbols in all of them. test_failed_fill.py gives 3 skipped ("llama-server was built without -DLLAMA_SERVER_TEST_FAULTS=ON"), even with the variables set.
• Test build (=ON), positive control: libllama-server-impl.dylib and libserver-context.a each carry 5 fault strings. The functions are inlined, so there are no symbols.
• Test build on the full fix: 3 passed.
• Test build with the fix removed (5683571's fill and checkpoint code with these gated hooks, plus an uncommitted throw at the unguarded speculative copy): 3 failed.
  • Refused prompt checkpoint: both slots get got exception: std::bad_alloc.
  • Failed prompt fill: the server aborts mid-request.
  • Refused speculative copy: std::bad_alloc on slot 0, then GGML_ASSERT(task) failed.

The runs below were taken before the gate, with the same tests:

On 5683571 (444f386 + the test hook only), the two fill tests fail. The first is the node's crash, line for line:

E slot iterate: id 0 | task 0 | got exception: std::bad_alloc
I slot release: id 0 | task 0 | stop processing: n_tokens = 37
E slot iterate: id 1 | task 1 | got exception: std::bad_alloc
I slot release: id 1 | task 1 | stop processing: n_tokens = 37
server-context.cpp:3452: GGML_ASSERT(batch.slot_batched || batch.size() == 0) failed

The second aborts on the same assert after test fault: the prompt fill failed after adding its tokens on slot 0.

Speculative path, red: on eae7df7 (the fill fix without the shared owner), with a temporary throw std::bad_alloc() at the unguarded ckpt.update_tgt (not committed), the speculative test fails:

E slot iterate: id 0 | task 0 | got exception: std::bad_alloc
I slot release: id 0 | task 0 | stop processing: n_tokens = 92
server-context.cpp:446: GGML_ASSERT(task) failed

On eaacc70 (this PR), 3 passed:

W slot copy_checkpo: id 0 | context checkpoint refused: its state copy could not be allocated (n_tokens = 33), a restore from here will be cold
W slot copy_checkpo: id 1 | context checkpoint refused: ...
E slot iterate: id 0 | got exception: test fault: the prompt fill failed after adding its tokens
I slot release: id 0 | stop processing: n_tokens = 0      <- was 37 phantom tokens before
W slot copy_checkpo: id 0 | task 0 | speculative checkpoint refused: its state copy could not be allocated (n_tokens = 92), the draft is dropped
I slot release: id 0 | task 0 | stop processing: n_tokens = 139

Mutant check: with the keep_first(n_written) line removed, the fill test fails. The faulted slot's retry trusts 32 unwritten tokens (n_past = 32, pos_min = -1) and aborts.

Neighbouring suites (-m "not slow"): test_failed_fill, test_completion, test_ctx_shift, test_kv_keep_only_active, test_slot_save, test_basic, test_chat_completion, test_speculative, test_stream, test_vision_api: 144 passed, 2 skipped, 1 failed (test_slot_save_while_slot_is_processing).

The test_slot_save flakes are timing, not this change

Two tests sometimes fail on this machine with or without this PR:

• test_slot_save_while_slot_is_processing ("slot 1 never entered processing state")
• test_deferred_restore_is_bound_to_its_slot ("both slots never processing concurrently")

Both poll /slots every 20 ms to see a slot mid-generation. The tiny stories260K model generates hundreds of tokens in well under that window on a fast CPU, so the poll can miss the busy state entirely. The second test even carries a skip for this ("runner too fast to separate the slots"). Neither test throws inside a fill or a checkpoint copy, so neither reaches the changed code. Receipt: on an unmodified 444f386 build on this machine, unit/test_slot_save.py ran 3 times and gave 2 failed, 9 passed / 10 passed, 1 skipped / 2 failed, 9 passed, failing the same two tests with the same messages. This PR's build flips between those outcomes the same way.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo