[joelteply]

Slice S2 of continuum's SERVING-ENGINE.md ("S2 in detail: reconfigure, grow or shrink live"; section 3, "a relaunch is a pause").

Why. The fork can change its context size or slot count only by relaunching: the sleep/reload path is destroy() then load_model(), which frees and reloads the weights. On the live node on 2026-11-10 the core relaunched the engine 13 times in 4 minutes, and each relaunch threw away every resident's KV. With this change, grow and shrink are one live path at a safe point between batched decode steps. The model and its mmap are never freed.

What.

• Lifetimes split. common_init_result::init_context / free_context and common_speculative_init_result::init_context / free_context create a context apart from its model, and each kind of context is created in exactly one function. common_load_from_params loads the model and its adapters without a context. In the server, load_model is load_weights + build_contexts, and destroy is destroy_contexts + free_weights.
• server-slot-page.{h,cpp} is the one state mover: measure / read / write of a sequence's target and draft state, plus the .ckpt sidecar codec. server_slot::prompt_save, server_prompt_cache::load, the reconfigure, and /slots?action=save|restore (its sidecar) all go through it.
• server-reconfigure.{h,cpp} holds the logic: server_reconfigure(engine, state, Geometry) -> Applied | Refused, typed, against a server_reconfigure_engine that the server context implements (server-context-reconfigure.inc). JSON appears only at the HTTP edge.
• The hook. SERVER_TASK_TYPE_RECONFIGURE runs in the task phase, the same safe point SLOT_SAVE uses.
  • Every slot holding state is paged out, unfinished generations included.
  • The contexts are rebuilt and everything that fits is paged back in.
  • Samplers are re-attached and drafting is begun again.
  • Deferred work is offered the free slots.
• Shrink never truncates. A slot past a smaller n_seq, or a state the pool cannot hold, is parked whole and named as {slot, tokens_held, tokens_fit}. A later reconfigure that holds it places it back, and a parked generation then continues.
• Refusals by name: cache_type_change (a KV type change is the cold start, support SYCL backend windows build ggml-org/llama.cpp#5188), invalid_geometry, learning_bound, fan_out_in_progress, page_out_failed, and context_build_failed. On context_build_failed the previous geometry is rebuilt and every state is put back.
• /props reports the live geometry (context_geometry, total_slots, n_ctx), the weight-load counter, and the last outcome with its cost: pause ms against tokens and bytes held.

Validator: tools/server/tests/unit/test_reconfigure.py, run on stories260K by the existing server CI.

• Four generations in progress across a grow and a shrink finish token-identical to an unreconfigured control at temperature 0.
• 100 grow/shrink cycles (n_ctx and n_seq both change) keep four held states. Each slot's next tokens and its reused prefix match a control, and n_weight_loads stays 1. VmRSS before and after is printed on Linux.
• Requests queued across many reconfigures all complete.
• A smaller n_seq names slots 2 and 3, and a unified pool too small for four states names the rest. Both come back on the grow and continue identically.
• A cache type change and a bad geometry are refused by name.

A test-only LLAMA_SERVER_TEST_STEP_DELAY_MS (server-test-faults.h) paces decode steps, so the first test can act mid-generation on a model that would otherwise finish in milliseconds.

Measured

Pending: CI is the build and test runner, since no build runs beside the serving M5. The pause ms per reconfigure on a 27B-class model is measured after S4 wires the core's relaunch sites to /reconfigure (owner: Codex, S4).

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo