llama-server: support echo+prompt-logprobs for loglikelihood scoring

[zeeshanhaque21]

Why

lm-evaluation-harness's local-completions backend scores multiple_choice tasks (HellaSwag, ARC, raw MMLU, TruthfulQA-mc) by POSTing /v1/completions with echo=true, logprobs=<int>, max_tokens=1, then summing per-token logprobs over the echoed continuation to pick the highest-scoring answer choice. llama-server currently hard-rejects this: server-common.cpp:830 throws "Only no echo is supported". This blocks loglikelihood-type benchmarks for every GGUF-backed model (7 of 14 in the local llm-bench harness).

mlx_lm ships its own local (non-HTTP) teacher-forcing scorer (mlx_lm/evaluate.py, MLXLM class) that GGUF models can't use — GGUF models only run via llama-server, so a server-side echo+logprobs endpoint is the only path to score them.

What's needed

OpenAI completions API contract: echo=true returns the prompt tokens themselves in the response; logprobs=<int> (top-N) returns per-position logprobs, including for the echoed prompt tokens, not just newly generated ones.

Design notes from investigation (llama.cpp @ 7529fda, tools/server/)

1. Reject site: server-common.cpp:828-831, oaicompat_completion_params_parse() — currently throws on any truthy echo. Needs to set a flag through to task_params instead (server-task.h task_params struct, parsed in server-task.cpp params_from_json_cmpl ~line 260-330) alongside existing n_probs/post_sampling_probs fields.

2. Prompt-cache conflict: [TAG_PROMPT_LOGITS] block (server-context.cpp ~3053) already special-cases a fully-cached prompt by forcibly decoding at least 1 token, because logits aren't stored in the KV cache — only K/V vectors are. Echo needs logits at every prompt position, not the fallback single token. This means cache_prompt must be forced off (or the cached prefix forced to reprocess) whenever echo=true, or scores will silently be wrong for any prompt that partially or fully cache-hits. This is a correctness bug risk, not just a perf one — needs explicit handling, not just a TODO.

3. Per-position logit extraction: today only the last token of a prompt chunk gets logits extracted (batch.logits[batch.n_tokens - 1] = true, server-context.cpp:3243). Echo mode needs batch.logits[i] = true for every prompt position in the slot's span. Long prompts get split across multiple n_batch-sized chunks (server-context.cpp prompt-fill loop ~2900-3250), so results need accumulating across chunk boundaries into one per-position logprob array before the response is built.

4. Scoring/formatting: mirror the existing populate_token_probs() (server-context.cpp:1832) which does top-N logprob extraction for generated tokens — same math (softmax over logits, top-N via existing helper), but applied to prompt positions and merged into the OAI-format choices[0].logprobs alongside (or instead of, depending on how lm_eval parses it — see lm_eval/models/openai_completions.py parse_logprobs(): expects token_logprobs and top_logprobs arrays covering the full echoed sequence, sliced by ctxlen on the client side) the 1-token generation response.

5. Continuous batching / speculative decoding: orthogonal in principle — echo only touches the prompt-fill phase (SLOT_STATE_PROCESSING_PROMPT/DONE_PROMPT), speculative decoding and context-shift only engage in SLOT_STATE_GENERATING. But capture-type drafters (dspark) stage per-sequence state during prompt decode (common_speculative_begin called pre-prompt-eval when common_speculative_need_embd_capture) — needs verifying nothing downstream assumes "only last-position logits exist."

6. No prior art found upstream (ggml-org/llama.cpp) — searched issues/PRs for echo/prompt-logprobs, nothing to adapt from. This is a from-scratch design.

Scope note

Full production-grade version (correct under continuous batching + speculative decoding + context-shift + prompt-cache checkpoint restore, for arbitrary concurrent load) is the target — this is a shared inference binary, not a single-purpose eval harness. Expect multi-session effort with its own test plan against the existing server test suite (tools/server/tests/).

Immediate workaround in use

~/llm-bench (local benchmarking harness) ships MMLU/HellaSwag/ARC/TruthfulQA-mc today for MLX-backend models only, via mlx_lm.evaluate's local (non-HTTP) MLXLM scorer — bypasses this issue entirely for the 7 MLX models, but the 7 GGUF-backed models stay blocked on this.