feat(vllm-cpp): wire the full engine config surface through engine_args by localai-bot · Pull Request #11159 · mudler/LocalAI

localai-bot · 2026-07-28T08:28:46Z

Refreshed 2026-08-05. This branch had gone 144 commits stale. It is now merged
up to master and reconciled with #11386. Two changes to what is written below:
the pin is master's 0757cac2 (newer than the eec09bed/6d4fb5e7 this PR
originally carried, and the one the MLX-on-metal default is gated against), and
the mirrors are at ABI v10, not v9, so enable_jump_forward is exposed too.

Merge order: #11386 is the minimal unbreak for #11379 and should land first so
it can be cherry-picked onto 4.8.x. This branch already contains that fix, so once
#11386 lands the diff here narrows to the config surface. It keeps #11386's
make abi-check build guard alongside this branch's env-gated handshake spec, and
it keeps this branch's tri-state constants over #11386's plainer field comments.

Verification after the merge: make lint clean against origin/master,
backend/go/vllm-cpp unit specs green, make abi-check agrees at v10, and the
"not verified" caveat at the bottom is now closed - see the Testing section.

What

The vllm-cpp backend could configure four of the engine's knobs — block size, KV block count, max sequence length, max concurrent sequences — out of a config surface that is considerably larger. Speculative decoding, prefix caching, the chunked-prefill token budget, the scheduling policy and the external KV connector (LMCache) were reachable from vllm.cpp's own HTTP server and from nothing LocalAI could write in a model config.

Part of that gap was the C ABI itself, which carried strictly less than EngineParams does. That is fixed upstream in vllm.cpp ABI v9 (merged to main as eec09bed, which this PR pins). The rest was here: the backend parsed a flat options: list with five recognised keys and had no way to express a nested JSON document at all.

Configuration

Config now goes through engine_args: — the same map the vLLM and SGLang backends already take — with keys spelled as vLLM's own CLI flags, so a speculative_config or kv_transfer_config block written for vLLM works verbatim:

name: qwen36-dflash
backend: vllm-cpp
context_size: 8192
engine_args:
  max_num_seqs: 32
  max_num_batched_tokens: 8192
  enable_prefix_caching: true
  scheduling_policy: lpm
  speculative_config:
    method: dflash
    model: z-lab/Qwen3.6-27B-DFlash
    num_speculative_tokens: 4
  kv_transfer_config:
    kv_connector: LMCacheConnector
    kv_role: kv_both
    kv_connector_extra_config: {host: 127.0.0.1, port: 65432}

The options: list keeps working and now reads every key too, so no existing config breaks; engine_args wins where both set the same key.

All three speculative methods the engine supports are reachable: mtp (draft head inside the target checkpoint, safetensors only), dflash (separate draft checkpoint, model: required), and ngram (draft-free).

Two details worth reviewer attention

enable_prefix_caching: false maps to the ABI tri-state force-OFF (2), not 0. 0 means "let the model capability decide", and dense architectures default the cache on — collapsing the two would silently enable it against an explicit false.

cSamplingParams grows the ABI v8 logits-processor tail. LocalAI installs no processor, but the C side reads those fields off the pointer we hand it, so a Go struct that stopped at StructuredJSONObject (120 bytes vs C's 136) would have had the engine read past our allocation and call whatever sat there. Latent only because the old v5 ABI gate refused to load a v8 library.

Importer

A vllm-cpp import of a HuggingFace repo now probes config.json and writes speculative_config: {method: mtp} into the generated engine_args when the checkpoint declares an MTP head — the safetensors analogue of the llama-cpp importer's GGUF header probe. An explicit speculative_config is never overwritten, and every probe failure is non-fatal.

Two asymmetries versus the llama.cpp hook, both deliberate:

DFlash draft repositories are refused with a warning rather than auto-configured. A drafter cannot serve alone, and the target/draft pairing is not derivable from either repo in isolation.
The llama-cpp importer stops applying spec_type:draft-mtp when the chosen backend is vllm-cpp. Those are llama.cpp option keys vllm-cpp does not read, and vllm.cpp rejects MTP over a GGUF source outright because the mtp.* draft tensors do not survive GGUF conversion. This was a pre-existing bug — a GGUF import with the vllm-cpp preference emitted dead llama.cpp options.

Docs

docs/content/features/text-generation.md gains a vllm.cpp section covering the engine_args table, all three speculative methods, LMCache, and the legacy options: list. The backend previously had no documentation page at all.

Testing

make lint clean.
core/config, core/gallery/importers, backend/go/vllm-cpp all green. 9 new importer specs, 11 new engine_args specs, plus updated struct-offset assertions for the v9 / v8 layouts.
Built a CPU-only libvllm.so at the pinned commit and confirmed purego binds all 19 symbols with vllm_abi_version() returning 9. That check is now a spec: set VLLM_CPP_LIBRARY and it runs without needing model weights, so a future pin bump with a stale struct mirror fails in CI rather than at a user's first load.
Upstream: 31/31 capi test cases on the merged vllm.cpp tree.

Re-run after the merge to master (2026-08-05), now at ABI v10 and pin 0757cac2:

make lint LINT_NEW_FROM=origin/master: 0 issues.
backend/go/vllm-cpp, core/config, core/gallery/importers: green.
make abi-check (from fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again #11386) reports v10 on both sides, and fails as intended
against a header doctored to v11.
End-to-end against a live engine, which the original body listed as not
verified: full suite green in 1330s against a CPU libvllm.so reporting ABI
v10 and Qwen_Qwen3.5-0.8B-Q4_K_M.gguf - load, blocking completion, streaming,
and the chat / tool-call paths. Weights loaded, tokens generated.

Still not covered: a live generation with the new knobs actually engaged
(speculative decoding, LMCache, a non-fcfs scheduling policy). Those are verified
at the mapping and ABI-handshake level; the e2e above exercises the default path.

🤖 Generated with Claude Code

The vllm-cpp backend could configure four of the engine's knobs - block size, KV block count, max sequence length and max concurrent sequences - out of a config surface that is considerably larger. Speculative decoding, prefix caching, the chunked-prefill token budget, the scheduling policy and the external KV connector were all reachable from vllm.cpp's own HTTP server and from nothing LocalAI could write in a model config. Part of that gap was the C ABI itself, which carried strictly less than EngineParams does; that is fixed upstream in vllm.cpp ABI v9 (this bumps the pin to it). The rest was here: the backend parsed a flat `options:` list with five recognised keys and had no way to express a nested JSON document at all. Configuration now goes through `engine_args:`, the same map the vLLM and SGLang backends already take, with keys spelled as vLLM's own CLI flags - so a `speculative_config` or `kv_transfer_config` block written for vLLM works verbatim: engine_args: max_num_batched_tokens: 8192 enable_prefix_caching: true scheduling_policy: lpm speculative_config: method: dflash model: z-lab/Qwen3.6-27B-DFlash num_speculative_tokens: 4 kv_transfer_config: kv_connector: LMCacheConnector kv_role: kv_both kv_connector_extra_config: {host: 127.0.0.1, port: 65432} The `options:` list keeps working, and now reads every key too, so no existing config breaks; engine_args wins where both set the same key. Two details worth calling out. `enable_prefix_caching: false` maps to the ABI's force-OFF state (2), not the 0 that means "let the model capability decide" - collapsing them would silently turn the cache ON for the dense architectures that default it on. And cSamplingParams grows the ABI v8 logits-processor tail: LocalAI installs no processor, but the C side reads those fields off the pointer we hand it, so a Go struct that stopped short would have had the engine read 16 bytes past our allocation and call whatever sat there. The importer gets the safetensors counterpart of the llama-cpp MTP hook: a `vllm-cpp` import of a HuggingFace repo probes config.json and, on a checkpoint that declares an MTP head, writes speculative_config {method: mtp} into the generated engine_args. DFlash draft repositories are detected and refused with a warning rather than configured as standalone models, since a drafter cannot serve alone. The llama-cpp importer stops applying its own `spec_type:draft-mtp` options when the chosen backend is vllm-cpp: those are llama.cpp option keys vllm-cpp does not read, and vllm.cpp rejects MTP over a GGUF source anyway because the `mtp.*` draft tensors only exist in the safetensors checkpoint. docs/content/features/text-generation.md gains a vllm.cpp section - the backend had no documentation page at all - covering the engine_args table, all three speculative methods, LMCache, and the legacy options list. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

The Go PODs in govllmcpp.go are hand-written against one VLLM_ABI_VERSION and the Makefile pins the vllm.cpp commit that produces it. Nothing checked those two agree short of the e2e suite, which needs a model to run at all, so a pin bump could land with a stale mirror and only fail at a user's first load. VLLM_CPP_LIBRARY now drives a handshake spec that dlopens a built libvllm, binds every symbol, and compares the library's reported ABI against the mirrors'. No weights required. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

974d9d72 was the pre-merge commit on the feature branch. eec09bed is the merged main tip carrying the same ABI v9 surface, and is the tree the capi suite (31/31) and the Go ABI handshake were actually verified against. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

…F cache The engine resolves speculative_config.model against a directory containing config.json, or against ~/.cache/huggingface/hub/models--<org>--<repo>/ snapshots/*, and it never downloads. LocalAI keeps models in its own directory, so the repo-id spelling the vLLM docs teach - "z-lab/Qwen3.6-27B-DFlash" - misses the HF cache and dies deep inside the load with "draft checkpoint not found", which reads like a broken checkpoint rather than a model nobody fetched. Resolve it before the load call: the reference as given, then its last path segment under LocalAI's models dir (what LocalAI's own downloader produces), then the whole reference under the models dir. When none resolve, fail there naming both what was asked for and every location tried, so the message says what to do about it. mtp and ngram pass through untouched - neither has a separate draft checkpoint. A speculative_config that does not parse also passes through, because the engine owns config validation and produces the better error. Docs also gain the two limits that were missing and are easy to lose an afternoon to: speculation is Qwen3.5/3.6-only at this engine pin regardless of format, and mtp/dflash need a safetensors target. The latter is a gap in the engine's GGUF loader rather than a property of GGUF - the format carries MTP weights fine, llama.cpp reads them as nextn.* tensors plus a <arch>.nextn_predict_layers key - so the docs say that rather than implying GGUF cannot express it. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

…ding vllm.cpp landed ABI v10 on main while this branch was open. The backend's runtime handshake refuses any library whose reported ABI differs from the mirrors', so the pin and the Go PODs move together or not at all. v10 appends one int32, `enable_jump_forward`, AFTER the v9 fields. Nothing else in the config surface changed: the SGLang reconciliation that carried it explicitly dropped its own duplicate scheduler_policy int in favour of the v9 `scheduling_policy` string this branch already wires, and a diff of EngineParams and the server flags across the window turns up jump forward and nothing else. So the exposure is one new knob, `engine_args.enable_jump_forward` - SGLang's grammar-speed subset, which emits grammar-forced tokens without spending a model step and therefore only affects constrained requests. It is the SECOND tri-state on this struct, and it repeats the trap the first one had: 0 is not "off", it is "defer" (to the environment here, to the model capability for prefix caching), so an explicit `false` has to reach the engine as 2. The bool->tri-state helper and the log renderer are now shared rather than duplicated per field, and named for the encoding instead of for prefix caching, since the next tri-state will want them too. The docs say this outright, because "omitting the key" and "setting it false" being different is not guessable. The Go mirror grows the field plus an EXPLICIT trailing pad: the struct is 8-aligned and now ends on a lone int32, so it is 88 bytes rather than 84. The offset assertions cover it, and the real-library handshake spec (VLLM_CPP_LIBRARY against a CPU libvllm.so built at the new pin) confirms the version agrees: 43 specs pass, ABI reported 10. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

…ding is real now The pin sat at f384edcd while vllm.cpp main moved a long way. The ABI is unchanged at v10 and both POD structs are field-identical to the pinned commit (verified by diffing vllm_model_params and vllm_sampling_params across the range), so the Go mirror needs no edit and this is a clean bump. What it picks up matters for this backend: - MTP speculative decoding from a GGUF target, gated end to end on GPU. - DFlash speculative decoding with a GGUF draft AND a GGUF target. - NVFP4 GGUF: dequant, plus a native fp4 compute path for dense and full-attention projections. On the 27B that closed a cross-container divergence entirely (the GGUF and safetensors builds of the same quantization run now emit identical tokens) and halved peak RSS. - A real engine fix: the GDN speculative state gather/scatter was mis-striding the widened conv row, so speculation silently corrupted the target's own recurrent state on CPU. Docs corrected accordingly. The section previously told users that mtp and dflash are rejected on a .gguf target and called it a gap in the engine's GGUF loader. That is no longer true, and leaving it would send people to safetensors for no reason. A head-less GGUF is still refused, and the text now says so with the actual cause. The real-library ABI handshake was re-run against a libvllm.so built at the exact pinned commit rather than a stale one: 43 specs pass, reported ABI 10. make lint clean. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

The vllm-cpp backend could configure four of the engine's knobs - block size, KV block count, max sequence length and max concurrent sequences - out of a config surface that is considerably larger. Speculative decoding, prefix caching, the chunked-prefill token budget, the scheduling policy and the external KV connector were all reachable from vllm.cpp's own HTTP server and from nothing LocalAI could write in a model config. Part of that gap was the C ABI itself, which carried strictly less than EngineParams does; that is fixed upstream in vllm.cpp ABI v9 (this bumps the pin to it). The rest was here: the backend parsed a flat `options:` list with five recognised keys and had no way to express a nested JSON document at all. Configuration now goes through `engine_args:`, the same map the vLLM and SGLang backends already take, with keys spelled as vLLM's own CLI flags - so a `speculative_config` or `kv_transfer_config` block written for vLLM works verbatim: engine_args: max_num_batched_tokens: 8192 enable_prefix_caching: true scheduling_policy: lpm speculative_config: method: dflash model: z-lab/Qwen3.6-27B-DFlash num_speculative_tokens: 4 kv_transfer_config: kv_connector: LMCacheConnector kv_role: kv_both kv_connector_extra_config: {host: 127.0.0.1, port: 65432} The `options:` list keeps working, and now reads every key too, so no existing config breaks; engine_args wins where both set the same key. Two details worth calling out. `enable_prefix_caching: false` maps to the ABI's force-OFF state (2), not the 0 that means "let the model capability decide" - collapsing them would silently turn the cache ON for the dense architectures that default it on. And cSamplingParams grows the ABI v8 logits-processor tail: LocalAI installs no processor, but the C side reads those fields off the pointer we hand it, so a Go struct that stopped short would have had the engine read 16 bytes past our allocation and call whatever sat there. The importer gets the safetensors counterpart of the llama-cpp MTP hook: a `vllm-cpp` import of a HuggingFace repo probes config.json and, on a checkpoint that declares an MTP head, writes speculative_config {method: mtp} into the generated engine_args. DFlash draft repositories are detected and refused with a warning rather than configured as standalone models, since a drafter cannot serve alone. The llama-cpp importer stops applying its own `spec_type:draft-mtp` options when the chosen backend is vllm-cpp: those are llama.cpp option keys vllm-cpp does not read, and vllm.cpp rejects MTP over a GGUF source anyway because the `mtp.*` draft tensors only exist in the safetensors checkpoint. docs/content/features/text-generation.md gains a vllm.cpp section - the backend had no documentation page at all - covering the engine_args table, all three speculative methods, LMCache, and the legacy options list. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

The Go PODs in govllmcpp.go are hand-written against one VLLM_ABI_VERSION and the Makefile pins the vllm.cpp commit that produces it. Nothing checked those two agree short of the e2e suite, which needs a model to run at all, so a pin bump could land with a stale mirror and only fail at a user's first load. VLLM_CPP_LIBRARY now drives a handshake spec that dlopens a built libvllm, binds every symbol, and compares the library's reported ABI against the mirrors'. No weights required. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

…F cache The engine resolves speculative_config.model against a directory containing config.json, or against ~/.cache/huggingface/hub/models--<org>--<repo>/ snapshots/*, and it never downloads. LocalAI keeps models in its own directory, so the repo-id spelling the vLLM docs teach - "z-lab/Qwen3.6-27B-DFlash" - misses the HF cache and dies deep inside the load with "draft checkpoint not found", which reads like a broken checkpoint rather than a model nobody fetched. Resolve it before the load call: the reference as given, then its last path segment under LocalAI's models dir (what LocalAI's own downloader produces), then the whole reference under the models dir. When none resolve, fail there naming both what was asked for and every location tried, so the message says what to do about it. mtp and ngram pass through untouched - neither has a separate draft checkpoint. A speculative_config that does not parse also passes through, because the engine owns config validation and produces the better error. Docs also gain the two limits that were missing and are easy to lose an afternoon to: speculation is Qwen3.5/3.6-only at this engine pin regardless of format, and mtp/dflash need a safetensors target. The latter is a gap in the engine's GGUF loader rather than a property of GGUF - the format carries MTP weights fine, llama.cpp reads them as nextn.* tensors plus a <arch>.nextn_predict_layers key - so the docs say that rather than implying GGUF cannot express it. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

…ding vllm.cpp landed ABI v10 on main while this branch was open. The backend's runtime handshake refuses any library whose reported ABI differs from the mirrors', so the pin and the Go PODs move together or not at all. v10 appends one int32, `enable_jump_forward`, AFTER the v9 fields. Nothing else in the config surface changed: the SGLang reconciliation that carried it explicitly dropped its own duplicate scheduler_policy int in favour of the v9 `scheduling_policy` string this branch already wires, and a diff of EngineParams and the server flags across the window turns up jump forward and nothing else. So the exposure is one new knob, `engine_args.enable_jump_forward` - SGLang's grammar-speed subset, which emits grammar-forced tokens without spending a model step and therefore only affects constrained requests. It is the SECOND tri-state on this struct, and it repeats the trap the first one had: 0 is not "off", it is "defer" (to the environment here, to the model capability for prefix caching), so an explicit `false` has to reach the engine as 2. The bool->tri-state helper and the log renderer are now shared rather than duplicated per field, and named for the encoding instead of for prefix caching, since the next tri-state will want them too. The docs say this outright, because "omitting the key" and "setting it false" being different is not guessable. The Go mirror grows the field plus an EXPLICIT trailing pad: the struct is 8-aligned and now ends on a lone int32, so it is 88 bytes rather than 84. The offset assertions cover it, and the real-library handshake spec (VLLM_CPP_LIBRARY against a CPU libvllm.so built at the new pin) confirms the version agrees: 43 specs pass, ABI reported 10. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

…ding is real now The pin sat at f384edcd while vllm.cpp main moved a long way. The ABI is unchanged at v10 and both POD structs are field-identical to the pinned commit (verified by diffing vllm_model_params and vllm_sampling_params across the range), so the Go mirror needs no edit and this is a clean bump. What it picks up matters for this backend: - MTP speculative decoding from a GGUF target, gated end to end on GPU. - DFlash speculative decoding with a GGUF draft AND a GGUF target. - NVFP4 GGUF: dequant, plus a native fp4 compute path for dense and full-attention projections. On the 27B that closed a cross-container divergence entirely (the GGUF and safetensors builds of the same quantization run now emit identical tokens) and halved peak RSS. - A real engine fix: the GDN speculative state gather/scatter was mis-striding the widened conv row, so speculation silently corrupted the target's own recurrent state on CPU. Docs corrected accordingly. The section previously told users that mtp and dflash are rejected on a .gguf target and called it a gap in the engine's GGUF loader. That is no longer true, and leaving it would send people to safetensors for no reason. A head-less GGUF is still refused, and the text now says so with the actual cause. The real-library ABI handshake was re-run against a libvllm.so built at the exact pinned commit rather than a stale one: 43 specs pass, reported ABI 10. make lint clean. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

Keep the GNU constant-folding diagnostics visible on Darwin without allowing vllm.cpp's global -Werror to fail the Metal backend build. Assisted-by: Codex:gpt-5 [systematic-debugging]

The previous no-error flag is overridden by vllm.cpp's later target-local -Werror. Disable only the Apple Clang folding diagnostic so the Metal build can complete while all other warnings remain fatal. Assisted-by: Codex:gpt-5 [systematic-debugging]

The Metal source that triggers Apple Clang's constant-folding diagnostic is compiled as Objective-C. Pass the targeted suppression through CMAKE_OBJC_FLAGS as well as the C++ language flags so vllm.cpp's -Werror no longer promotes it. Assisted-by: Codex:gpt-5

The Go bindings mirror vllm.h by hand and refuse a library whose vllm_abi_version differs from what they were written against. Two automated pin bumps (#11174, #11352) moved VLLM_CPP_VERSION onto engines declaring ABI v10 while govllmcpp.go still mirrored v5, so every vllm-cpp image built since then panics at startup on every platform: panic: vllm-cpp: ABI mismatch: library reports v10, backend built against v5 Grow both PODs to the v10 layout: vllm_model_params gains speculative_config, enable_prefix_caching, max_num_batched_tokens, scheduling_policy, kv_transfer_config and enable_jump_forward (88 bytes), vllm_sampling_params gains the v8 logits-processor pair (136 bytes). The offsets in the specs come from offsetof() against the pinned header. All of the new fields are inert when zeroed, so the engine behaves exactly as it did under v5; the backend sets none of them. Nothing cross-checked the two files, which is why a blind pin bump could ship a backend that cannot load. The library build now runs abi-check first: it compares VLLM_ABI_VERSION in the fetched header against abiVersion in govllmcpp.go and fails the build naming both, instead of leaving the mismatch for a user's runtime. Fixes #11379 Assisted-by: Claude:claude-fable-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

…e-args # Conflicts: # backend/go/vllm-cpp/Makefile

Both branches moved the mirrors to ABI v10 independently: this one as part of exposing the v6-v10 config surface, #11386 as the minimal unbreak for issue #11379. The layouts agreed field for field, so the reconcile keeps this branch's tri-state constants and padding comments, plus #11386's Makefile abi-check target and its abiVersion spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

The remote copy of this branch was rebased and carries three Darwin build fixes (the Apple Clang gnu-folding-constant diagnostic on C++, Objective-C and Objective-C++) that the local worktree did not have. Merge rather than force over them; the only conflicts were the ABI version in two comments, where v10 wins. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

The merge commit 76f7959 was created with `git merge --no-edit`, which takes git's default merge message and therefore carried no Signed-off-by. DCO flags it. Remediate forward rather than rewriting the branch, which would drop the Darwin fixes other people already pulled from it. I, Ettore Di Giacinto <mudler@localai.io>, hereby add my Signed-off-by to this commit: 76f7959 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

localai-bot · 2026-08-05T23:53:44Z

Refreshed and reconciled: merged up to master (144 commits), then merged #11386 in so the two independent ABI v10 moves became one. Kept this branch's tri-state constants plus #11386's make abi-check build guard. The remote copy of this branch had been rebased and carried three Darwin gnu-folding-constant fixes the local worktree lacked, so those were merged in rather than force-pushed over.

Pin is now master's 0757cac2 rather than the older 6d4fb5e7 this branch carried: it is newer, contains it, and the MLX-on-metal default is gated on being at or past 89c46aeb, so moving the pin backwards would have quietly un-gated that.

Post-merge verification: lint clean, backend/go/vllm-cpp + core/config + core/gallery/importers green, and the full e2e now passes in 1330s against a live CPU libvllm.so at ABI v10 with Qwen_Qwen3.5-0.8B-Q4_K_M.gguf (load, completion, streaming, chat/tool-calls) - that closes the "not verified end-to-end" caveat the original description carried. The CI backend build for -cpu-vllm-cpp also exercised the new guard: vllm-cpp: ABI v10 matches the pinned engine.

Still not covered: a live generation with the new knobs actually engaged (speculative decoding, LMCache, non-fcfs scheduling). Those are verified at the mapping and handshake level only.

localai-bot · 2026-08-06T06:27:14Z

On the red DCO check: it flags three commits, all pre-dating today's refresh and all authored by localai-org-maint-bot (20e537b, c7c6edf, a8fadc5, the Apple Clang folding-diagnostic fixes). Merge commits are ignored by the app, so nothing from the merge-forward is implicated.

Deliberately not fixing it by rebasing: that would rewrite commits already pushed to this shared branch. The plan is to squash-merge, which collapses the branch into a single commit authored and signed off by the merger and satisfies the DCO in substance. Repo settings already disallow merge commits, so squash is the path anyway.

One wart to ignore: commit 578e784 is an empty DCO remediation commit I added before the app told me which commits it was actually complaining about. It is inert under a squash merge. If anyone does decide to rebase this branch instead, drop that commit first, since DCO's own guidance says an empty commit breaks the rebase remedy.

mudler added 5 commits July 29, 2026 09:06

localai-org-maint-bot force-pushed the feat/vllm-cpp-engine-args branch from f90497a to df5e2a6 Compare July 29, 2026 09:07

localai-org-maint-bot and others added 10 commits July 29, 2026 15:04

Merge branch 'master' into feat/vllm-cpp-engine-args

f5b38e2

fix(vllm-cpp): tolerate Apple Clang folding warning

20e537b

Keep the GNU constant-folding diagnostics visible on Darwin without allowing vllm.cpp's global -Werror to fail the Metal backend build. Assisted-by: Codex:gpt-5 [systematic-debugging]

localai-org-maint-bot force-pushed the feat/vllm-cpp-engine-args branch from 08b7a0c to a8fadc5 Compare August 4, 2026 07:10

mudler added 5 commits August 5, 2026 22:38

Merge remote-tracking branch 'origin/master' into feat/vllm-cpp-engin…

76f7959

…e-args # Conflicts: # backend/go/vllm-cpp/Makefile

localai-bot mentioned this pull request Aug 5, 2026

fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again #11386

Merged

Navigation Menu

Uh oh!

feat(vllm-cpp): wire the full engine config surface through engine_args - #11159

feat(vllm-cpp): wire the full engine config surface through engine_args#11159
localai-bot wants to merge 20 commits into
masterfrom
feat/vllm-cpp-engine-args

localai-bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

localai-bot commented Aug 5, 2026

Uh oh!

localai-bot commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

localai-bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Configuration

Two details worth reviewer attention

Importer

Docs

Testing

Uh oh!

localai-bot commented Aug 5, 2026

Uh oh!

localai-bot commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

localai-bot commented Jul 28, 2026 •

edited

Loading