fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again - #11386
Open
localai-bot wants to merge 1 commit into
Open
fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again#11386localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
The Go bindings mirror vllm.h by hand and refuse a library whose vllm_abi_version differs from what they were written against. Two automated pin bumps (#11174, #11352) moved VLLM_CPP_VERSION onto engines declaring ABI v10 while govllmcpp.go still mirrored v5, so every vllm-cpp image built since then panics at startup on every platform: panic: vllm-cpp: ABI mismatch: library reports v10, backend built against v5 Grow both PODs to the v10 layout: vllm_model_params gains speculative_config, enable_prefix_caching, max_num_batched_tokens, scheduling_policy, kv_transfer_config and enable_jump_forward (88 bytes), vllm_sampling_params gains the v8 logits-processor pair (136 bytes). The offsets in the specs come from offsetof() against the pinned header. All of the new fields are inert when zeroed, so the engine behaves exactly as it did under v5; the backend sets none of them. Nothing cross-checked the two files, which is why a blind pin bump could ship a backend that cannot load. The library build now runs abi-check first: it compares VLLM_ABI_VERSION in the fetched header against abiVersion in govllmcpp.go and fails the build naming both, instead of leaving the mismatch for a user's runtime. Fixes #11379 Assisted-by: Claude:claude-fable-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler
added a commit
that referenced
this pull request
Aug 5, 2026
Both branches moved the mirrors to ABI v10 independently: this one as part of exposing the v6-v10 config surface, #11386 as the minimal unbreak for issue #11379. The layouts agreed field for field, so the reconcile keeps this branch's tri-state constants and padding comments, plus #11386's Makefile abi-check target and its abiVersion spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #11379.
The bug is not L4T-specific
Every
vllm-cppimage built since #11174 refuses to start, on every platform:govllmcpp.gohand-mirrorsvllm.handregisterLibrejects any library whosevllm_abi_versiondiffers from what the mirrors were written against. Twoautomated pin bumps (#11174, then #11352 into v4.8.0) moved
VLLM_CPP_VERSIONonto engines declaring ABI v10 while the bindings still declared v5. The
bindings had not been touched since #11100. The reporter happened to hit it on
an L4T image first.
The fix
v6 through v10 grew both PODs by appending only, so the mirrors just have to
grow with them:
vllm_model_params(48 -> 88 bytes):speculative_config(v6),enable_prefix_caching(v7),max_num_batched_tokens/scheduling_policy/kv_transfer_config(v9),enable_jump_forward(v10).vllm_sampling_params(120 -> 136 bytes): the v8logits_processor/logits_processor_user_datapair.Every one of them is inert when zeroed and the backend sets none of them, so the
engine behaves exactly as it did under v5. This restores the backend, it does not
change inference. Exposing the new knobs as backend options is a separate
follow-up.
The offsets asserted in
vllmcpp_test.goare not eyeballed: they come fromcompiling the pinned header and printing
offsetoffor each field.Keeping it from happening again
Nothing cross-checked the pin against the bindings. The bump PRs did build the
backend image (
backend_pr.ymlruns the path-filtered matrix on pull requests),but a stale mirror compiles perfectly well: the only thing that ever compared the
two was
registerLib, at runtime, on the user's machine. A green build meantnothing here.
The library build now runs
make abi-checkfirst, comparingVLLM_ABI_VERSIONin the fetched header against
abiVersioningovllmcpp.goand failing with bothversions named. That moves the comparison to where the image is produced, so a
future blind bump turns its own PR red.
The other backends using this hand-mirrored purego pattern were checked too. Only
trellis2cppandmagpie-tts-cppenforce a hard ABI equality, and both currentlymatch their pins (11 vs 11, 1 vs 1); the rest probe symbols and degrade
gracefully. Wiring the same guard into those two is worth a follow-up.
Verification
of the suite green) and pass after.
make abi-checkpasses on the matching pin, fails on a simulated v11 header.make lint LINT_NEW_FROM=origin/master: 0 issues.libvllm.so(CPU build) andQwen_Qwen3.5-0.8B-Q4_K_M.gguf: load, blocking completion, streaming, and thechat / tool-call paths. That same run died at
registerLibbefore the fix.abi-checkhas since run inside a real CI backend image build (on feat(vllm-cpp): wire the full engine config surface through engine_args #11159,which carries this commit):
vllm-cpp: ABI v10 matches the pinned engine.The
tests-linuxfailure on the first run of this PR is unrelated: HuggingFaceanswered the runner with HTTP 429, taking down the
pkg/huggingface-apiintegration specs and the importer specs that call HF. This PR touches only
backend/go/vllm-cpp. Re-run requested.Related
#11159 exposes these engine knobs through
engine_args:and already contains thisfix (the two branches moved to v10 independently and have been reconciled). Land
this one first so it can be cherry-picked onto 4.8.x; #11159's diff then narrows to
the config surface.