Skip to content

fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again - #11386

Open
localai-bot wants to merge 1 commit into
masterfrom
fix/vllm-cpp-abi-v10-11379
Open

fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again#11386
localai-bot wants to merge 1 commit into
masterfrom
fix/vllm-cpp-abi-v10-11379

Conversation

@localai-bot

@localai-bot localai-bot commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Fixes #11379.

The bug is not L4T-specific

Every vllm-cpp image built since #11174 refuses to start, on every platform:

panic: vllm-cpp: ABI mismatch: library reports v10, backend built against v5

govllmcpp.go hand-mirrors vllm.h and registerLib rejects any library whose
vllm_abi_version differs from what the mirrors were written against. Two
automated pin bumps (#11174, then #11352 into v4.8.0) moved VLLM_CPP_VERSION
onto engines declaring ABI v10 while the bindings still declared v5. The
bindings had not been touched since #11100. The reporter happened to hit it on
an L4T image first.

The fix

v6 through v10 grew both PODs by appending only, so the mirrors just have to
grow with them:

  • vllm_model_params (48 -> 88 bytes): speculative_config (v6),
    enable_prefix_caching (v7), max_num_batched_tokens / scheduling_policy /
    kv_transfer_config (v9), enable_jump_forward (v10).
  • vllm_sampling_params (120 -> 136 bytes): the v8 logits_processor /
    logits_processor_user_data pair.

Every one of them is inert when zeroed and the backend sets none of them, so the
engine behaves exactly as it did under v5. This restores the backend, it does not
change inference. Exposing the new knobs as backend options is a separate
follow-up.

The offsets asserted in vllmcpp_test.go are not eyeballed: they come from
compiling the pinned header and printing offsetof for each field.

Keeping it from happening again

Nothing cross-checked the pin against the bindings. The bump PRs did build the
backend image (backend_pr.yml runs the path-filtered matrix on pull requests),
but a stale mirror compiles perfectly well: the only thing that ever compared the
two was registerLib, at runtime, on the user's machine. A green build meant
nothing here.

The library build now runs make abi-check first, comparing VLLM_ABI_VERSION
in the fetched header against abiVersion in govllmcpp.go and failing with both
versions named. That moves the comparison to where the image is produced, so a
future blind bump turns its own PR red.

The other backends using this hand-mirrored purego pattern were checked too. Only
trellis2cpp and magpie-tts-cpp enforce a hard ABI equality, and both currently
match their pins (11 vs 11, 1 vs 1); the rest probe symbols and degrade
gracefully. Wiring the same guard into those two is worth a follow-up.

Verification

  • The layout specs fail behaviorally before the change (3 value mismatches, rest
    of the suite green) and pass after.
  • make abi-check passes on the matching pin, fails on a simulated v11 header.
  • make lint LINT_NEW_FROM=origin/master: 0 issues.
  • Full e2e green in 1000s against a real ABI v10 libvllm.so (CPU build) and
    Qwen_Qwen3.5-0.8B-Q4_K_M.gguf: load, blocking completion, streaming, and the
    chat / tool-call paths. That same run died at registerLib before the fix.
  • abi-check has since run inside a real CI backend image build (on feat(vllm-cpp): wire the full engine config surface through engine_args #11159,
    which carries this commit): vllm-cpp: ABI v10 matches the pinned engine.

The tests-linux failure on the first run of this PR is unrelated: HuggingFace
answered the runner with HTTP 429, taking down the pkg/huggingface-api
integration specs and the importer specs that call HF. This PR touches only
backend/go/vllm-cpp. Re-run requested.

Related

#11159 exposes these engine knobs through engine_args: and already contains this
fix (the two branches moved to v10 independently and have been reconciled). Land
this one first so it can be cherry-picked onto 4.8.x; #11159's diff then narrows to
the config surface.

The Go bindings mirror vllm.h by hand and refuse a library whose
vllm_abi_version differs from what they were written against. Two
automated pin bumps (#11174, #11352) moved VLLM_CPP_VERSION onto engines
declaring ABI v10 while govllmcpp.go still mirrored v5, so every
vllm-cpp image built since then panics at startup on every platform:

  panic: vllm-cpp: ABI mismatch: library reports v10, backend built against v5

Grow both PODs to the v10 layout: vllm_model_params gains
speculative_config, enable_prefix_caching, max_num_batched_tokens,
scheduling_policy, kv_transfer_config and enable_jump_forward (88 bytes),
vllm_sampling_params gains the v8 logits-processor pair (136 bytes). The
offsets in the specs come from offsetof() against the pinned header. All
of the new fields are inert when zeroed, so the engine behaves exactly as
it did under v5; the backend sets none of them.

Nothing cross-checked the two files, which is why a blind pin bump could
ship a backend that cannot load. The library build now runs abi-check
first: it compares VLLM_ABI_VERSION in the fetched header against
abiVersion in govllmcpp.go and fails the build naming both, instead of
leaving the mismatch for a user's runtime.

Fixes #11379

Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler added a commit that referenced this pull request Aug 5, 2026
Both branches moved the mirrors to ABI v10 independently: this one as part
of exposing the v6-v10 config surface, #11386 as the minimal unbreak for
issue #11379. The layouts agreed field for field, so the reconcile keeps
this branch's tri-state constants and padding comments, plus #11386's
Makefile abi-check target and its abiVersion spec.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vllm.cpp not working (L4T build)

2 participants