fix(serve): reject empty/falsy prompt input in format_prompts and AsyncEngine.generate - #4803
Open
SuperMarioYL wants to merge 1 commit into
Open
fix(serve): reject empty/falsy prompt input in format_prompts and AsyncEngine.generate#4803SuperMarioYL wants to merge 1 commit into
SuperMarioYL wants to merge 1 commit into
Conversation
…ne.generate
Empty/falsy prompt input ('', [], {}, ()) is not None, so it slipped past the 'messages is not None' XOR guard in AsyncEngine.generate. The falsy messages then took the input_ids else-branch, leaving input_ids at its default None and crashing later in len(input_ids) with a confusing 'TypeError: object of type NoneType has no len()'.
Add a shared MultimodalProcessor.validate_prompt that rejects None and the empty str/list/tuple/dict shapes with a clear ValueError, and apply it at the two non-redundant layers: format_prompts (the single pipeline-layer chokepoint every prompt passes through) and generate (the direct-SDK backstop reachable without a model). The HTTP path already filters empty input, so this is an offline-SDK backstop only.
A multimodal (prompt, image) pair tuple has len == 2 and is intentionally not rejected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(serve): reject empty/falsy prompt input in format_prompts and AsyncEngine.generate
Motivation
Passing an empty/falsy prompt (
'',[],{},()) to the offline SDK path(
AsyncEngine.generate/pipe.stream_infer) does not raise a clear error. Theprompt is not
None, so it slips past themessages is not NoneXOR guard inAsyncEngine.generate(async_engine.py:501). The falsymessagesthen takes theinput_idselse-branch, leavinginput_idsat its defaultNone, and the requestlater crashes in
len(input_ids)with a confusing:at
async_engine.py:566(and the same shape at:424). The HTTP path alreadyfilters empty messages (
api_server.py:443-445), so this only affects users of theoffline SDK /
pipelineAPI who pass malformed input — they get an opaqueTypeErrordeep inside the engine instead of an actionable error at the boundary.A regression test (
tests/test_lmdeploy/serve/test_empty_prompt_guard.py) reproducesthe real
generate()path (no model/GPU required) and is red onmain(raisesTypeError) / green on this branch (raises a clearValueError).Modification
lmdeploy/serve/processors/multimodal.py: add a sharedMultimodalProcessor.validate_prompt@staticmethodthat rejectsNoneand theempty str/list/tuple/dict shapes with a clear
ValueError; call it at the top offormat_prompts(root-cause boundary). A(prompt, image)multimodal pair haslen == 2and is intentionally not rejected.lmdeploy/serve/core/async_engine.py: callvalidate_promptingenerate()immediately after the existing XOR guard, covering both the
messagesandinput_idsfalsy shapes (direct-SDK backstop).tests/test_lmdeploy/serve/test_empty_prompt_guard.py: new regression test(red-on-
main/ green-on-branch) exercising the realgenerate()path viaAsyncEngine.__new__+ minimal mocks (no model/GPU), plus pure-functionformat_prompts('' / [] / None)assertions.Scope is intentionally offline-SDK only: the HTTP (
serve/openai,serve/anthropic), TurboMind C++, PyTorch engine,pipeline.py, and the reward/pplpaths are untouched.
BC-breaking (Optional)
No. The change only converts a previously-crashing input (
TypeErrordeep insidethe engine) into an early, clear
ValueErrorat the input boundary. All valid(non-empty) input flows through unchanged. Downstream projects passing valid prompts
see no behavioral change.
Use cases (Optional)
Offline SDK users who accidentally pass an empty prompt (
pipe(''),engine.generate(messages=''), an empty batch) now get an actionableValueError: ...at the boundary instead of an opaqueTypeError: object of type 'NoneType' has no len()from inside the engine.Checklist
ruff checkon all changed files — all checks passed (line-length 120,E/F/I/W/UP, py310), per
.pre-commit-config.yaml.python -m pytest tests/test_lmdeploy/serve/test_empty_prompt_guard.py -q→ 7 passed (green on branch; 7 failed onmain);python -m pytest tests/test_lmdeploy/serve/test_session_cleanup.py -q→ 9 passed (regression forthe touched
generate()path, no regressions).error for previously-crashing invalid input).