Skip to content

fix(serve): reject empty/falsy prompt input in format_prompts and AsyncEngine.generate - #4803

Open
SuperMarioYL wants to merge 1 commit into
InternLM:mainfrom
SuperMarioYL:fix/empty-prompt-crash-offline
Open

fix(serve): reject empty/falsy prompt input in format_prompts and AsyncEngine.generate#4803
SuperMarioYL wants to merge 1 commit into
InternLM:mainfrom
SuperMarioYL:fix/empty-prompt-crash-offline

Conversation

@SuperMarioYL

Copy link
Copy Markdown

fix(serve): reject empty/falsy prompt input in format_prompts and AsyncEngine.generate

Motivation

Passing an empty/falsy prompt ('', [], {}, ()) to the offline SDK path
(AsyncEngine.generate / pipe.stream_infer) does not raise a clear error. The
prompt is not None, so it slips past the messages is not None XOR guard in
AsyncEngine.generate (async_engine.py:501). The falsy messages then takes the
input_ids else-branch, leaving input_ids at its default None, and the request
later crashes in len(input_ids) with a confusing:

TypeError: object of type 'NoneType' has no len()

at async_engine.py:566 (and the same shape at :424). The HTTP path already
filters empty messages (api_server.py:443-445), so this only affects users of the
offline SDK / pipeline API who pass malformed input — they get an opaque
TypeError deep inside the engine instead of an actionable error at the boundary.

A regression test (tests/test_lmdeploy/serve/test_empty_prompt_guard.py) reproduces
the real generate() path (no model/GPU required) and is red on main (raises
TypeError) / green on this branch (raises a clear ValueError).

Why guard format_prompts + generate() rather than a one-line assert in
pipeline.stream_infer?
MultimodalProcessor.format_prompts is the single
pipeline-layer chokepoint that every offline caller (stream_infer/infer/
__call__/chat via _request_generator, see pipeline.py:123,170,343-378)
passes through, so guarding it once fixes the whole offline path. generate() is
the direct-SDK backstop reachable without going through pipeline at all (the
reproduction exercises it directly). A guard only in stream_infer would miss
direct generate() callers and would be strictly redundant with the
format_prompts guard — hence the two non-redundant layers, no third.

Modification

  • lmdeploy/serve/processors/multimodal.py: add a shared
    MultimodalProcessor.validate_prompt @staticmethod that rejects None and the
    empty str/list/tuple/dict shapes with a clear ValueError; call it at the top of
    format_prompts (root-cause boundary). A (prompt, image) multimodal pair has
    len == 2 and is intentionally not rejected.
  • lmdeploy/serve/core/async_engine.py: call validate_prompt in generate()
    immediately after the existing XOR guard, covering both the messages and
    input_ids falsy shapes (direct-SDK backstop).
  • tests/test_lmdeploy/serve/test_empty_prompt_guard.py: new regression test
    (red-on-main / green-on-branch) exercising the real generate() path via
    AsyncEngine.__new__ + minimal mocks (no model/GPU), plus pure-function
    format_prompts('' / [] / None) assertions.

Scope is intentionally offline-SDK only: the HTTP (serve/openai,
serve/anthropic), TurboMind C++, PyTorch engine, pipeline.py, and the reward/ppl
paths are untouched.

BC-breaking (Optional)

No. The change only converts a previously-crashing input (TypeError deep inside
the engine) into an early, clear ValueError at the input boundary. All valid
(non-empty) input flows through unchanged. Downstream projects passing valid prompts
see no behavioral change.

Use cases (Optional)

Offline SDK users who accidentally pass an empty prompt (pipe(''),
engine.generate(messages=''), an empty batch) now get an actionable
ValueError: ... at the boundary instead of an opaque TypeError: object of type 'NoneType' has no len() from inside the engine.

Checklist

  1. Lint: ruff check on all changed files — all checks passed (line-length 120,
    E/F/I/W/UP, py310), per .pre-commit-config.yaml.
  2. Tests: python -m pytest tests/test_lmdeploy/serve/test_empty_prompt_guard.py -q → 7 passed (green on branch; 7 failed on main); python -m pytest tests/test_lmdeploy/serve/test_session_cleanup.py -q → 9 passed (regression for
    the touched generate() path, no regressions).
  3. No new/downstream dependency introduced.
  4. No documentation change needed (no user-facing API surface added — only a clearer
    error for previously-crashing invalid input).

…ne.generate

Empty/falsy prompt input ('', [], {}, ()) is not None, so it slipped past the 'messages is not None' XOR guard in AsyncEngine.generate. The falsy messages then took the input_ids else-branch, leaving input_ids at its default None and crashing later in len(input_ids) with a confusing 'TypeError: object of type NoneType has no len()'.

Add a shared MultimodalProcessor.validate_prompt that rejects None and the empty str/list/tuple/dict shapes with a clear ValueError, and apply it at the two non-redundant layers: format_prompts (the single pipeline-layer chokepoint every prompt passes through) and generate (the direct-SDK backstop reachable without a model). The HTTP path already filters empty input, so this is an offline-SDK backstop only.

A multimodal (prompt, image) pair tuple has len == 2 and is intentionally not rejected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant