refactor: stream tool parameters incrementally - #4802
Open
lvhan028 wants to merge 2 commits into
Open
Conversation
* feat: stream tool arguments incrementally * refactor: simplify XML tool parser streaming * fix: restore empty stream delta fallback * fix: suppress empty streaming deltas * perf: stream xml tool parser incrementally * refactor: stream json tool parser incrementally * refactor: simplify XML tool parser consumers * refactor: rename XML tool parser stream state * refactor: extract JSON tool parser base Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors tool-call streaming so tool parameters/arguments can be emitted incrementally (instead of waiting for the full payload), spanning JSON, XML-like, and DeepSeek DSML tool formats, and adds/updates tests to validate incremental emission and bounded buffering. It also updates OpenAI streaming to buffer token metadata across parser steps that produce no visible deltas.
Changes:
- Introduce
JsonToolParser(incremental JSON argument streaming) and refactor Qwen/Llama3/InternLM parsers to reuse it. - Refactor
XmlToolParserinto a phase-driven incremental engine and update Qwen3Coder/GLM47 parsers to implement adapter-style consumption. - Update streaming plumbing/tests: incremental argument assertions, metadata buffering across “no visible delta” steps, and new regression coverage for bounded buffers.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_lmdeploy/test_deepseek_v4_encoding.py | Update DSML tests to join streamed argument fragments before JSON parsing. |
| tests/test_lmdeploy/test_deepseek_v32_encoding.py | Same as above for DeepSeek v3.2. |
| tests/test_lmdeploy/serve/parsers/test_tool_parser_incremental.py | New tests covering incremental argument emission + bounded buffering across parsers. |
| tests/test_lmdeploy/serve/parsers/test_qwen3_parser.py | Update streaming reference assertions for incremental tool argument fragments; add regressions. |
| tests/test_lmdeploy/serve/parsers/test_qwen3_5_parser.py | Restructure streaming expectations into flattened events; add many incremental/complete-parse equivalence tests. |
| tests/test_lmdeploy/serve/parsers/test_llama3_parser.py | Add regression ensuring arguments are emitted before JSON completes. |
| tests/test_lmdeploy/serve/parsers/test_glm47_parser.py | Restructure streaming assertions for incremental fragments; add incremental/complete-parse equivalence tests. |
| tests/test_lmdeploy/serve/openai/chat_completions/test_streaming_metadata.py | New tests for buffering token_ids/logprobs when parser returns no visible delta. |
| tests/test_lmdeploy/serve/openai/chat_completions/test_delta_tool_call_id.py | Update/extend incremental JSON streaming tests via JsonToolParser. |
| tests/test_lmdeploy/serve/openai/chat_completions/test_arguments_validation.py | Switch validation tests to JsonToolParser.parse_tool_call_complete. |
| lmdeploy/serve/parsers/tool_parser/xml_tool_parser.py | Major refactor: shared incremental XML-like parse state + streaming JSON emission/coercion. |
| lmdeploy/serve/parsers/tool_parser/tool_parser.py | Remove embedded JSON parsing helpers; simplify base lifecycle state. |
| lmdeploy/serve/parsers/tool_parser/qwen3coder_tool_parser.py | Rework incremental parsing using XmlToolParser adapter methods. |
| lmdeploy/serve/parsers/tool_parser/glm47_tool_parser.py | Rework incremental parsing using XmlToolParser adapter methods. |
| lmdeploy/serve/parsers/tool_parser/json_tool_parser.py | New incremental JSON tool payload parser with early argument streaming. |
| lmdeploy/serve/parsers/tool_parser/qwen3_tool_parser.py | Refactor to inherit JsonToolParser. |
| lmdeploy/serve/parsers/tool_parser/qwen2d5_tool_parser.py | Refactor to inherit JsonToolParser. |
| lmdeploy/serve/parsers/tool_parser/llama3_tool_parser.py | Refactor to inherit JsonToolParser (no close tag). |
| lmdeploy/serve/parsers/tool_parser/internlm2_tool_parser.py | Refactor to inherit JsonToolParser. |
| lmdeploy/serve/parsers/tool_parser/deepseek_v32_tool_parser.py | Implement incremental DSML emission across invocations/parameters. |
| lmdeploy/serve/parsers/tool_parser/deepseek_v4_tool_parser.py | Point V4 parser at the refactored V3.2 incremental implementation. |
| lmdeploy/serve/parsers/tool_parser/init.py | Export JsonToolParser. |
| lmdeploy/serve/parsers/response_parser.py | Update tool streaming to use _payload_closed instead of JSONDecoder completeness checks. |
| lmdeploy/serve/openai/api_server.py | Buffer token metadata across parser steps that emit no visible deltas. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+191
to
+195
| def _consume_arg_delta(self, raw: str, json_fragments: list[str]) -> None: | ||
| arg_name = self._state.arg_name | ||
| if arg_name is None: | ||
| return | ||
|
|
Comment on lines
557
to
561
| out = self.tool_parser.decode_tool_incremental(added_text=emit, final=False) | ||
| if (self.profile.tool_payload_format == 'json' | ||
| and self._is_complete_json_object(self.tool_parser._tool_payload)): | ||
| if self.profile.tool_payload_format == 'json' and self.tool_parser._payload_closed: | ||
| out.extend(self.tool_parser.decode_tool_incremental(added_text='', final=True)) | ||
| self.tool_parser.finish_tool_call() | ||
| self._mode = self.MODE_PLAIN |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.