[integrations][plan] Handle finish_reason in OpenAI-family connections - #1040
Open
weiqingy wants to merge 5 commits into
Open
[integrations][plan] Handle finish_reason in OpenAI-family connections#1040weiqingy wants to merge 5 commits into
weiqingy wants to merge 5 commits into
Conversation
added 5 commits
August 22, 2026 20:56
convert_to_openai_message merged the whole extra_args dict into outbound system, user and assistant message params. The inbound path writes completion metadata into that same dict, and chat_model_action replays the stored assistant message on the next turn of a tool loop, so model_name, promptTokens, completionTokens and structured_output were sent back to the provider as top-level message fields. structured_output holds a Pydantic model or a pyflink Row, which is not serialisable as a message field at all. Send only the fields OpenAI defines for each role, matching what Java's OpenAIChatCompletionsUtils.convertToOpenAIMessage already produces: content alone for system and user; content, tool calls and a string refusal for assistant; the tool branch unchanged. The isinstance guard on refusal mirrors Java's instanceof String on the same outbound path. Azure OpenAI and vLLM share this converter and pick up the same fix. The function had no test coverage before. The new tests were driven by mutation testing and kill every mutant in scope, including reinstating the blanket merge. Generated-by: Claude Code 2.1.234 (Claude Opus 5)
…onses The provider's finish reason was never recorded, so a completion truncated by a token limit came back as ordinary partial content with nothing marking it incomplete. Record it on the returned message as extra_args["finish_reason"] in both OpenAI-family connections. Nothing reads it yet; acting on it under structured output is a separate change. The capture sits outside the token-metrics guard in both connections. A response can carry a finish reason with no usage block, and Azure's guard additionally depends on a deployment name unrelated to the reason, so writing it inside either guard would drop it silently. The value is stored verbatim, including reasons outside OpenAI's documented set, and an absent or null reason writes no key and raises nothing. OpenAI-compatible servers omit the field and emit vendor-specific values, and VLLMChatModelConnection subclasses the OpenAI connection, so this path serves them too. Tests mock only the transport and drive real SDK response objects. Choice declares finish_reason as a required Literal, so strict validation would reject an unknown vendor value before this code ran; the client itself defaults to lenient response construction, which is what the tests use. Generated-by: Claude Code 2.1.234 (Claude Opus 5)
…onse A response truncated by a token limit was parsed like any other. When the partial content was invalid JSON the caller got a syntax error pointing at the parser rather than at the truncation, and when it happened to still parse, the caller silently received wrong structured output. Check why the model stopped before parsing against an output schema. A length or content_filter reason raises with the cause named; stop and tool_calls parse as before; an unrecognized reason or an absent key also parses, so providers that never record one are unaffected. The check runs before the parser execution is reported, not inside the parse. The reporting wrapper records any exception from the parse as model_output_parse_error, so gating inside it would report a truncation as a parse failure, which is the same misleading diagnosis this change removes. Rejecting beforehand records no parser execution at all, which is accurate because none was attempted. ValueError keeps callers that already catch the JSONDecodeError symptom working, and adds no public API. Truncated content that still parses previously produced wrong output and now raises, so callers using the RETRY strategy will see such responses retried. Generated-by: Claude Code 2.1.234 (Claude Opus 5)
Mirror the Python capture on the Java side: record the provider's finish reason on the returned message as extraArgs["finish_reason"] in both OpenAI-family connections. Nothing reads it yet; acting on it under structured output is a separate change. Read the value through _finishReason().asKnown() rather than finishReason(). The latter resolves through JsonField.getRequired and throws OpenAIInvalidDataException when the member is absent or JSON-null. VLLMChatModelConnection extends OpenAICompletionsConnection, so OpenAI-compatible servers that omit the member use this path, and the plain accessor would fail a call that previously succeeded. FinishReason is an open value wrapper rather than an enum, so the wire string is taken with asString() and vendor-specific reasons are stored verbatim. known() throws on an unrecognized value and value() collapses it, so neither is used. The capture sits outside the token-usage guard in both connections. A response can carry a finish reason with no usage block, and the Azure guard additionally requires a deployment name unrelated to the reason. Azure is tested through its existing package-private seam. The OpenAI connection stashes inline in a private method, so its tests drive a loopback endpoint, which also covers a vendor reason surviving real deserialization. An omitted member and a null member are distinct wire shapes and both are covered. Generated-by: Claude Code 2.1.234 (Claude Opus 5)
Java mirror of the Python gate. Check why the model stopped before parsing against an output schema: a length or content_filter reason throws with the cause named, while stop, tool_calls, an unrecognized reason and an absent key all parse as before. The check runs before the parser execution is reported, not inside the parse. The reporting wrapper records any exception from the parse as model_output_parse_error, so gating inside it would report a truncation as a parse failure, which is the same misleading diagnosis this change removes. Rejecting beforehand records no parser execution at all, which is accurate because none was attempted. IllegalStateException is what plan already uses for this class of failure, and adds no public API. The two messages are kept individually identifiable so a truncation cannot be reported as content filtering or the reverse. Also seed the structured-output result from the inbound extra args instead of building a fresh map. The previous behavior discarded everything the connection recorded, so Java dropped finish_reason exactly when an output schema was in play, while Python preserved it. Token metrics are recorded before this point from the only call site, and the result cannot re-enter the parser because the path is guarded on tool calls being empty. Generated-by: Claude Code 2.1.234 (Claude Opus 5)
weiqingy
force-pushed
the
936-finish-reason
branch
from
August 23, 2026 04:08
1ffaa41 to
9e72b51
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked issue: #936
Purpose of change
finish_reasonwas never read in either language, so a truncated completion came back as ordinary partial content. Under structured output that content went straight to a JSON parser, so the caller got an error naming the parser instead of the truncation. Worse, when the truncated content happened to still be valid JSON, nothing raised at all and the caller silently received wrong data.Cases 1 and 3 of #936 shipped in #952 and #989. This is the last one.
The change has three parts:
finish_reasonin message metadata. Unknown values are stored verbatim, and an absent reason writes no key.lengthandcontent_filterfail with the cause named.stop,tool_calls, unknown values and an absent key parse as before. Non-structured calls are untouched, so a truncated answer is still returned and the caller decides.extra_argsinto outbound messages, and since the chat action replays the assistant response on the next tool-call turn,model_name,promptTokens,completionTokensandstructured_outputwere reaching the provider as message fields. Each role now sends only the fields OpenAI defines, matching Java.Four things worth knowing when reading the diff:
_finishReason().asKnown()rather thanfinishReason(). The plain accessor throwsOpenAIInvalidDataExceptionwhen the member is absent or null, andVLLMChatModelConnectionextendsOpenAICompletionsConnection, so OpenAI-compatible servers that omit it would start failing calls that work today.model_output_parse_error, so gating inside it would report a truncation as a parse failure, which is the same misleading diagnosis this change removes. The trade is that a truncation no longer produces aPARSERfailure report. The action-level failure is still reported.finish_reasonexactly when an output schema was in play, while Python kept it.extra_argsto the provider, which Java never allowed and nothing in the repo relies on. And truncated content that still parses now raises instead of returning wrong data, so callers usingRETRYwill see those responses retried.Out of scope: the Responses API reports
statusandincomplete_details.reason, a different vocabulary. Providers that record no reason are unaffected and pick the behavior up if they later write the same key.Tests
./tools/build.shand./tools/ut.share green: 33/33 modules, Java 1592 run with 0 failures, Python 938 passed. Each of the five commits also builds and passes on its own.Coverage was driven by mutation testing, and every mutant in scope is killed in both languages: swapping in the throwing accessor, moving either capture inside the token-usage guard, moving the gate inside the reporting wrapper, dropping either terminal reason, and reinstating the blanket merge.
Two test properties are load-bearing and easy to lose in a later refactor. The throwing tests use valid JSON, because truncated JSON raises on its own and a malformed fixture would pass with no gate at all. And one test asserts that a rejected response produces no parser execution report, which is the only thing stopping the gate from drifting back inside the wrapper.
Both languages were also driven through identical inputs to confirm they agree, including that neither sends the metadata keys to a provider.
API
No public API change.
finish_reasonis a metadata key rather than a field, and no new exception type or problem category is introduced. Failures reuseValueErrorin Python andIllegalStateExceptionin Java, both already used for this class of failure in their packages. Sincejson.JSONDecodeErrorsubclassesValueError, Python callers catching today's symptom keep working and simply get a clearer message.Documentation
doc-neededdoc-not-neededdoc-includedWas this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code 2.1.234 (Claude Opus 5)