Skip to content

[integrations][plan] Handle finish_reason in OpenAI-family connections - #1040

Open
weiqingy wants to merge 5 commits into
apache:mainfrom
weiqingy:936-finish-reason
Open

[integrations][plan] Handle finish_reason in OpenAI-family connections#1040
weiqingy wants to merge 5 commits into
apache:mainfrom
weiqingy:936-finish-reason

Conversation

@weiqingy

@weiqingy weiqingy commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Linked issue: #936

Purpose of change

finish_reason was never read in either language, so a truncated completion came back as ordinary partial content. Under structured output that content went straight to a JSON parser, so the caller got an error naming the parser instead of the truncation. Worse, when the truncated content happened to still be valid JSON, nothing raised at all and the caller silently received wrong data.

Cases 1 and 3 of #936 shipped in #952 and #989. This is the last one.

The change has three parts:

  1. Both languages record the provider's reason as finish_reason in message metadata. Unknown values are stored verbatim, and an absent reason writes no key.
  2. The shared structured-output path checks it before parsing. length and content_filter fail with the cause named. stop, tool_calls, unknown values and an absent key parse as before. Non-structured calls are untouched, so a truncated answer is still returned and the caller decides.
  3. Python's outbound converter had to be fixed first. It merged all of extra_args into outbound messages, and since the chat action replays the assistant response on the next tool-call turn, model_name, promptTokens, completionTokens and structured_output were reaching the provider as message fields. Each role now sends only the fields OpenAI defines, matching Java.

Four things worth knowing when reading the diff:

  • Java reads through _finishReason().asKnown() rather than finishReason(). The plain accessor throws OpenAIInvalidDataException when the member is absent or null, and VLLMChatModelConnection extends OpenAICompletionsConnection, so OpenAI-compatible servers that omit it would start failing calls that work today.
  • The check runs before the parser execution is reported, not inside the parse. The reporting wrapper labels any parse exception model_output_parse_error, so gating inside it would report a truncation as a parse failure, which is the same misleading diagnosis this change removes. The trade is that a truncation no longer produces a PARSER failure report. The action-level failure is still reported.
  • Java's structured-output result now carries inbound metadata forward instead of starting from an empty map. It previously dropped finish_reason exactly when an output schema was in play, while Python kept it.
  • Two visible behavior changes. A Python caller can no longer forward arbitrary extra_args to the provider, which Java never allowed and nothing in the repo relies on. And truncated content that still parses now raises instead of returning wrong data, so callers using RETRY will see those responses retried.

Out of scope: the Responses API reports status and incomplete_details.reason, a different vocabulary. Providers that record no reason are unaffected and pick the behavior up if they later write the same key.

Tests

./tools/build.sh and ./tools/ut.sh are green: 33/33 modules, Java 1592 run with 0 failures, Python 938 passed. Each of the five commits also builds and passes on its own.

Coverage was driven by mutation testing, and every mutant in scope is killed in both languages: swapping in the throwing accessor, moving either capture inside the token-usage guard, moving the gate inside the reporting wrapper, dropping either terminal reason, and reinstating the blanket merge.

Two test properties are load-bearing and easy to lose in a later refactor. The throwing tests use valid JSON, because truncated JSON raises on its own and a malformed fixture would pass with no gate at all. And one test asserts that a rejected response produces no parser execution report, which is the only thing stopping the gate from drifting back inside the wrapper.

Both languages were also driven through identical inputs to confirm they agree, including that neither sends the metadata keys to a provider.

API

No public API change. finish_reason is a metadata key rather than a field, and no new exception type or problem category is introduced. Failures reuse ValueError in Python and IllegalStateException in Java, both already used for this class of failure in their packages. Since json.JSONDecodeError subclasses ValueError, Python callers catching today's symptom keep working and simply get a clearer message.

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

Generated-by: Claude Code 2.1.234 (Claude Opus 5)

@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Aug 23, 2026
@weiqingy weiqingy changed the title [python][java][integrations][plan] Handle finish_reason in OpenAI-family connections [integrations][java][python] Handle finish_reason in OpenAI-family connections Aug 23, 2026
@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs and removed doc-not-needed Your PR changes do not impact docs labels Aug 23, 2026
@weiqingy weiqingy changed the title [integrations][java][python] Handle finish_reason in OpenAI-family connections [integrations][plan] Handle finish_reason in OpenAI-family connections Aug 23, 2026
@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs and removed doc-not-needed Your PR changes do not impact docs labels Aug 23, 2026
weiqingy added 5 commits August 22, 2026 20:56
convert_to_openai_message merged the whole extra_args dict into outbound
system, user and assistant message params. The inbound path writes completion
metadata into that same dict, and chat_model_action replays the stored
assistant message on the next turn of a tool loop, so model_name,
promptTokens, completionTokens and structured_output were sent back to the
provider as top-level message fields. structured_output holds a Pydantic model
or a pyflink Row, which is not serialisable as a message field at all.

Send only the fields OpenAI defines for each role, matching what Java's
OpenAIChatCompletionsUtils.convertToOpenAIMessage already produces: content
alone for system and user; content, tool calls and a string refusal for
assistant; the tool branch unchanged. The isinstance guard on refusal mirrors
Java's instanceof String on the same outbound path.

Azure OpenAI and vLLM share this converter and pick up the same fix.

The function had no test coverage before. The new tests were driven by
mutation testing and kill every mutant in scope, including reinstating the
blanket merge.

Generated-by: Claude Code 2.1.234 (Claude Opus 5)
…onses

The provider's finish reason was never recorded, so a completion truncated by
a token limit came back as ordinary partial content with nothing marking it
incomplete. Record it on the returned message as extra_args["finish_reason"]
in both OpenAI-family connections. Nothing reads it yet; acting on it under
structured output is a separate change.

The capture sits outside the token-metrics guard in both connections. A
response can carry a finish reason with no usage block, and Azure's guard
additionally depends on a deployment name unrelated to the reason, so writing
it inside either guard would drop it silently.

The value is stored verbatim, including reasons outside OpenAI's documented
set, and an absent or null reason writes no key and raises nothing.
OpenAI-compatible servers omit the field and emit vendor-specific values, and
VLLMChatModelConnection subclasses the OpenAI connection, so this path serves
them too.

Tests mock only the transport and drive real SDK response objects. Choice
declares finish_reason as a required Literal, so strict validation would
reject an unknown vendor value before this code ran; the client itself
defaults to lenient response construction, which is what the tests use.

Generated-by: Claude Code 2.1.234 (Claude Opus 5)
…onse

A response truncated by a token limit was parsed like any other. When the
partial content was invalid JSON the caller got a syntax error pointing at the
parser rather than at the truncation, and when it happened to still parse, the
caller silently received wrong structured output.

Check why the model stopped before parsing against an output schema. A length
or content_filter reason raises with the cause named; stop and tool_calls
parse as before; an unrecognized reason or an absent key also parses, so
providers that never record one are unaffected.

The check runs before the parser execution is reported, not inside the parse.
The reporting wrapper records any exception from the parse as
model_output_parse_error, so gating inside it would report a truncation as a
parse failure, which is the same misleading diagnosis this change removes.
Rejecting beforehand records no parser execution at all, which is accurate
because none was attempted.

ValueError keeps callers that already catch the JSONDecodeError symptom
working, and adds no public API.

Truncated content that still parses previously produced wrong output and now
raises, so callers using the RETRY strategy will see such responses retried.

Generated-by: Claude Code 2.1.234 (Claude Opus 5)
Mirror the Python capture on the Java side: record the provider's finish
reason on the returned message as extraArgs["finish_reason"] in both
OpenAI-family connections. Nothing reads it yet; acting on it under structured
output is a separate change.

Read the value through _finishReason().asKnown() rather than finishReason().
The latter resolves through JsonField.getRequired and throws
OpenAIInvalidDataException when the member is absent or JSON-null.
VLLMChatModelConnection extends OpenAICompletionsConnection, so
OpenAI-compatible servers that omit the member use this path, and the plain
accessor would fail a call that previously succeeded.

FinishReason is an open value wrapper rather than an enum, so the wire string
is taken with asString() and vendor-specific reasons are stored verbatim.
known() throws on an unrecognized value and value() collapses it, so neither
is used.

The capture sits outside the token-usage guard in both connections. A response
can carry a finish reason with no usage block, and the Azure guard
additionally requires a deployment name unrelated to the reason.

Azure is tested through its existing package-private seam. The OpenAI
connection stashes inline in a private method, so its tests drive a loopback
endpoint, which also covers a vendor reason surviving real deserialization. An
omitted member and a null member are distinct wire shapes and both are
covered.

Generated-by: Claude Code 2.1.234 (Claude Opus 5)
Java mirror of the Python gate. Check why the model stopped before parsing
against an output schema: a length or content_filter reason throws with the
cause named, while stop, tool_calls, an unrecognized reason and an absent key
all parse as before.

The check runs before the parser execution is reported, not inside the parse.
The reporting wrapper records any exception from the parse as
model_output_parse_error, so gating inside it would report a truncation as a
parse failure, which is the same misleading diagnosis this change removes.
Rejecting beforehand records no parser execution at all, which is accurate
because none was attempted.

IllegalStateException is what plan already uses for this class of failure, and
adds no public API. The two messages are kept individually identifiable so a
truncation cannot be reported as content filtering or the reverse.

Also seed the structured-output result from the inbound extra args instead of
building a fresh map. The previous behavior discarded everything the
connection recorded, so Java dropped finish_reason exactly when an output
schema was in play, while Python preserved it. Token metrics are recorded
before this point from the only call site, and the result cannot re-enter the
parser because the path is guarded on tool calls being empty.

Generated-by: Claude Code 2.1.234 (Claude Opus 5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant