Skip to content

Cancel the HTTP call when a model stream is closed, drop the openai-java hold - #58

Merged
codeministry merged 1 commit into
mainfrom
fix/openai-stream-cancel
Nov 9, 2026
Merged

codeministry merged 1 commit into
mainfrom
fix/openai-stream-cancel

Conversation

@codeministry

Copy link
Copy Markdown
Owner

Replaces the openai-java hold at 4.65.0 with a fix on our side, so #55 is no longer needed.

Cause

From openai-java 4.66.0 on (openai/openai-java#1027), StreamResponse.close() closes the OkHttp response before the reader. A chat turn is stopped, deleted or timed out by closing its stream from a thread other than the one blocked reading it. That close then runs under the read still in flight: okio throws "Unbalanced enter/exit", the socket stays open, and the model keeps generating an answer nobody reads.

Fix

  • llm/CancelOnClose, an OkHttp interceptor: closing a response body calls Call.cancel() first, which is OkHttp's own thread-safe way to end a call from outside.
  • ChatModels hands it to the sync and the async OpenAI client through OpenAiSetup's customizer list.
  • A body that was read to its end has nothing left to cancel, so the judge and the classifier, which call rather than stream, are unaffected.
  • The strictly constraint, the openaiJava catalog entry and openai-java's own Renovate rule are gone. openai-java comes from the Spring AI BOM again (4.80.0).
  • docs/decisions/chat.md § Stop and regenerate has the reasoning, backend/CLAUDE.md a one-line rule, and the CHANGELOG is corrected.

Verification

  • Without the interceptor, on 4.80.0: ChatStreamTest, ChatTurnDeadlineTest and two ChatConversationTest cases are red ("the stub saw its connection closed", "the model call was cancelled").
  • With it: green on 4.80.0, and on 4.101.0 with a temporary constraint.
  • ./gradlew build: 1304 backend cases, 96 unit and 22 browser test files, all green; the 13 skipped are the corpus tests without .eml samples.
  • No "Unbalanced" in any test report.

After the merge, #55 points at a catalog entry that no longer exists and can be closed.

…ava hold

From openai-java 4.66.0 on, closing a stream from another thread closes the
OkHttp response under the in-flight read, so a stopped, deleted or timed-out
chat turn left the model generating. CancelOnClose makes closing a response
body cancel its call first; ChatModels hands it to both OpenAI clients. The
four stop tests are green on 4.80.0 and 4.101.0, so openai-java comes from the
Spring AI BOM again and its Renovate rule goes.
@codeministry codeministry self-assigned this Nov 9, 2026
@codeministry
codeministry merged commit e6ae977 into main Nov 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant