fix(streaming): add heartbeat keepalive + connection pooling#27
Open
garywhat wants to merge 2 commits into
Open
fix(streaming): add heartbeat keepalive + connection pooling#27garywhat wants to merge 2 commits into
garywhat wants to merge 2 commits into
Conversation
Upstream JoyCode API buffers the entire response before sending anything
(TTFB 10-30s for reasoning models). Without keepalive, downstream clients
(Claude Code, OpenAI clients) time out or show "no response" during this
gap.
- Add SSE heartbeat goroutines for OpenAI and Anthropic streaming paths
that send comment lines (": ...") every 15s while waiting for upstream
- Parse usage from SSE data lines in OpenAI streaming for token logging
- Add shared http.Transport with connection pooling in joycode client
- Add graceful shutdown via stopCh for background goroutines in serve
- Add pkg/common.IsTimeoutError helper for timeout detection
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Resolve conflict in pkg/joycode/client.go: keep both defaultTransport (connection pooling from this PR) and envOr (configurable BaseURL from upstream vibe-coding-labs#24). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upstream JoyCode API buffers the entire response before sending anything (TTFB 10-30s for reasoning models). Without keepalive, downstream clients (Claude Code, OpenAI clients) time out or show "no response" during this gap.
pkg/openai/chat.go) and Anthropic (pkg/anthropic/handler.go) streaming paths that send comment lines (": ...") every 15s while waiting for upstreamusagefrom SSE data lines in OpenAI streaming so token consumption is logged correctly (previously missing for streaming requests)http.Transportwith connection pooling inpkg/joycode/client.goso clients reuse TCP connections instead of dialing anew for every requeststopChfor background goroutines incmd/JoyCode2Api/serve.gopkg/common.IsTimeoutErrorhelper for timeout detection across packagesBackground
This addresses the streaming stability issues reported in #2 and #12 (the "stream failed / read first line: EOF" symptom that occurs when the upstream TTFB exceeds client timeouts).
Test plan
pingevents during upstream waitGenerated with Devin