fix(ai-proxy-multi): keep the client request body intact across fallback retries - #13793
Open
janiussyafiq wants to merge 1 commit into
Open
fix(ai-proxy-multi): keep the client request body intact across fallback retries#13793janiussyafiq wants to merge 1 commit into
janiussyafiq wants to merge 1 commit into
Conversation
…ack retries The retry loop in before_proxy re-reads the parsed request body from the per-request cache, but build_request rewrites that shared table in place with per-instance options. A fallback attempt therefore started from the previous instance's rewritten body: $request_llm_model was clobbered with the first instance's model, and the first instance's options (e.g. temperature) leaked into the retry request. Deep-copy the parsed body at the start of every attempt so each instance builds its request from the client's original body, and drop the redundant request_llm_model re-assignment (detect_request_type already captures it from the client body in the access phase).
shreemaan-abhishek
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The retry loop in
before_proxyre-reads the parsed request body from the per-request cache (ctx._request_body_table), butbuild_requestrewrites that shared table in place with per-instance options. A fallback attempt therefore starts from the previous instance's rewritten body:$request_llm_modelis clobbered with the first instance's model, and the first instance's options (e.g.temperature) leak into the retry request sent to the fallback instance.This PR deep-copies the parsed body at the start of every attempt so each instance builds its request from the client's original body, and drops the redundant
request_llm_modelre-assignment inbefore_proxy(detect_request_typealready captures it from the client body in the access phase, for bothai-proxyandai-proxy-multi).The copy is O(body) and cheaper than the JSON decode of the same body already performed in the access phase. Keeping the shared cache pristine also fixes log-phase
post_arg.*variable reads and the semantic balancer's re-pick on retry, both of which previously saw the instance-rewritten body instead of the client's request.Which issue(s) this PR fixes:
Fixes #13769
Checklist