Skip to content

fix(ai-proxy-multi): keep the client request body intact across fallback retries - #13793

Open
janiussyafiq wants to merge 1 commit into
apache:masterfrom
janiussyafiq:fix/ai-proxy-multi-fallback-body
Open

fix(ai-proxy-multi): keep the client request body intact across fallback retries#13793
janiussyafiq wants to merge 1 commit into
apache:masterfrom
janiussyafiq:fix/ai-proxy-multi-fallback-body

Conversation

@janiussyafiq

Copy link
Copy Markdown
Contributor

Description

The retry loop in before_proxy re-reads the parsed request body from the per-request cache (ctx._request_body_table), but build_request rewrites that shared table in place with per-instance options. A fallback attempt therefore starts from the previous instance's rewritten body: $request_llm_model is clobbered with the first instance's model, and the first instance's options (e.g. temperature) leak into the retry request sent to the fallback instance.

This PR deep-copies the parsed body at the start of every attempt so each instance builds its request from the client's original body, and drops the redundant request_llm_model re-assignment in before_proxy (detect_request_type already captures it from the client body in the access phase, for both ai-proxy and ai-proxy-multi).

The copy is O(body) and cheaper than the JSON decode of the same body already performed in the access phase. Keeping the shared cache pristine also fixes log-phase post_arg.* variable reads and the semantic balancer's re-pick on retry, both of which previously saw the instance-rewritten body instead of the client's request.

Which issue(s) this PR fixes:

Fixes #13769

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

…ack retries

The retry loop in before_proxy re-reads the parsed request body from the
per-request cache, but build_request rewrites that shared table in place
with per-instance options. A fallback attempt therefore started from the
previous instance's rewritten body: $request_llm_model was clobbered with
the first instance's model, and the first instance's options (e.g.
temperature) leaked into the retry request.

Deep-copy the parsed body at the start of every attempt so each instance
builds its request from the client's original body, and drop the
redundant request_llm_model re-assignment (detect_request_type already
captures it from the client body in the access phase).
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. bug Something isn't working labels Aug 7, 2026

@membphis membphis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: ai-proxy-multi: Variable $request_llm_model reset in fallback_strategy

3 participants