perf(core): skip the X-Forwarded-* rewrite when there is nothing to neutralize - #13783
perf(core): skip the X-Forwarded-* rewrite when there is nothing to neutralize#13783AlinsRan wants to merge 1 commit into
Conversation
…eutralize `is_trusted()` returns false whenever `trusted_addresses` is unset, and unset is the default, so every request on a default install runs the "untrusted peer" branch of `handle_x_forwarded_headers`: four `ngx.req.set_header()` calls plus a string parse. Configuring a trust boundary was the only way out, i.e. you had to opt into the feature to stop paying for it. A second cost is hidden behind that branch. `set_upstream_x_forwarded_headers` reads three `http_x_forwarded_*` variables through `api_ctx.var`, and `core/ctx.lua`'s `__index` runs a `key:lower()` and an `ngx.re.gsub()` for every `http_*` key and then does not cache a nil result. A request carrying no X-Forwarded-* header -- the common case -- pays three PCRE calls on every request, on the trusted path too. Neither is needed. The security requirement is to neutralize a value the client supplied; when the client supplied none there is nothing to neutralize. So: - `handle_x_forwarded_headers` returns early when the request carries none of X-Forwarded-Proto/Host/Port/For and Forwarded. It still writes `var_x_forwarded_host` / `var_x_forwarded_port` in the two cases where the `set $var_x_forwarded_* $host/$server_port` defaults in ngx_tpl.lua would lose information: an explicit port in the Host header (the port the client actually connected to, and the only correct X-Forwarded-Port on a port-mapped deployment), and a Host that differs from the lower-cased `$host`. - The `$realip_remote_addr` lookup and the IP match are skipped when no trust boundary is configured; the header reads sit below the trusted early-return; Forwarded and X-Forwarded-For are only cleared when actually present. - Both functions read the request headers through `ngx.var`, skipping the `api_ctx.var` wrapper's per-key regex. Single worker, wrk2 at saturation, upstream on localhost, master and this commit measured back to back, two rounds: | scenario | master | this commit | |----------------------------------------|-------------|-------------| | default, client sends no X-Forwarded-* | 79.4k 77.5k | 89.9k 90.5k | Trust semantics are unchanged: a forged X-Forwarded-* or Forwarded from an untrusted peer is still overridden or cleared. An empty-valued header (`X-Forwarded-Proto:`) reads back as "", which is truthy in Lua, so it takes the slow path -- the fast path cannot be entered while carrying a forgeable value. Two behaviour changes, both covered by new tests: Requests reaching `@grpc_pass`, `@dubbo_pass` or the mirror subrequest read X-Forwarded-* from `r->headers_in` rather than from `$var_x_forwarded_*` (ngx_http_grpc_module always passes request headers, `dubbo_pass_all_headers` is on, and `ngx_http_subrequest` copies the header list), so one that carries no X-Forwarded-* now reaches its backend without them. This is what those backends saw before apache#12551, which added the injection; closing the gap would mean adding `grpc_set_header`/`proxy_set_header` to those locations, which would also start sending these headers for trusted peers that send none -- a separate change. A plugin-set X-Forwarded-* now reaches the upstream in the untrusted states. `core.request.set_header` invalidates `ctx.var` under `http_x-forwarded-port` (hyphens) while `ctx.lua`'s `__index` caches `http_x_forwarded_port` (underscores), so the value cached by `handle_x_forwarded_headers` won over the plugin's and the upstream got the gateway's own value. Reading `ngx.var` bypasses the stale cache; the underlying key mismatch in `core/request.lua` is left for a separate fix.
There was a problem hiding this comment.
Pull request overview
This PR optimizes APISIX’s handling of X-Forwarded-* / Forwarded by adding a fast path when there is no client-supplied forgeable value to neutralize, and by avoiding expensive api_ctx.var lookups (including regex work) when reading request headers. It keeps the existing trust semantics while improving default-install request performance and adds regression tests for the edge cases called out in the PR description.
Changes:
- Add a fast path in
handle_x_forwarded_headersto skip rewrites when noX-Forwarded-*/Forwardedheaders are present, while still preserving upstream-visible Host-derived details (explicit port and Host casing) where needed. - Avoid per-request trusted-address matching when no trust boundary is configured; read request headers via
ngx.varto avoidapi_ctx.var’shttp_*lookup overhead. - Add/extend tests to validate trust-boundary behavior, plugin overrides, and mirror/grpc/dubbo header-copy semantics.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| apisix/init.lua | Adds fast path + switches header reads to ngx.var to reduce per-request overhead while preserving trust semantics. |
| t/core/trusted-addresses.t | Extends coverage for “no trust boundary” vs configured trust boundary cases, including no-header and partial-header scenarios. |
| t/plugin/proxy-rewrite2.t | Adds regression tests ensuring plugin-set X-Forwarded-Port propagates when the client sends none (including when trusted_addresses is unset). |
| t/plugin/proxy-mirror4.t | Adds tests validating that mirror subrequests observe injected headers only when the client sent forgeable values (and that forged values are neutralized). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
membphis
left a comment
There was a problem hiding this comment.
[P1] Preserve the established X-Forwarded-* visibility
This fast path changes observable behavior when an untrusted request carries none of these headers. Route variables, plugins that forward the request header table, and gRPC, Dubbo, or mirror paths that read r->headers_in can now see a different header set. Existing routes or policies may stop matching, and some upstreams may lose headers. Please preserve the existing request/header visibility while optimizing the expensive lookups, or treat this as an explicit breaking change with a migration path and compatibility coverage for every affected exit.
Description
is_trusted()returnsfalsewheneverapisix.trusted_addressesis unset, andunset is the default. So since #12551 every request on a default install runs
the "untrusted peer" branch of
handle_x_forwarded_headers: fourngx.req.set_header()calls plus a string parse. You had to opt into thefeature to stop paying for it.
Behind that branch,
set_upstream_x_forwarded_headersreads threehttp_x_forwarded_*variables throughapi_ctx.var, andcore/ctx.lua's__indexrunskey:lower()and anngx.re.gsub()for everyhttp_*keywithout caching a nil result — three PCRE calls per request, on the trusted path
too.
The security requirement is to neutralize a value the client supplied. When the
client supplied none, there is nothing to neutralize.
Change
X-Forwarded-Proto/Host/Port/Forand
Forwarded. The two Host-derived values theset $var_x_forwarded_* $host/$server_portdefaults would lose are still written: an explicit port inHost (the port the client actually connected to, and the only correct
X-Forwarded-Porton a port-mapped deployment), and a Host differing from thelower-cased
$host.$realip_remote_addrlookup and the IP match when no trust boundaryis configured; clear
Forwarded/X-Forwarded-Foronly when present.ngx.var, avoiding theapi_ctx.varregex.Benchmark — ~10% on a default install (single worker, wrk2 at saturation,
upstream on localhost,
masterand this branch back to back). No measurablechange once a trust boundary is configured, or on the slow path.
Behaviour
Trust semantics are unchanged. A forged
X-Forwarded-*orForwardedfroman untrusted peer is still overridden or cleared (
trusted-addresses.t1, 8, 9unchanged, 13/14 new;
proxy-mirror4.tTEST 2). An empty-valued header readsback as
"", truthy in Lua, so it takes the slow path — the fast path cannot beentered while carrying a forgeable value. Values reaching the upstream through
location /and@disable_proxy_bufferingare identical to today in all truststates.
Three deliberate deviations:
gRPC, Dubbo and mirrored backends no longer receive injected
X-Forwarded-*when the client sends none. Those exits readr->headers_inrather than$var_x_forwarded_*:ngx_http_grpc_modulealways passes request headers and
@grpc_passhas nogrpc_set_header,@dubbo_passsetsdubbo_pass_all_headers on, andngx_http_subrequestcopies the header list. This restores pre-fix: only trust
X-Forwarded-*headers fromtrusted_addresses#12551 behaviour. Closing the gapneeds
grpc_set_header/proxy_set_headeron those locations, which wouldalso start sending these headers to trusted peers that send none — a
separate behaviour change.
proxy-mirror4.tTEST 1 pins the currentbehaviour; it fails on
master.A plugin-set
X-Forwarded-*now reaches the upstream in the untrustedstates.
core.request.set_headerinvalidatesctx.varunderhttp_x-forwarded-port(hyphens) whilectx.luacacheshttp_x_forwarded_port(underscores), so the gateway's cached value won overthe plugin's. Reading
ngx.varbypasses the stale cache. Not a trustweakening — the value comes from plugin configuration, not client input.
proxy-rewrite2.tTEST 10 fails onmasterand passes here. The underlyingkey mismatch in
core/request.luais untouched; it is also the mechanismbehind help request: Unable to remove or override
X-Forwarded-Hostheader per-route using any plugin #13753.Request headers are no longer synthesised when the client sends none.
ctx.var.http_x_forwarded_protoreturnsnilrather than a value APISIX madeup — pre-fix: only trust
X-Forwarded-*headers fromtrusted_addresses#12551 behaviour, and what a trusted peer already gets.redirect.luafalls back toget_scheme()so it is unaffected. Pluginsforwarding the whole header table (
opa,authz-casbin,batch-requests,serverless/generic-upstream,soap) now send a different set — an OPApolicy keyed on
x-forwarded-protowould see it absent. A route matching onvars: [["http_x_forwarded_proto", "==", "http"]]stops matching.Also:
set_upstream_x_forwarded_headersnow readsngx.var, so a plugin thatassigned
ctx.var.http_x_forwarded_protoas cache only would no longer bepropagated. No in-tree plugin does this.
Which issue(s) this PR fixes
Follow-up to #12551. Deviation 2 is the mechanism behind #13753, though this PR
only bypasses it rather than fixing the
core/request.luacache-key mismatch.Checklist
trusted_addressesis not documented anywhere in this repo — absent fromconf/config.yaml.exampleand fromdocs/— so there was nothing to update.That gap predates this PR; the option is currently discoverable only from
apisix/cli/schema.luaand the #12551 changelog.