Skip to content

perf(core): skip the X-Forwarded-* rewrite when there is nothing to neutralize - #13783

Open
AlinsRan wants to merge 1 commit into
apache:masterfrom
AlinsRan:perf/x-forwarded-fast-path
Open

perf(core): skip the X-Forwarded-* rewrite when there is nothing to neutralize#13783
AlinsRan wants to merge 1 commit into
apache:masterfrom
AlinsRan:perf/x-forwarded-fast-path

Conversation

@AlinsRan

@AlinsRan AlinsRan commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Description

is_trusted() returns false whenever apisix.trusted_addresses is unset, and
unset is the default. So since #12551 every request on a default install runs
the "untrusted peer" branch of handle_x_forwarded_headers: four
ngx.req.set_header() calls plus a string parse. You had to opt into the
feature to stop paying for it.

Behind that branch, set_upstream_x_forwarded_headers reads three
http_x_forwarded_* variables through api_ctx.var, and core/ctx.lua's
__index runs key:lower() and an ngx.re.gsub() for every http_* key
without caching a nil result — three PCRE calls per request, on the trusted path
too.

The security requirement is to neutralize a value the client supplied. When the
client supplied none, there is nothing to neutralize.

Change

  • Fast path when the request carries none of X-Forwarded-Proto/Host/Port/For
    and Forwarded. The two Host-derived values the set $var_x_forwarded_* $host/$server_port defaults would lose are still written: an explicit port in
    Host (the port the client actually connected to, and the only correct
    X-Forwarded-Port on a port-mapped deployment), and a Host differing from the
    lower-cased $host.
  • Skip the $realip_remote_addr lookup and the IP match when no trust boundary
    is configured; clear Forwarded / X-Forwarded-For only when present.
  • Read request headers through ngx.var, avoiding the api_ctx.var regex.

Benchmark~10% on a default install (single worker, wrk2 at saturation,
upstream on localhost, master and this branch back to back). No measurable
change once a trust boundary is configured, or on the slow path.

Behaviour

Trust semantics are unchanged. A forged X-Forwarded-* or Forwarded from
an untrusted peer is still overridden or cleared (trusted-addresses.t 1, 8, 9
unchanged, 13/14 new; proxy-mirror4.t TEST 2). An empty-valued header reads
back as "", truthy in Lua, so it takes the slow path — the fast path cannot be
entered while carrying a forgeable value. Values reaching the upstream through
location / and @disable_proxy_buffering are identical to today in all trust
states.

Three deliberate deviations:

  1. gRPC, Dubbo and mirrored backends no longer receive injected
    X-Forwarded-*
    when the client sends none. Those exits read
    r->headers_in rather than $var_x_forwarded_*: ngx_http_grpc_module
    always passes request headers and @grpc_pass has no grpc_set_header,
    @dubbo_pass sets dubbo_pass_all_headers on, and ngx_http_subrequest
    copies the header list. This restores pre-fix: only trust X-Forwarded-* headers from trusted_addresses #12551 behaviour. Closing the gap
    needs grpc_set_header / proxy_set_header on those locations, which would
    also start sending these headers to trusted peers that send none — a
    separate behaviour change. proxy-mirror4.t TEST 1 pins the current
    behaviour; it fails on master.

  2. A plugin-set X-Forwarded-* now reaches the upstream in the untrusted
    states.
    core.request.set_header invalidates ctx.var under
    http_x-forwarded-port (hyphens) while ctx.lua caches
    http_x_forwarded_port (underscores), so the gateway's cached value won over
    the plugin's. Reading ngx.var bypasses the stale cache. Not a trust
    weakening — the value comes from plugin configuration, not client input.
    proxy-rewrite2.t TEST 10 fails on master and passes here. The underlying
    key mismatch in core/request.lua is untouched; it is also the mechanism
    behind help request: Unable to remove or override X-Forwarded-Host header per-route using any plugin #13753.

  3. Request headers are no longer synthesised when the client sends none.
    ctx.var.http_x_forwarded_proto returns nil rather than a value APISIX made
    up — pre-fix: only trust X-Forwarded-* headers from trusted_addresses #12551 behaviour, and what a trusted peer already gets.
    redirect.lua falls back to get_scheme() so it is unaffected. Plugins
    forwarding the whole header table (opa, authz-casbin, batch-requests,
    serverless/generic-upstream, soap) now send a different set — an OPA
    policy keyed on x-forwarded-proto would see it absent. A route matching on
    vars: [["http_x_forwarded_proto", "==", "http"]] stops matching.

Also: set_upstream_x_forwarded_headers now reads ngx.var, so a plugin that
assigned ctx.var.http_x_forwarded_proto as cache only would no longer be
propagated. No in-tree plugin does this.

Which issue(s) this PR fixes

Follow-up to #12551. Deviation 2 is the mechanism behind #13753, though this PR
only bypasses it rather than fixing the core/request.lua cache-key mismatch.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change

trusted_addresses is not documented anywhere in this repo — absent from
conf/config.yaml.example and from docs/ — so there was nothing to update.
That gap predates this PR; the option is currently discoverable only from
apisix/cli/schema.lua and the #12551 changelog.

  • I have verified that this change is backward compatible (if not, please discuss on the APISIX mailing list first)

…eutralize

`is_trusted()` returns false whenever `trusted_addresses` is unset, and unset
is the default, so every request on a default install runs the "untrusted peer"
branch of `handle_x_forwarded_headers`: four `ngx.req.set_header()` calls plus
a string parse. Configuring a trust boundary was the only way out, i.e. you had
to opt into the feature to stop paying for it.

A second cost is hidden behind that branch. `set_upstream_x_forwarded_headers`
reads three `http_x_forwarded_*` variables through `api_ctx.var`, and
`core/ctx.lua`'s `__index` runs a `key:lower()` and an `ngx.re.gsub()` for every
`http_*` key and then does not cache a nil result. A request carrying no
X-Forwarded-* header -- the common case -- pays three PCRE calls on every
request, on the trusted path too.

Neither is needed. The security requirement is to neutralize a value the client
supplied; when the client supplied none there is nothing to neutralize. So:

- `handle_x_forwarded_headers` returns early when the request carries none of
  X-Forwarded-Proto/Host/Port/For and Forwarded. It still writes
  `var_x_forwarded_host` / `var_x_forwarded_port` in the two cases where the
  `set $var_x_forwarded_* $host/$server_port` defaults in ngx_tpl.lua would lose
  information: an explicit port in the Host header (the port the client actually
  connected to, and the only correct X-Forwarded-Port on a port-mapped
  deployment), and a Host that differs from the lower-cased `$host`.
- The `$realip_remote_addr` lookup and the IP match are skipped when no trust
  boundary is configured; the header reads sit below the trusted early-return;
  Forwarded and X-Forwarded-For are only cleared when actually present.
- Both functions read the request headers through `ngx.var`, skipping the
  `api_ctx.var` wrapper's per-key regex.

Single worker, wrk2 at saturation, upstream on localhost, master and this
commit measured back to back, two rounds:

| scenario                               | master      | this commit |
|----------------------------------------|-------------|-------------|
| default, client sends no X-Forwarded-* | 79.4k 77.5k | 89.9k 90.5k |

Trust semantics are unchanged: a forged X-Forwarded-* or Forwarded from an
untrusted peer is still overridden or cleared. An empty-valued header
(`X-Forwarded-Proto:`) reads back as "", which is truthy in Lua, so it takes
the slow path -- the fast path cannot be entered while carrying a forgeable
value.

Two behaviour changes, both covered by new tests:

Requests reaching `@grpc_pass`, `@dubbo_pass` or the mirror subrequest read
X-Forwarded-* from `r->headers_in` rather than from `$var_x_forwarded_*`
(ngx_http_grpc_module always passes request headers, `dubbo_pass_all_headers`
is on, and `ngx_http_subrequest` copies the header list), so one that carries no
X-Forwarded-* now reaches its backend without them. This is what those backends
saw before apache#12551, which added the injection; closing the gap would mean adding
`grpc_set_header`/`proxy_set_header` to those locations, which would also start
sending these headers for trusted peers that send none -- a separate change.

A plugin-set X-Forwarded-* now reaches the upstream in the untrusted states.
`core.request.set_header` invalidates `ctx.var` under `http_x-forwarded-port`
(hyphens) while `ctx.lua`'s `__index` caches `http_x_forwarded_port`
(underscores), so the value cached by `handle_x_forwarded_headers` won over the
plugin's and the upstream got the gateway's own value. Reading `ngx.var`
bypasses the stale cache; the underlying key mismatch in `core/request.lua` is
left for a separate fix.
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. performance generate flamegraph for the current PR labels Aug 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes APISIX’s handling of X-Forwarded-* / Forwarded by adding a fast path when there is no client-supplied forgeable value to neutralize, and by avoiding expensive api_ctx.var lookups (including regex work) when reading request headers. It keeps the existing trust semantics while improving default-install request performance and adds regression tests for the edge cases called out in the PR description.

Changes:

  • Add a fast path in handle_x_forwarded_headers to skip rewrites when no X-Forwarded-*/Forwarded headers are present, while still preserving upstream-visible Host-derived details (explicit port and Host casing) where needed.
  • Avoid per-request trusted-address matching when no trust boundary is configured; read request headers via ngx.var to avoid api_ctx.var’s http_* lookup overhead.
  • Add/extend tests to validate trust-boundary behavior, plugin overrides, and mirror/grpc/dubbo header-copy semantics.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
apisix/init.lua Adds fast path + switches header reads to ngx.var to reduce per-request overhead while preserving trust semantics.
t/core/trusted-addresses.t Extends coverage for “no trust boundary” vs configured trust boundary cases, including no-header and partial-header scenarios.
t/plugin/proxy-rewrite2.t Adds regression tests ensuring plugin-set X-Forwarded-Port propagates when the client sends none (including when trusted_addresses is unset).
t/plugin/proxy-mirror4.t Adds tests validating that mirror subrequests observe injected headers only when the client sent forgeable values (and that forged values are neutralized).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@membphis membphis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve the established X-Forwarded-* visibility

This fast path changes observable behavior when an untrusted request carries none of these headers. Route variables, plugins that forward the request header table, and gRPC, Dubbo, or mirror paths that read r->headers_in can now see a different header set. Existing routes or policies may stop matching, and some upstreams may lose headers. Please preserve the existing request/header visibility while optimizing the expensive lookups, or treat this as an explicit breaking change with a migration path and compatibility coverage for every affected exit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance generate flamegraph for the current PR size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants