diff --git a/docs/en/solutions/How_to_Preserve_a_Non_Default_Backend_Port_in_the_Upstream_Host_Header_with_Envoy_Gateway_on_ACP.md b/docs/en/solutions/How_to_Preserve_a_Non_Default_Backend_Port_in_the_Upstream_Host_Header_with_Envoy_Gateway_on_ACP.md new file mode 100644 index 000000000..604cace9f --- /dev/null +++ b/docs/en/solutions/How_to_Preserve_a_Non_Default_Backend_Port_in_the_Upstream_Host_Header_with_Envoy_Gateway_on_ACP.md @@ -0,0 +1,472 @@ +--- +kind: + - Solution +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x and later +tags: + - LB +--- + +# How to Preserve a Non-Default Backend Port in the Upstream Host Header with Envoy Gateway on ACP + +## Issue + +An Envoy Gateway route forwards traffic to an HTTPS backend that listens on a non-default port, such as `7448`: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: Backend +metadata: + name: external-model + namespace: model-serving +spec: + type: Endpoints + endpoints: + - fqdn: + hostname: model.example.com + port: 7448 +``` + +The route also uses an Envoy Gateway `HTTPRouteFilter` that rewrites the upstream hostname to the selected backend: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: HTTPRouteFilter +metadata: + name: rewrite-host-to-backend + namespace: model-serving +spec: + urlRewrite: + hostname: + type: Backend +``` + +The real TCP/TLS connection reaches port `7448`, but the upstream HTTP `Host` header or HTTP/2 `:authority` does not contain `:7448`. A strict upstream virtual host or inner gateway may reject the request because it expects: + +```text +Host: model.example.com:7448 +``` + +Possible upstream errors include: + +```text +403 Host forbidden model.example.com:443 +``` + +or an error that reports only the hostname without a port. + +The error text alone does not prove that Envoy connected to TCP port `443`. The network destination port and the HTTP authority are independent values and must be verified separately. + +## Environment + +This solution applies when all of the following conditions are true: + +- Alauda Container Platform 4.3.x or later is used. +- The traffic is handled by Envoy Gateway. +- The backend is represented by `Backend` from `gateway.envoyproxy.io/v1alpha1`. +- The backend listens on a non-default port. +- `HTTPRouteFilter.spec.urlRewrite.hostname.type` is `Backend`, either configured directly or generated by another controller such as Envoy AI Gateway. +- The upstream server requires the non-default port to be present in `Host` or `:authority`. + +The workaround in this document was validated on an ACP 4.3 test cluster with: + +```text +Envoy Gateway: v1.8.0 +Envoy Proxy: v1.38.0 +``` + +The exact `EnvoyPatchPolicy` resource and JSON Patch paths shown in this document are validated for Envoy Gateway v1.8.0. Do not copy the paths unchanged to Envoy Gateway v1.7.x, v1.9.x, or another version. First inspect that version's generated RouteConfiguration and adapt the resource name and JSON Patch paths. + +`EnvoyPatchPolicy` modifies generated xDS resources and is an unstable advanced API. Inspect the generated configuration again after changing routes or upgrading Envoy Gateway, including upgrades between v1.8.x patch releases. + +## Root Cause + +Four different values participate in this request path: + +| Value | Example | Purpose | +| --- | --- | --- | +| Client entry port | `32080` | NodePort or other port used by the client to reach Envoy | +| Backend socket port | `7448` | TCP/TLS destination from Envoy to the backend | +| HTTP `Host` or `:authority` | `model.example.com` | HTTP virtual-host selection at the upstream server | +| TLS SNI | `model.example.com` | TLS certificate and virtual-server selection; it never contains a port | + +The Backend endpoint port is translated into the Envoy Cluster socket address: + +```json +{ + "address": "model.example.com", + "port_value": 7448 +} +``` + +However, `HTTPRouteFilter` with `hostname.type: Backend` is translated into the following Envoy RouteAction: + +```yaml +auto_host_rewrite: true +``` + +Envoy's automatic host rewrite uses the endpoint hostname. It does not append the endpoint socket port. Therefore, the generated authority and the socket destination can be: + +```text +TCP destination: model.example.com:7448 +HTTP Host: model.example.com +TLS SNI: model.example.com +``` + +An upstream component may normalize a missing HTTPS port to `443` when producing its error message. Capture the request at the upstream server or inspect Envoy xDS before concluding that Envoy connected to port `443`. + +## Diagnostic Steps + +### 1. Confirm the Backend Endpoint Port + +Inspect the Backend: + +```bash +kubectl get backend external-model -n model-serving -o yaml +``` + +Confirm that the endpoint contains the required port: + +```yaml +spec: + endpoints: + - fqdn: + hostname: model.example.com + port: 7448 +``` + +Do not change the hostname to `model.example.com:7448`. The Backend CRD expects a DNS hostname in this field and does not accept a colon and port. + +### 2. Confirm That Backend Host Rewrite Is Enabled + +Inspect the generated or manually created filter: + +```bash +kubectl get httproutefilter -n model-serving -o yaml +``` + +Look for: + +```yaml +urlRewrite: + hostname: + type: Backend +``` + +Then identify the `HTTPRoute` that references the filter: + +```bash +kubectl get httproute -n model-serving -o yaml +``` + +### 3. Inspect the Generated RouteConfiguration + +Set the Gateway namespace and name: + +```bash +export GATEWAY_NAMESPACE=model-serving +export GATEWAY_NAME= +``` + +Find the Envoy data-plane Pod: + +```bash +export ENVOY_POD=$( + kubectl get pod -n "$GATEWAY_NAMESPACE" \ + -l gateway.envoyproxy.io/owning-gateway-name="$GATEWAY_NAME" \ + -o jsonpath='{.items[0].metadata.name}' +) +``` + +Forward the Envoy administration port in a separate terminal: + +```bash +kubectl port-forward -n "$GATEWAY_NAMESPACE" pod/"$ENVOY_POD" 19000:19000 +``` + +Read the dynamic route configuration: + +```bash +curl -sS 'http://127.0.0.1:19000/config_dump?resource=dynamic_route_configs' \ + | jq ' + .configs[].route_config + | . as $route_configuration + | range(0; (.virtual_hosts | length)) as $virtual_host_index + | .virtual_hosts[$virtual_host_index] as $virtual_host + | range(0; ($virtual_host.routes | length)) as $route_index + | $virtual_host.routes[$route_index] + | select(.route.auto_host_rewrite? == true) + | { + route_configuration: $route_configuration.name, + virtual_host: $virtual_host.name, + virtual_host_index: $virtual_host_index, + route_index: $route_index, + route_name: .name, + route: .route + } + ' +``` + +An affected route contains: + +```json +{ + "route_configuration": "model-serving/model-gateway/http", + "virtual_host": "model-serving/model-gateway/http/*", + "virtual_host_index": 0, + "route_index": 0, + "route_name": "httproute/model-serving/model-route/rule/0/match/0/*", + "route": { + "cluster": "httproute/model-serving/model-route/rule/0", + "auto_host_rewrite": true + } +} +``` + +Record all of the following values. They are required to construct a narrowly scoped patch: + +- RouteConfiguration name, normally `//`. +- Virtual-host array index containing the affected route. +- Route array index containing the affected rule. +- Generated route name, normally beginning with `httproute/`. + +### 4. Confirm the Real Backend Socket Port + +With the administration port still forwarded, inspect active clusters: + +```bash +curl -sS 'http://127.0.0.1:19000/config_dump?resource=dynamic_active_clusters' \ + | jq '.. | objects | select(has("socket_address")) | .socket_address' +``` + +The affected endpoint should contain: + +```json +{ + "address": "model.example.com", + "port_value": 7448 +} +``` + +This proves that Envoy's TCP/TLS destination is port `7448`, regardless of the HTTP authority reported by the upstream application. + +### 5. Verify the Upstream Host Requirement Directly + +If the upstream IP is known, use curl to separate the TLS destination from the HTTP Host value. + +Send a request to port `7448` with a Host header that does not include the port: + +```bash +curl -vk \ + --resolve model.example.com:7448: \ + 'https://model.example.com:7448/' \ + -H 'Host: model.example.com' +``` + +Then send the expected Host header: + +```bash +curl -vk \ + --resolve model.example.com:7448: \ + 'https://model.example.com:7448/' \ + -H 'Host: model.example.com:7448' +``` + +If only the second request succeeds, the upstream virtual host is performing strict authority matching. + +When inspecting NGINX logs, use `$http_host` to preserve the raw Host header. `$host` can normalize the value and omit the port. + +## Resolution + +### Option 1: Make the Upstream Host Match Port-Independent + +When the inner gateway or upstream virtual host can be changed, configure it to match the hostname independently of the port or accept both of these authorities: + +```text +model.example.com +model.example.com:7448 +``` + +This avoids coupling the application configuration to Envoy's generated xDS structure and is the preferred long-term solution. + +### Option 2: Set host_rewrite_literal with EnvoyPatchPolicy + +Use this workaround when the upstream server cannot be changed and must receive exactly: + +```text +model.example.com:7448 +``` + +The configuration in this section is for Envoy Gateway v1.8.0. For any other Envoy Gateway version, repeat the diagnostic steps and construct the patch from that version's xDS output. + +#### Step 1: Enable EnvoyPatchPolicy + +`EnvoyPatchPolicy` is disabled by default. Add `enableEnvoyPatchPolicy: true` to the `EnvoyGatewayCtl` instance that manages the affected GatewayClass. + +If the same instance uses Backend resources, retain `enableBackend: true`: + +```yaml +apiVersion: envoy-gateway.alauda.io/v1 +kind: EnvoyGatewayCtl +metadata: + name: + namespace: +spec: + config: + envoyGateway: + extensionApis: + enableBackend: true + enableEnvoyPatchPolicy: true +``` + +Merge these fields into the existing `EnvoyGatewayCtl`; do not replace its other configuration and do not directly edit the generated Envoy Gateway ConfigMap or Deployment. + +Wait until the Envoy Gateway control-plane Deployment is ready again before applying the policy. + +#### Step 2: Create a Route-Specific EnvoyPatchPolicy + +The following example assumes that the diagnostic configuration dump found: + +| Item | Example value | +| --- | --- | +| Gateway namespace | `model-serving` | +| Gateway name | `model-gateway` | +| Listener name | `http` | +| RouteConfiguration | `model-serving/model-gateway/http` | +| Virtual-host index | `0` | +| Route index | `0` | +| Required authority | `model.example.com:7448` | + +Create the policy in the same namespace as the Gateway: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: EnvoyPatchPolicy +metadata: + name: upstream-authority-with-port + namespace: model-serving +spec: + targetRef: + group: gateway.networking.k8s.io + kind: Gateway + name: model-gateway + type: JSONPatch + jsonPatches: + - type: type.googleapis.com/envoy.config.route.v3.RouteConfiguration + name: model-serving/model-gateway/http + operation: + op: remove + path: /virtual_hosts/0/routes/0/route/auto_host_rewrite + - type: type.googleapis.com/envoy.config.route.v3.RouteConfiguration + name: model-serving/model-gateway/http + operation: + op: add + path: /virtual_hosts/0/routes/0/route/host_rewrite_literal + value: model.example.com:7448 +``` + +The `remove` operation is required because `auto_host_rewrite` and `host_rewrite_literal` are mutually exclusive RouteAction settings. + +Do not copy the array indexes from this example without checking the target Gateway's xDS dump. A Gateway with multiple virtual hosts, routes, matches, or generated controller routes can use different indexes. + +Apply the policy: + +```bash +kubectl apply -f upstream-authority-with-port.yaml +``` + +#### Step 3: Verify Policy Status + +Inspect the policy: + +```bash +kubectl get envoypatchpolicy upstream-authority-with-port \ + -n model-serving -o yaml +``` + +Both conditions must be true: + +```text +Accepted=True +Programmed=True +``` + +If `Programmed=False`, use the condition message as the primary error. Common causes include an incorrect RouteConfiguration name, an incorrect JSON Patch path, or an Envoy Gateway instance that does not have EnvoyPatchPolicy enabled. + +#### Step 4: Verify the Patched xDS Configuration + +Read the RouteConfiguration again. The affected RouteAction should now contain: + +```json +{ + "cluster": "httproute/model-serving/model-route/rule/0", + "host_rewrite_literal": "model.example.com:7448" +} +``` + +It must no longer contain: + +```yaml +auto_host_rewrite: true +``` + +#### Step 5: Verify the Request at the Backend + +Send a request through the Gateway and inspect the raw authority at the backend. + +The expected values are: + +```text +Backend server port: 7448 +HTTP Host: model.example.com:7448 +TLS SNI: model.example.com +``` + +The following result was observed in the validation environment: + +```json +{ + "server_port": "7448", + "http_host": "model.example.com:7448", + "tls_sni": "model.example.com" +} +``` + +The active Envoy Cluster continued to use port `7448`: + +```json +{ + "address": "model-backend.model-serving.svc.cluster.local", + "port_value": 7448 +} +``` + +## Rollback + +Delete only the workaround policy: + +```bash +kubectl delete envoypatchpolicy upstream-authority-with-port -n model-serving +``` + +Envoy Gateway will regenerate the route from the original Gateway API and `HTTPRouteFilter` resources, restoring `auto_host_rewrite` behavior. The Backend, Gateway, HTTPRoute, and Envoy Gateway instance do not need to be deleted. + +## Limitations and Operational Considerations + +- `EnvoyPatchPolicy` operates on generated xDS and is intentionally an unstable, advanced API. +- RouteConfiguration names and array indexes can change after an Envoy Gateway upgrade or after editing Gateway and HTTPRoute resources. +- A policy targeting a Gateway can modify any xDS resource generated for that Gateway. Keep every patch restricted to the exact RouteConfiguration and route path identified from the configuration dump. +- Recheck `Accepted` and `Programmed` after every Gateway, HTTPRoute, AI Gateway route, or Envoy Gateway version change. +- A controller such as Envoy AI Gateway may continue reconciling the source `HTTPRouteFilter`. The PatchPolicy is applied to the resulting xDS configuration and does not modify that source object. +- TLS SNI must remain a hostname without a port. Configure the expected TLS hostname through `BackendTLSPolicy`; do not include `:7448` in SNI-related fields. +- If the backend authority can be made port-independent, prefer changing the upstream matching rule instead of retaining a long-term xDS patch. + +## References + +- [ACP Envoy Gateway Operator: Advanced Config Via EnvoyGatewayCtl](https://docs-dev.alauda.cn/container_platform/main/networking/operators/envoy_gateway_operator#envoygatewayctl) +- [Envoy Gateway v1.8: Envoy Patch Policy](https://gateway.envoyproxy.io/v1.8/tasks/extensibility/envoy-patch-policy/) +- [Envoy issue #26022: auto_host_rewrite does not preserve the endpoint port](https://github.com/envoyproxy/envoy/issues/26022) +- [Envoy Gateway issue #8823: Support for Port in auto_host_rewrite](https://github.com/envoyproxy/gateway/issues/8823) +- [Envoy AI Gateway issue #2500: Generated Backend host rewrite drops non-default port](https://github.com/envoyproxy/ai-gateway/issues/2500)