Preserve Client Source IP in Virtual Router

[senthilnathan-am]

Why virtual router is not preserving source IP during port forwarding?

[DaanHoogland]

Ithought it was @senthilnathan-am . can you add the scenario you are using here, for people to reproduce the possible issue, please?

[weizhouapache]

Why virtual router is not preserving source IP during port forwarding?

Do you use vpc ?

[zap51]

Hi @senthilnathan-am,
Please try capturing "X-Forwarded-For" header sent by envoy on your backend server which should have the end-client IP.
Due to the nature of Kubernetes & depending on the configuration, you'll likely not see end-client IPs due to Source NAT being done by nodes and pods acting as reverse proxies.

Regards,
Jayanth

[senthilnathan-am]

Hi @senthilnathan-am, Please try capturing "X-Forwarded-For" header sent by envoy on your backend server which should have the end-client IP. Due to the nature of Kubernetes & depending on the configuration, you'll likely not see end-client IPs due to Source NAT being done by nodes and pods acting as reverse proxies.

Regards, Jayanth

Thanks for the reply @zap51

I have already used XFF and also enabled real-ip module in the nginx which is the server running behind istio. Below are the directives used in the configuration and also provided the sample log.

log_format  main  '$http_x_forwarded_for $remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$remote_addr"';


    real_ip_header X-Forwarded-For;
    real_ip_recursive on;
    set_real_ip_from 127.0.0.6;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

10.40.0.0 10.40.0.0 - - [16/Aug/2024:07:00:21 +0000] "GET /css/ajax-loader.gif HTTP/1.1" 404 548 "http://ams.com/css/slick-theme.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "10.40.0.0"

In the above log, you could see both XFF and real-ip headers are showing same IP which is Weave interface IP not the actual client IP.

[zap51]

@senthilnathan-am,
Thanks for the update. Ignoring proxy_set_header in your backend server at the moment, I see that $http_x_forwarded_for is still being read as only one address.
So, please make sure the XFF configuration is properly configured at Istio. You should try increasing numTrustedProxies, which ideally adds the request with what's that is actually coming with X-Forwarded-For to the first and then appends the SNAT IP separated by a comma like the below

<ip_in_xff_req_header>, 10.40.0.0 10.40.0.0 ...

Also, VR is not considered a proxy, hence the amount in numTrustedProxies makes less sense unless there are more number of http-parsing proxies which add XFF request headers in front of the VR.

[senthilnathan-am]

Finally found a solution to preserve client IP for Kubernetes platform in ACS.

Below are the things to be done:

1. For K8s, if it setup manually then it is recommended to use Load Balancing rules instead of port forwarding rules for network traffic diversion.
2. If Istio service mesh is used to expose the cluster ports then proxy protocol has to be enabled in VR haproxy service in the appropriate listen section, just by adding 'send-proxy-v2' after check.
3. Restart the haproxy service in VR. Also, make sure the proxy protocol is enabled in Istio level.

Now, actual client source IP will get displayed in the backend server/pod log as expected.

[zap51]

@senthilnathan-am,
That's correct. However, I would recommend you to use annotations instead of manually tweaking the VR. Please see https://docs.cloudstack.apache.org/en/latest/plugins/cloudstack-kubernetes-provider.html#usage
This is one of the recommended ways to have the applications load-balanced when you're using CKS or when deploying the provider manually.

Correct, PROXY protocol will contain additional information which your applications should be configured to accept.

Regards,
Jayanth

[senthilnathan-am]

@senthilnathan-am, That's correct. However, I would recommend you to use annotations instead of manually tweaking the VR. Please see https://docs.cloudstack.apache.org/en/latest/plugins/cloudstack-kubernetes-provider.html#usage This is one of the recommended ways to have the applications load-balanced when you're using CKS or when deploying the provider manually.

Correct, PROXY protocol will contain additional information which your applications should be configured to accept.

Regards, Jayanth

Thanks for the suggestions @zap51 . I am not sure whether that annotation will support in Istio, anyway will give a try.

[senthilnathan-am]

@zap51
As expected, annotation is not working for Istio.

[zap51]

@zap51
As expected, annotation is not working for Istio.

This only works when using CloudStack Kubernetes Provider. I see that you'd manually configured the LB and its rules.

Regards,
Jayanth

[weizhouapache]

haproxy supports transparent mode, however, it requires some changes in route table, iptables rules and haproxy config
refer to https://www.haproxy.com/blog/howto-transparent-proxying-and-binding-with-haproxy-and-aloha-load-balancer

there is also a related PR #7484

[senthilnathan-am]

Closing this...