VPC VR: static-NAT SNAT rule can precede the site-to-site VPN NAT exemption, so a static-NAT VM's VPN traffic leaves with its public IP

[skorchir]

problem

On a VPC virtual router, the per-VM static-NAT rule

-A POSTROUTING -o <public dev> -s <vm>/32 -j SNAT --to-source <public ip>

and the site-to-site VPN NAT exemption

-A POSTROUTING -o <public dev> -m mark --mark 0x525 -j ACCEPT

are both inserted at the front of the nat POSTROUTING chain, and whichever is created later ends up first. When static NAT is enabled on a VM after a site-to-site VPN connection already exists, the SNAT rule precedes the exemption and every packet that VM sends into the tunnel carries its public address instead of its private one. The remote side sees traffic from the public IP, its rules for the VPC's guest CIDR do not match, and that VM cannot reach the remote network while VMs without static NAT can. Nothing in the API or UI indicates a problem: the connection stays Connected.

Where the other end of the tunnel is does not matter (another VPC, another zone, another cloud, a customer gateway); the translation happens on the local VPC router before encryption.

Observed:

• Ping and TCP/22 from the static-NAT VM (A) to a VM behind the peer (B) fail. B's kernel receives no echo requests (/proc/net/snmp InEchos does not move).
• Temporarily allowing all ingress on B's tier makes the traffic arrive, and a listener on B reports the connection from A's static-NAT public address, not from the VPC's guest CIDR.
• Traffic initiated by B toward A is not affected (connection tracking keeps the reply un-translated), so the failure is one-directional and only for the VM with static NAT.
• The order of operations decides the outcome. Measured on the same setup:
  • static NAT enabled on A before the VPN gateway, customer gateway and connection are created: traffic crosses with private addresses;
  • static NAT disabled and re-enabled after the connection exists: ICMP and TCP/22 from A both fail;
  • the connection deleted and recreated with static NAT left in place: traffic crosses again.
• Replacing the static NAT with a port-forwarding rule also avoids it, since port forwarding only source-translates hairpin traffic to the VM's own public IP.

Analysis (4.22.1.1 sources):

• systemvm/debian/opt/cloud/bin/configure.py, CsForwardingRules.processStaticNatRule, lines 1559-1560 (4.23.0.0: line 1685):
  ["nat", "front", "-A POSTROUTING -o %s -s %s/32 -j SNAT --to-source %s"]
• configure.py, CsSite2SiteVpn.configure_iptables, line 1059 (4.23.0.0: line 1184):
  ["nat", "front", "-A POSTROUTING -t nat -o %s -m mark --mark 0x525 -j ACCEPT"]
• systemvm/debian/opt/cloud/bin/cs/CsNetfilter.py, lines 164-176: front becomes iptables -I <chain> <rule> (insert at position 1), and a rule that already exists is skipped (has_rule).

So on a fresh apply the databags are processed forwardingrules first and site2sitevpn second (configure.py lines 1591-1595), the exemption is inserted last and lands on top, and everything works. On an incremental apply only the new rule is inserted: enabling static NAT after the connection exists puts the SNAT rule at position 1 above the existing exemption, so the VM's VPN-bound packets are translated before the exemption can match them. The order then persists until the connection is deleted and recreated (measured) or the router is rebuilt (expected from the fresh-apply order).

The VPN mark itself is set correctly (mangle FORWARD -s <vpc cidr> -d <peer cidr> -j MARK --set-xmark 0x525), so the exemption matches whenever it is reached. The VPC-wide source-NAT rule is appended rather than front-inserted and is not affected; only per-VM static NAT is.

versions

• Apache CloudStack 4.22.1.1, KVM, advanced zone, VPC with "Default VPC offering" (NAT mode, VPC virtual router), DefaultIsolatedNetworkOfferingForVpcNetworks tiers, IKEv2 site-to-site connections between two VPC virtual routers.
• The same rules and insertion logic are present in 4.23.0.0 (line numbers above).

The steps to reproduce the bug

1. Two VPCs in one zone with non-overlapping CIDRs, e.g. 10.90.0.0/22 (A) and 10.91.0.0/22 (B), one tier each, one VM each. Tier ACLs on both sides allow ICMP and TCP/22 from the peer VPC CIDR. (A second VPC is only the simplest peer; any site-to-site endpoint reproduces it.)
2. createVpnGateway on each VPC, a createVpnCustomerGateway on each side describing the other (peer gateway IP, peer VPC CIDR, IKEv2), and createVpnConnection on each side. Send traffic once; both connections reach Connected.
3. associateIpAddress on VPC A and enableStaticNat it to VM A.
4. From VM A, ping VM B's private address and open TCP/22 to it.

Expected: traffic between VM A and VM B crosses the tunnel with private source and destination addresses, as it does for a VM without static NAT. Static NAT should apply to internet-bound traffic only.

Actual: both fail; with the peer tier temporarily open, VM B sees the connection arriving from VM A's public address.

What to do about it?

Make the static-NAT rule itself skip VPN-marked traffic, so chain order stops mattering:

-A POSTROUTING -o <dev> -s <vm>/32 -m mark ! --mark 0x525 -j SNAT --to-source <public ip>

Alternatives: re-assert the VPN exemption at position 1 whenever forwarding rules are (re)applied, or stop inserting the per-VM SNAT rule at the front. The first option is the smallest change, and it matches what a Neutron router does for floating IPs, which are exempted from source translation by an IPsec policy match.

Remote-access VPN was not checked.

[github-actions]

🎯 Triage report

On a VPC virtual router, the per-VM static-NAT SNAT rule and the site-to-site VPN NAT exemption are both inserted at the front of the nat POSTROUTING chain, so whichever is applied later wins. Enabling static NAT on a VM after a VPN connection already exists puts the SNAT rule ahead of the exemption, causing that VM's VPN-bound traffic to leave with its public IP instead of its private one — breaking one-directional connectivity to the VPN peer while other VMs remain unaffected. The report includes a clear root cause analysis (with file/line references in configure.py and CsNetfilter.py) and a concrete proposed fix (adding a -m mark ! --mark 0x525 exclusion to the SNAT rule).

📊 Assessment

Dimension	Value	Reasoning
Type	type:bug	Clear functional defect causing incorrect NAT behavior/connectivity failure
Component	component:virtual-router, component:networking	Issue is in the VPC virtual router's iptables rule generation (configure.py, CsNetfilter.py) affecting site-to-site VPN/static NAT networking
Severity	Severity:Major	Breaks VPN connectivity for static-NAT VMs silently (no error surfaced), but has a workaround (recreate the connection, or use port forwarding instead of static NAT)
Labels	type:bug, component:virtual-router, component:networking, Severity:Major	See reasoning above
Coding agent	Suitable	Root cause is precisely identified with file/line references, reproduction steps are detailed, and a specific minimal fix is proposed (adding a mark exclusion to the SNAT rule in configure.py)

No similar or duplicate issues were found in the repository.

💡 Notes and suggestions

• The reporter's proposed fix — appending -m mark ! --mark 0x525 to the static-NAT SNAT rule in CsForwardingRules.processStaticNatRule (systemvm/debian/opt/cloud/bin/configure.py) — looks like the smallest, most robust change since it makes the SNAT rule order-independent rather than relying on insertion order at apply time.
• Alternative approaches worth considering during review: re-asserting the VPN exemption at position 1 on every forwarding-rule reapplication, or switching the per-VM SNAT rule from front/insert to append so it no longer races with the VPN exemption for chain position.
• Suggest a maintainer or the reporter verify the fix against both a fresh VR rebuild and an incremental apply (the scenario described) to confirm the ordering issue no longer occurs either way.
• Remote-access VPN interaction was explicitly not checked by the reporter and may warrant a follow-up test once the static NAT fix lands.

Generated by Daily Issue Triage · sonnet50 84.6K · ◷

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9