Summary
galactic-nat is configured almost entirely by env vars on its DaemonSet, and the shard-identity and datapath values are set as raw env overrides:
| Env var |
Field |
GALACTIC_NAT_SHARD_SID |
shard's SRv6 uSID (forward-leg identity) |
GALACTIC_NAT_SHARD_PUB_ADDR6 |
NAT66 masquerade source |
GALACTIC_NAT_SHARD_PUB_ADDR4 / GALACTIC_NAT_NAT64_PREFIX |
NAT64 masquerade source + prefix |
GALACTIC_NAT_UPLINK_INTERFACES |
the fabric uplinks the XDP datapath attaches to |
The first four are fabric-wide allocation decisions — the SID must be reserved and disjoint from the node's BGPRouter locator (locator_matches in internal/plumbing/ebpf/natprog/nat.c), and the masquerade addresses are public addressing other components reference. All four today are only settable via a hand-written per-shard patch on the DaemonSet — and because the DaemonSet env is cluster-wide, the operator writes a hand-crafted Kustomize patch per node (see the per-site node-patch.yaml in this repo's deploy/containerlab).
The uplinks, by contrast, are auto-detectable — the CNI's SRv6 datapath already derives the same physical uplinks from the node itself rather than being told. attach.ResolveInterfaces() in internal/plumbing/ebpf/attach/interfaces.go returns the interfaces carrying the IPv6 default route and BGP-learned routes (with tunnel/VRF/loopback exclusions and bond-slave expansion), and GALACTIC_CNI_EBPF_INTERFACES is only an override for ambiguous multi-homed cases. galactic-nat runs on the same nodes and attaches to the same interfaces as the CNI, so it can reuse that derivation instead of being configured — which removes the per-node uplink patch entirely in the common case, and keeps the shard and the CNI on a node converging on the same physical uplinks by construction.
The machinery to make the identity a CRD already exists but is inverted: galactic-nat is already a controller-runtime process with an EgressShardReconciler (internal/controller/egressshard_controller.go), and the EgressShard CRD already carries the four identity fields — but only in Status, echoed from what the process was started with (EgressShardSpec has no field but TargetRef; EgressShardStatus holds shardSID, shardAddressIPv6, shardAddressIPv4, NAT64Prefix). So the CRD is a status/observability mirror and the advertisement source today, not the config authority. galactic-nat is the odd one out: galactic-router and galactic-gateway already drive their datapaths from CRD spec via in-process reconcilers.
Proposal
- Promote the four identity fields out of
EgressShardStatus and into EgressShardSpec (shardSID, shardAddressIPv6, shardAddressIPv4, nat64Prefix), so the operator's intent lives in spec.
- Do not add
uplinkInterfaces to EgressShardSpec. Derive the uplinks instead: drive the datapath's XDP attach from the same attach.ResolveInterfaces() auto-detection the CNI uses, so the shard and the CNI on a node consistently converge on the same physical uplinks with no per-node configuration. Keep it overridable — GALACTIC_NAT_UPLINK_INTERFACES (or, optionally, a spec field) remains the escape hatch for a genuinely ambiguous multi-homed node where auto-detection cannot be confident — but fall back to auto-detection when unset rather than requiring it.
- Have
EgressShardReconciler drive the datapath from spec on reconcile — attach the XDP program to the auto-detected uplinks and program shard_config_table from the identity fields — instead of the process reading env and only echoing values into status. Status becomes the observed output (configured / Ready).
- Keep only genuinely process-local config as env/downward-API, which do not belong in the CRD:
GALACTIC_NAT_NODE_NAME — who the process is (the CRD's join key via spec.targetRef).
GALACTIC_NAT_METRICS_PORT / GALACTIC_NAT_GRPC_HEALTH_PORT — process ports.
Bootstrap ordering to handle: today the datapath must know its identity before first reconcile (shard_config_table is an array map, and the dispatcher fails open XDP_PASS until configured), and it must know its uplinks before attach (the datapath only claims packets on interfaces it is attached to, so too few uplinks is a silent blackhole). Flipping to spec-based identity means the process starts with env only for node name, then the first reconcile of its own EgressShard attaches the datapath to the auto-detected uplinks and configures the tables from spec. Uplinks stay derivable at attach time from the node's routing state; identity comes from spec. This is the same bootstrapping pattern the router and gateway reconcilers already use.
Note (cross-repo): the EgressShard CRD lives in the separately-released go.datum.net/network module (galactic pins it; see #574), so promoting the fields into EgressShardSpec is a two-repo stacked change — network module PR + release, then this repo consuming it. The in-cluster/EgressShard node-patch.yaml deploy resources also move from Status-echoing to Spec-configured as part of this.
Summary
galactic-natis configured almost entirely by env vars on its DaemonSet, and the shard-identity and datapath values are set as raw env overrides:GALACTIC_NAT_SHARD_SIDGALACTIC_NAT_SHARD_PUB_ADDR6GALACTIC_NAT_SHARD_PUB_ADDR4/GALACTIC_NAT_NAT64_PREFIXGALACTIC_NAT_UPLINK_INTERFACESThe first four are fabric-wide allocation decisions — the SID must be reserved and disjoint from the node's
BGPRouterlocator (locator_matchesininternal/plumbing/ebpf/natprog/nat.c), and the masquerade addresses are public addressing other components reference. All four today are only settable via a hand-written per-shard patch on the DaemonSet — and because the DaemonSet env is cluster-wide, the operator writes a hand-crafted Kustomize patch per node (see the per-sitenode-patch.yamlin this repo'sdeploy/containerlab).The uplinks, by contrast, are auto-detectable — the CNI's SRv6 datapath already derives the same physical uplinks from the node itself rather than being told.
attach.ResolveInterfaces()ininternal/plumbing/ebpf/attach/interfaces.goreturns the interfaces carrying the IPv6 default route and BGP-learned routes (with tunnel/VRF/loopback exclusions and bond-slave expansion), andGALACTIC_CNI_EBPF_INTERFACESis only an override for ambiguous multi-homed cases.galactic-natruns on the same nodes and attaches to the same interfaces as the CNI, so it can reuse that derivation instead of being configured — which removes the per-node uplink patch entirely in the common case, and keeps the shard and the CNI on a node converging on the same physical uplinks by construction.The machinery to make the identity a CRD already exists but is inverted:
galactic-natis already a controller-runtime process with anEgressShardReconciler(internal/controller/egressshard_controller.go), and theEgressShardCRD already carries the four identity fields — but only inStatus, echoed from what the process was started with (EgressShardSpechas no field butTargetRef;EgressShardStatusholdsshardSID,shardAddressIPv6,shardAddressIPv4,NAT64Prefix). So the CRD is a status/observability mirror and the advertisement source today, not the config authority.galactic-natis the odd one out:galactic-routerandgalactic-gatewayalready drive their datapaths from CRD spec via in-process reconcilers.Proposal
EgressShardStatusand intoEgressShardSpec(shardSID,shardAddressIPv6,shardAddressIPv4,nat64Prefix), so the operator's intent lives in spec.uplinkInterfacestoEgressShardSpec. Derive the uplinks instead: drive the datapath's XDP attach from the sameattach.ResolveInterfaces()auto-detection the CNI uses, so the shard and the CNI on a node consistently converge on the same physical uplinks with no per-node configuration. Keep it overridable —GALACTIC_NAT_UPLINK_INTERFACES(or, optionally, a spec field) remains the escape hatch for a genuinely ambiguous multi-homed node where auto-detection cannot be confident — but fall back to auto-detection when unset rather than requiring it.EgressShardReconcilerdrive the datapath fromspecon reconcile — attach the XDP program to the auto-detected uplinks and programshard_config_tablefrom the identity fields — instead of the process reading env and only echoing values into status. Status becomes the observed output (configured / Ready).GALACTIC_NAT_NODE_NAME— who the process is (the CRD's join key viaspec.targetRef).GALACTIC_NAT_METRICS_PORT/GALACTIC_NAT_GRPC_HEALTH_PORT— process ports.Bootstrap ordering to handle: today the datapath must know its identity before first reconcile (
shard_config_tableis an array map, and the dispatcher fails openXDP_PASSuntil configured), and it must know its uplinks before attach (the datapath only claims packets on interfaces it is attached to, so too few uplinks is a silent blackhole). Flipping to spec-based identity means the process starts with env only for node name, then the first reconcile of its ownEgressShardattaches the datapath to the auto-detected uplinks and configures the tables fromspec. Uplinks stay derivable at attach time from the node's routing state; identity comes fromspec. This is the same bootstrapping pattern the router and gateway reconcilers already use.