Skip to content

Drive galactic-nat shard identity from the EgressShard CRD spec, not env vars #581

Description

@privateip

Summary

galactic-nat is configured almost entirely by env vars on its DaemonSet, and the shard-identity and datapath values are set as raw env overrides:

Env var Field
GALACTIC_NAT_SHARD_SID shard's SRv6 uSID (forward-leg identity)
GALACTIC_NAT_SHARD_PUB_ADDR6 NAT66 masquerade source
GALACTIC_NAT_SHARD_PUB_ADDR4 / GALACTIC_NAT_NAT64_PREFIX NAT64 masquerade source + prefix
GALACTIC_NAT_UPLINK_INTERFACES the fabric uplinks the XDP datapath attaches to

The first four are fabric-wide allocation decisions — the SID must be reserved and disjoint from the node's BGPRouter locator (locator_matches in internal/plumbing/ebpf/natprog/nat.c), and the masquerade addresses are public addressing other components reference. All four today are only settable via a hand-written per-shard patch on the DaemonSet — and because the DaemonSet env is cluster-wide, the operator writes a hand-crafted Kustomize patch per node (see the per-site node-patch.yaml in this repo's deploy/containerlab).

The uplinks, by contrast, are auto-detectable — the CNI's SRv6 datapath already derives the same physical uplinks from the node itself rather than being told. attach.ResolveInterfaces() in internal/plumbing/ebpf/attach/interfaces.go returns the interfaces carrying the IPv6 default route and BGP-learned routes (with tunnel/VRF/loopback exclusions and bond-slave expansion), and GALACTIC_CNI_EBPF_INTERFACES is only an override for ambiguous multi-homed cases. galactic-nat runs on the same nodes and attaches to the same interfaces as the CNI, so it can reuse that derivation instead of being configured — which removes the per-node uplink patch entirely in the common case, and keeps the shard and the CNI on a node converging on the same physical uplinks by construction.

The machinery to make the identity a CRD already exists but is inverted: galactic-nat is already a controller-runtime process with an EgressShardReconciler (internal/controller/egressshard_controller.go), and the EgressShard CRD already carries the four identity fields — but only in Status, echoed from what the process was started with (EgressShardSpec has no field but TargetRef; EgressShardStatus holds shardSID, shardAddressIPv6, shardAddressIPv4, NAT64Prefix). So the CRD is a status/observability mirror and the advertisement source today, not the config authority. galactic-nat is the odd one out: galactic-router and galactic-gateway already drive their datapaths from CRD spec via in-process reconcilers.

Proposal

  1. Promote the four identity fields out of EgressShardStatus and into EgressShardSpec (shardSID, shardAddressIPv6, shardAddressIPv4, nat64Prefix), so the operator's intent lives in spec.
  2. Do not add uplinkInterfaces to EgressShardSpec. Derive the uplinks instead: drive the datapath's XDP attach from the same attach.ResolveInterfaces() auto-detection the CNI uses, so the shard and the CNI on a node consistently converge on the same physical uplinks with no per-node configuration. Keep it overridable — GALACTIC_NAT_UPLINK_INTERFACES (or, optionally, a spec field) remains the escape hatch for a genuinely ambiguous multi-homed node where auto-detection cannot be confident — but fall back to auto-detection when unset rather than requiring it.
  3. Have EgressShardReconciler drive the datapath from spec on reconcile — attach the XDP program to the auto-detected uplinks and program shard_config_table from the identity fields — instead of the process reading env and only echoing values into status. Status becomes the observed output (configured / Ready).
  4. Keep only genuinely process-local config as env/downward-API, which do not belong in the CRD:
    • GALACTIC_NAT_NODE_NAME — who the process is (the CRD's join key via spec.targetRef).
    • GALACTIC_NAT_METRICS_PORT / GALACTIC_NAT_GRPC_HEALTH_PORT — process ports.

Bootstrap ordering to handle: today the datapath must know its identity before first reconcile (shard_config_table is an array map, and the dispatcher fails open XDP_PASS until configured), and it must know its uplinks before attach (the datapath only claims packets on interfaces it is attached to, so too few uplinks is a silent blackhole). Flipping to spec-based identity means the process starts with env only for node name, then the first reconcile of its own EgressShard attaches the datapath to the auto-detected uplinks and configures the tables from spec. Uplinks stay derivable at attach time from the node's routing state; identity comes from spec. This is the same bootstrapping pattern the router and gateway reconcilers already use.

Note (cross-repo): the EgressShard CRD lives in the separately-released go.datum.net/network module (galactic pins it; see #574), so promoting the fields into EgressShardSpec is a two-repo stacked change — network module PR + release, then this repo consuming it. The in-cluster/EgressShard node-patch.yaml deploy resources also move from Status-echoing to Spec-configured as part of this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions