Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,6 @@ metadata:
spec:
template:
spec:
# Image override only -- this overlay deliberately inherits
# config/fabric-router/'s own affinity (galactic.datumapis.com/fabric=router,
# which every lab worker carries) rather than narrowing it. It used to
# narrow to node in (compute, edge) so that iad's route-reflector node
# could be served by a second DaemonSet (resources/fabric-control/iad/,
# removed) carrying its own frr.conf; that split existed only because a
# shared fabric-config ConfigMap couldn't serve two nodes' configs.
# frr-init now selects a frr.conf.<nodename> key via NODE_NAME, so one
# DaemonSet plus one ConfigMap covers every role -- the route
# reflector included (../iad/kustomization.yaml's configMapGenerator
# carries iad-worker3's key alongside the compute and edge nodes').
# Nothing about FRR itself differs by role; only the per-node
# frr.conf each pod picks.
initContainers:
- name: frr-init
image: fabric-router:latest
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@ namespace: galactic-system
resources:
- ../base

# One frr.conf.<nodename> key per node this cluster's fabric DaemonSet
# matches -- frr-init selects its own via the NODE_NAME downward-API env
# var. dfw has three: one compute worker and two edge workers.
# One frr.conf.<nodename> key per node (dfw: compute, two edge workers)
configMapGenerator:
- name: fabric-config
files:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@ namespace: galactic-system
resources:
- ../base

# One key per fabric-carrying node -- see ../dfw/kustomization.yaml. iad
# has three: compute, edge, and the lab's single EVPN route reflector.
# One key per fabric-carrying node (iad: compute, edge, route reflector)
configMapGenerator:
- name: fabric-config
files:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ namespace: galactic-system
resources:
- ../base

# One key per fabric-carrying node -- see ../dfw/kustomization.yaml.
# One key per fabric-carrying node
configMapGenerator:
- name: fabric-config
files:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,7 @@ metadata:
spec:
template:
spec:
# Both of dfw-worker's uplinks. dfw-worker2/dfw-worker3 also have an
# eth2 (facing the compute node), so this value is valid on every node
# in this cluster -- which matters, since the list is DaemonSet-wide
# and an interface a node does not have fails that node's attach.
#
# This covers the SRv6 uSID decap hook only. galactic-nat's shard XDP
# program is configured separately and names the same two uplinks
# (GALACTIC_NAT_UPLINK_INTERFACES, resources/galactic-nat/dfw/
# node-patch.yaml) -- the two lists have to agree on a dual-homed node,
# or traffic arriving on an uplink only one of them covers is
# decapsulated but never translated, or vice versa.
# Both uplinks
initContainers:
- name: install-cni
env:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# dfw: the one site whose compute node is dual-homed, so the uSID decap hook
# has to attach to both of its uplinks.
# dfw: dual-homed compute node
namespace: galactic-system
resources:
- ../shared
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# iad: single-homed compute node, so the shared eth1-only interface list
# applies unchanged. See ../dfw/ for the dual-homed case.
# iad: single-homed compute node
namespace: galactic-system
resources:
- ../shared
Original file line number Diff line number Diff line change
Expand Up @@ -10,65 +10,20 @@ spec:
image: galactic-cni:latest
imagePullPolicy: Never
env:
# Fabric-wide NAT66 shard membership list (design plan 搂3,
# internal/plumbing/srv6.EgressDefaultRouteAdd's own doc
# comment) -- the same three shard SIDs
# resources/galactic-nat/{dfw,sjc,iad}/node-patch.yaml
# configure each shard with, shared identically across every
# site here since this list is currently operator-supplied,
# not learned in-cluster (see EnvCNIEgressShardSIDs's own doc
# comment). Written into the static conflist by
# internal/installer.Bootstrap (this init container), read
# back by internal/cnibgp at every CNI ADD.
# Fabric-wide NAT66 shard SIDs
- name: GALACTIC_CNI_EGRESS_SHARD_SIDS
value: "2001:db8:ff01:9:e001::,2001:db8:ff02:9:e001::,2001:db8:ff03:9:e001::"
# The fabric-wide NAT64 prefix, identical to every shard's own
# GALACTIC_NAT_NAT64_PREFIX. Setting it is what makes each CNI ADD
# install the tenant VRF's route for the prefix, so an IPv6-only
# tenant reaches an IPv4 destination by addressing its synthesized
# form (RFC 6052: the IPv4 address in the low 32 bits). Written
# into the static conflist by this init container and read back by
# internal/cnibgp on every ADD, exactly like the shard list above.
# Fabric-wide NAT64 prefix
- name: GALACTIC_CNI_NAT64_PREFIX
value: "2001:db8:64::/96"
# Every lab node is dual-homed: eth0 carries the IPv6 default
# route but only reaches the Kind/ContainerLab management
# bridge, while eth1 is the dedicated point-to-point link to
# the transit fabric that actual SRv6-encapsulated VPC traffic
# arrives/departs on -- see credential-refresh's own identical
# env var below for the full history of why this override
# exists at all. This init container needs its own copy, not
# just credential-refresh's: internal/installer.Bootstrap
# (this container) is what resolves and writes it into the
# static conflist's own "ebpf_interfaces" field for
# internal/cnibgp to read back on every CNI ADD (mirroring
# NAT66_SHARD_SIDS just above) -- found live, without it here,
# Bootstrap's own resolution fell back to the same broken
# eth0 auto-detection this override exists to fix, producing
# a wrong-but-plausible srv6.ResolvePublicUplink/
# ResolveNodeSourceAddress result for every CNI ADD with no
# error at all (this env var being set only on
# credential-refresh, a separate container in the same pod,
# never propagates to this one).
# Fabric uplink the eBPF hook attaches to
- name: GALACTIC_CNI_EBPF_INTERFACES
value: eth1
containers:
- name: credential-refresh
image: galactic-cni:latest
imagePullPolicy: Never
env:
# Every lab node is dual-homed: eth0 carries the IPv6 default
# route but only reaches the Kind/ContainerLab management
# bridge (kubectl/API-server traffic), while eth1 is the
# dedicated point-to-point link to the transit fabric (tr1-4)
# that actual SRv6-encapsulated VPC traffic arrives on.
# ResolveInterfaces' auto-detection (default-IPv6-route
# heuristic, internal/plumbing/ebpf/attach/interfaces.go) picks
# eth0 here since it's ambiguous between the two -- confirmed
# live by tcpdump: cross-region SRv6 packets arrive on eth1 but
# the eBPF usid_ingress filter was only ever attached to eth0,
# so decapsulation never ran and every VPC ping between sites
# silently blackholed. This override forces the correct
# interface for this topology; see docs/cni/configuration.md.
# Fabric uplink the eBPF hook attaches to
- name: GALACTIC_CNI_EBPF_INTERFACES
value: eth1
Original file line number Diff line number Diff line change
@@ -1,12 +1,4 @@
# The lab's shared galactic-cni DaemonSet: config/galactic-cni/ (copied onto
# the node at deploy time as ../base -- see scripts/deploy-cni.sh) plus the
# lab-only image/env patch every site needs.
#
# Referenced as a directory by each site overlay rather than being applied
# directly, because one env var has to differ per site: dfw's compute node is
# dual-homed and needs GALACTIC_CNI_EBPF_INTERFACES to name both uplinks,
# while a node without eth2 fails its attach outright if eth2 is listed
# (internal/plumbing/ebpf/attach.attachOne errors on an unknown link).
# Lab-shared galactic-cni layer: base DaemonSet plus the image/env patch every site applies.
resources:
- ../base
patches:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# sjc: single-homed compute node, so the shared eth1-only interface list
# applies unchanged. See ../dfw/ for the dual-homed case.
# sjc: single-homed compute node
namespace: galactic-system
resources:
- ../shared
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# Reflector-side session toward dfw-worker (dfw compute). The client side of
# this same session lives in resources/galactic-router/, pointed
# back at this node's fc00:0:8::1:1790. Every galactic-router in the lab --
# compute and edge alike, in all three clusters -- is a client of this one
# reflector; compute and edge nodes never peer with each other directly.
# Reflector-side session toward dfw-worker (dfw compute)
apiVersion: network.datumapis.com/v1alpha1
kind: BGPPeer
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# Reflector-side session toward dfw-worker2 (dfw edge). The client side of
# this same session lives in resources/galactic-gateway/, pointed
# back at this node's fc00:0:8::1:1790. Every galactic-router in the lab --
# compute and edge alike, in all three clusters -- is a client of this one
# reflector; compute and edge nodes never peer with each other directly.
# Reflector-side session toward dfw-worker2 (dfw edge)
apiVersion: network.datumapis.com/v1alpha1
kind: BGPPeer
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# Reflector-side session toward dfw-worker3 (dfw edge). The client side of
# this same session lives in resources/galactic-gateway/, pointed
# back at this node's fc00:0:8::1:1790. Every galactic-router in the lab --
# compute and edge alike, in all three clusters -- is a client of this one
# reflector; compute and edge nodes never peer with each other directly.
# Reflector-side session toward dfw-worker3 (dfw edge)
apiVersion: network.datumapis.com/v1alpha1
kind: BGPPeer
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# Reflector-side session toward iad-worker (iad compute). The client side of
# this same session lives in resources/galactic-router/, pointed
# back at this node's fc00:0:8::1:1790. Every galactic-router in the lab --
# compute and edge alike, in all three clusters -- is a client of this one
# reflector; compute and edge nodes never peer with each other directly.
# Reflector-side session toward iad-worker (iad compute)
apiVersion: network.datumapis.com/v1alpha1
kind: BGPPeer
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# Reflector-side session toward iad-worker2 (iad edge). The client side of
# this same session lives in resources/galactic-gateway/, pointed
# back at this node's fc00:0:8::1:1790. Every galactic-router in the lab --
# compute and edge alike, in all three clusters -- is a client of this one
# reflector; compute and edge nodes never peer with each other directly.
# Reflector-side session toward iad-worker2 (iad edge)
apiVersion: network.datumapis.com/v1alpha1
kind: BGPPeer
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# Reflector-side session toward sjc-worker (sjc compute). The client side of
# this same session lives in resources/galactic-router/, pointed
# back at this node's fc00:0:8::1:1790. Every galactic-router in the lab --
# compute and edge alike, in all three clusters -- is a client of this one
# reflector; compute and edge nodes never peer with each other directly.
# Reflector-side session toward sjc-worker (sjc compute)
apiVersion: network.datumapis.com/v1alpha1
kind: BGPPeer
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# Reflector-side session toward sjc-worker2 (sjc edge). The client side of
# this same session lives in resources/galactic-gateway/, pointed
# back at this node's fc00:0:8::1:1790. Every galactic-router in the lab --
# compute and edge alike, in all three clusters -- is a client of this one
# reflector; compute and edge nodes never peer with each other directly.
# Reflector-side session toward sjc-worker2 (sjc edge)
apiVersion: network.datumapis.com/v1alpha1
kind: BGPPeer
metadata:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,6 @@ spec:
localASN: 65000
routerID: "10.255.255.4"
srv6Locator: "2001:db8:ff03::/48"
# nodeID 4, not 1: this router shares iad's locator with iad-worker
# (nodeID 1) and iad-worker2 (nodeID 2), and a uSID's Node-ID is what
# separates one node's SID space from another's within a site. The
# reflector originates no tenant path of its own, so nothing is derived
# from this today -- but a duplicate would be indistinguishable from
# iad-worker's own SIDs the moment it did.
nodeID: 4
addressFamilies:
- afi: l2vpn
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,7 @@ metadata:
spec:
template:
spec:
# Image override only. This must NOT carry a galactic-router entry:
# `containers` is a strategic-merge list keyed on `name`, so an entry
# for a container config/galactic-gateway/base/daemonset.yaml doesn't
# define is *added* rather than overridden. This patch used to list
# one, from when galactic-router ran as a sidecar in the gateway pod;
# once that moved to its own DaemonSet, the entry stayed behind and
# silently resurrected a galactic-router container with an image and
# nothing else -- no GALACTIC_ROUTER_NODE_NAME, no volumes -- which
# exited 1 ("node name is required") into CrashLoopBackOff on both
# gateway nodes while the gateway container itself stayed Ready (pod
# stuck at 1/2). galactic-router reaches these nodes via
# config/galactic-router/overlays/router/ like every other node,
# since they carry galactic.datumapis.com/galactic=router.
# Image override only.
containers:
- name: galactic-gateway
image: galactic-gateway:latest
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,3 @@
# Mirrors resources/galactic-router/base/'s pattern exactly, pointed at
# config/galactic-gateway/base instead of config/galactic-router/base: "gateway" is copied
# onto the node at deploy time (see scripts/deploy-galactic-router.sh's
# copy_router_gateway_config), nested here so this kustomization's
# "gateway" resource reference resolves. Unlike resources/galactic-router/
# base/'s "tenant" resource, config/galactic-gateway/base is self-contained (its own
# full single-container DaemonSet spec, not a patch onto config/galactic-router/base),
# so there is no separate "base" resource to nest alongside it here.
resources:
- gateway
patches:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
# This edge node's own client session to the lab's single EVPN route
# reflector (iad-worker3, fc00:0:8::1 port 1790) -- the same reflector every
# compute node peers with. Edge and compute exchange EVPN paths through it,
# never directly.
# Client session to the lab's single EVPN route reflector.
apiVersion: network.datumapis.com/v1alpha1
kind: BGPPeer
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
# dfw-worker2's instantiation of config/galactic-gateway/base (see that dir's
# kustomization.yaml doc comment for why this role needs one instance per
# gateway node rather than one shared DaemonSet).
# dfw-worker2's edge gateway overlay.
namespace: galactic-system
resources:
- ../base
Expand All @@ -12,13 +10,7 @@ patches:
target:
kind: DaemonSet
name: galactic-gateway
# Strategic-merge patches can't rename a resource (Kustomize keeps the
# target's original identity regardless of what metadata.name the patch
# body says), so the rename needs a JSON6902 patch: without a distinct
# name, a site with two edge nodes would have both DaemonSets named
# "galactic-gateway" in the same namespace, silently clobbering each
# other. Applied uniformly even where a site has only one edge node, so
# the DaemonSet name always names the node it runs on.
# Renames the DaemonSet to this node's name.
- target:
kind: DaemonSet
name: galactic-gateway
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,7 @@ metadata:
spec:
template:
spec:
# kubernetes.io/hostname pins this DaemonSet instance to exactly one
# node: GALACTIC_GATEWAY_SRV6_ADDRESS below must be unique per gateway
# node (see config/galactic-gateway/base/kustomization.yaml's doc
# comment), so every edge node gets its own DaemonSet rather than one
# DaemonSet matching every edge-labeled node with identical env. dfw
# has two edge nodes and so two instances; sjc and iad have one each.
# Hostname pin: one DaemonSet instance per edge node.
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
Expand All @@ -29,36 +24,12 @@ spec:
containers:
- name: galactic-gateway
env:
# eth1 is this lab's dedicated transit-fabric-facing uplink on
# every node (see resources/galactic-cni/daemonset-patch.yaml's
# comment) -- the same interface edgedsr.c's edge_lb XDP program
# attaches to for both underlay BGP and ingress traffic.
# Transit-fabric-facing uplink (underlay BGP + ingress).
- name: GALACTIC_GATEWAY_PUBLIC_INTERFACE
value: eth1
# eth2 is this node's compute-facing link (see
# gvpc.clab.yaml's links section): dfw's compute node is
# dual-homed to both of dfw's edge nodes, and every site's
# compute node reaches the fabric only through its edge tier.
# A backend's reply to a VIP therefore crosses this node,
# where connection tracking holds no record of the forward
# half -- which went to the backend encapsulated, through XDP
# -- and kube-proxy drops it as invalid. edge_return forwards
# those replies before netfilter sees them.
# Compute-facing link.
- name: GALACTIC_GATEWAY_INTERNAL_INTERFACES
value: eth2
# uFMT 48+16 uSID over this site's shared locator (2001:db8:ff01::/48,
# see bgprouter.yaml's srv6Locator). nodeID=2,
# Function=End.DT46 (arbitrary -- Argument 0 always misses
# vrf_table, so no Function value is ever consulted for it; see
# internal/plumbing/ebpf/prog/usid.c), Argument=0 (reserved).
# Computed via internal/plumbing/ebpf/uformat.Encode directly,
# bypassing srv6.ComputeSID's argument==0 guard (that guard
# exists for tenant-VRF SID derivation specifically). This is
# this node's own plain SRv6-reachable address, used only as the
# DSR datapath's outer-header encap source -- never a NAT/SNAT
# source and never compared against anything on a receive path
# (see internal/config/gateway.go's EnvGatewaySRv6Address doc
# comment). No in-cluster mechanism yet derives this
# automatically, so it is supplied statically per node here.
# This node's SRv6 address: DSR outer-header encap source.
- name: GALACTIC_GATEWAY_SRV6_ADDRESS
value: "2001:db8:ff01:2:e000::"
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
# This edge node's own client session to the lab's single EVPN route
# reflector (iad-worker3, fc00:0:8::1 port 1790) -- the same reflector every
# compute node peers with. Edge and compute exchange EVPN paths through it,
# never directly.
# Client session to the lab's single EVPN route reflector.
apiVersion: network.datumapis.com/v1alpha1
kind: BGPPeer
metadata:
Expand Down
Loading
Loading