Edge datapath attach permanently breaks a bonded uplink

[scotwells]

Summary

Starting the edge datapath on a node whose public interface is a bond leaves every attached link unable to transmit.

The links stay up and keep receiving, but send nothing, so the switch times the LACP session out and drops them from the aggregate. A node with both of its links attached loses all connectivity, and neither stopping the datapath nor waiting restores it.

This took two production machines offline and a third has been running on a single uplink since it was deployed.

Acceptance criteria

• A node keeps its uplinks when the datapath starts on a bonded public interface
• Stopping the datapath returns any affected link to service without a power cycle
• A node that cannot attach safely fails visibly rather than going offline

[scotwells]

Evidence, gathered today against production and staging nodes running v0.83.121. All observations are from GALACTIC_GATEWAY_PUBLIC_INTERFACE set to a bonding master, which edgeattach.ResolveTargets expands to the bond's slaves.

The transmit path stops. A staging node with the datapath attached to one slave, sampled over 90 seconds. eno2 is the untouched slave, enp3s0f1 is the attached one:

iface         rx_pkts   rx_mcast    tx_pkts
eno2           +31681       +365     +63484
enp3s0f1          +91        +91         +0

The attached interface has carrier, is listed in the aggregator, and receives normally. It transmits zero packets in 90 seconds, so it emits no LACPDUs. The partner record then defaults, which is the switch correctly timing out a peer that went silent. This is our side going quiet first, not the switch withdrawing.

The resulting bonding state, with actor 13 meaning synchronized but neither collecting nor distributing, and partner 69 carrying the defaulted bit:

Slave Interface: eno2         actor 61   partner 61
Slave Interface: enp3s0f1     actor 13   partner 69

What the driver does on attach. Kernel log from an Intel X550 at the moment of attach:

15:48:54.062  ixgbe 0000:01:00.1: Multiqueue Enabled: Rx 16, Tx 16, XDP Queue count = 16
15:48:54.521  bond0: (slave enp1s0f1): speed changed to 0 on port 2
15:48:59.623  ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps

The attach reallocates Tx rings and takes the link down for 5.6 seconds. On 10GBASE-T copper that relink cost is unavoidable, so no bonding debounce covers it.

Controlled reproduction. Two identical machines, same rack, same NICs down to the PCI addresses and firmware:

Attached to	Result
both bond slaves	node unreachable 17s after the pod starts, never recovers
one bond slave	node survives on the other slave, attached slave stuck at 13/69
a NIC outside the bond	node healthy, pod 1/1 for 7 minutes, no side effects
nothing (control node)	unaffected throughout

It does not recover. After deleting the DaemonSet I watched the affected slave for 150 seconds and it stayed at 13/69. Detach restores XDP Queue count = 0 but not the transmit path. Only a cold power cycle returns it to 61/61.

It is not driver specific. The signature above is from igb. I reproduced it identically on ixgbe. Across seven nodes running the datapath, the two on ixgbe fail outright while four on bnxt_en and one on igb survive, but at least one bnxt_en node is also sitting with a slave at 13/69, so the difference looks like how much the node tolerates rather than whether the link breaks.

What I could not determine. Whether the transmit path breaks first and the LACP session dies as a consequence, or the port leaves the aggregate first and transmit stops after. Both fit the measurements. Settling it means reading the driver XDP setup path against the bonding code rather than more black-box testing. The observable that is not in doubt is a synced port, carrier up, in the aggregator, transmitting nothing.

Separate, noticed while reading pod logs. The FIB-lookup sysctls never get set, and the failure is logged as non-fatal:

WARN failed to set sysctl (non-fatal) sysctl=net.ipv6.conf.enp1s0f0.forwarding
     err="open /proc/sys/net/ipv6/conf/enp1s0f0/forwarding: read-only file system"
WARN failed to set sysctl (non-fatal) sysctl=net.ipv6.conf.all.forwarding
     err="read-only file system"

Per the architecture doc, without these bpf_fib_lookup() returns BPF_FIB_LKUP_RET_NOT_FWDED for every packet. That appears to affect every node running this, including the healthy ones, and probably deserves its own issue.

The affected production nodes are held out of service in datum-cloud/infra#5752 until this is resolved.

[privateip]

Reading the driver XDP setup path against the bonding code, as the evidence comment called for. It settles the ordering question, and it puts most of the permanence outside this repo.

Our side goes silent first. The partner's DEFAULTED bit is the consequence, about 90 seconds later.

The chain, against Linux 6.12:

1. The attach reallocates rings and drops carrier. bond_netdev_event sees NETDEV_CHANGE, bond_update_speed_duplex() fails while the link is down, and 802.3ad parks the slave at BOND_LINK_FAIL — bond_main.c:4036, whose comment reads "let link-monitoring (miimon) set it right when correct speeds/duplex are available."
2. The same event runs ad_update_actor_keys(). __get_link_speed() and __get_duplex() both return 0 because slave->link != BOND_LINK_UP, which logs bond_3ad.c:2639 "speed changed to 0 on port %d" — your speed changed to 0 on port 2 — and clears AD_PORT_LACP_ENABLED.
3. ad_periodic_machine then parks at AD_NO_PERIODIC, so LACPDUs stop. Separately bond_slave_can_tx() requires BOND_LINK_UP, so the slave leaves the transmit hash array. That is your +0 tx_pkts.
4. Carrier returns and nothing acts on it. The only code that sets port->is_enabled back to true is bond_3ad_handle_link_change(BOND_LINK_UP), reachable in steady state solely from bond_miimon_commit() → bond_mii_monitor() — never scheduled when miimon is 0.

Your actor/partner numbers confirm the direction: 13 is activity|aggregation|sync with collecting and distributing clear, a local mux decision, and DEFAULTED in 69 is only ever set by our own receive timeout of the partner.

So the permanence — "neither stopping the datapath nor waiting restores it", the fourteen-hour outage, the power cycle — is miimon=0, not this datapath. datum-cloud/infra#5748 sets miimon: 100 on every bond and should resolve acceptance criterion 2 on its own; galactic cannot, since nothing here can make a bond notice a link coming back. Worth landing that first, and worth knowing the trigger is not special: any carrier bounce does this. A cable, an SFP reseat, a switch port flap. The datapath guarantees one per start and one per stop.

Two defects here are ours, and I am fixing both on fix/issue-583:

• Every slave is attached back to back, so both members go down inside the same second and no uplink survives — attach.go:167-177. Now each bond slave is waited back into its aggregate, on MII status plus collecting|distributing, before the next interface is touched. Acceptance criterion 1.
• An unsupported driver is discovered by attaching. edgepreflight.Check() probes kernel-wide capability only, nothing per-NIC, so the healthy slaves ahead of the failure have already paid a bounce and the rollback costs them another. Now the netdev generic netlink family is asked which interfaces support a native program, and the process refuses with nothing touched. Acceptance criterion 3.

On the bnxt_en node sitting at 13/69: that fits the same mechanism. Nothing about it is driver-specific once carrier drops — the drivers differ only in whether their ring realloc drops carrier long enough to be noticed.

Confirmed separately that tg3 has no XDP support at all in 6.12 (no ndo_bpf, zero xdp references in the driver), which also answers the question left open on #498: reverting the infra#4495 stopgap to bond0 would crash-loop until this lands.

The sysctl observation is real and is now #586.