Repository navigation
NAS-145291 / 28.0.0-BETA.1 / truenas-appliance-runner: network: bridge, the appliance on a bridge with the runner - #30
Merged
Conversation
…with the runner Under qemu's user-mode network the appliance is 10.0.2.15 behind qemu's NAT, and only its ports are forwarded, to 8443, 8080 and 2222 on the runner's loopback. A TrueNAS Connect stack on the runner cannot work with that: registration publishes the NAS's own IPs, so its truenas.direct name resolves to 10.0.2.15, and a client then connects to it on 443 (truenas-connect/cui#121). network: bridge (APPLIANCE_NETWORK) puts the appliance on a bridge, tnapp-br, with the runner at 10.232.0.1. qemu's NIC is a tap on it with a fixed MAC, and dnsmasq gives that MAC 10.232.0.10 with a lease that never ends, so a revert keeps the address. dnsmasq also answers the appliance's DNS, forwarding to the runner's resolvers, which may be a loopback stub the appliance cannot reach. iptables accepts the bridge's traffic through Docker's FORWARD policy and NATs the appliance out. TN_HOST is then 10.232.0.10:443. Both modes now print TN_NETWORK and TN_RUNNER_IP: 10.0.2.2 under qemu's NAT, which is the runner's loopback, and 10.232.0.1 on the bridge. Bridge also prints TN_APPLIANCE_IP. Nothing is forwarded to 127.0.0.1 with bridge, so consumers that hard-code it keep user, the default, which is unchanged. stop now also takes the bridge network down, each part only if it is there, from what start noted before it changed anything, and start refuses while tnapp-br exists. The NIC stays a -nic in both modes, so it keeps its PCI slot and its name in the guest.
|
No findings. No human review required. I checked
|
Contributor
Contributor
|
This PR has been merged and conversations have been locked. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Under qemu's user-mode network the appliance is
10.0.2.15behind qemu's NAT, and only its ports are forwarded, to8443,8080and2222on the runner's loopback. That works for@nassuites, but not for a TrueNAS Connect stack on the runner:truenas.directname resolves to10.0.2.15, which nothing on the runner can reach.443, not on the forwarded8443.network: bridgegives the appliance an address the runner can route to, asha.shalready does for an HA pair. Tracked from truenas-connect/cui#121.Changes
New input
network(APPLIANCE_NETWORK):userorbridge, defaulting touser, which is unchanged.bridge_up(inappliance.sh) builds the network:tnapp-br, with the runner at10.232.0.1/24. This doesn't collide with HA's10.231.0.0/24, Docker's172.17.0.0/16or compose's networks.tnapp-tap, owned by the job's user, so qemu still runs without sudo.FORWARDaccepts traffic in and out of the bridge, past theDROPpolicy Docker sets.MASQUERADENATs10.232.0.0/24out of anything but the bridge.ip_forwardis turned on if it was off.52:54:00:e2:e0:00the address10.232.0.10. The lease is infinite, so a revert (which restores the guest's lease and moves its clock on) never finds it expired.10.232.0.1, forwarding to whatever the runner's/etc/resolv.confnames. HA runs--port=0instead, but this runner's resolver may be a loopback stub the appliance can't reach.--no-hostskeeps the runner's/etc/hostsout of the answers.The NIC stays a
-nicin both modes, using the tap backend withbridge. It is made the same way, so it keeps its PCI slot and its name in the guest.spare-nicsstay on their qemu hubs.addr <guest port>says where the runner reaches a port of the appliance.url,boot-report's ssh and the outputs use it.state.envrecords the mode, so laterrevert,callandprobecalls reach the right address.Outputs:
userbridgeTN_HOST127.0.0.1:844310.232.0.10:443TN_HOST_HTTP127.0.0.1:808010.232.0.10:80TN_SSH127.0.0.1:222210.232.0.10:22TN_NETWORK(new)userbridgeTN_RUNNER_IP(new)10.0.2.210.232.0.1TN_APPLIANCE_IP(new)10.232.0.10TN_NETWORKandTN_RUNNER_IP, so a consumer can be written once.user,10.0.2.2is qemu's alias for the runner's loopback.network,runner-ipandappliance-ip.stopnow waits for qemu to exit, then takes the bridge network down (bridge_down):network.env, whichbridge_upwrites before it changes anything, so it can clean up after a start that failed halfway.ip_forward.userappliance there is nothing to do, and it needs no sudo.startrefusals: it refuses whiletnapp-brexists, and refuses anetworkother thanuserorbridge.bridgeinstallsdnsmasq-baseandiptablesif they are missing, and requires passwordless sudo.HA:
networkis an error withha: true, asspare-nicsis.README: the input, an outputs table for both modes, and a section on
bridge.bridge, nothing is forwarded to127.0.0.1. A consumer that hard-codes it, or reaches the appliance ashost.docker.internal(webui's nginx container), keepsuser.10.232.0.1or on every address, while a port published only on127.0.0.1is out of reach.Testing
This repo's checks pass:
-x appliance.sh ha.sh lib.sh)python3 test_tnctl.pynode scripts/check-input-refs.jsThe network, in a privileged Debian 13 container (not a runner). The test ran
bridge_up, then put a network namespace on a veth in the bridge with the appliance's MAC, standing in for qemu's tap, and ranudhcpcin it:bridge_down, the links, the filter and nat rules, the dnsmasq processes andip_forward(0 before the test) were identical to a snapshot taken beforebridge_up.bridge_downis a no-op.Refusals:
startrefusesAPPLIANCE_NETWORK=bogus, and refuses whiletnapp-brexists.stopthen removes the leftover bridge.Not yet run on a GARM runner with a real appliance. A throwaway workflow in cui checks every acceptance criterion of truenas-connect/cui#121 there:
snapshot x/revert xprobestopuserunchangedIt's still to run. I'll add its output here before this leaves draft.
This directory changing makes github-truenas-runner build a new appliance image. Its smoke test boots with the default network, which is unchanged. Its
docs/e2e-appliances.mdmentions the network only for webui's move tousermode, which still holds, so it needs no change.Needs a NAS ticket in the title for the ticket check.