Skip to content

NAS-145291 / 28.0.0-BETA.1 / truenas-appliance-runner: network: bridge, the appliance on a bridge with the runner - #30

Merged
william-gr merged 1 commit into
masterfrom
appliance-bridge-network
Nov 17, 2026
Merged

william-gr merged 1 commit into
masterfrom
appliance-bridge-network

Conversation

@william-gr

Copy link
Copy Markdown
Contributor

Under qemu's user-mode network the appliance is 10.0.2.15 behind qemu's NAT, and only its ports are forwarded, to 8443, 8080 and 2222 on the runner's loopback. That works for @nas suites, but not for a TrueNAS Connect stack on the runner:

  • Registration publishes the NAS's own IPs, so its truenas.direct name resolves to 10.0.2.15, which nothing on the runner can reach.
  • A client then connects to that name on 443, not on the forwarded 8443.

network: bridge gives the appliance an address the runner can route to, as ha.sh already does for an HA pair. Tracked from truenas-connect/cui#121.

Changes

  • New input network (APPLIANCE_NETWORK): user or bridge, defaulting to user, which is unchanged.

  • bridge_up (in appliance.sh) builds the network:

    • Bridge: tnapp-br, with the runner at 10.232.0.1/24. This doesn't collide with HA's 10.231.0.0/24, Docker's 172.17.0.0/16 or compose's networks.
    • Tap: tnapp-tap, owned by the job's user, so qemu still runs without sudo.
    • iptables:
      • FORWARD accepts traffic in and out of the bridge, past the DROP policy Docker sets.
      • MASQUERADE NATs 10.232.0.0/24 out of anything but the bridge.
      • ip_forward is turned on if it was off.
    • dnsmasq:
      • DHCP: static, giving the fixed MAC 52:54:00:e2:e0:00 the address 10.232.0.10. The lease is infinite, so a revert (which restores the guest's lease and moves its clock on) never finds it expired.
      • DNS: it also answers DNS on 10.232.0.1, forwarding to whatever the runner's /etc/resolv.conf names. HA runs --port=0 instead, but this runner's resolver may be a loopback stub the appliance can't reach. --no-hosts keeps the runner's /etc/hosts out of the answers.
  • The NIC stays a -nic in both modes, using the tap backend with bridge. It is made the same way, so it keeps its PCI slot and its name in the guest. spare-nics stay on their qemu hubs.

  • addr <guest port> says where the runner reaches a port of the appliance. url, boot-report's ssh and the outputs use it. state.env records the mode, so later revert, call and probe calls reach the right address.

  • Outputs:

    Variable user bridge
    TN_HOST 127.0.0.1:8443 10.232.0.10:443
    TN_HOST_HTTP 127.0.0.1:8080 10.232.0.10:80
    TN_SSH 127.0.0.1:2222 10.232.0.10:22
    TN_NETWORK (new) user bridge
    TN_RUNNER_IP (new) 10.0.2.2 10.232.0.1
    TN_APPLIANCE_IP (new) not set 10.232.0.10
    • Both modes print TN_NETWORK and TN_RUNNER_IP, so a consumer can be written once.
    • Under user, 10.0.2.2 is qemu's alias for the runner's loopback.
    • The action also has the new outputs network, runner-ip and appliance-ip.
  • stop now waits for qemu to exit, then takes the bridge network down (bridge_down):

    • It removes each part only if it is there.
    • It works from network.env, which bridge_up writes before it changes anything, so it can clean up after a start that failed halfway.
    • It restores ip_forward.
    • For a user appliance there is nothing to do, and it needs no sudo.
  • start refusals: it refuses while tnapp-br exists, and refuses a network other than user or bridge. bridge installs dnsmasq-base and iptables if they are missing, and requires passwordless sudo.

  • HA: network is an error with ha: true, as spare-nics is.

  • README: the input, an outputs table for both modes, and a section on bridge.

    • No loopback forward: with bridge, nothing is forwarded to 127.0.0.1. A consumer that hard-codes it, or reaches the appliance as host.docker.internal (webui's nginx container), keeps user.
    • What the appliance reaches on the runner differs: on the bridge it reaches what listens on 10.232.0.1 or on every address, while a port published only on 127.0.0.1 is out of reach.

Testing

  • This repo's checks pass:

    • shellcheck 0.10.0 (-x appliance.sh ha.sh lib.sh)
    • python3 test_tnctl.py
    • node scripts/check-input-refs.js
  • The network, in a privileged Debian 13 container (not a runner). The test ran bridge_up, then put a network namespace on a veth in the bridge with the appliance's MAC, standing in for qemu's tap, and ran udhcpc in it:

    guest DHCP: lease=4294967295 router=10.232.0.1 dns=10.232.0.1 ip=10.232.0.10
    guest DNS via 10.232.0.1: github.com -> 4.228.31.150
    guest ping 1.1.1.1: 2 packets transmitted, 2 received, 0% packet loss
    guest -> runner 10.232.0.1 and runner -> guest: both answer
    DHCPOFFER(tnapp-br) 10.232.0.10 52:54:00:e2:e0:00 / DHCPACK(tnapp-br) 10.232.0.10 52:54:00:e2:e0:00
    
    • After bridge_down, the links, the filter and nat rules, the dnsmasq processes and ip_forward (0 before the test) were identical to a snapshot taken before bridge_up.
    • A second bridge_down is a no-op.
  • Refusals:

    • start refuses APPLIANCE_NETWORK=bogus, and refuses while tnapp-br exists.
    • stop then removes the leftover bridge.
  • Not yet run on a GARM runner with a real appliance. A throwaway workflow in cui checks every acceptance criterion of truenas-connect/cui#121 there:

    • runner, container and NAS both ways
    • DNS, internet and NTP
    • snapshot x / revert x
    • probe
    • nothing left after stop
    • user unchanged

    It's still to run. I'll add its output here before this leaves draft.

This directory changing makes github-truenas-runner build a new appliance image. Its smoke test boots with the default network, which is unchanged. Its docs/e2e-appliances.md mentions the network only for webui's move to user mode, which still holds, so it needs no change.

Needs a NAS ticket in the title for the ticket check.

…with the runner

Under qemu's user-mode network the appliance is 10.0.2.15 behind qemu's
NAT, and only its ports are forwarded, to 8443, 8080 and 2222 on the
runner's loopback. A TrueNAS Connect stack on the runner cannot work with
that: registration publishes the NAS's own IPs, so its truenas.direct
name resolves to 10.0.2.15, and a client then connects to it on 443
(truenas-connect/cui#121).

network: bridge (APPLIANCE_NETWORK) puts the appliance on a bridge,
tnapp-br, with the runner at 10.232.0.1. qemu's NIC is a tap on it with a
fixed MAC, and dnsmasq gives that MAC 10.232.0.10 with a lease that never
ends, so a revert keeps the address. dnsmasq also answers the
appliance's DNS, forwarding to the runner's resolvers, which may be a
loopback stub the appliance cannot reach. iptables accepts the bridge's
traffic through Docker's FORWARD policy and NATs the appliance out.
TN_HOST is then 10.232.0.10:443.

Both modes now print TN_NETWORK and TN_RUNNER_IP: 10.0.2.2 under qemu's
NAT, which is the runner's loopback, and 10.232.0.1 on the bridge. Bridge
also prints TN_APPLIANCE_IP. Nothing is forwarded to 127.0.0.1 with
bridge, so consumers that hard-code it keep user, the default, which is
unchanged.

stop now also takes the bridge network down, each part only if it is
there, from what start noted before it changed anything, and start
refuses while tnapp-br exists. The NIC stays a -nic in both modes, so it
keeps its PCI slot and its name in the guest.
@github-actions

Copy link
Copy Markdown

No findings. No human review required.

I checked network: bridge against the paths a caller would take:

  • Default unchanged: with user the default, the -nic string is built exactly as before, and TN_HOST, TN_HOST_HTTP and TN_SSH resolve through addr to the same 127.0.0.1:<port> values. stop on a user-mode appliance still needs no sudo, because bridge_down returns early when there is no network.env and no tnapp-br.
  • Later commands reach the right address: state.env records APPLIANCE_NETWORK. revert, probe, call and boot-report all call state before url/addr, so they use the mode start recorded.
  • Nothing leaks into the env file: bridge_up writes only through say (stderr), sysctl -q and a daemonised dnsmasq, so nothing gets into the TN_* block that start prints on stdout.
  • Cleanup survives a half-finished start: bridge_down is idempotent. It checks each iptables rule with -C before deleting it, removes links only if they exist, and puts ip_forward back to the value saved in network.env before anything changed.
  • HA and invalid values are refused: network with ha: true and values other than user/bridge both fail the step instead of being ignored.
  • Docs match: README and action.yml document the new input, the new outputs, and how the outputs differ between the two modes.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. No findings.

Nothing blocks and no human review is needed.

@bugclerk bugclerk changed the title truenas-appliance-runner: network: bridge, the appliance on a bridge with the runner NAS-145291 / 28.0.0-BETA.1 / truenas-appliance-runner: network: bridge, the appliance on a bridge with the runner Nov 17, 2026
@bugclerk

Copy link
Copy Markdown
Contributor

@william-gr
william-gr marked this pull request as ready for review November 17, 2026 11:39
@william-gr
william-gr merged commit fa57177 into master Nov 17, 2026
8 of 9 checks passed
@william-gr
william-gr deleted the appliance-bridge-network branch November 17, 2026 11:39
@bugclerk

Copy link
Copy Markdown
Contributor

This PR has been merged and conversations have been locked.
If you would like to discuss more about this issue please use our forums or raise a Jira ticket.

@iXsystems iXsystems locked as resolved and limited conversation to collaborators Nov 17, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants