Skip to content

NAS-145202 / 28.0.0-BETA.1 / truenas-appliance-runner: set the appliance's clock after a revert - #29

Merged
william-gr merged 2 commits into
masterfrom
appliance-revert-clock
Nov 10, 2026
Merged

william-gr merged 2 commits into
masterfrom
appliance-revert-clock

Conversation

@william-gr

Copy link
Copy Markdown
Contributor

truenas-appliance revert does loadvm, which restores the appliance's memory, its clock included. After a revert the appliance ran minutes behind the runner, with no NTP to catch up with, so a suite's two-factor codes, tokens and timestamps made on the runner's clock were refused or wrong. truenas-connect/cui#79's two-factor spec hit it on a retry: the right code was refused.

Changes

  • qemu gives the guest agent a channel: virtio-serial-pci with a virtserialport named org.qemu.guest_agent.0 on $APPLIANCE_DIR/qga.sock. TrueNAS ships qemu-guest-agent (scale-build's conf/build.manifest), which starts once the port appears.
  • tnctl.py set-time SOCKET: guest-sync with a fresh id first (a loadvm restores the channel's buffers too), then guest-set-time with the runner's time, then reads the clock back and fails if it is over 2 s off. Logs how far off it was.
    • On TrueNAS guest-set-time sets the system clock and then fails, because qemu-ga runs /sbin/hwclock and TrueNAS does not ship it. The hardware clock does not matter here (loadvm restores it, the guest reads it only at boot), so the read-back decides.
    • qga() closes its socket's file too: otherwise the connection stays open after an error, and qemu's chardev serves one client at a time.
  • revert runs set-time after loadvm. If it fails, the revert still succeeds, with a ::warning:: that the clock is behind.
  • probe checks the clock through middleware's system.info after its revert, and fails if it is over 5 s off.
  • README: the clock behaviour, next to revert.

Testing

  • python3 test_tnctl.py: 10 tests, four new against a fake guest agent (a stale reply in the channel, the clock set, a guest without hwclock, a clock that does not take, no agent).
  • shellcheck clean. qemu 10.0 (Debian 13, the runner image's base) starts with the new devices.
  • On a real appliance, through truenas-connect/cui#82, which pointed e2e.yml at this branch and ran truenas-appliance probe seeded after the suite (run):
    tnctl: guest clock set: it was -158.9s off
    appliance: revert to seeded took 6.3s, undid the change, and left the clock -0.2s off
    

This directory changing makes github-truenas-runner build a new appliance image; its appliance.yml probe then checks the clock on every revert.

Needs a NAS ticket in the title for the ticket check.

loadvm restores the appliance's memory, its clock included, so after a
revert the appliance runs minutes behind the runner until it catches up,
and it has no NTP to catch up with. A suite's two-factor codes, tokens
and timestamps, made on the runner's clock, are then refused or wrong
(truenas-connect/cui#79's two-factor spec after a retry).

qemu now gives the guest agent a channel (virtio-serial,
org.qemu.guest_agent.0 on $DIR/qga.sock); TrueNAS ships qemu-ga, which
starts once the port appears. tnctl's set-time syncs the channel with
guest-sync, sets the clock with guest-set-time, and checks it took.
revert runs it after loadvm and warns, without failing, if it cannot;
probe checks the clock after its revert and fails if it is over 5 s off.
On a TrueNAS appliance guest-set-time sets the system clock and then
fails: qemu-ga runs /sbin/hwclock to copy it to the hardware clock, and
TrueNAS does not ship hwclock. revert warned with the clock 0.4 s off
(truenas-connect/cui#82). The hardware clock does not matter here, so the
clock check after the call decides, and the call's error is reported only
when the clock did not take or the error is about something else.

qga() also closes its socket's file: socket.close() leaves the connection
open while the file lives, an error's traceback kept it alive, and
qemu's chardev serves one client at a time, so the next call hung.
@github-actions

Copy link
Copy Markdown

No findings. No human review needed.

What I checked:

  • qemu wiring (appliance.sh start): the virtio-serial-pci device, the virtserialport and the qga.sock chardev are always added. savevm and loadvm run in the same job against the same qemu command line, so no existing snapshot has to match the new device set. The image build fetches these scripts at the commit that changed this directory, so the image and the scripts stay in step.
  • tnctl.py qga(): the guest-sync with a random id skips anything stale, including the 0xFF-prefixed replies guest-sync-delimited sends. An agent that is missing or stopped shows up as a socket timeout (TimeoutError, which is an OSError), and main() turns that into a non-zero exit. Closing the file and the socket together keeps the next call from hanging on qemu's one-client chardev.
  • cmd_set_time: the read-back decides whether the clock was set, not what guest-set-time returns. That is correct whichever way a given qemu-ga build orders settimeofday and the hwclock check. When the clock still has not taken, the call exits non-zero.
  • revert: if set-time fails, you get a ::warning:: on stderr and the revert still succeeds. stdout still carries only the elapsed time, which probe captures. probe fails when the skew is over 5 s, which matches what the README says.
  • Tests: the new fake-agent tests can fail. They cover a stale reply in the channel, a clock that does not take, a guest without hwclock, and no agent at all.
  • CI: the only failure is check-ticket, because the title has no NAS ticket yet. The PR description already says it needs one, so this is not a code finding.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. No findings.

Nothing blocks and no human review is needed.

@bugclerk bugclerk changed the title truenas-appliance-runner: set the appliance's clock after a revert NAS-145202 / 28.0.0-BETA.1 / truenas-appliance-runner: set the appliance's clock after a revert Nov 10, 2026
@bugclerk

Copy link
Copy Markdown
Contributor

@william-gr
william-gr merged commit 9c129b2 into master Nov 10, 2026
8 of 9 checks passed
@william-gr
william-gr deleted the appliance-revert-clock branch November 10, 2026 20:16
@bugclerk

Copy link
Copy Markdown
Contributor

This PR has been merged and conversations have been locked.
If you would like to discuss more about this issue please use our forums or raise a Jira ticket.

@iXsystems iXsystems locked as resolved and limited conversation to collaborators Nov 10, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants