setup: integrate reusable runner setup and Azure VMSS bootstrap

[Enrique Saurez (esaurez)]

Summary

• Extend provider-neutral Linux and Windows runner setup with validated additional labels while retaining the required platform/backend labels. Accept an otherwise fresh Windows runner directory containing only a prewarmed _work directory.
• Isolate VMSS identity checks, managed-disk and host preparation, reboot continuation, and managed-identity/Key Vault registration in scripts/setup/azure/. One example configuration per OS documents the Azure-specific inputs.
• Build deterministic, hash-verified Azure guest packages from the repository's authoritative runner setup scripts and adjacent tool manifest. Reject mismatched Rust toolchains, unsafe configurations, and custom data over Azure's decoded size limit. Packaging does not modify Azure resources or GitHub runners.
• Cover the adapters with offline configuration, package, identity, and installer tests, and wire Python, shell, and PowerShell checks into the existing CI actions.

[Copilot]

🟡 Changes recommended

Linux jobs can access bootstrap credentials through IMDS, and Windows NVMe selection can target the wrong managed disk.

5 open findings

• Block Linux runner access to IMDS after registration · New
• Require exact Azure NVMe disk LUN matching · New
• Reject root as the Linux admin username · New
• Prevent stale platform artifacts in reused output directories · New
• Make offline label validation case-insensitive · New

What changed in this PR

Integrates reusable runner setup with deterministic Azure VMSS bootstrap packages for Linux and Windows.

Changes:

• Adds validated runner labels and Windows prewarmed-directory support.
• Adds Azure identity, disk, reboot, registration, and packaging adapters.
• Extends offline tests and CI validation.

File	Description
scripts/​test_tool_versions.py	Tests runner labels and Windows directory handling.
scripts/​test_setup_azure.py	Tests Azure configuration and packaging.
scripts/​test_nvx_tools.py	Covers Azure setup change classification.
scripts/​setup/​setup-windows-whp.ps1	Supports extra labels and prewarmed _work.
scripts/​setup/​setup-linux-runner.sh	Supports extra runner labels.
scripts/​setup/​README.md	Documents reusable runner setup.
scripts/​setup/​azure/​windows/​vmss-bootstrap.ps1	Configures Windows VMSS host access.
scripts/​setup/​azure/​windows/​setup-data-disk.ps1	Prepares Windows managed disks.
scripts/​setup/​azure/​windows/​install-package.ps1	Verifies and installs bootstrap packages.
scripts/​setup/​azure/​windows/​config.example.json	Provides Windows VMSS example configuration.
scripts/​setup/​azure/​windows/​bootstrap-runner.ps1	Registers and resumes Windows runners.
scripts/​setup/​azure/​requirements.txt	Pins the packaging dependency.
scripts/​setup/​azure/​README.md	Documents Azure provisioning.
scripts/​setup/​azure/​linux/​verify-mshv.sh	Verifies MSHV host readiness.
scripts/​setup/​azure/​linux/​setup-mshv.sh	Prepares Azure Linux MSHV boot.
scripts/​setup/​azure/​linux/​setup-data-disk.sh	Prepares Linux managed disks.
scripts/​setup/​azure/​linux/​config.example.json	Provides Linux VMSS example configuration.
scripts/​setup/​azure/​linux/​bootstrap-runner.sh	Registers Linux VMSS runners.
scripts/​setup/​azure/​build_package.py	Builds deterministic Azure payloads.
requirements-dev.txt	Includes Azure packaging requirements.
.github/​actions/​validate-nvx/​action.yml	Runs Azure tests on both platforms.
.github/​actions/​check-quality/​action.yml	Checks new shell and PowerShell scripts.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

A stale Windows disk completion marker can allow runner registration before reboot-based disk setup finishes.

1 open finding

• Clear completion marker before scheduling a pending reboot · New

5 resolved since last review

• Require exact Azure NVMe disk LUN matching
• Block Linux runner access to IMDS after registration
• Make offline label validation case-insensitive
• Prevent stale platform artifacts in reused output directories
• Reject root as the Linux admin username

Previously missed (2)

In code that hasn't changed since last review

Add offline tests for Network Service firewall rule enforcement

scripts/​setup/​azure/​windows/​bootstrap-runner.ps1:86

This function is the security boundary preventing Network Service jobs from reaching IMDS, but the offline Windows tests never execute it; they only extract Get-RunnerName and Get-ExtraRunnerLabels. Add mocked coverage for rule creation and for rejecting a wrong principal, address, direction, action, or disabled rule so a firewall regression cannot pass CI.

Correct unsupported enableSsh guidance in failure advice

scripts/​setup/​azure/​windows/​vmss-bootstrap.ps1:151

The failure advice points operators to enableSsh=false, but this adapter has no such parameter or configuration field and always runs the OpenSSH setup before the packaged installer. Recommend only an actionable option supported here, such as selecting an image with OpenSSH capability support.

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Bootstrap ownership protection and provisioning correctness issues remain unresolved.

8 open issues

• Install fdisk before disk partitioning · New
• Set a trusted owner on the bootstrap directory · New
• Clear completion marker before scheduling a pending reboot
• Reject scratch paths overlapping the runner directory · New
• Avoid restarting an already active runner · New
• Disable provisioning after successful registration · New
• Validate mount ownership before overwriting fstab · New
• Verify GRUB uses the configured default entry · New

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Firewall validation and mounted-disk identity checks leave unresolved security and correctness risks.

9 open issues

• Incomplete metadata firewall rule validation · New
• Set a trusted owner on the bootstrap directory
• Install fdisk before disk partitioning
• Clear completion marker before scheduling a pending reboot
• Verify GRUB uses the configured default entry
• Validate mount ownership before overwriting fstab
• Disable provisioning after successful registration
• Avoid restarting an already active runner
• Reject scratch paths overlapping the runner directory

1 previously missed issue

In code that hasn't changed since last review

Verify mounted device identity after first mount

scripts/​setup/​azure/​linux/​setup-data-disk.sh:247

The first-mount path skips the device-identity check above. Attached disks restored from the same snapshot can share a PARTUUID or UUID, so mounting through the generated fstab entry can select a disk other than the configured LUN and still report success. Compare the mounted device's MAJ:MIN with TARGETDEV after mounting too, before bootstrap uses the volume.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Firewall reconciliation and Windows scratch-path validation leave unresolved security gaps.

6 open issues

• Reject scratch directories that contain the runner directory · New
• Avoid deleting metadata deny rules before replacement is verified · New
• Install fdisk before disk partitioning
• Clear completion marker before scheduling a pending reboot
• Reject trailing-period paths before runner overlap validation · New
• Set ZipInfo create_system for reproducible archives · New

7 resolved since last review

• Incomplete metadata firewall rule validation
• Set a trusted owner on the bootstrap directory
• Verify GRUB uses the configured default entry
• Validate mount ownership before overwriting fstab
• Disable provisioning after successful registration
• Avoid restarting an already active runner
• Reject scratch paths overlapping the runner directory

_{Review effort: Balanced}

[Pedro Henrique Penna (ppenna)]

A significant portion of this file is written in python. Why not simply write the entire file in python?

[Enrique Saurez (esaurez)]

It can be Python directly. The payload generator already installs python3 and orders bootstrap after cloud-final.service. It embeds all guest files in one cloud-init payload, so additional helper files do not require separate downloads. Neither distribution nor interpreter bootstrapping requires this Bash wrapper. A direct Python controller can handle configuration, identity, HTTP/token requests, reboot coordination, and runner orchestration while retaining the existing OS-specific Bash helpers. That structural conversion remains a separate, deferred refactor; the current correctness fixes do not change the controller language.

[Copilot]

🟡 Changes recommended

Disk-preservation and effective firewall-enforcement gaps remain, alongside provisioning validation and package reproducibility issues.

6 open issues

• Prevent repartitioning disks with unrecognized existing data · New
• Verify effective firewall enforcement before runner registration · New
• Install fdisk before disk partitioning
• Clear completion marker before scheduling a pending reboot
• Reject reserved nvx-runner administrator username · New
• Normalize PowerShell line endings for reproducible packages · New

4 resolved since last review

• Avoid deleting metadata deny rules before replacement is verified
• Reject scratch directories that contain the runner directory
• Set ZipInfo create_system for reproducible archives
• Reject trailing-period paths before runner overlap validation

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Packaging fixes remain necessary, and disk initialization, reboot recovery, and credential isolation require isolated-VM validation and human review.

8 open issues

• Verify effective firewall enforcement before runner registration
• Prevent repartitioning disks with unrecognized existing data
• Install fdisk before disk partitioning
• Clear completion marker before scheduling a pending reboot
• Validation accepts reserved Windows DOS device names in paths · New
• Gzip output embeds host-specific OS byte, breaking cross-platform determinism · New
• Normalize PowerShell line endings for reproducible packages
• Reject reserved nvx-runner administrator username

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Windows disk preparation has unresolved data-preservation risks and an unsupported formatting parameter that blocks provisioning.

11 open issues

• Validate entire disk before allowing initialization · New
• Avoid claiming unmarked GPT disks without verified ownership · New
• Pass resolved volume to Format-Volume via InputObject · New
• Verify effective firewall enforcement before runner registration
• Prevent repartitioning disks with unrecognized existing data
• Install fdisk before disk partitioning
• Clear completion marker before scheduling a pending reboot
• Gzip output embeds host-specific OS byte, breaking cross-platform determinism
• Validation accepts reserved Windows DOS device names in paths
• Normalize PowerShell line endings for reproducible packages
• Reject reserved nvx-runner administrator username

_{Review effort: Balanced}

[Pedro Henrique Penna (ppenna)]

LGTM

[Copilot]

🔵 Needs a closer look

Privileged disk operations, credential isolation, and reboot orchestration require human review and isolated platform validation.

7 open issues

• Verify effective firewall enforcement before runner registration
• Prevent repartitioning disks with unrecognized existing data
• Install fdisk before disk partitioning
• Clear completion marker before scheduling a pending reboot
• Resolve default scratch path before overlap validation · New
• Normalize PowerShell line endings for reproducible packages
• Reject reserved nvx-runner administrator username

5 resolved since last review

• Pass resolved volume to Format-Volume via InputObject
• Avoid claiming unmarked GPT disks without verified ownership
• Validate entire disk before allowing initialization
• Gzip output embeds host-specific OS byte, breaking cross-platform determinism
• Validation accepts reserved Windows DOS device names in paths

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Privileged disk initialization, credential isolation, and cross-reboot provisioning require isolated-host validation and final human review.

7 open issues

• Verify effective firewall enforcement before runner registration
• Prevent repartitioning disks with unrecognized existing data
• Install fdisk before disk partitioning
• Clear completion marker before scheduling a pending reboot
• Resolve default scratch path before overlap validation
• Normalize PowerShell line endings for reproducible packages
• Reject reserved nvx-runner administrator username

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Privileged disk preparation, credential isolation, and reboot handling need isolated Azure validation and human review, including the unresolved extension timeout risk.

6 open issues

• Verify effective firewall enforcement before runner registration
• Prevent repartitioning disks with unrecognized existing data
• Install fdisk before disk partitioning
• Clear completion marker before scheduling a pending reboot
• Normalize PowerShell line endings for reproducible packages
• Reject reserved nvx-runner administrator username

1 previously missed issue

In code that hasn't changed since last review

Move long-running bootstrap out of Custom Script Extension timeout

scripts/​setup/​azure/​windows/​install-package.ps1:76

This synchronous call keeps initial bootstrap inside Windows Custom Script Extension, including the full-device RAW-disk scan and prerequisite installation. The extension has a 90-minute execution limit, so provisioning that exceeds it is reported as failed despite the continuation task's four-hour budget. Registering an AtStartup task does not move the current invocation outside that limit. Stage bootstrap in an independent SYSTEM scheduled task and start it immediately, allowing the extension to return after successful handoff. Keep bootstrap completion and failure reporting separate from extension staging.

1 resolved since last review

• Resolve default scratch path before overlap validation

_{Review effort: Balanced}

[Enrique Saurez (esaurez)]

Addressed the extension-timeout finding from the latest review in 06b90fc: the extension now verifies/stages the package and starts the persistent SYSTEM task without waiting for bootstrap. The startup trigger, four-hour budget, and retries are preserved. Staging acknowledgement is separate from final readiness; bootstrap logs errors, emits a failure marker, and rethrows failures.

e4e1420 fixes the Linux CI failure in the CLI replay fixture: Unix roots now link the built fake OpenVMM instead of creating writable copies while other test threads launch children, eliminating that ETXTBSY exposure. Production launch behavior, Windows copies, parallelism, and recorded expectations are unchanged.

The repaired Linux CLI target (13 tests), Rust lint configurations, Azure package suite, and native Windows mocked cases pass locally. As of 14:11 PDT, CI run 39324816007 has passed the Linux/Windows Rust checks, Linux/Windows CLI checks, and lint/type/format. The overall matrix is still in progress with no failed jobs.

[Copilot]

🔵 Needs a closer look

Privileged disk initialization, metadata isolation, and reboot recovery need final human review and validation on isolated Azure instances.

6 open issues

• Verify effective firewall enforcement before runner registration
• Prevent repartitioning disks with unrecognized existing data
• Install fdisk before disk partitioning
• Clear completion marker before scheduling a pending reboot
• Normalize PowerShell line endings for reproducible packages
• Reject reserved nvx-runner administrator username

_{Review effort: Balanced}