[Pedro Henrique Penna (ppenna)]

Fixes #1030

NVX CLI / Windows failed twice in a month, in the dev push run 40946801274 and in the first run of #797, because a GitHub-hosted runner held up the unittest process's first powershell.exe launch for longer than the 30 s that test_windows_acceptance_forwards_benchmark_scratch gave it. Neither failure showed what PowerShell had done. As the issue suggests, the PowerShell launches of the CLI tests now get one generous limit from a shared constant by default, and a launch that reaches its limit reports what it printed. The tests neither warm PowerShell up nor retry a launch.

Changes

1. scripts: give the CLI tests' PowerShell launches a shared 120 s limit
   • The new scripts/powershell_testing.py holds POWERSHELL_TIMEOUT, 120 s, the limit that test_tool_versions.py's launches already had, and run_powershell(), which runs a command line under it, or under a limit that its test passes, and captures its output as text.
   • All 12 PowerShell launches of the CLI tests go through it, with their arguments, working directories, and environments unchanged: 8 in test_nvx_tools.py and 3 in test_setup_azure.py, which had 30 s, or 90 s for the installer test, and _run_powershell() in test_tool_versions.py.
   • Only the storage events test passes a limit of its own, its 300 s: its save step runs the real performance storage-events, which queries the host's event logs for up to 120 s each.
   • The new scripts/test_powershell_testing.py, which both CLI validation steps run, checks what run_powershell() passes to subprocess.run(), and that no scripts/test_*.py module starts PowerShell through subprocess itself. At dev, that check finds exactly the 12 launches above.
   • doc/ci.md describes the limit and its exception, and doc/project-structure.md the module.
2. scripts: report what a timed-out PowerShell launch of the CLI tests printed
   • run_powershell() turns a TimeoutExpired into an AssertionError that names the program and the limit, then gives what PowerShell wrote to stdout and to stderr until the kill, or nothing. The TimeoutExpired, which names the command line, stays its cause.
   • subprocess.run() collects a killed child's output as text on Windows, but leaves bytes, or None, elsewhere, where the tool manifest tests run PowerShell 7, so the report decodes bytes.
   • unittest reports failures only once the whole run ends, but NVX CLI / Windows has a 15-minute limit, which a PowerShell that hung at every launch would exceed. So run_powershell() also writes each report to stderr as soon as it kills the launch, and a job that its limit cancels keeps it.

A launch that a stall holds past the limit now fails like this, on prometheus28 below. The report reaches the log when the launch is killed, before the subtest's FAIL, and again in the failure's details:

test_windows_acceptance_forwards_benchmark_scratch (scripts.test_nvx_tools.CiConfigurationTests.test_windows_acceptance_forwards_benchmark_scratch) ...
powershell.exe did not exit within 120 s; until it was killed, it printed:
stdout: nothing
stderr: nothing

  test_windows_acceptance_forwards_benchmark_scratch (scripts.test_nvx_tools.CiConfigurationTests.test_windows_acceptance_forwards_benchmark_scratch) (scratch='') ... FAIL

======================================================================
FAIL: test_windows_acceptance_forwards_benchmark_scratch (scripts.test_nvx_tools.CiConfigurationTests.test_windows_acceptance_forwards_benchmark_scratch) (scratch='')
----------------------------------------------------------------------
Traceback (most recent call last):
  ...
subprocess.TimeoutExpired: Command '['powershell.exe', '-NoProfile', '-NonInteractive', '-Command', ...]' timed out after 120 seconds

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  ...
AssertionError: powershell.exe did not exit within 120 s; until it was killed, it printed:
stdout: nothing
stderr: nothing

Validation

prometheus28 (Windows Server 2025 Datacenter, bare metal, Python 3.14.7, Windows PowerShell 5.1.26100), in a separate worktree, with PyYAML in a separate virtual environment:

• The stall did not occur on its own: in the full Windows CLI suite at dev, the 76 PowerShell launches took 0.29–1.77 s, the first 0.52 s.
• To reproduce it, a helper ran the tests and suspended the powershell.exe processes that they started, with NtSuspendProcess, for a set time. At dev e42b17959, holding the forwards test's first launch for 40 s reproduced the incident: its scratch='' subtest raised TimeoutExpired after 30.03 s, with no output, and its second subtest passed in 0.41 s.
• At the first commit, the same 40 s stall passed, the launch taking 40.40 s, and so did 35 s stalls of each of the first three launches of the first three PowerShell tests, which recalls the slow later launches of 2026-10-23.
• At 49ec8f1f8, the head before the rebase onto dev 4434b1af0, which changed none of these files, and before the review fixes, a 130 s stall failed after 120.03 s with the report. A stall that began 1 s into a script, after it printed a line to each stream, reported both lines.
• The full Windows CLI unittest command ran 1,491 tests at dev and 1,494 at 49ec8f1f8, with the same 55 failures and errors: test_setup_azure's Linux and MSHV shell tests, which run bash scripts that fail on this host at dev too.
• At the head, 70306c00e, the 130 s stall produced the output above.

nvx-kvm-test (Azure Linux 3, Python 3.12.16), in a separate worktree at 49ec8f1f8:

• The Linux CLI unittest command: 1,494 tests, OK, 35 skipped.
• The host has no PowerShell 7, so run_powershell() ran sh under a 3 s limit to check the report on POSIX, where run() leaves bytes or None: it reported the decoded lines, and nothing when nothing arrived.

Local, Windows 11 (Python 3.14.7, Windows PowerShell 5.1):

• Each commit: ruff check and ruff format --check, pyright for Linux and for Windows, compileall, the Markdown link check, test_powershell_testing.py, and the test_windows_* tests of test_nvx_tools.py; at the head, also the Windows tests of test_setup_azure.py, and a unittest -v run of a real powershell.exe that hangs, which printed the report before FAIL.
• The full Windows CLI unittest command at 49ec8f1f8: 1,494 tests. Its 27 failures and errors are in 8 tests that run shell scripts, test_microvm_tests' host loopback and private network ingress tests and test_setup_azure's Linux and MSHV setup tests, and are the same as those 8 tests' at dev on this workstation.

Left to CI: the hosted windows-latest image, on which the stall happened, and PowerShell 7 on the hosted Ubuntu runner. The stall's cause stays unknown, as in the issue: this change makes a stall shorter than 120 s harmless and a longer one visible.

[Copilot]

🟡 Changes recommended

Repeated timeouts can exhaust the Windows CLI job’s 15-minute limit and lose deferred failure reports because they are not flushed to stderr before raising.

1 open issue

• Flush timeout reports before Windows CLI job cancellation · New

What changed in this PR

Reduces Windows CLI test flakes caused by slow PowerShell launches and improves diagnostics when a launch times out, addressing #1030.

Changes:

• Centralizes PowerShell launches under a 120 s default timeout, preserving the storage events test’s 300 s limit.
• Reports captured stdout and stderr on timeout, decoding byte output when needed.
• Adds helper and launch-policy tests to both CLI validation steps and documents the behavior.

File	Description
scripts/​test_tool_versions.py	Uses the shared PowerShell launcher.
scripts/​test_setup_azure.py	Migrates Azure setup tests to the shared launcher.
scripts/​test_powershell_testing.py	Tests launch options, timeout diagnostics, and shared-helper use.
scripts/​test_nvx_tools.py	Migrates CLI tests while retaining the storage events timeout.
scripts/​powershell_testing.py	Defines the default timeout and captured-output reporting.
doc/​project-structure.md	Describes the new testing helper.
doc/​ci.md	Documents PowerShell test limits and timeout reporting.
.github/​actions/​validate-nvx/​action.yml	Runs the new tests on Linux and Windows.

_{Review effort: Max}

[Copilot]

🟡 Changes recommended

Repeated 120-second waits can exhaust the CLI job's 15-minute timeout before captured-output diagnostics reach the log, so reports must be emitted and flushed before raising.

1 open issue

• Flush timeout reports before raising to preserve diagnostics · New

1 resolved since last review

• Flush timeout reports before Windows CLI job cancellation

_{Review effort: Max}

[Copilot]

🟡 Changes recommended

The timeout documentation should distinguish the shared 120-second default from explicit overrides, notably the storage events test’s 300-second timeout.

1 open issue

• Clarify 120 s timeout is a default with a 300 s exception · New

1 resolved since last review

• Flush timeout reports before raising to preserve diagnostics

_{Review effort: Max}

[Copilot]

🟢 Approval recommended

No active review issues block approval of the PowerShell test timeout and diagnostic changes.

No open issues 🎉

1 resolved since last review

• Clarify 120 s timeout is a default with a 300 s exception

_{Review effort: Max}