Draft
fix: add shell-level timeout to perf stat capability probes in metadata collection#717
Conversation
Co-authored-by: romirdes <86635949+romirdes@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix indefinite hanging during metadata collection on large VMs
fix: add shell-level timeout to perf stat capability probes in metadata collection
Aug 31, 2026
…arge virtualized instances perf list --json can also stall indefinitely on some hypervisors (e.g. m6i.16xlarge), just like perf stat. The stderr.txt from the failing test showed PerfSpect was stuck at "collecting metadata", and perf list is the remaining unguarded perf command in that stage. Wrap both scriptPerfSupportedEvents and scriptPerfAllSupportedEvents with timeout 30 to bound them consistently with the perf stat probes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On large virtualized x86 instances (e.g.
m6i.16xlarge), the hypervisor stallsperf_event_openindefinitely rather than returning an error. Since metadata probe scripts run with no timeout (timeout := 0in the controller), PerfSpect hangs forever at "collecting metadata" and never reaches "collecting metrics". Smaller VMs (e.g.c6i.2xl) fail fast with a non-zero exit → expected error; bare-metal works normally.Changes
cmd/metrics/metadata.go: Prefix all 9perf statcapability probeScriptTemplatevalues withtimeout 30:Affected probes:
scriptPerfStatInstructions,scriptPerfStatRefCycles,scriptPerfStatPEBS,scriptPerfStatOCR,scriptPerfStatTMA,scriptPerfStatAMDUncoreProbe,scriptPerfStatFixedInstr,scriptPerfStatFixedCycles,scriptPerfStatFixedRefCycles.Each probe normally completes in ~1 s (it runs
sleep 1as the workload). All probes run concurrently, so the 30 s budget is ~30× the normal wall-clock cost and poses no risk of false failures on loaded instances. On a hung instance,timeoutexits with code 124 (non-zero) → probe treated as unsupported → fast-fail with a clear error instead of an indefinite hang.