Skip to content

[BugFix]: dataloader_multithread test to propagate shard thread failures as non-zero exit code - #505

Open
levxn wants to merge 1 commit into
ROCm:developfrom
levxn:fix/dataloader-multithread-exit-code
Open

[BugFix]: dataloader_multithread test to propagate shard thread failures as non-zero exit code#505
levxn wants to merge 1 commit into
ROCm:developfrom
levxn:fix/dataloader-multithread-exit-code

Conversation

@levxn

@levxn levxn commented Aug 10, 2026

Copy link
Copy Markdown
Member

Motivation

Fixes #499: Dataloader tests don't check return value, can return false positive

Technical Details

  • Replaced std::vector<std::thread> with std::vector<std::future<int>>, launching each shard via std::async(std::launch::async, thread_func, ...) instead of std::thread.
  • After launching all shards, the results are collected via future::get(), which blocks until each shard completes and returns its actual status.
  • If any shard returns non-zero, its shard id and status are printed, and main() now returns a non-zero exit code instead of always returning 0.

Test Plan

Built rocAL with the HIP backend, then manually forced a shard failure by requesting more GPU ids than physically exist (num_shards=3 num_gpus=3 on a machine with 2 GPUs), so the third shard's rocalCreate fails cleanly. Compared exit codes before and after the fix using the same repro command.

Test Result

Before the fix, the shard failure is printed but the process still exits 0:

[ERR] {rocalCreate} Failed to init the Rocal context, { MasterGraph } ERROR: HIP Device(%d) out of range2 Could not create the Rocal context exit code: 0

After the fix, the same failure is now detected and surfaced as a non-zero exit code:

[ERR] {rocalCreate} Failed to init the Rocal context, { MasterGraph } ERROR: HIP Device(%d) out of range2 Could not create the Rocal context shard_id: 2 failed with status -1 exit code: 255

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Issue]: Dataloader tests don't check return value, can return false positive

2 participants