Skip to content

[fp8]: Read detected local world size environment variable - #6438

Open
tandede wants to merge 1 commit into
hpcaitech:mainfrom
tandede:hotfix/fp8-local-world-size
Open

[fp8]: Read detected local world size environment variable#6438
tandede wants to merge 1 commit into
hpcaitech:mainfrom
tandede:hotfix/fp8-local-world-size

Conversation

@tandede

@tandede tandede commented Aug 20, 2026

Copy link
Copy Markdown

📌 Checklist before creating the PR

  • I have created an issue for this PR for traceability
  • The title follows the standard format: [doc/gemini/tensor/...]: A concise description
  • I have installed pre-commit: pip install pre-commit && pre-commit install

🚨 Issue number

Fixed #6437

📝 What does this PR do?

process_group_is_intranode() previously detected the first available local-world-size environment variable but always read LOCAL_WORLD_SIZE. Jobs launched with OpenMPI or Slurm therefore raised KeyError before FP8 collectives could select the intranode path.

This change reads the detected variable and stops at the first match, preserving the existing priority order: torchrun, OpenMPI, then Slurm.

Regression tests cover:

  • torchrun, OpenMPI, and Slurm environment variables independently;
  • intranode and cross-node process groups;
  • deterministic priority when several variables are present.

Validation:

  • pytest -q --confcutdir=tests/test_fp8 tests/test_fp8/test_fp8_utils.py — 7 passed
  • pre-commit run --files colossalai/quantization/fp8.py tests/test_fp8/test_fp8_utils.py — all hooks passed

The repository-level test fixture invokes GPU accelerator cache hooks during collection, so the full GPU suite is left to CI; the new regression itself is CPU-safe.

💥 Checklist before requesting a review

  • I have linked my PR to an issue
  • My issue clearly describes the problem and includes a minimal reproduction
  • I have performed a self-review of my code
  • I have added thorough tests
  • I have added docstrings for all the functions/methods I implemented (no new functions or methods)

⭐️ Do you enjoy contributing to Colossal-AI?

  • 🌝 Yes, I do.
  • 🌚 No, I don't.

@tandede
tandede marked this pull request as ready for review August 20, 2026 08:35
@tandede
tandede requested a review from a team as a code owner August 20, 2026 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: FP8 intranode detection ignores OpenMPI and Slurm local size values

1 participant