Skip to content

fix(train): select XCCL as the XPU distributed backend instead of NCCL - #3786

Open
li-lizhe wants to merge 1 commit into
modelscope:mainfrom
li-lizhe:fix/xpu-dist-backend-xccl
Open

li-lizhe wants to merge 1 commit into
modelscope:mainfrom
li-lizhe:fix/xpu-dist-backend-xccl

Conversation

@li-lizhe

Copy link
Copy Markdown
Contributor

Summary

get_default_dist_backend() returns "nccl" for every device that is not Ascend NPU. Since
get_accelerator_device_type() does select XPU, an XPU run reaches
init_process_group(backend="nccl") and cannot start at all on a build without NCCL
(RuntimeError: Distributed package doesn't have NCCL built in) - the same class of failure #3771
fixed for Ascend NPU. This is the follow-up tracked in #3777, which was kept out of #3771 only
because that change had no XPU hardware to exercise the new path.

Change

  • add the XPU entry to _ACCELERATOR_DIST_BACKENDS: "xpu": ("xccl", "ccl");
  • the existing probe logic does the rest: torch.distributed.is_xccl_available() wins, a build that
    only exposes the former oneCCL name (is_ccl_available()) resolves to "ccl", and a build that
    exposes no probe keeps the first (canonical) name, "xccl" - the same convention already used for
    HCCL on Ascend;
  • docstring/comment updated: the "XPU is deliberately absent" note that this follow-up was waiting
    on is gone;
  • tests/test_distributed_device_setup.py: the case that pinned XPU to the historical "nccl" is
    replaced by three: XPU resolves to "xccl", "xccl" wins when both probes report available, and
    "ccl" is used when only that probe does.

Fixes #3777.

Verification

probe state on the build get_default_dist_backend() for XPU
before this change "nccl"
is_xccl_available() true "xccl"
only is_ccl_available() true "ccl"
no probe exposed "xccl" (first/canonical name)

Local CPU-only venv (torch 2.8.0+cpu, pytest 9.1.1), pytest tests/test_distributed_device_setup.py:
10 passed with this change; with only the _ACCELERATOR_DIST_BACKENDS line reverted and the new
tests kept: 3 failed, 7 passed (the three new XPU cases), so the cases do lock the behaviour.

Not tested: a real 2-rank torchrun on Intel XPU with XCCL. No XPU device exists in this
environment, so the runtime path is only verified as far as the backend selection itself - I am
flagging that here rather than claiming an XPU run. The CUDA and Ascend NPU paths are untouched. If
you would rather hold this until someone with XPU hardware confirms it end to end, the change is a
one-line mapping plus tests and is easy to revert.

get_default_dist_backend() returned the historical "nccl" for every device
that is not Ascend NPU, but get_accelerator_device_type() does select XPU, so an
XPU run reached init_process_group(backend="nccl") and could not start on a
build without NCCL. Resolve XPU to XCCL (legacy oneCCL name "ccl" where that is
the only probe the build exposes).

Fixes modelscope#3777.

Signed-off-by: li-lizhe <147392333@qq.com>
@li-lizhe

Copy link
Copy Markdown
Contributor Author

您好,这条 PR 已经静置 7 天没有新的进展,想确认一下是否还有需要我这边补充或调整的地方?如果有其他更合适的推进方式,也请告知。谢谢!

@LauraGPT

Copy link
Copy Markdown
Collaborator

感谢跟进。我重新审阅了当前 exact head 564839b02a212bc31f4fc90444143ced70213825 及其相对最新 main@dd00b0e42261b628466037f3a522d713399360b6 的集成候选。XPU 映射到 ("xccl", "ccl")、优先当前名称并在构建只暴露旧探针时回退到 ccl,与新增的选择器测试一致;目前没有新的静态代码修改要求。

不过,这个 PR 仍受 #3777 已明确记录的硬件验收门约束:在合并前,需要基于 exact head 的真实 Intel XPU 两进程 torchrun,记录每个 rank 的 XPU 绑定、all_reduce、DDP forward/backward、epoch 末 cache release,以及准确的 PyTorch / oneAPI / 设备版本。当前 exact-head 的 hosted CI 只有成功的 checkpoint-selection,没有执行该分布式设备测试;PR 中的 CPU 模拟测试只证明后端选择逻辑,不能替代 XPU runtime 验证,上海 L20Z CUDA 也不能替代 Intel XPU。

如果你能接触 XPU 硬件,请按 #3777 的清单补充完整回执;如果暂时没有,则无需继续改代码,可以保持 PR 打开,等待具备 XPU 环境的维护者完成验证。本次不会在缺少该证据时合并或关闭。

@li-lizhe

Copy link
Copy Markdown
Contributor Author

明白,谢谢确认。代码这边不再改动,PR 保持打开等 XPU 环境验证。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[dist] XPU distributed backend selection (XCCL / oneCCL) is unverified — needs XPU hardware

2 participants