CollectiveX: b300 deepep-v2 LL EP8 (single-node HCA pin) + H-series LL EP16 bring-up findings

[Oseltamivir]

What merges

b300 gains its first production-backend low-latency row: deepep-v2 [8].

DeepEP's legacy LL Buffer self-enables IBGDA even single-node, and the image's baked NVSHMEM_HCA_PE_MAPPING steers that init onto b300's GPU-fabric RoCE rails, where ibv_create_ah fails at every GID index (ibgda.cpp "Unable to create ah" → create DCT share err, rc255 on all ranks). The storage-IB rails (mlx5_12-15) accept AH/DCT creation, and with allow_nvlink_for_low_latency_mode all steady-state traffic stays on NVLink — the rails carry init-time control traffic only.

Mechanism: new optional registry network field single_node_rdma_devices, exported as NVSHMEM_HCA_LIST on single-node shards only (seam-tested; scale-out shards keep resolving from the ordinary selector).

Validation (run 31179442983): both precisions GREEN with healthy NVLink-class numbers — bf16 roundtrip p50 58.5µs @t=1 → 146µs @t=256, fp8 51.8µs → 118µs, correctness clean at every rung. Comparable to the h200 EP8 LL control (52µs @t=1).

What was tried and deliberately held (H-series LL EP16)

The middle commit enabled deepep-v2 ll [8,16] on h100-dgxc/h200-dgxc; the last commit reverts it after bring-up:

• h200 (run 31179010881): 4/4 GREEN, correctness clean — but EP16 periods are 20-120× off a healthy internode LL reference (dispatch p50 1.84ms @t=64, 7.7ms @t=256, linear in tokens; bare-metal b200 runs the identical cells at ~50-130µs).
• h100 (run 31179008559): EP8 controls green; both EP16 shards red — bf16-n2 ran 15 min into the 900s hang guard (same degradation, RoCE flavor), fp8-n2 additionally hit the known TCPStore rendezvous flake.
• Root cause candidate verified on-node: /dev/gdrdrv is absent on both pools (nvidia_peermem loaded) — the one delta vs b200-nscale where these cells are ±1% of hand campaigns. SRE has been asked to install gdrcopy; once it lands, re-flipping the two registry rows re-runs this bring-up.

b300 LL EP16 remains excluded: cross-node IBGDA needs AH creation on the RoCE rails — an SRE-level fabric fix (kernel GID→DMAC/L3 resolution), not a harness change. NCCL RC and ib_write_bw pass on those rails because neither creates AHs from GIDs, which is why health checks miss it.

---

Note

Medium Risk
Changes b300 benchmark networking env (NVSHMEM_HCA_LIST) and scheduling exclusions for a new production LL backend row; scope is SKU-specific but affects GPU job bring-up.

Overview
Enables b300 low-latency deepep-v2 EP8 in platform_config.json and tightens operator baselines (more exclude_nodes).

Adds an optional platform network field single_node_rdma_devices, loaded as COLLX_SINGLE_NODE_RDMA_DEVICES and, in collx_apply_network_profile, exported as NVSHMEM_HCA_LIST only when nodes <= 1 so single-node DeepEP LL init can use storage-IB HCAs instead of the image’s default GPU-fabric mapping. Scale-out runs still use the normal rdma_devices selector.

b300 pins mlx5_12–15:1 for that path. Registry typing/tests cover the new field, plus shell tests that single-node honors the override, omits it when unset, and ignores it on multi-node.

^{Reviewed by Cursor Bugbot for commit 13f85b4. Bugbot is set up for automated code reviews on this repo. Configure here.}

[claude]

LGTM. Small, well-scoped harness change confined to experimental/CollectiveX, with unit tests covering the new single-node HCA override behavior directly.

What was reviewed: the new single_node_rdma_devices -> NVSHMEM_HCA_LIST override path in collx_apply_network_profile (early-return placement, regex validation, scale-out non-interference); the b300 registry entry enabling deepep-v2 LL EP8; and the config.py/test allowlist additions for the new network field. Beyond the tests, I checked that the override applies uniformly to both LL and normal-mode single-node runs (not gated on LL) — this was raised and refuted as a real bug, since the storage-IB rails still just carry init-time control traffic in either mode. I also confirmed there's no pre-flight fabric check added for the new rails before an LL EP8 run — this was raised too, but the existing srun-based validation only runs for scale-out (nodes>1), so it's consistent with existing single-node behavior, not a regression.

Extended reasoning...

Overview

The PR adds a new optional per-SKU registry field, single_node_rdma_devices (experimental/CollectiveX/configs/platform_config.json, runtime/config.py, runtime/common.sh), used to pin an alternate HCA list for single-node DeepEP low-latency runs on b300, where the baked NVSHMEM PE mapping otherwise steers IBGDA init onto RoCE rails that reject AH/DCT creation. It also flips b300's ll_backends to include deepep-v2: [8]. The bulk of the new logic is a single conditional block in collx_apply_network_profile in common.sh, with matching allowlist additions in config.py and three new focused unit tests plus a registry-schema test update in test_runtime.py.

Security risks

None identified. This is cluster/fabric configuration for an experimental GPU benchmarking harness — no auth, crypto, or permission logic is touched. The new value is regex-validated before being exported as an env var (mirroring the existing COLLX_RDMA_DEVICES validation pattern), so it doesn't introduce a new injection surface.

Level of scrutiny

Low-to-moderate. The change is entirely within experimental/CollectiveX (a benchmarking harness, not a production serving path), is small and mechanically similar to existing patterns in the same file, and ships with direct unit tests for exactly the new branch (override present, override absent, scale-out ignores it). The PR description is unusually thorough about the on-metal root cause and validation runs backing the change.

Other factors

No CODEOWNERS entry restricts this path. No prior reviews exist on this PR. The bug-hunting pass found no issues, and the two candidate concerns raised during verification (override not gated to LL-only, no new pre-flight fabric check) were examined and are not defects — they match existing single-node/scale-out behavioral boundaries in the file.