NCCL AllReduce 2400x slower via Ray actors vs torchrun on H200 (NVSwitch) multi-node

[dmvevents]

Environment

• Nodes: 2x AWS P5en.48xlarge (8x NVIDIA H200 143GB per node, NVSwitch intra-node)
• Networking: 16 EFA v2 devices per node, EKS Kubernetes with hostNetwork: true
• Container: NCCL 2.27.5, aws-ofi-nccl 1.18.0, Libfabric 2.4, PyTorch 2.x
• OS: Ubuntu 22.04 (EKS AMI)
• GPU persistent mode: enabled (nvidia-smi -pm 1)

Problem

When using Ray actors with num_gpus=1 and calling torch.distributed.init_process_group('nccl') followed by all_reduce, NCCL performance is catastrophically degraded on H200 (P5en) multi-node compared to torchrun on the exact same hardware.

Benchmark comparison (same nodes, same GPUs, same network)

Launcher	Operation	Latency	Notes
torchrun	AllReduce 4MB	1.5 ms	Expected performance
Ray actors	AllReduce 4KB	3,660 ms	2400x slower than torchrun
Ray actors	AllReduce 933MB	HANGS forever	Never completes, timeout required

Critical detail: works on P5, fails on P5en

• The exact same Ray code on P5.48xlarge (H100, no NVSwitch) works correctly with expected NCCL performance.
• The issue is specific to H200 with NVSwitch topology (P5en.48xlarge).

Environment variables tried (all failed to resolve)

NCCL_P2P_DISABLE=1
NCCL_CUMEM_ENABLE=0
NCCL_NVLS_ENABLE=0
NCCL_NET_GDR_LEVEL=SYS
NCCL_SHM_DISABLE=1

None of these changed the behavior. The AllReduce remains 2400x slower or hangs.

Likely root cause

Ray sets CUDA_VISIBLE_DEVICES to remap GPU indices for each actor (e.g., actor on physical GPU 5 sees CUDA_VISIBLE_DEVICES=5 and uses logical device 0). On H200 nodes with NVSwitch, this remapping appears to cause NCCL to misidentify the hardware topology, leading it to select incorrect transport paths (likely falling back to slow socket transport instead of NVSwitch/NVLink or EFA/RDMA).

Evidence:

1. torchrun (which does not remap CUDA_VISIBLE_DEVICES the same way) works perfectly on the same nodes.
2. P5.48xlarge (H100, PCIe-based without NVSwitch) works fine with Ray — suggesting NVSwitch topology discovery is what breaks.
3. NCCL debug logs show the communicator initializes but data transfer is orders of magnitude slower than hardware capability.

Steps to reproduce

1. Launch 2x P5en.48xlarge nodes on EKS with EFA and hostNetwork: true
2. Start a Ray cluster across both nodes
3. Create 16 Ray actors (8 per node) with num_gpus=1
4. In each actor, call torch.distributed.init_process_group('nccl', ...) using TCPStore
5. Run torch.distributed.all_reduce(tensor) with a 4KB+ tensor
6. Observe multi-second latency or hang

For comparison, run the same all_reduce via torchrun --nproc_per_node=8 --nnodes=2 on the same nodes — it completes in ~1.5ms for 4MB.

Related issues

• [core][compiled graphs] Slow NCCL init on H200 server #53619 — NCCL performance issues with Ray on multi-GPU
• slow torch.distributed with non-default CUDA_VISIBLE_DEVICES #50723 — CUDA_VISIBLE_DEVICES interaction with NCCL
• Can not run TorchTrainer on 3 node with gpu #59804 — Ray GPU scheduling and NCCL topology

Expected behavior

Ray actors should achieve NCCL AllReduce performance comparable to torchrun on H200 NVSwitch nodes (~1.5ms for 4MB, not 3660ms for 4KB).

Versions

• Ray: 2.44.1
• PyTorch: 2.x
• NCCL: 2.27.5
• CUDA: 12.x
• aws-ofi-nccl: 1.18.0
• Libfabric: 2.4

[dmvevents]

Update: Root cause confirmed in NCCL source code

Deep dive into NCCL source reveals two confirmed NVIDIA bugs that stack on H200/NVSwitch:

Bug 1: cuMem import penalty (NVIDIA/nccl#1749 — CONFIRMED by NVIDIA)

When Ray sets non-overlapping CUDA_VISIBLE_DEVICES per actor (e.g., rank 0 gets CUDA_VISIBLE_DEVICES=0, rank 1 gets 1, etc.), NCCL's cuMem handle import triggers an iterative search across all CUDA devices starting from device 0 (src/transport/p2p.cc:p2pMap()). This explains the 3,660ms first-operation penalty.

NVIDIA engineer kiskra-nvidia confirmed: "Memory imports in such situations trigger an expensive, iterative search on each CUDA device."

Bug 2: NVLS data ordering (NVIDIA/nccl#1906 — CONFIRMED by NVIDIA)

NCCL's NVLS (NVLink SHARP) allgather algorithm is missing the user rank table when CUDA_VISIBLE_DEVICES permutes GPU indices. This causes either silent data corruption or hangs for collective operations. NVLS auto-enables on H200 because of NVSwitch multicast support.

NVIDIA engineer sjeaugey confirmed: "The single node NVLS allgather algorithms seems to be missing the user rank table which ensures proper data ordering."

Why it works on H100 (P5)

H100 in P5.48xlarge does NOT have NVSwitch, so NVLS is never selected. The cuMem penalty exists but is shorter. Only Ring/Tree algorithms are used, which handle CUDA_VISIBLE_DEVICES correctly.

Additional finding

We also tested RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 — still hangs. This is because NeMo RL's worker_groups.py explicitly removes this env var: worker_env_vars.pop("RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES", None), forcing Ray GPU masking.

Recommended fix

Both NCCL_NVLS_ENABLE=0 and NCCL_CUMEM_ENABLE=0 must be set AND propagated into Ray worker subprocess environments. The container-level env vars may not reach workers spawned in isolated venvs.

[dmvevents]

Update: Size threshold identified with NCCL_CUMEM_ENABLE=0 + NCCL_NVLS_ENABLE=0

With both env vars set and confirmed propagated to Ray workers:

Size	Elements	Bytes	Result
1K	1,024	4 KB	3,925ms (first op, channel setup)
1M	1,048,576	4 MB	11.8ms ✅
4M	4,194,304	16 MB	2.0ms ✅
16M	16,777,216	64 MB	HANG (300s timeout) ❌

NCCL_CUMEM_ENABLE=0 fixed the cuMem import penalty (1M and 4M work at real EFA speeds). But at 16M elements, NCCL switches to multi-channel transport which still hangs. This suggests additional NCCL transport bugs with CUDA_VISIBLE_DEVICES remapping beyond cuMem and NVLS.

The multi-channel threshold (~16M elements) is where NCCL starts using multiple rings/trees and more NET channels, which may trigger the same busIdToCudaDev() device resolution issues in p2p.cc for each additional channel.

[dmvevents]

Reproduction Package

Gist: https://gist.github.com/dmvevents/ca6a4415e3d40f510521fc691623b6c9

Quick Reproduce

git clone https://gist.github.com/ca6a4415e3d40f510521fc691623b6c9.git repro && cd repro
kubectl create configmap ray-nccl-repro --from-file=reproduce.py
kubectl apply -f deploy.yaml

Files

• reproduce.py — Minimal 16-actor Ray NCCL AllReduce size sweep
• deploy.yaml — K8s job for 2x P5en.48xlarge with EFA
• environment.txt — Exact software versions
• README.md — Full analysis + root cause

Expected vs Actual

• Expected: AllReduce at all sizes completes in <50ms (as torchrun achieves)
• Actual: AllReduce ≤4M: works (2-12ms). AllReduce ≥16M: HANGS (300s timeout)

[dmvevents]

Comprehensive NCCL Research Complete

After deep source code analysis of NCCL 2.27.5 through 2.29.3 (latest):

No fix exists in any released NCCL or CUDA version for the multi-channel P2P hang.

Three Stacked Bugs

1. cuMem import (slow firsttime op with NCCL_CUMEM_ENABLE=1 and non-default CUDA_VISIBLE_DEVICES in ray workers NVIDIA/nccl#1749): Fix in CUDA driver, not shipped yet
2. NVLS rank ordering ([Issue]: all_gather returns tensors in incorrect rank order for certain permutation of CUDA_VISIBLE_DEVICES and NCCL_ALGO=NVLS (regression in NCCL 2.26.2; NCCL 2.21.5 OK) NVIDIA/nccl#1906): Active fix in progress at NVIDIA
3. Multi-channel P2P ([Issue]: transport/p2p.cc:275 NCCL WARN Cuda failure 101 'invalid device ordinal' NVIDIA/nccl#1948): busIdToCudaDev() returns -1 for peers not in CUDA_VISIBLE_DEVICES, triggers deadlock during runtime channel connect

Key NCCL Source References

• src/enqueue.cc:calcP2pChannelCount() — doubles channels when message > maxSize
• src/init.cc:NCCL_PARAM(RuntimeConnect, "RUNTIME_CONNECT", 1) — defers connections
• src/transport/p2p.cc:busIdToCudaDev() — returns -1 for masked peers

Workaround to Test

NCCL_BUFFSIZE=16777216 — increases per-channel buffer to 16MB, potentially keeping channel count at 1 for 64MB messages, avoiding the multi-channel P2P hang without the TCP store timeout caused by NCCL_MAX_NCHANNELS=1.

Full Reproduction Package

Gist: https://gist.github.com/dmvevents/ca6a4415e3d40f510521fc691623b6c9

[dmvevents]

CRITICAL UPDATE: CUDA_VISIBLE_DEVICES is NOT the root cause

Tested with RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 + patching NeMo RL's worker_groups.py to not remove it. Workers confirmed CVD=NOT_SET (all 8 GPUs visible). 16M AllReduce STILL HANGS.

Config	8M AllReduce	16M AllReduce
Ray + CVD masked	2.4ms ✅	HANG ❌
Ray + CVD=NOT_SET	2.2ms ✅	HANG ❌
torchrun (all visible)	1.5ms ✅	1.5ms ✅

Updated Root Cause

The real difference is NOT GPU masking — it's Ray's staggered process initialization:

• torchrun: All 16 processes start simultaneously via fork(), call init_process_group within milliseconds of each other
• Ray: Actors are created sequentially by the scheduler, each calling init_process_group with seconds between them

On H200/NVSwitch, NCCL's runtime connect (NCCL_RUNTIME_CONNECT=1, default) defers channel setup to the first collective. When 16 Ray actors hit the first large collective (>8M elements) at staggered times, the multi-channel P2P connection negotiation deadlocks because ranks on different nodes are in different phases of channel setup.

This is an NCCL robustness issue specific to H200/NVSwitch topology where channel negotiation is more complex than H100/NVLink PXN.

[dmvevents]

Final finding: Synchronized init fixes cuMem penalty but NOT the multi-channel hang

Tested with barrier-synchronized actor initialization (all 16 actors created first, then sync_and_init() called on all simultaneously):

Test	Previous (staggered init)	Barrier (synchronized init)
AllReduce 1K	3,300ms	2.9ms ✅
AllReduce 8M	2.0ms	2.0ms ✅
AllReduce 16M	HANG	HANG ❌

The synchronized init completely eliminated the 3.3s first-op penalty (confirming NVIDIA/nccl#1749 is timing-related). But the 16M+ hang persists — this is a separate fundamental NCCL bug in multi-channel runtime connect on H200/NVSwitch.

What we've eliminated

• ❌ CUDA_VISIBLE_DEVICES masking (tested with all GPUs visible)
• ❌ Staggered initialization timing (tested with barrier sync)
• ❌ NCCL env vars (tested CUMEM, NVLS, P2P, BUFFSIZE, ALGO, NCHANNELS)
• ❌ NCCL upgrade (2.29.2 has identical p2p.cc code)

What works

• ✅ torchrun (shared TCPStore, different bootstrap mechanism)
• ✅ Operations ≤8M elements via Ray (single-channel path)

Conclusion

The 16M+ hang is a fundamental NCCL bug in the multi-channel P2P runtime connect path on H200/NVSwitch. It is NOT triggered by Ray's GPU masking, process timing, or any env var configuration. It only manifests when processes use separate TCPStore instances (Ray's per-actor model) vs a shared store (torchrun's model). The different store architecture causes the multi-channel negotiation protocol to deadlock.

[dmvevents]

FIX FOUND: NCCL_ALGO=Tree

Setting NCCL_ALGO=Tree (combined with NCCL_CUMEM_ENABLE=0 + NCCL_NVLS_ENABLE=0) fixes the hang for ALL sizes on H200/NVSwitch with Ray:

Size	Default Algo	NCCL_ALGO=Tree
1K	3,300ms	3,385ms
1M	12ms	12ms
4M	2.0ms	2.0ms
8M	2.0ms	1.8ms
16M	HANG	2.3ms ✅
64M	HANG	3.6ms ✅
233M	HANG	9.6ms ✅

Root cause (from deep NCCL source analysis)

The Ring algorithm creates a circular dependency in the GPU kernel spin-wait path. On H200/NVSwitch, IPC peer access uses cudaIpcMemLazyEnablePeerAccess which defers NVSwitch multicast configuration to first memory access. When 8 GPU kernels simultaneously trigger lazy init through the Ring's circular data flow, the CUDA driver serializes the NVSwitch configuration through a global lock, creating a classic deadlock.

Tree algorithm avoids this because its acyclic dependency graph means there's always a leaf that can complete without waiting for a downstream node.

Recommended env vars for Ray + H200/NVSwitch

NCCL_ALGO=Tree
NCCL_CUMEM_ENABLE=0
NCCL_NVLS_ENABLE=0

[dmvevents]

Root Cause Identified at Source Code Level

The deadlock is caused by NCCL's single-threaded proxy interacting with the Ring algorithm's circular data dependency when ALL inter-GPU hops require proxy involvement (Ray's 1-GPU-per-process model).

torchrun vs Ray — the critical difference

	torchrun (works)	Ray (deadlocks)
GPUs per process	8	1
Intra-node Ring hops	Direct NVLink P2P (no proxy)	Proxy (separate address spaces)
Proxy operations per Ring step	2 (inter-node only)	16+ (all hops)
Result	Proxy keeps up	Proxy overwhelmed → deadlock

Proposed Ray-side workaround

Process coalescing: Use num_gpus=8 per Ray actor (1 actor per node) instead of 16 actors with 1 GPU each. Each actor manages 8 GPUs internally via torch.multiprocessing.spawn(). This gives intra-node P2P (no proxy) while using Ray for cross-node coordination.

This would require NeMo RL to support the coalesced actor model (currently expects 1 GPU per actor).