Skip to content

fix: reuse PyTorch NCCL comm only when all ranks own one (#726) - #767

Open
Dashener2 wants to merge 1 commit into
deepseek-ai:mainfrom
Dashener2:fix/comm-handle-null-comm-reuse
Open

Dashener2 wants to merge 1 commit into
deepseek-ai:mainfrom
Dashener2:fix/comm-handle-null-comm-reuse

Conversation

@Dashener2

Copy link
Copy Markdown

Fixes #726.

Problem

get_nccl_comm_handle() cached whatever PyTorch's _comm_ptr() returned, including the
null pointer it yields before the group's NCCL communicator is lazily materialized
(eager creation only happens when init_process_group received device_id=...). The
cached null handle is later dereferenced in C++ — e.g. ncclTeamWorld reads
comm->nRanks — crashing the rank during buffer construction. This is the
vllm serve 16/16-rank failure reported in the issue.

Root cause

deep_ep/comm/handle.py:

backend = group._get_backend(torch.device('cuda'))
if not force_new_comm and hasattr(backend, '_comm_ptr') and int(os.getenv('EP_REUSE_NCCL_COMM', '1')):
    _storage[group] = NCCLCommHandle(backend._comm_ptr(), False)
    return _storage[group]

The pointer is used without checking that it is non-null. A per-rank != 0 check would
not be enough either: ranks can disagree on _comm_ptr() (a peer may have materialized
its communicator via an earlier point-to-point op, or be on a different current device),
and then some ranks would return early while others enter the group-wide unique-id
exchange below — a hang rather than an error.

Fix

Decide reuse with a group-wide all_gather_object so every rank takes the same branch,
and fall back to a DeepEP-managed comm when any rank has not materialized its
communicator yet:

comm_ptr = backend._comm_ptr()
have_comm = [None] * group.size()
dist.all_gather_object(have_comm, comm_ptr != 0, group)
if all(have_comm):
    _storage[group] = NCCLCommHandle(comm_ptr, False)
    return _storage[group]

The collective is reached unconditionally on the reuse path, so ranks cannot split
between "return PyTorch's comm" and "run the unique-id exchange".

Relationship to #727

#727 addresses the same bug but targets deep_ep/utils/comm.py, which has since moved
to deep_ep/comm/handle.py on main, so it is stale/conflicting. This ports the fix to
the current path.

Testing

tests/comm/test_comm_reuse.py adds 5 mocked cases:

  • all ranks hold a live comm → PyTorch's comm is reused, exactly one collective
  • local _comm_ptr() is null → fall through to a managed comm
  • rank skew (this rank live, a peer null) → every rank falls through
  • EP_REUSE_NCCL_COMM=0 → forces a managed comm regardless of pointer state
  • cache hit on the same group → no collective at all

The collectives are monkeypatched, so the test needs no GPU. Importing
deep_ep.comm.handle still requires the host extension to be built, like the rest of the
suite.

python tests/comm/test_comm_reuse.py
# All comm-reuse tests passed

…#726)

get_nccl_comm_handle() cached whatever PyTorch _comm_ptr() returned,
including the null pointer it yields before the group NCCL communicator
is lazily materialized. A cached null handle is later dereferenced in
C++ (ncclTeamWorld), crashing the rank.

Decide reuse with a group-wide all_gather_object so every rank takes the
same branch, and fall back to a DeepEP-managed comm when any rank has not
materialized its communicator yet. Adds a GPU-free regression test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant