Conversation
…#726) get_nccl_comm_handle() cached whatever PyTorch _comm_ptr() returned, including the null pointer it yields before the group NCCL communicator is lazily materialized. A cached null handle is later dereferenced in C++ (ncclTeamWorld), crashing the rank. Decide reuse with a group-wide all_gather_object so every rank takes the same branch, and fall back to a DeepEP-managed comm when any rank has not materialized its communicator yet. Adds a GPU-free regression test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #726.
Problem
get_nccl_comm_handle()cached whatever PyTorch's_comm_ptr()returned, including thenull pointer it yields before the group's NCCL communicator is lazily materialized
(eager creation only happens when
init_process_groupreceiveddevice_id=...). Thecached null handle is later dereferenced in C++ — e.g.
ncclTeamWorldreadscomm->nRanks— crashing the rank during buffer construction. This is thevllm serve16/16-rank failure reported in the issue.Root cause
deep_ep/comm/handle.py:The pointer is used without checking that it is non-null. A per-rank
!= 0check wouldnot be enough either: ranks can disagree on
_comm_ptr()(a peer may have materializedits communicator via an earlier point-to-point op, or be on a different current device),
and then some ranks would return early while others enter the group-wide unique-id
exchange below — a hang rather than an error.
Fix
Decide reuse with a group-wide
all_gather_objectso every rank takes the same branch,and fall back to a DeepEP-managed comm when any rank has not materialized its
communicator yet:
The collective is reached unconditionally on the reuse path, so ranks cannot split
between "return PyTorch's comm" and "run the unique-id exchange".
Relationship to #727
#727 addresses the same bug but targets
deep_ep/utils/comm.py, which has since movedto
deep_ep/comm/handle.pyonmain, so it is stale/conflicting. This ports the fix tothe current path.
Testing
tests/comm/test_comm_reuse.pyadds 5 mocked cases:_comm_ptr()is null → fall through to a managed commEP_REUSE_NCCL_COMM=0→ forces a managed comm regardless of pointer stateThe collectives are monkeypatched, so the test needs no GPU. Importing
deep_ep.comm.handlestill requires the host extension to be built, like the rest of thesuite.
python tests/comm/test_comm_reuse.py # All comm-reuse tests passed