Skip to content

fix: get_nvlink_gbs may return 0 as a divisor #740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cxxlink wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: get_nvlink_gbs may return 0 as a divisor #740

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion deep_ep/buffers/elastic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -806,7 +806,15 @@ def get_expected_topk(num_groups: int) -> float:
rdma_traffic += (self.num_ranks - self.num_nvlink_ranks) / self.num_ranks

# Found the bounded one
if self.num_scaleout_ranks > 1 and (rdma_traffic / rdma_gbs) > (nvlink_traffic / nvlink_gbs):
if nvlink_traffic > 0 and nvlink_gbs == 0:
raise ValueError("NVLink connection speed is necessary")
nvlink_time = 0.0 if nvlink_traffic == 0 else nvlink_traffic / nvlink_gbs

if self.num_scaleout_ranks > 1 and rdma_traffic > 0 and rdma_gbs == 0:
raise ValueError("RDMA connection speed is necessary")
rdma_time = 0.0 if rdma_traffic == 0 else rdma_traffic / rdma_gbs
Comment on lines +809 to +815

@ds-review-bot ds-review-bot Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 warning: 带宽校验和除法被提到了 if self.num_scaleout_ranks > 1 分支之外，导致不参与瓶颈比较的路径也会强制要求带宽。例如 allow_hybrid_mode=False 时 num_scaleout_ranks 恒为 1，rdma_time 根本不会进入比较，最终只以 NVLink 为瓶颈；但多机环境下 rdma_traffic > 0 且 get_rdma_gbs() 探测失败返回 0（如容器内缺少 ibstat）时，这里会抛出 ValueError，而旧实现不会执行除法、可正常回退到 NVLink 瓶颈继续计算。nvlink 侧同理也会影响单节点多卡且 nvidia-smi nvlink -s 探测失败的场景。建议只在 self.num_scaleout_ranks > 1 且对应 traffic 真正参与比较时做零带宽校验/除法，或在该分支外保留旧有的回退行为。

🤖 v4p


if self.num_scaleout_ranks > 1 and rdma_time > nvlink_time:
bounded_traffic, bounded_gbs = rdma_traffic, rdma_gbs
else:
bounded_traffic, bounded_gbs = nvlink_traffic, nvlink_gbs
Expand Down