Network gIB Issues

[samueljmello]

I'm trying to run any of these tests and can't. As far as I can tell, I have everything configured according to the prerequisites and documentation. Everything seems to work until the containers spin up and run. The best I can tell, the actual error is this:

torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/NCCLUtils.hpp:328, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.25.1
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
Last error:
Error: network gIB not found.

Does anyone have any advice? I'm trying to do this with FlexStart node pools and I'm not sure if there is a network configuration difference between DENSE provisioned nodes (mentioned in the readme) and FlexStart. Any help is appreciated.

[junjieqian]

Hi @samueljmello , which recipe are you trying to run? This error seems indicating incorrect NCCL setup.

[capri-xiyue]

I encountered the same issue here, I used https://github.com/AI-Hypercomputer/gpu-recipes/tree/main/RL/a4/recipes/qwen2.5-1.5b/nemoRL. I'm not sure whether it's because I didn't set up the GKE cluster using https://github.com/AI-Hypercomputer/gpu-recipes/blob/main/docs/configuring-environment-gke-a4.md#create-a-gke-cluster-environment-with-a4-node-pools as I can't make a reservation for such gpu resources.

However, the GKE cluster I created has satisfied such requirements

A [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview) cluster configured as follows:
[A GKE regional standard cluster](https://cloud.google.com/kubernetes-engine/docs/concepts/configuration-overview) version: v1.32.4-gke.1236000 or later.
A GPU node pool with the user specified number of [a4-highgpu-8g](https://cloud.google.com/compute/docs/gpus#h200-gpus) provisioned using the DENSE deployment type.
[Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) enabled.
[Cloud Storage FUSE CSI driver for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/cloud-storage-fuse-csi-driver) enabled.
[DCGM metrics](https://cloud.google.com/kubernetes-engine/docs/how-to/dcgm-metrics) enabled.
[Kueue](https://kueue.sigs.k8s.io/docs/reference/kueue.v1beta1/) and [JobSet](https://jobset.sigs.k8s.io/docs/overview/) APIs installed.
Kueue configured to support [Topology Aware Scheduling](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/).