Repository navigation
Running the GPU Operator on a kind cluster #662
Description
Activity
Of note, I have also tried without KinD and instead using k0s with the exact same result.
Could you confirm that you're able to run
nvidia-smiin the Kind worker node?I can confirm that it does not run inside kind:
on the bare metal:
nvidia-smi Tue Jan 23 17:10:33 2024 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... On | 00000000:01:00.0 On | N/A | | 0% 50C P8 12W / 220W | 260MiB / 8192MiB | 5% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 3883 G /usr/lib/xorg/Xorg 146MiB | | 0 N/A N/A 4041 G /usr/bin/gnome-shell 67MiB | | 0 N/A N/A 6091 G /usr/bin/nautilus 16MiB | | 0 N/A N/A 78264 G ...b/firefox-esr/firefox-esr 10MiB | | 0 N/A N/A 702357 G vlc 6MiB | +-----------------------------------------------------------------------------+from a container inside of k0s:
k logs nv-5dc699dbc6-xwhwt ========== == CUDA == ========== CUDA Version 12.3.1 Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. This container image and its contents are governed by the NVIDIA Deep Learning Container License. By pulling and using the container, you accept the terms and conditions of this license: https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience. WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available. Use the NVIDIA Container Toolkit to start this container with GPU support; see https://docs.nvidia.com/datacenter/cloud-native/ . /opt/nvidia/nvidia_entrypoint.sh: line 67: exec: nvidia-smi: not foundand from inside kind:
k logs nv-5df8456f86-9gkwf ========== == CUDA == ========== CUDA Version 12.3.1 Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. This container image and its contents are governed by the NVIDIA Deep Learning Container License. By pulling and using the container, you accept the terms and conditions of this license: https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience. WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available. Use the NVIDIA Container Toolkit to start this container with GPU support; see https://docs.nvidia.com/datacenter/cloud-native/ . /opt/nvidia/nvidia_entrypoint.sh: line 67: exec: nvidia-smi: not foundwith this as my deployment:
cat nv-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: kompose.cmd: ./kompose convert -f docker-compose.yml kompose.version: 1.22.0 (955b78124) labels: io.kompose.service: nv name: nv spec: replicas: 1 selector: matchLabels: io.kompose.service: nv template: metadata: annotations: kompose.cmd: ./kompose convert -f docker-compose.yml kompose.version: 1.22.0 (955b78124) labels: io.kompose.network/noworky-default: "true" io.kompose.service: nv spec: containers: - args: - nvidia-smi image: nvidia/cuda:12.3.1-devel-centos7 name: nv restartPolicy: AlwaysWhat are you doing to inject GPU support into the docker container that kind starts to represent the k8s node?
Something like this is necessary:
kubernetes-sigs/kind#3257 (comment)Using the example config you supplied I get the same results:
========== == CUDA == ========== CUDA Version 12.3.1 Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. This container image and its contents are governed by the NVIDIA Deep Learning Container License. By pulling and using the container, you accept the terms and conditions of this license: https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience. WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available. Use the NVIDIA Container Toolkit to start this container with GPU support; see https://docs.nvidia.com/datacenter/cloud-native/ . /opt/nvidia/nvidia_entrypoint.sh: line 67: exec: nvidia-smi: not foundI forgot to include that config file:
/etc/nvidia-container-runtime/config.toml
accept-nvidia-visible-devices-as-volume-mounts = true #accept-nvidia-visible-devices-envvar-when-unprivileged = true disable-require = false supported-driver-capabilities = "compat32,compute,display,graphics,ngx,utility,video" #swarm-resource = "DOCKER_RESOURCE_GPU" [nvidia-container-cli] #debug = "/var/log/nvidia-container-toolkit.log" environment = [] #ldcache = "/etc/ld.so.cache" ldconfig = "@/sbin/ldconfig" load-kmods = true #no-cgroups = false #path = "/usr/bin/nvidia-container-cli" #root = "/run/nvidia/driver" #user = "root:video" [nvidia-container-runtime] #debug = "/var/log/nvidia-container-runtime.log" log-level = "info" mode = "auto" runtimes = ["docker-runc", "runc"] [nvidia-container-runtime.modes] [nvidia-container-runtime.modes.cdi] annotation-prefixes = ["cdi.k8s.io/"] default-kind = "nvidia.com/gpu" spec-dirs = ["/etc/cdi", "/var/run/cdi"] [nvidia-container-runtime.modes.csv] mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d" [nvidia-container-runtime-hook] path = "nvidia-container-runtime-hook" skip-mode-detection = false [nvidia-ctk] path = "nvidia-ctk"I even gave that create-cluster.sh script a try:
+++ local 'value=VERSION ?= v0.1.0' +++ echo v0.1.0 ++ DRIVER_IMAGE_VERSION=v0.1.0 ++ : k8s-dra-driver ++ : ubuntu20.04 ++ : v0.1.0 ++ : nvcr.io/nvidia/cloud-native/k8s-dra-driver:v0.1.0 ++ : v1.27.1 ++ : k8s-dra-driver-cluster ++ : /home/thoth/k8s-dra-driver/demo/clusters/kind/scripts/kind-cluster-config.yaml ++ : v20230515-01914134-containerd_v1.7.1 ++ : gcr.io/k8s-staging-kind/base:v20230515-01914134-containerd_v1.7.1 ++ : kindest/node:v1.27.1-v20230515-01914134-containerd_v1.7.1 + kind create cluster --retain --name k8s-dra-driver-cluster --image kindest/node:v1.27.1-v20230515-01914134-containerd_v1.7.1 --config /home/thoth/k8s-dra-driver/demo/clusters/kind/scripts/kind-cluster-config.yaml Creating cluster "k8s-dra-driver-cluster" ... ✓ Ensuring node image (kindest/node:v1.27.1-v20230515-01914134-containerd_v1.7.1) 🖼 ✓ Preparing nodes 📦 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 ✓ Joining worker nodes 🚜 Set kubectl context to "kind-k8s-dra-driver-cluster" You can now use your cluster with: kubectl cluster-info --context kind-k8s-dra-driver-cluster Thanks for using kind! 😊 + docker exec -it k8s-dra-driver-cluster-worker umount -R /proc/driver/nvidia ++ docker images --filter reference=nvcr.io/nvidia/cloud-native/k8s-dra-driver:v0.1.0 -q + EXISTING_IMAGE_ID= + '[' '' '!=' '' ']' + set +x Cluster creation complete: k8s-dra-driver-clusterSame results though.
appears to be the same issue here NVIDIA/k8s-device-plugin#478
Backing up … what about running with GPUs under docker in general (I.e. without kind).
docker run -e NVIDIA_VISIBLE_DEVICES=all ubuntu:22.04 nvidia-smi
If things are not configured properly to have that work, then kind will not work either.
To be clear, that will work so long as accept-nvidia-visible-devices-as-volume-mounts = false
Once that is configured to true you would need to run:
docker run -v /dev/null:/var/run/nvidia-container-devices/all ubuntu:22.04 nvidia-smi
Both seem to work:
docker run -e NVIDIA_VISIBLE_DEVICES=all ubuntu:22.04 nvidia-smi 24-01-23 - 22:08:54 Wed Jan 24 04:09:07 2024 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... On | 00000000:01:00.0 On | N/A | | 0% 49C P8 12W / 220W | 156MiB / 8192MiB | 7% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| +-----------------------------------------------------------------------------+ (base)docker run -v /dev/null:/var/run/nvidia-container-devices/all ubuntu:22.04 nvidia-smi 24-01-23 - 22:09:08 Wed Jan 24 04:09:15 2024 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... On | 00000000:01:00.0 On | N/A | | 0% 49C P8 12W / 220W | 156MiB / 8192MiB | 5% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| +-----------------------------------------------------------------------------+ (base)grep accept-nvidia-visible-devices-as /etc/nvidia-container-runtime/config.toml accept-nvidia-visible-devices-as-volume-mounts = trueOK. That’s encouraging.
So you’re saying that even with that configured properly if you run the cluster-create.sh script from the k8s-dra-driver repo, docker exec into the worker node created by kind, and run nvidia-smi, it doesn’t work?
well at the moment ./create-cluster.sh ends with this error:
+ kind load docker-image --name k8s-dra-driver-cluster nvcr.io/nvidia/cloud-native/k8s-dra-driver:v0.1.0 Image: "nvcr.io/nvidia/cloud-native/k8s-dra-driver:v0.1.0" with ID "sha256:9c74ea73db6f97a5e7287e11888757504b1e5ecfde4d2e5aa8396a25749ae046" not yet present on node "k8s-dra-driver-cluster-control-plane", loading... Image: "nvcr.io/nvidia/cloud-native/k8s-dra-driver:v0.1.0" with ID "sha256:9c74ea73db6f97a5e7287e11888757504b1e5ecfde4d2e5aa8396a25749ae046" not yet present on node "k8s-dra-driver-cluster-worker", loading... ERROR: failed to load image: command "docker exec --privileged -i k8s-dra-driver-cluster-control-plane ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1 Command Output: unpacking nvcr.io/nvidia/cloud-native/k8s-dra-driver:v0.1.0 (sha256:e9df1b5622ca4f042dcff02f580a0a18ecad4b740fe639df2349c55067ef35b7)...time="2024-01-24T04:21:59Z" level=info msg="apply failure, attempting cleanup" error="wrong diff id calculated on extraction \"sha256:f344b08ff6c5121d786112e0f588c627da349e4289e409d1fde1b3ad8845fa66\"" key="extract-191866144-_8aF sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9" ctr: wrong diff id calculated on extraction "sha256:f344b08ff6c5121d786112e0f588c627da349e4289e409d1fde1b3ad8845fa66"and
./install-dra-driver.sh
now fails with:+ kubectl label node k8s-dra-driver-cluster-control-plane --overwrite nvidia.com/dra.controller=true node/k8s-dra-driver-cluster-control-plane labeled + helm upgrade -i --create-namespace --namespace nvidia-dra-driver nvidia /home/thoth/k8s-dra-driver/deployments/helm/k8s-dra-driver --wait Release "nvidia" does not exist. Installing it now. Error: client rate limiter Wait returned an error: context deadline exceededthe build is successful from:
./build-dra-driver.shso I'm kind of confused at what is wrong.
I tried doing an equivalent ctr run with:
sudo ctr run --env NVIDIA_VISIBLE_DEVICES=all docker.io/library/ubuntu:22.04 nvidia-smibut it is just hanging here with no output.
I figured out the equivalent ctr command ( I had nvidiacontainer missing above):
sudo ctr run --env NVIDIA_VISIBLE_DEVICES=all docker.io/library/ubuntu:22.04 nvidiacontainer nvidia-smi ctr: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/usr/bin/nvidia-smi": stat /usr/bin/nvidia-smi: no such file or directory: unknownin comparison to the docker:
docker run -v /dev/null:/var/run/nvidia-container-devices/all ubuntu:22.04 nvidia-smi Wed Jan 24 05:23:32 2024 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... On | 00000000:01:00.0 On | N/A | | 0% 50C P8 12W / 220W | 117MiB / 8192MiB | 6% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| +-----------------------------------------------------------------------------+which I'm kind of uncertain why that file exists here, but not in the ctr form?
docker run -v /dev/null:/var/run/nvidia-container-devices/all ubuntu:22.04 which nvidia-smi /usr/bin/nvidia-smisudo ctr run --env NVIDIA_VISIBLE_DEVICES=all docker.io/library/ubuntu:22.04 nvidia-smi500 which nvidia-smi sudo ctr run --env NVIDIA_VISIBLE_DEVICES=all docker.io/library/ubuntu:22.04 nvidia-smi501 ls /usr/bin/nvidia-smi ls: cannot access '/usr/bin/nvidia-smi': No such file or directoryprobably some magic I'm unaware of.
ctr does not use the nvidia-container-runtime even if you have configured the CRI plugin in the containerd config to use it. The ctr command does not use CRI so it would need to be configured elsewhere to use the nvidia runtime (but that wouldn’t help with your current problem anyway of trying to get k8s to work — which does communicate with containerd over CRI).
Since I don't have k0s experience, let's start out assuming that your goal is to install the GPU Operator in a Kind cluster with GPU support. This involves two stages:
- Starting a kind cluster with GPUs and the driver injected
- Installing the GPU Operator in this cluster.
I've tried to provide more details for each of the stages below. In order to get to the bottom of this issue we would need to identify which of these is not working as expected. Once we've run through the steps for kind it may be possible to map the steps to something like k0s.
Note that as prerequisites:
- the CUDA driver needs to be installed on the host. Since you're able to run
nvidia-smithere, that seems to already be the case. - The NVIDIA Container Toolkit needs to be installed on the host. The latest release (v1.14.4) is recommended.
Starting a kind cluster with GPUs and drivers injected.
This needs to be set up as described in kubernetes-sigs/kind#3257 (comment)
This means that we need to do the following:
- The
nvidiaruntime is configured as the default runtime in the docker daemon config. (Note that the Daemon needs to be restarted to apply this config).sudo nvidia-ctk runtime configure --runtime=docker --set-as-default sudo systemctl restart docker - The NVIDIA Container Runtime Hook is configured to allow device selection by mounts. (
accept-nvidia-visible-devices-as-volume-mounts = trueis set in/etc/nvidia-container-runtime/config.toml) - The applicable volume mount (
/dev/null:/var/run/nvidia-container-devices/all) is added to the Kind config for the nodes that require GPU access.
In order to verify that the nodes have the GPU devices and Driver installed correctly one can exec into the Kind worker node and run
nvidia-smi:docker exec -ti <node-cluster> nvidia-smi -LThis should give the same output as on the host. I noted in your example that you are starting a single node Kind cluster. This should not affect the behaviour, but is a difference between our cluster definitions and the ones that you use.
Installing the GPU Operator on the Kind cluster
At this point, the Kind cluster represents a k8s cluster with ony the GPU Driver installed. Even though the NVIDIA Container Toolkit is installed on the host, it has not been injected into the nodes.
This means that we should do one of the following:
- Explicitly install the NVIDIA Container Toolkit on each of the nodes
- Ensure that
--set toolkit.enabled=true(the default) is specified when installing the GPU Operator. (Note that your description mentions that--set toolkit.enabled=falsewas specified).
For the Kind demo included in this repo, we don't use the GPU operator and as such we install the container toolkit when creating the cluster: https://github.com/NVIDIA/k8s-device-plugin/blob/2bef25804caf5924f35a164158f097f954fe4c74/demo/clusters/kind/scripts/create-kind-cluster.sh#L38-L47
Note that since the Kind nodes themselves are effectively Debian nodes and are not officially supported. Most of this might be due to driver cotainer limitations and may not be applicable in this case, since we are dealing with a preinstalled driver.
34 remaining items
- changed the title
[-]no runtime for "nvidia" is configured[/-][+]Running the GPU Operator on a kind cluster[/+]on Jan 15, 2026 - linked a pull request that will close this issueexamples: add gpu-operator kind cluster walkthrough #71
on Jun 30, 2026
1. Issue or feature description
When following the quickstart I end up with this error in
k describe po -n gpu-operator gpu-feature-discovery-6tk4hWarning FailedCreatePodSandBox 0s (x5 over 49s) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox runtime: no runtime for "nvidia" is configured
2. Steps to reproduce the issue
3. Information to attach (optional if deemed irrelevant)
with my kind-config.yaml
Common error checking:
nvidia-smi -aon your hostand
docker run --rm nvidia/cuda:12.3.1-devel-centos7 nvidia-smi/etc/docker/daemon.json)and /etc/docker/daemon.json
and /etc/containerd/config.toml
sudo journalctl -r -u kubelet)Additional information that might help better understand your environment and reproduce the bug:
docker versionand the helm below fails as well:
uname -auname -a
Linux saruman 6.1.0-17-amd64 NVIDIA/k8s-device-plugin#1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux
dmesgnone that I see?
dpkg -l '*nvidia*'orrpm -qa '*nvidia*'nvidia-container-cli -Vnvidia-container-cli -V
cli-version: 1.14.3
lib-version: 1.14.3
build date: 2023-10-19T11:32+00:00
build revision: 1eb5a30a6ad0415550a9df632ac8832bf7e2bbba
build compiler: x86_64-linux-gnu-gcc-7 7.5.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
the above page no longer exists.
sudo journalctl -u nvidia-container-toolkit
-- No entries --