The amd-gpu-operator's discuss from yevgeny-shnaidman

KMM Issue to create amd-gpu driver image

Hello,

First of all thank you very much for this operator.

I'm trying to use this solution on my OKD cluster in version 4.14 to install an AMD RADEON RX 7900XT card as GPU to test AI models.

Unfortunately I have a problem creating the image on the internal registry. I have this problem on kmm operator pod :

E0507 13:44:31.120907 1 controller.go:324] kmm "msg"="Reconciler error" "error"="failed to handle build for kernel version 6.5.5-200.fc38.x86_64: could not synchronize the build: could not make Build template: could not get DTK image for kernel 6.5.5-200.fc38.x86_64: could not find kernel 6.5.5-200.fc38.x86_64 in kernel --> OS mapping" "Module"={"name":"dc-harbor","namespace":"openshift-amd-gpu"} "controller"="BuildSignReconciler" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "name"="dc-harbor" "namespace"="openshift-amd-gpu" "reconcileID"="ae042701-0f38-4dfe-802b-56795b3f5691"

If I remove DeviceConfig to add the DTK image value of my cluster version on the Module object (I remove the owner reference), and if I change DTK_AUTO in the configuration map to DTK, I can start the build .

Even with this option, I have to add a new variable named "KVERS" and replace it with the kernel version of the DTK image. (And apply this in the config map to replace all KERNEL_VERSION variables with KVERS). (6.5.5-200.fc38.x86_64 to 5.14.0-284.36.1.el9_2.x86_64)

You will find my custom files attached.
cm_and_modude.txt

If all those change are mades, I can build the image and use it by recreating the DeviceConfig file but It will not work as intended because of those logs :

I0507 13:56:47.805280       1 main.go:100] kmm-worker "msg"="Starting worker" "git commit"="e1625b2" "version"="2.0.1"
I0507 13:56:47.805292       1 main.go:101] kmm-worker "msg"="Reading pull secrets" "base dir"="/var/run/kmm/pull-secrets"
I0507 13:56:47.805362       1 pullsecrets.go:39] kmm-worker "msg"="Reading file" "path"="/var/run/kmm/pull-secrets/amd-gpu-operator-kmm-module-loader-dockercfg-wjwn5/..2024_05_07_13_31_23.2974230544/.dockercfg" "type"="kubernetes.io/dockercfg"
I0507 13:56:47.805388       1 pullsecrets.go:35] kmm-worker "msg"="Unhandled file name; ignoring" "path"="/var/run/kmm/pull-secrets/amd-gpu-operator-kmm-module-loader-dockercfg-wjwn5/..data"
I0507 13:56:47.805395       1 pullsecrets.go:39] kmm-worker "msg"="Reading file" "path"="/var/run/kmm/pull-secrets/amd-gpu-operator-kmm-module-loader-dockercfg-wjwn5/.dockercfg" "type"="kubernetes.io/dockercfg"
I0507 13:56:47.807301       1 mirror.go:43] kmm-worker "msg"="No configuration found for registry" "image name"="<myregistry>/public/amddriver:v1.0"
I0507 13:56:47.807314       1 worker.go:190] kmm-worker "msg"="Pulling image" "image name"="<myregistry>/public/amddriver:v1.0"
time="2024-05-07T13:57:12Z" level=warning msg="ignored xattrs in archive: underlying filesystem doesn't support them" errors="[operation not permitted]"
time="2024-05-07T13:57:12Z" level=warning msg="ignored xattrs in archive: underlying filesystem doesn't support them" errors="[operation not permitted]"
I0507 13:57:16.304671       1 worker.go:198] kmm-worker "msg"="Image pulled successfully" "dir"="/var/run/kmm/images/<myregistry>/public/amddriver:v1.0/fs" "image name"="<myregistry>/public/amddriver:v1.0"
I0507 13:57:16.304743       1 worker.go:49] kmm-worker "msg"="Unloading in-tree module" "name"="amdgpu"
I0507 13:57:16.304831       1 modprobe.go:33] kmm-worker "msg"="Running modprobe" "command"="/usr/sbin/modprobe -rv amdgpu"
I0507 13:57:16.306316       1 worker.go:59] kmm-worker "msg"="preparing firmware for loading" "host mount directory"="/var/lib/firmware" "image directory"="/var/run/kmm/images/harbor.kakor.ovh/public/amddriver:v1.0/fs/firmwareDir/updates"
I0507 13:57:16.614772       1 modprobe.go:33] kmm-worker "msg"="Running modprobe" "command"="/usr/sbin/modprobe -vd /var/run/kmm/images/<myregistry>/public/amddriver:v1.0/fs/opt amdgpu"
I0507 13:57:16.615904       1 cmdlog.go:71] kmm-worker/modprobe/stderr "msg"="modprobe: FATAL: Module amdgpu not found in directory /var/run/kmm/images/<myregistry>/public/amddriver:v1.0/fs/opt/lib/modules/6.5.5-200.fc38.x86_64"
E0507 13:57:16.616009       1 cmdutils.go:11] kmm-worker "msg"="Fatal error" "error"="error while waiting on the command: exit status 1"

Thank you in advance :)

Regards

Airgapped installation

Hi Yevgeny, thanks for your effort on the amd-gpu-operator.

IHAC that needs an airgapped installation of this operator - can you advise on how I can go about that? for e.g. I noted that building the amdgpu driver container from source requires image quay.io/yshnaidm/amd_gpu_sources, which reaches out to an external registry. If whitelisting is not allowed in customer environment, what are the steps (like mirroring/caching) do we take?

Thanks in advance.

yevgeny-shnaidman / amd-gpu-operator Goto Github PK

amd-gpu-operator's Issues

KMM Issue to create amd-gpu driver image

Airgapped installation

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent