gpu plugin device can only be used by one pod about intel-device-plugins-for-kubernetes HOT 8 CLOSED

Sorry for the delay. I'm just back from holidays.

Currently over-commitment of extended resources isn't supported by K8s. Once allocated a device stops being available, the scheduler simply can't schedule a pod on the same node (given there is one device on the node). Semantically it would make more sense if a user could request a fraction of a device, but this contradicts to the spec for K8s extended resources.

But we could advertise one physical device as many virtual GPUs though, e.g. 10 of gpu.intel.com/i915-one-tenth. I think the exact number of such virtual devices per physical one could be configured with the plugin's command line options or node attributes.

from intel-device-plugins-for-kubernetes.

Well GPU device can serve many clients simultaneously, so restrict it for only one user doesn't serve practical usage.

I'm not sure what you mean by exposing "virtual GPU", does current k8s device plugin have support for virtual device entry or that's just a way to provide number of devices can be used for pods?

from intel-device-plugins-for-kubernetes.

Well GPU device can serve many clients simultaneously, so restrict it for only one user doesn't serve practical usage.

Understandable.
I'm not very knowledgeable about GPUs: is it possible to guarantee that a pod won't abuse a shared GPU and exhaust its resources (e.g. by allocating memory for a huge texture) so other pods won't starve. Then can a process access and read a GEM BO created by another process from inside a different pod? This is important especially in multi-tenant installations.

I'm not sure what you mean by exposing "virtual GPU", does current k8s device plugin have support for virtual device entry or that's just a way to provide number of devices can be used for pods?

The latter. The same real device node (/dev/dri/card0) can be mounted to many containers in different pods as /dev/dri/card0, /dev/dri/card1,..., /dev/dri/cardXX.

from intel-device-plugins-for-kubernetes.

DRM linux has no memory control on GPU now, as current intel gpu just use host memory for GPU, so depend on shmem fs for allocation. Usage for pod is similar to usage on native GPU processes.

BO sharing requires authenticated client which is normally handled between display manager vs. gfx library in applications. So there's no explicit way for between pods BO access.

I think when you expose to kubelet, you needs to provide a device list named e.g card0, card1, card2...but for device access path if for current one intel gpu device should all be /dev/dri/card0 and /dev/dri/renderD128.

So I think first step we can add an option to gpu plugin to set max gpu instances for pods which would generate that number of device nodes to kubelet.

from intel-device-plugins-for-kubernetes.

So I think first step we can add an option to gpu plugin to set max gpu instances for pods which would generate that number of device nodes to kubelet.

Agree. This might be useful for users running their own private clusters.

Please disregard my last sentence in the previous comment. kubelet doesn't really care about the names of devices nodes. It just needs different device IDs like card0-0, card0-1,..., card0-X corresponding to the same content of DeviceSpec.HostPath and DeviceSpec.ContainerPath, so containers sharing the same GPU would see it as /dev/card0, /dev/renderD128.

from intel-device-plugins-for-kubernetes.

Just wonder if user assigns multiple intel.com/gpu resources in pod yaml, would that cause problem? My guess is not, as dev path is same so container runtime should still apply for it, right?

And I'd like to know if you like to implement new option for this, or like to see a PR.

from intel-device-plugins-for-kubernetes.

Just wonder if user assigns multiple intel.com/gpu resources in pod yaml, would that cause problem? My guess is not, as dev path is same so container runtime should still apply for it, right?

Right, that's my expectation too.

And I'd like to know if you like to implement new option for this, or like to see a PR.

I'll implement it.

from intel-device-plugins-for-kubernetes.

Thanks. Looks good to me!

from intel-device-plugins-for-kubernetes.