Comments (8)
Sorry for the delay. I'm just back from holidays.
Currently over-commitment of extended resources isn't supported by K8s. Once allocated a device stops being available, the scheduler simply can't schedule a pod on the same node (given there is one device on the node). Semantically it would make more sense if a user could request a fraction of a device, but this contradicts to the spec for K8s extended resources.
But we could advertise one physical device as many virtual GPUs though, e.g. 10 of gpu.intel.com/i915-one-tenth
. I think the exact number of such virtual devices per physical one could be configured with the plugin's command line options or node attributes.
from intel-device-plugins-for-kubernetes.
Well GPU device can serve many clients simultaneously, so restrict it for only one user doesn't serve practical usage.
I'm not sure what you mean by exposing "virtual GPU", does current k8s device plugin have support for virtual device entry or that's just a way to provide number of devices can be used for pods?
from intel-device-plugins-for-kubernetes.
Well GPU device can serve many clients simultaneously, so restrict it for only one user doesn't serve practical usage.
Understandable.
I'm not very knowledgeable about GPUs: is it possible to guarantee that a pod won't abuse a shared GPU and exhaust its resources (e.g. by allocating memory for a huge texture) so other pods won't starve. Then can a process access and read a GEM BO created by another process from inside a different pod? This is important especially in multi-tenant installations.
I'm not sure what you mean by exposing "virtual GPU", does current k8s device plugin have support for virtual device entry or that's just a way to provide number of devices can be used for pods?
The latter. The same real device node (/dev/dri/card0
) can be mounted to many containers in different pods as /dev/dri/card0
, /dev/dri/card1
,..., /dev/dri/cardXX
.
from intel-device-plugins-for-kubernetes.
DRM linux has no memory control on GPU now, as current intel gpu just use host memory for GPU, so depend on shmem fs for allocation. Usage for pod is similar to usage on native GPU processes.
BO sharing requires authenticated client which is normally handled between display manager vs. gfx library in applications. So there's no explicit way for between pods BO access.
I think when you expose to kubelet, you needs to provide a device list named e.g card0, card1, card2...but for device access path if for current one intel gpu device should all be /dev/dri/card0 and /dev/dri/renderD128.
So I think first step we can add an option to gpu plugin to set max gpu instances for pods which would generate that number of device nodes to kubelet.
from intel-device-plugins-for-kubernetes.
So I think first step we can add an option to gpu plugin to set max gpu instances for pods which would generate that number of device nodes to kubelet.
Agree. This might be useful for users running their own private clusters.
Please disregard my last sentence in the previous comment. kubelet doesn't really care about the names of devices nodes. It just needs different device IDs like card0-0
, card0-1
,..., card0-X
corresponding to the same content of DeviceSpec.HostPath
and DeviceSpec.ContainerPath
, so containers sharing the same GPU would see it as /dev/card0
, /dev/renderD128
.
from intel-device-plugins-for-kubernetes.
Just wonder if user assigns multiple intel.com/gpu resources in pod yaml, would that cause problem? My guess is not, as dev path is same so container runtime should still apply for it, right?
And I'd like to know if you like to implement new option for this, or like to see a PR.
from intel-device-plugins-for-kubernetes.
Just wonder if user assigns multiple intel.com/gpu resources in pod yaml, would that cause problem? My guess is not, as dev path is same so container runtime should still apply for it, right?
Right, that's my expectation too.
And I'd like to know if you like to implement new option for this, or like to see a PR.
I'll implement it.
from intel-device-plugins-for-kubernetes.
Thanks. Looks good to me!
from intel-device-plugins-for-kubernetes.
Related Issues (20)
- FPGA: Issue with .aocx bitstream parser HOT 3
- QAT: enable 420xx
- QAT VF kernel driver name is c62xvt, not c6xxvt for C62X device HOT 5
- Intel QAT service script issues and restriction HOT 3
- intel device plugins gpu : failed to call webhook, context deadline exceeded HOT 10
- GPU product node label supports only one product type HOT 17
- operator: service selectors are vague and overlap with other operators HOT 1
- e2e-sgx: admissionwebhook tests are not run anymore
- [QAT] Why kernel driver passthrough all uio files HOT 9
- QAT testing/deployment updates HOT 3
- GPU label not added to nodes on Talos because i915 is built into the kernel HOT 1
- Publish accel-config-demo image to docker hub HOT 2
- intel-idxd-config-initcontainer cannot use UBI base images. HOT 5
- SGX-enabled pods sometimes get created without SGX device mounted HOT 5
- [QAT] No devices found in container without privilege HOT 2
- move openssl-qat-engine image to Ubuntu 24.04
- Intel 14th-gen Meteor Lake HOT 2
- Not compatible device: version HOT 2
- workflow: upgrade codecov upload method
- Prepare 0.30.0 release HOT 22
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from intel-device-plugins-for-kubernetes.