Comments (8)
I think we should use ALIYUN_COM_GPU_MEM_IDX env value inside the container to get the index of the assigned GPU (for the current container)
Reference from the here and here
from gpushare-scheduler-extender.
Given a server with 8 GPUs, if we start a pod with "aliyun.com/gpu-count:2", and the scheduler assign GPU3 and GPU7 to this pod, what is the GPU number for these 2 GPU cards in the pod? 0 and 1?
aliyun.com/gpu-count
is not used for scheduling, it's only for calculate the number of GPUs. And if you are concerned about GPU ID, as @nareshganesan said, it's environment variable ALIYUN_COM_GPU_MEM_IDX.
from gpushare-scheduler-extender.
So, the value of ALIYUN_COM_GPU_MEM_IDX should be the actual ID of the host GPU server, such as 3 or 7 in my example, right? And in the container, the application will access the GPU according to the real GPU ID of the server.
from gpushare-scheduler-extender.
@tjliupeng , yes exactly as per the code, will verify it once in container (inside a multi gpu host machine ) and confirm here 👍
from gpushare-scheduler-extender.
Great work... We have servers with 8 GPUs which can be mixed variety (eg. P100, V100, RTX 6000). It would be a nice enhancement to be able to specify GPU model preference. Since P100 is not MPS architecture, by default we want pod to choose GPU model V100 or RTX 6000.
from gpushare-scheduler-extender.
I think you can add node label to specify the GPU model, and use node selector to choose the node.
from gpushare-scheduler-extender.
I think you can add node label to specify the GPU model, and use node selector to choose the node.
This would work if we have different GPUs residing in different nodes. If we have two types of GPUs residing in the same node/machine, is there currently any way to tell your pod which GPU to pick? Can we request GPU by their ID (ALIYUN_COM_GPU_MEM_IDX)?
from gpushare-scheduler-extender.
I think the current implementation doesn't work for your scenario. The assumption of the design is the GPUs in the same node are the same types.
from gpushare-scheduler-extender.
Related Issues (20)
- 显存与真实情况不符 HOT 1
- Not able to use gpushare-scheduler-extender on EKS cluster with Kubernetes v1.24 HOT 2
- plugin does not evenly distribute the pods. 这个插件无法均匀分配Pod。 HOT 2
- GPU cores scheduling / GPU核心调度
- 你好,kubectl logs这个命令在gpu容器上无效,在普通容器上却可以
- Back-off restarting failed container: gpushare-device-plugin-ds-xxxxx HOT 1
- 这个GPU共享插件支持使用dcgm-exporter做监控吗 HOT 4
- 如果想要指定使用两张显卡多卡训练 该怎么做 HOT 1
- 如果一个机器上有两张卡,第一张卡的内存使满了,之后的任务会调度到另一张卡上吗
- Support for Horizontal Pod Autoscaling (HPA) with GPU Pods? 是否支持使用GPU Pods的水平Pod自动扩展(HPA)? HOT 1
- 该项目还在维护吗
- 关于显存申请基本单位改为MiB但不起作用的问题
- 调度层有bug吧,请求8G,实际设备最大7G,居然最终能创建成功pod
- 这个项目目前在使用过程中存在的问题
- ALIYUN_COM_GPU_MEM_IDX in the annotation is different than ALIYUN_COM_GPU_MEM_IDX inside the pod
- 方案只能在阿里云上的机器里使用吗 HOT 1
- kubelet版本問題
- [AKS] kube-scheduler static POD not running for Aliyun GPU Scheduler Extender HOT 1
- 版本兼容性问题
- k8s上安装好插件,无法识别到集群GPU资源 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gpushare-scheduler-extender.