Question: how does the GPU device number count in the container with "gpu-count"? about gpushare-scheduler-extender HOT 8 CLOSED

I think we should use ALIYUN_COM_GPU_MEM_IDX env value inside the container to get the index of the assigned GPU (for the current container)

Reference from the here and here

from gpushare-scheduler-extender.

Given a server with 8 GPUs, if we start a pod with "aliyun.com/gpu-count:2", and the scheduler assign GPU3 and GPU7 to this pod, what is the GPU number for these 2 GPU cards in the pod? 0 and 1?

aliyun.com/gpu-count is not used for scheduling, it's only for calculate the number of GPUs. And if you are concerned about GPU ID, as @nareshganesan said, it's environment variable ALIYUN_COM_GPU_MEM_IDX.

from gpushare-scheduler-extender.

So, the value of ALIYUN_COM_GPU_MEM_IDX should be the actual ID of the host GPU server, such as 3 or 7 in my example, right? And in the container, the application will access the GPU according to the real GPU ID of the server.

from gpushare-scheduler-extender.

@tjliupeng , yes exactly as per the code, will verify it once in container (inside a multi gpu host machine ) and confirm here 👍

from gpushare-scheduler-extender.

Great work... We have servers with 8 GPUs which can be mixed variety (eg. P100, V100, RTX 6000). It would be a nice enhancement to be able to specify GPU model preference. Since P100 is not MPS architecture, by default we want pod to choose GPU model V100 or RTX 6000.

from gpushare-scheduler-extender.

I think you can add node label to specify the GPU model, and use node selector to choose the node.

from gpushare-scheduler-extender.

I think you can add node label to specify the GPU model, and use node selector to choose the node.

This would work if we have different GPUs residing in different nodes. If we have two types of GPUs residing in the same node/machine, is there currently any way to tell your pod which GPU to pick? Can we request GPU by their ID (ALIYUN_COM_GPU_MEM_IDX)?

from gpushare-scheduler-extender.

I think the current implementation doesn't work for your scenario. The assumption of the design is the GPUs in the same node are the same types.

from gpushare-scheduler-extender.