Comments (24)
nvidia 驱动 410.48 ; NVIDIA Corporation GP102 [TITAN X] ; nvidia-docker2-2.2.2 ; Centos7.6
go 1.13.5 ;
我使用的集群是1.16版本,按照教程安装完成后,aliyun.com/gpu-count: 2 gpu-mem:22
创建pod时aliyun.com/gpu-mem: 10,但是会报错stderr: nvidia-container-cli: device error: unknown device id: no-gpu-has-10MiB-to-run; 如果指定gpu-count: 1 报错Back-off restarting failed container可能是什么原因?
环境:
docker 19.03.5
可以尝试一下在container中添加env,如下:
containers:
- name: cuda
image: nvidia/cuda:latest
env:
- name: NVIDIA_VISIBLE_DEVICES
value: "all"
resources:
limits:
# GiB
aliyun.com/gpu-mem: 1
这对于我的类似错误有效。
from gpushare-scheduler-extender.
nvidia 驱动 410.48 ; NVIDIA Corporation GP102 [TITAN X] ; nvidia-docker2-2.2.2 ; Centos7.6
go 1.13.5 ;我使用的集群是1.16版本,按照教程安装完成后,aliyun.com/gpu-count: 2 gpu-mem:22
创建pod时aliyun.com/gpu-mem: 10,但是会报错stderr: nvidia-container-cli: device error: unknown device id: no-gpu-has-10MiB-to-run; 如果指定gpu-count: 1 报错Back-off restarting failed container
可能是什么原因?
环境:
docker 19.03.5可以尝试一下在container中添加env,如下:
containers: - name: cuda image: nvidia/cuda:latest env: - name: NVIDIA_VISIBLE_DEVICES value: "all" resources: limits: # GiB aliyun.com/gpu-mem: 1
这对于我的类似错误有效。
在我们的尝试中,加上 NVIDIA_VISIBLE_DEVICES=all
这个环境变量看起来并不是一个合理的解决方法;加上之后会导致容器实际使用的显卡与 scheduler
分配的显卡并不匹配,从而导致其他问题。这一点可以通过 k exec [pod] -- env
查看正常的 pod 环境变量确认;可以发现 scheduler
会通过类似于 NVIDIA_VISIBLE_DEVICES=GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
的环境变量控制 container 使用的显卡。而且加上 =all
的设置之后,通过 nvidia-smi
检查也会发现实际使用的显卡和分配的不匹配。
我们碰到的问题是 container 实际使用的内存大小超出了 k8s 设置的限制(或者是本身运行了一些非 k8s 管理的程序占用了显存),导致虽然 scheduler
认为卡上还有显存(于是调度到了该卡),但是实际上已经没有足够的显存。
最后我们的解决方法是严格保证实际使用的显存小于限制值,并且将非 k8s 管理的程序迁移到其他服务器(或集群中)。
In our case, adding the NVIDIA_VISIBLE_DEVICES=all
env config is not a reasonable solution to this problem. Add NVIDIA_VISIBLE_DEVICES
will cause that the GPU card the container actually used doesn't match the GPU card scheduler
assigned to the container, and this will cause other issues. And this can be verified by k exec [pod] -- env
, you will find a NVIDIA_VISIBLE_DEVICES=GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
even you don't specify one. Also, the nvidia-smi
can verify the actually used GPU card is not matched with the assigned one if you added NVIDIA_VISIBLE_DEVICES=all
.
The cause of our situation is that the container is using more memory than we give it (or there are some processes unmanaged by k8s using this card). Although the scheduler
thinks there's enough memory (so assign the process to that card), it finds that there isn't enough memory left when it actually runs.
And finally, we ensure all containers will use less memory than we assign to them and move all processes which are not managed by k8s to other machines.
from gpushare-scheduler-extender.
分享下我的解决方案以供大家参考。我的问题出在修改 kube-scheduler.yaml
上。
之前错误的方式:
cd /etc/kubernetes/manifests
cp kube-scheduler.yaml kube-scheduler.yaml.backup
- 按照安装教程直接修改
kube-scheduler.yaml
问题就出在第 2 步: 在 /etc/kubernetes/manifests
下现在有两个文件 kube-scheduler.yaml
和 kube-scheduler.yaml.backup
,kubernetes 会把它们全部加载,可能因为后者会覆盖前者导致配置修改不生效。
正确的方式:
cd /etc/kubernetes
mv manifests/kube-scheduler.yaml .
,把kube-scheduer.yaml
先移出manifests
目录。cp kube-scheduler.yaml kube-scheduler.yaml.backup
,如果不想备份可以跳过这一步。- 按照安装教程直接修改
kube-scheduler.yaml
mv kube-scheduler.yaml manifests
,将修改好的文件重新放入manifests
文件夹。
from gpushare-scheduler-extender.
I have the same issue. it shows:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 28s default-scheduler Successfully assigned kong/binpack-2-54df84c8d7-nknfx to 192.168.3.4
Normal Pulled <invalid> (x3 over <invalid>) kubelet, 192.168.3.4 Container image "cheyang/gpu-player:v2" already present on machine
Normal Created <invalid> (x3 over <invalid>) kubelet, 192.168.3.4 Created container binpack-2
Warning Failed <invalid> (x3 over <invalid>) kubelet, 192.168.3.4 Error: failed to start container "binpack-2": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:407: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: device error: unknown device id: no-gpu-has-2MiB-to-run\\\\n\\\"\"": unknown
I also checked gpushare-device-plugin. The log is here:
I0220 10:16:27.337772 1 main.go:18] Start gpushare device plugin
I0220 10:16:27.337870 1 gpumanager.go:28] Loading NVML
I0220 10:16:27.365052 1 gpumanager.go:37] Fetching devices.
I0220 10:16:27.365082 1 gpumanager.go:43] Starting FS watcher.
I0220 10:16:27.365204 1 gpumanager.go:51] Starting OS watcher.
I0220 10:16:27.392614 1 nvidia.go:64] Deivce GPU-5f44aae0-ca45-9038-5202-a033fa4f471a's Path is /dev/nvidia0
I0220 10:16:27.392682 1 nvidia.go:69] # device Memory: 12036
I0220 10:16:27.392691 1 nvidia.go:40] set gpu memory: 11
I0220 10:16:27.392699 1 nvidia.go:76] # Add first device ID: GPU-5f44aae0-ca45-9038-5202-a033fa4f471a-_-0
I0220 10:16:27.392713 1 nvidia.go:79] # Add last device ID: GPU-5f44aae0-ca45-9038-5202-a033fa4f471a-_-10
I0220 10:16:27.421574 1 nvidia.go:64] Deivce GPU-28071eed-0993-c165-e123-ea818a546f14's Path is /dev/nvidia1
I0220 10:16:27.421596 1 nvidia.go:69] # device Memory: 12036
I0220 10:16:27.421604 1 nvidia.go:76] # Add first device ID: GPU-28071eed-0993-c165-e123-ea818a546f14-_-0
I0220 10:16:27.421628 1 nvidia.go:79] # Add last device ID: GPU-28071eed-0993-c165-e123-ea818a546f14-_-10
I0220 10:16:27.453463 1 nvidia.go:64] Deivce GPU-203884e8-4afc-ad03-1a15-2ff5b34fc01c's Path is /dev/nvidia2
I0220 10:16:27.453482 1 nvidia.go:69] # device Memory: 12036
I0220 10:16:27.453490 1 nvidia.go:76] # Add first device ID: GPU-203884e8-4afc-ad03-1a15-2ff5b34fc01c-_-0
I0220 10:16:27.453502 1 nvidia.go:79] # Add last device ID: GPU-203884e8-4afc-ad03-1a15-2ff5b34fc01c-_-10
I0220 10:16:27.480145 1 nvidia.go:64] Deivce GPU-d0cd36b5-9221-facd-203c-b2342b207439's Path is /dev/nvidia3
I0220 10:16:27.480166 1 nvidia.go:69] # device Memory: 12036
I0220 10:16:27.480172 1 nvidia.go:76] # Add first device ID: GPU-d0cd36b5-9221-facd-203c-b2342b207439-_-0
I0220 10:16:27.480190 1 nvidia.go:79] # Add last device ID: GPU-d0cd36b5-9221-facd-203c-b2342b207439-_-10
I0220 10:16:27.501184 1 nvidia.go:64] Deivce GPU-c3ea63e8-ccde-4e49-7093-d570e14d82c2's Path is /dev/nvidia4
I0220 10:16:27.501203 1 nvidia.go:69] # device Memory: 12036
I0220 10:16:27.501209 1 nvidia.go:76] # Add first device ID: GPU-c3ea63e8-ccde-4e49-7093-d570e14d82c2-_-0
I0220 10:16:27.501216 1 nvidia.go:79] # Add last device ID: GPU-c3ea63e8-ccde-4e49-7093-d570e14d82c2-_-10
I0220 10:16:27.524208 1 nvidia.go:64] Deivce GPU-b663687a-4219-c707-1c3c-8ddcc59b9dbd's Path is /dev/nvidia5
I0220 10:16:27.524226 1 nvidia.go:69] # device Memory: 12036
I0220 10:16:27.524231 1 nvidia.go:76] # Add first device ID: GPU-b663687a-4219-c707-1c3c-8ddcc59b9dbd-_-0
I0220 10:16:27.524243 1 nvidia.go:79] # Add last device ID: GPU-b663687a-4219-c707-1c3c-8ddcc59b9dbd-_-10
I0220 10:16:27.547600 1 nvidia.go:64] Deivce GPU-8d72b13d-0942-e5f4-e2bc-e77d29e88be1's Path is /dev/nvidia6
I0220 10:16:27.547627 1 nvidia.go:69] # device Memory: 12036
I0220 10:16:27.547635 1 nvidia.go:76] # Add first device ID: GPU-8d72b13d-0942-e5f4-e2bc-e77d29e88be1-_-0
I0220 10:16:27.547653 1 nvidia.go:79] # Add last device ID: GPU-8d72b13d-0942-e5f4-e2bc-e77d29e88be1-_-10
I0220 10:16:27.573674 1 nvidia.go:64] Deivce GPU-7d1c58db-fc7d-8042-f86e-f37297c2a1c9's Path is /dev/nvidia7
I0220 10:16:27.573696 1 nvidia.go:69] # device Memory: 12036
I0220 10:16:27.573704 1 nvidia.go:76] # Add first device ID: GPU-7d1c58db-fc7d-8042-f86e-f37297c2a1c9-_-0
I0220 10:16:27.573718 1 nvidia.go:79] # Add last device ID: GPU-7d1c58db-fc7d-8042-f86e-f37297c2a1c9-_-10
I0220 10:16:27.573736 1 server.go:43] Device Map: map[GPU-b663687a-4219-c707-1c3c-8ddcc59b9dbd:5 GPU-8d72b13d-0942-e5f4-e2bc-e77d29e88be1:6 GPU-7d1c58db-fc7d-8042-f86e-f37297c2a1c9:7 GPU-5f44aae0-ca45-9038-5202-a033fa4f471a:0 GPU-28071eed-0993-c165-e123-ea818a546f14:1 GPU-203884e8-4afc-ad03-1a15-2ff5b34fc01c:2 GPU-d0cd36b5-9221-facd-203c-b2342b207439:3 GPU-c3ea63e8-ccde-4e49-7093-d570e14d82c2:4]
I0220 10:16:27.573807 1 server.go:44] Device List: [GPU-5f44aae0-ca45-9038-5202-a033fa4f471a GPU-28071eed-0993-c165-e123-ea818a546f14 GPU-203884e8-4afc-ad03-1a15-2ff5b34fc01c GPU-d0cd36b5-9221-facd-203c-b2342b207439 GPU-c3ea63e8-ccde-4e49-7093-d570e14d82c2 GPU-b663687a-4219-c707-1c3c-8ddcc59b9dbd GPU-8d72b13d-0942-e5f4-e2bc-e77d29e88be1 GPU-7d1c58db-fc7d-8042-f86e-f37297c2a1c9]
I0220 10:16:27.592888 1 podmanager.go:68] No need to update Capacity aliyun.com/gpu-count
I0220 10:16:27.593476 1 server.go:222] Starting to serve on /var/lib/kubelet/device-plugins/aliyungpushare.sock
I0220 10:16:27.594247 1 server.go:230] Registered device plugin with Kubelet
I0220 16:33:58.048842 1 allocate.go:46] ----Allocating GPU for gpu mem is started----
I0220 16:33:58.048863 1 allocate.go:57] RequestPodGPUs: 2
I0220 16:33:58.048868 1 allocate.go:61] checking...
I0220 16:33:58.063205 1 podmanager.go:112] all pod list [{{ } {binpack-2-54df84c8d7-nknfx binpack-2-54df84c8d7- kong /api/v1/namespaces/kong/pods/binpack-2-54df84c8d7-nknfx ea316ce3-fd64-413e-b5fe-c06baff6444b 366992 0 2020-02-20 16:32:54 +0000 UTC <nil> <nil> map[app:binpack-2 pod-template-hash:54df84c8d7] map[] [{apps/v1 ReplicaSet binpack-2-54df84c8d7 a0faf71e-cd14-4614-a9f4-d87a236badd1 0xc4203bfe6a 0xc4203bfe6b}] nil [] } {[{default-token-kj8jt {nil nil nil nil nil &SecretVolumeSource{SecretName:default-token-kj8jt,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}] [] [{binpack-2 cheyang/gpu-player:v2 [] [] [] [] [] {map[aliyun.com/gpu-mem:{{2 0} {<nil>} 2 DecimalSI}] map[aliyun.com/gpu-mem:{{2 0} {<nil>} 2 DecimalSI}]} [{default-token-kj8jt true /var/run/secrets/kubernetes.io/serviceaccount <nil>}] [] nil nil nil /dev/termination-log File IfNotPresent nil false false false}] Always 0xc4203bff08 <nil> ClusterFirst map[] default default <nil> 192.168.3.4 false false false <nil> &PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[],} [] nil default-scheduler [{node.kubernetes.io/not-ready Exists NoExecute 0xc4203bffa0} {node.kubernetes.io/unreachable Exists NoExecute 0xc4203bffc0}] [] 0xc4203bffd0 nil []} {Pending [{PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-02-20 16:32:54 +0000 UTC }] <nil> [] [] BestEffort}}]
I0220 16:33:58.063418 1 podmanager.go:123] list pod binpack-2-54df84c8d7-nknfx in ns kong in node 192.168.3.4 and status is Pending
I0220 16:33:58.063430 1 podutils.go:81] No assume timestamp for pod binpack-2-54df84c8d7-nknfx in namespace kong, so it's not GPUSharedAssumed assumed pod.
W0220 16:33:58.063439 1 allocate.go:152] invalid allocation requst: request GPU memory 2 can't be satisfied.
from gpushare-scheduler-extender.
Fixed my issue by re-creating /etc/kubernetes/manifests/kube-scheduler.yaml
to make static scheduler pod use correct configuration.
from gpushare-scheduler-extender.
@Svendegroote91 in my case, it was the wrong configuration for kube-scheduler.yaml
. I recommend you to check it and read installation guide again
from gpushare-scheduler-extender.
I solved it - I had to update all my master nodes with the instruction from the installation guide.
from gpushare-scheduler-extender.
@Mhs-220 my cluster version is v1.15.11 (because I am using Kubeflow on top of Kubernetes and v1.15 is fully supported)
from gpushare-scheduler-extender.
nvidia 驱动 410.48 ; NVIDIA Corporation GP102 [TITAN X] ; nvidia-docker2-2.2.2 ; Centos7.6
go 1.13.5 ;
from gpushare-scheduler-extender.
你要看一下gpushare-device-plugin的日志。我怀疑gpushare-scheduler-extender没有正确配置。
from gpushare-scheduler-extender.
Hi guys, I also have this issue when i use demo1:
Warning Failed 3m41s (x5 over 5m20s) kubelet, k8s-master Error: failed to start container "binpack-1": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:413: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: device error: unknown device id: no-gpu-has-2MiB-to-run\\\\n\\\"\"": unknown
My docker version is 19.3.3 nvidia driver vesion is 418.116.00, and i use kubectl-inspect-gpushare it shows:
NAME IPADDRESS GPU0(Allocated/Total) PENDING(Allocated) GPU Memory(GiB)
k8s-master 192.168.1.103 0/7 2 2/7
Allocated/Total GPU Memory In Cluster:
2/7 (28%)
The gpushare-device-plugin log is same as above.
Is there any solution can share with me,thanks.
from gpushare-scheduler-extender.
I tried what @Carlosnight says, but when I run kubectl inspect gpushare
I see a new column called PENDING(Allocated) and it seems that GPU isolation not happened.
I checked the value of ALIYUN_COM_GPU_MEM_IDX
inside my pod and it's -1.
P.S: I have a warning in gpushare-schd-extender
that says: Pod gpu-pod in ns production is not set the GPU ID -1 in node xxxxxxxxx
P.S: The DaemonSet log shows that it's found GPU but repeatedly saying about not able to assign 3GiB memory to the pod.
from gpushare-scheduler-extender.
Same issue here, any feedback on this?
from gpushare-scheduler-extender.
@Mhs-220
I attached the kube-scheduler.yaml.zip from my controller node.
I have 3 controller nodes but only did the kube-scheduler update on the controller node (node "ctr-1" in my case) through which I am using the KubeAPI (I have no HA on top of the controller nodes at the moment). That should be sufficient no? You can see that the kube-scheduler restarted after updating the file:
Maybe it helps if I share the logs of the gpushare-schd-extender together with the logs of the actual container:
What strikes me is that it first goes into "Pending" state and subsequently to "Running" but the status in the kubectl inspect gpushare
is not moving.
Can you elaborate what exactly you forgot to apply to the kube-scheduler.yaml file or point to the mistake in my attached file?
Your help would be appreciated a lot!
from gpushare-scheduler-extender.
Can somebody explain why the environment variable NVIDIA_VISIBLE_DEVICES
fixes this and why it is needed in the manifest file?
from gpushare-scheduler-extender.
The error is happening again after update kubernetes to 1.18. Any Idea?
@Svendegroote91 what is your cluster version?
from gpushare-scheduler-extender.
In my case, it does not work with v1.17.4, but works well with 1.15.11. Thanks to @Mhs-220 and @Svendegroote91.
I do not know why it works or why it does not work.
from gpushare-scheduler-extender.
v1.16.15 has the same problem. Seems that v1.16 is a version where API changes a lot. I have seen some deprecated "pre-1.16" versions in other projects, like shown here. Maybe @cheyang can help us out?
from gpushare-scheduler-extender.
versions
- docker 19.03.11
- kubernetes v1.19.8
processing
containers:
- name: cuda
image: nvidia/cuda:latest
env:
- name: NVIDIA_VISIBLE_DEVICES
value: "all"
resources:
limits:
# GiB
aliyun.com/gpu-mem: 1
- after add the environment variable NVIDIA_VISIBLE_DEVICES=all the program can be run but the result of nvidia-smi comand , gpu memory, may be not correct or something wrong
No running processes found
- 当我使用了NVIDIA_VISIBLE_DEVICES=all之后, 我的应用可以正常启动了, 但是nvidia-smi命令的输出结果显示的内存大小似乎不符合预期或者某个地方出现了什么问题, 例如:
No running processes found
[root@k8s243 models]# nvidia-smi
Thu Sep 9 17:21:00 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P4 Off | 00000000:00:10.0 Off | 0 |
| N/A 31C P8 6W / 75W | 0MiB / 7611MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
kubectl inspect gpushare
NAME IPADDRESS GPU0(Allocated/Total) GPU Memory(GiB)
k8s243 10.3.171.243 2/7 2/7
k8s25 10.3.144.25 0/14 0/14
------------------------------------------------
Allocated/Total GPU Memory In Cluster:
2/21 (9%)
from gpushare-scheduler-extender.
I encountered the same problem after installing according to the instructions. Later, I found that the other master had not been adjusted according to the instructions. After I modified the scheduler component configuration of all two masters, they worked normally.
from gpushare-scheduler-extender.
from gpushare-scheduler-extender.
Can somebody explain why the environment variable
NVIDIA_VISIBLE_DEVICES
fixes this and why it is needed in the manifest file?
seems like this is how it works AliyunContainerService/gpushare-device-plugin#55 (comment)
from gpushare-scheduler-extender.
from gpushare-scheduler-extender.
请教下您看的哪个安装教程吗
from gpushare-scheduler-extender.
Related Issues (20)
- 显存与真实情况不符 HOT 1
- Not able to use gpushare-scheduler-extender on EKS cluster with Kubernetes v1.24 HOT 2
- plugin does not evenly distribute the pods. 这个插件无法均匀分配Pod。 HOT 2
- GPU cores scheduling / GPU核心调度
- 你好,kubectl logs这个命令在gpu容器上无效,在普通容器上却可以
- Back-off restarting failed container: gpushare-device-plugin-ds-xxxxx HOT 1
- 这个GPU共享插件支持使用dcgm-exporter做监控吗 HOT 4
- 如果想要指定使用两张显卡多卡训练 该怎么做 HOT 1
- 如果一个机器上有两张卡,第一张卡的内存使满了,之后的任务会调度到另一张卡上吗
- Support for Horizontal Pod Autoscaling (HPA) with GPU Pods? 是否支持使用GPU Pods的水平Pod自动扩展(HPA)? HOT 1
- 该项目还在维护吗
- 关于显存申请基本单位改为MiB但不起作用的问题
- 调度层有bug吧,请求8G,实际设备最大7G,居然最终能创建成功pod
- 这个项目目前在使用过程中存在的问题
- ALIYUN_COM_GPU_MEM_IDX in the annotation is different than ALIYUN_COM_GPU_MEM_IDX inside the pod
- 方案只能在阿里云上的机器里使用吗 HOT 1
- kubelet版本問題
- [AKS] kube-scheduler static POD not running for Aliyun GPU Scheduler Extender HOT 1
- 版本兼容性问题
- k8s上安装好插件,无法识别到集群GPU资源 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gpushare-scheduler-extender.