Comments (3)
I can reproduce this, launched a g3.8x
instance with bottlerocket-aws-ecs-2-nvidia-x86_64-v1.19.2-29cc92cc
and followed the repro steps, docker logs <container>
gives me:
GPU 0: Tesla M60 (UUID: GPU-963a4569-ebb4-d876-14a7-c3c8491f8682)
GPU 1: Tesla M60 (UUID: GPU-cf45c479-5e3d-5316-42b5-1301bc4f4f6a)
Failed to initialize NVML: Unknown Error
Failed to initialize NVML: Unknown Error
Because the instance I was using had 2 GPUs, I also tried exposing each of the GPUs to two different containers:
docker run -d --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=0 -e NVIDIA_DRIVER_CAPABILITIES=all \
nvcr.io/nvidia/cuda:12.0.0-base-ubuntu20.04 bash -c "while [ true ]; do nvidia-smi -L; sleep 5; done"
docker run -d --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=1 -e NVIDIA_DRIVER_CAPABILITIES=all \
nvcr.io/nvidia/cuda:12.0.0-base-ubuntu20.04 bash -c "while [ true ]; do nvidia-smi -L; sleep 5; done"
The result was the same for both containers, but it showed a new error message in the logs:
GPU 0: Tesla M60 (UUID: GPU-963a4569-ebb4-d876-14a7-c3c8491f8682)
Unable to determine the device handle for gpu 0000:00:1D.0: Unknown Error
Failed to initialize NVML: Unknown Error
Failed to initialize NVML: Unknown Error
from bottlerocket.
Hi @modelbitjason , thanks for reporting this.
I just want to add some context on why the steps you follow trigger this failure.
When you enable-admin-container
/disable-admin-container
, a systemctl
command will be issued to reload containerd's configurations so that the admin container is started/stopped. Whenever this happens, systemd will undo any cgroups modifications done by libnvidia-container
while creating the containers and granting them access to the GPUs. This is a know issue, and the solution that was given was to run nvidia-ctk
whenever the GPUs are loaded. We already do that today in all Bottlerocket variants. There seems to be another fix in newer versions of libnvidia-container, however when we tried to update to v1.14.X, it broke the ECS aarch64 variant. I'll ask my coworkers to give that new fix a spin, and check if the problem persists (after we figure out why the new libnvidia-container version broke the ECS aarch64 variant).
That said, could you please expand on what do you mean with this?
If I tell ECS about the GPU, it won't let me gracefully roll a new instance on that host.
Are you having problems with task deployments? Or are you trying to "over-subscribe" the node so that you can run multiple tasks in the same host and share the one GPU?
from bottlerocket.
Thanks for the explanation @arnaldo2792 -- I saw some other tickets about cgroups but those fixes didn't help. This explains why.
re: ECS + GPUs, yes, but more specifically, I need to have the old instance hand-off data to the new one and do a graceful shutdown. I only have one ECS-managed task per EC2 instance (except during rollout).
When I roll a new task version, the newly started version finds the existing container, asks for some state, then tells it to drain + eventually shutdown. This way I can both be sure the new version comes up healthy before the old version on that host is removed.
This service manages some long-lived containers on the host and proxies requests to them. The old version passes control of the containers to the newly started version, but needs to stick around until all the outstanding requests it is currently proxying are complete.
from bottlerocket.
Related Issues (20)
- Bottlerocket compliance with IRS 1075 HOT 1
- v1.19.3 update eni-max-pods mapping file
- v1.19.3 Host container updates HOT 1
- v1.19.3 Go and Rust dependency updates HOT 1
- v1.19.3 💘 Tracking Issue HOT 2
- SELinux seems to be blocking spire csi driver installation. HOT 2
- Fix lints in host-ctr
- Can't use nvidia GPU since v1.19.1 HOT 4
- Kubernetes-1.30 package and variants Tracking Issue 🐳
- Intermittent pod attach issues with bottlerocket-aws-k8s-1.24-x86_64-v1.19.3-f097c617 AMI HOT 9
- exec probes such as readiness and liveness probes time out in v1.19.3 HOT 8
- Dogtag Hostname tool
- v1.19.4 💘 Tracking Issue HOT 2
- Upgrading containerd version to 1.7.x HOT 4
- dynamic models crate workaround HOT 1
- How to disable IPv6 DAD to reduce startup delay of pods on IPv6 cluster HOT 6
- consider defaulting the clock source for EC2 instances HOT 1
- OOTB: package builds should not be aware of image features HOT 3
- Remove metal and vmware k8s 1.25 variants by May 2024
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bottlerocket.