Comments (11)
Hi, I found that the issue with the initialization of wandb was due to its inability to retrieve the GPU model and CPU model during initialization. However, when I commented out that part of the code, wandb was able to initialize normally. It might be because I am using the latest version of the Nvidia driver, version 555.99, and wandb cannot get the corresponding GPU model from the driver, causing the error.
from wandb.
@Wissotsky @zzk2021
Hi,My solution is to avoid using pynvml to get GPU information. The downside is that you won't be able to see GPU and CPU-related information on the wandb website. In the code, you can comment out the code I showed in the probe function of the library file wandb/sdk/internal/system/assets, and then wandb will work normally.
from wandb.
I'm seeing this today after upgrading to the v555 of the NVIDIA drivers. I can confirm commenting out this block bypasses the error for now.
from wandb.
from wandb.
Can concur the same problem occurring with driver ver 555.99
from wandb.
Thank you for surfacing this. I will raise with our engineering team and we will keep you posted with the progress
from wandb.
WandB Internal User commented:
zzk2021 commented:
same problem, how to fix it?
from wandb.
WandB Internal User commented:
Wissotsky commented:
Can concur the same problem occurring with driver ver 555.99
from wandb.
WandB Internal User commented:
CYYJL commented:
@Wissotsky @zzk2021
Hi,My solution is to avoid using pynvml to get GPU information. The downside is that you won't be able to see GPU and CPU-related information on the wandb website. In the code, you can comment out the code I showed in the probe function of the library file wandb/sdk/internal/system/assets, and then wandb will work normally.
from wandb.
Thanks, looks like the fix worked.
In case anyone else is having this issue, make sure you upgrade to 0.17.2
from wandb.
Please upgrade to version 0.17.2 and turn on our new backend wandb-core
with wandb.require("core")
after importing wandb. No source code modification should be necessary.
from wandb.
Related Issues (20)
- Distributed training with wandb.sweep HOT 6
- does not exist and causes issue with numpy 2.0 release HOT 15
- [Solved][App]: Unable to see the runs in workspace even if the run is taking place successfully HOT 2
- [App]: runs showing from link but not in project page HOT 3
- [Q] Upgrade to gqlparser 2.5.14 in next release? HOT 1
- [App]: All tests fail to run because some dependencies are missing HOT 5
- [Feature]: Halving Random Grid Search for Hyperparameter Tuning HOT 1
- Issue showing runs in the groups from mobile browsers HOT 4
- [Q] "None" was logged when wandb.sweep using pytorch_lightning HOT 3
- [CLI]: wandb write data error HOT 2
- [Q] Download an artifact without the API/CLI HOT 4
- [CLI]: Offline wandb sync failed HOT 7
- [CLI]: init wandb sdk but the /tmp/code directory was not created HOT 4
- [Q] why some steps didn't be logged during training? HOT 2
- [Q] wandb stream ID error HOT 2
- [Q]wandb: ERROR Internal wandb error: file data was not synced wandb: While tearing down the service manager. The following error has occurred: Python int too large to convert to C long HOT 4
- [Q] Does per-sample logging bottleneck batching? HOT 2
- [CLI]: Logging an external artifact folder in Azure Storage Account (HNS) results in a directory stub being logged HOT 2
- [CLI]: wandb.errors.UsageError: Agent user not valid HOT 1
- [Q] Any docs on the settings argument to `wandb.init`? HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wandb.