Comments (14)
It's not still missing in DeepSpeed. See current official documentation: https://www.deepspeed.ai/docs/config-json/#bfloat16-options.
As a first step, are you running DeepSpeed with ZeRO stage 1 or 2? Have you updated the DeepSpeed config in addition to passing bf16
as a parameter to the training script?
Our testing was done with the following:
CUDA Driver 465.19.01
CUDA 11.3 Update 1 (11.3.1.005)
cuBLAS 11.5.1.109 (part of CUDA 11.3 U1)
CUDNN 8.2.1.32
NCCL 2.9.9
As for the log_sigmoid thing, I don't know why that call would succeed with DeepSpeed and fail without it. As a sanity check, I'd try just spelling out the sigmoid formula instead.
from openfold.
Stage 2. My DeepSpeed config:
{ "optimizer": { "type": "Adam", "params": { "lr": 0.0005, "eps": 1e-05 } }, "amp": { "enabled": false, "opt_level": "O2" }, "bfloat16": { "enabled": true }, "zero_optimization": { "stage": 2, "cpu_offload": true, "contiguous_gradients": true }, "activation_checkpointing": { "partition_activations": true, "cpu_checkpointing": false, "profile": false }, "gradient_clipping": 0.1 }
Driver Version: 460.27.04 CUDA Version: 11.2
Is there no automatic casting with DeepSpeed? Features are still in torch.float32...
logsigmoid error without DeepSpeed disappears if I replace it with log(sigmoid(.)). In this version all losses go to NaN immediately and never recover. Same OOM. Features still in float32.
from openfold.
Just to clarify: you're also passing --precision bf16
to the training script, right?
from openfold.
Yes
from openfold.
In pure pytorch (no deepspeed, no lightning) I get the same logsigmoid error and all losses go to NaN immediately. There is no autocasting, features stay in torch.float32. Maybe because of the dict argument? I have to cast all floating point features manually.
What's interesting is, if I run deepspeed (with bfloat16 enabled) and manually cast the model to bfloat16 it complains: ValueError: fp32 is enabled but the following parameters have dtype that is not fp32: module.model.input_embedder.linear_tf_z_i.weight [..]
If I only use pytorch-lightning it actually shows in the beginning that it is in bfloat16 mixed precision mode (no message like this with deepspeed), but still no autocasting and NaN losses.
from openfold.
Hm. Interesting. I won't be personally up on A100s for a few more days, so there's not much I can do at the moment to investigate. I'll loop back when I do.
What happens if you manually cast both the model and the input features when you're using DeepSpeed? Do you think casting the model interferes with DeepSpeed's master FP32 copy of the weights? Maybe this would make a good DeepSpeed issue too.
from openfold.
"ValueError: fp32 is enabled but the following parameters have dtype that is not fp32: module.model.input_embedder.linear_tf_z_i.weight [..]"
This happens when I cast the model to bfloat32. The error is triggered in the setup phase in pytorch-lightning. When I only convert the data, the error occurs in the network, asking for a float instead.
I'm not sure what's happening exactly. I have the feeling that the bfloat16 flag in the configuration file is just ignored for some reason. Or it's some miscommunication between deepspeed and pytorch-lightning?
from openfold.
FYI: Earlier today I removed a line from the training script that silently changed the value of "CUDA_VISIBLE_DEVICES". This is a long shot, but could it be that this was moving computation onto GPUs that aren't compatible with bfloat16?
I'm not sure what's happening exactly. I have the feeling that the bfloat16 flag in the configuration file is just ignored for some reason. Or it's some miscommunication between deepspeed and pytorch-lightning?
Both of these sound plausible to me.
from openfold.
FYI: Earlier today I removed a line from the training script that silently changed the value of "CUDA_VISIBLE_DEVICES". This is a long shot, but could it be that this was moving computation onto GPUs that aren't compatible with bfloat16?
No, the nodes are in different queues.
from openfold.
I'm up on an A100 now, and I can replicate your NaN issue. I'll circle back if I can figure out how to fix it.
from openfold.
I'm up on an A100 now, and I can replicate your NaN issue. I'll circle back if I can figure out how to fix it.
This is when only using pytorch-lightning and changing the loss calculation? Does the bfloat16 flag in DeepSpeed have any effect for you? I piggybacked on the DeepSpeed issue above in the hope we get some response from the DeepSpeed people.
Thanks for following up!
from openfold.
It's just PyTorch Lightning, but I haven't changed the loss calculation. What do you mean by that?
from openfold.
It's just PyTorch Lightning, but I haven't changed the loss calculation. What do you mean by that?
Replacing logsigmoid with log(sigmoid(.)).
from openfold.
This should have been resolved by this week's commits. Closing this for now.
from openfold.
Related Issues (20)
- Inference error using precomputed alignments
- Pickling error in Docker? HOT 1
- Is the pdb_mmcif.zip from OpenProteinSet on AWS suitable for RODA training data? HOT 1
- Finetune from AF_Multimer parameters HOT 7
- Unable to create conda environment HOT 5
- I notice there is only instructions for a Linux Install, can I run this on my Windows laptop? HOT 1
- Frequently failed in training. HOT 2
- Rigid.from_3_points comment HOT 1
- Colab broken by version skew HOT 1
- Docker build broken HOT 4
- Question: Can the geometry module and rigid_utils be converted to each other?
- Alignment error during inference HOT 1
- ModuleNotFoundError: No module named 'attn_core_inplace_cuda' HOT 3
- Enable Dropout in inference HOT 1
- Questions about the meaning of folder naming conventions in OpenProteinSet HOT 1
- ModuleNotFoundError: No module named 'attn_core_inplace_cuda' HOT 1
- RuntimeError: Error building extension 'evoformer_attn' HOT 1
- Multimer predicting a homomer HOT 2
- Docker container AttributeError: Did you mean: 'linear_a_p'? when trying to run multimer inference
- Unable to install OpenFold in Google Colaboratory
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openfold.