urinx / alphafold_pytorch Goto Github PK

Hi,
I am trying to run the PyTorch AlphaFold implementation on my server with Python 3.6.8, PyTorch 1.5.0, and TensorFlow 2.0.0-alpha0. But it cannot be successfully executed.
Here is the error message.

Device: cuda
Model parameters: 21182345
Load tf model cost time: 0:00:33.470121
Data:  T1019s2-l64_s24 64
Traceback (most recent call last):
  File "alphafold.py", line 223, in <module>
    run_eval(TARGET_PATH, MODEL_PATH, REPLICA, OUT_DIR, DEVICE)
  File "alphafold.py", line 54, in run_eval
    for x_2d, crop_x, crop_y in crops:
  File "dataset.py", line 317, in make_crops
    [0, 0]]
TypeError: pad() missing 1 required positional argument: 'mode'
Input file: test_data/T1019s2.pkl
Output dir: T1019s2_out/Torsion/0
Torsion model: model/941521
Replica: 0

Traceback (most recent call last):
  File "dataset.py", line 383, in <module>
    test()
  File "dataset.py", line 375, in test
    dataset = ProteinDataset('T1019s2.pkl', DISTOGRAM_MODEL, replica)
TypeError: __init__() takes 3 positional arguments but 4 were given

• Please give me the solution this problem.
• I want to test the dataset.py.

error: save: error compressing data element
error: called from
    plmDCA at line 165 column 5

/Users/mac/lab/guo/alphafold_pytorch
Traceback (most recent call last):
  File "feature.py", line 263, in <module>
    feature_generation(SEQ_FILE, OUT_FILE)
  File "feature.py", line 168, in feature_generation
    pseudo_bias = np.float32(mat['pseudo_bias'])
KeyError: 'pseudo_bias'

Can anyone help?

I have a cyclic peptide sequence. I put it into Alphafold2 Colab, but I didn't get a cyclic peptide structure. What should I do to connect the C-terminal and N-terminal for the next dynamic simulation (GROMACS)?

Should I do some processing on the structure obtained by Alphafold2 Colab for Gromacs dynamics? Or input sequence to Alphafold2 Colab need for some pre-processing? Or can Alphafold2 not predict the sequence of cyclic peptides?

It's really important for my research, thanks for any help.

In line 262 0f dataset.py, why not use the "exp_range_n_x" of line 261 for bin_x?
if binary_code_bits:
exp_range_n_y = np.expand_dims(range_n_y, 1)
bin_y = np.concatenate([exp_range_n_y // (1 << i) % 2 for i in range(binary_code_bits)], 1)
exp_range_n_x = np.expand_dims(range_n_x, 1)
bin_x = np.concatenate([exp_range_n_y // (1 << i) % 2 for i in range(binary_code_bits)], 1)

Hi,
I ran feature.sh with T1019s2.seq as input. It generated the files as follows:

But there wasn't T1019s2.pkl or T1019s2.tfrec.
Can you tell me how to correctly generate those files?
Thank you!

Can anyone help me with where the optimization is being done after predicting the torsion angles and distogram?

After generating features and loading those from the npy file, I'm getting this error when trying to run the alphafold.py script. Any ideas on the cause?

RuntimeError: Given groups=1, weight of size [264, 1878, 3, 3], expected input[1, 1880, 64, 64] to have 1878 channels, but got 1880 channels instead

I found that an error for some input sequences is produced if an alignment during the feature generation pipeline contains the amino acid place holder X:

KeyError: 'X' Traceback (most recent call last): File "feature.py", line 263, in <module> feature_generation(SEQ_FILE, OUT_FILE) File "feature.py", line 194, in feature_generation non_gapped_profile[i, mapping[j]] += 1

I could solve the problem by replacing line 194 in feature.py with:
if (j!='X'):
non_gapped_profile[i, mapping[j]] += 1

I guess this plmDCA.m can produce .mat file from .aln file. But, after I executed

octave plmDCA.m .aln

I got only the following message and did not get any .mat file

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features

I prepared the environment the following instruction with a few modifications

git clone https://github.com/magnusekeberg/plmDCA.git
mv plmDCA/plmDCA_asymmetric_v2 plmDCA/plmDCA
mkdir plmDCA/plmDCA_asymmetric_v2
cp plmDCA.m plmDCA/plmDCA_asymmetric_v2/
cp plmDCA/plmDCA/functions/ plmDCA/plmDCA_asymmetric_v2/
cp plmDCA/plmDCA/3rd_party_code/ plmDCA/plmDCA_asymmetric_v2/
cd plmDCA/plmDCA_asymmetric_v2/functions/; for i in *.c; do octave --eval "mex $i";done
cd ../3rd_party_code/minFunc/; for i in *.c; do octave --eval "mex $i"; done`
By using those code, I made two .mex files in function folder, and three .mex files in minFunc folder.

I also noticed that the message written above did not change even after I renamed the function and 3rd_party_code folders. So, I think my plmDCA.m does not recognize those mex or c files.

How can I fix this problem? Thanks in advance!

Hi, when I bash feature.sh,there was an error:
Traceback (most recent call last):
File "feature.py", line 275, in
feature_generation(SEQ_FILE, OUT_FILE)
File "feature.py", line 205, in feature_generation
non_gapped_profile[i, mapping[j]] += 1
KeyError: 'X'
How can I solve it?

First of all, thank you for the code! It helped me a lot about understanding AlphaFold's model and its code.

My question is about how you converted the tensorflow model(especially, one that also used sonnet library, which is an open-source library from Deepmind) to pytorch model. (Or did you train the model again with pytorch framework?)
I'm trying to make tensorflow2 version of the code, but it is difficult to load the model in tensorflow2.

Thank you in advance!

Hi!

I tried to feed a single sequence with the pipeline provided (feature.sh), but it is still running after 2 days straight. Also tried to feed into the pipeline an already created MSA, but it has been chopped into pieces and also ran for days and eventually I forced it to stop.

My question is that is there a way to directly input an already created MSA and convert into .tfrec or I must go through the pipeline provided in feature.sh?

And also furthermore is it normal that this hhblits and muscle ,psiblast and all these stuff is running like days straight?

Thank you in advance!

Hi

Thanks for sharing this repository.

For Blast+, do I need to download any of the databases for this?

Thanks

• How can I get the pkl and tfrec files from the pdb file?
• Does not support pdb file?
• Please give me the solutions.

from where install the utils module, the one from the:
python3 -m pip install utils

gives:

AttributeError: module 'utils' has no attribute 'generate_domains'

Hi, in order to run a prediction for other target using the alphafold.sh script, I need as input a pkl file, how can I get it?

I have a problem with the uniclust_30_2018_08_hhsuite.tar.gz file required to preprocessing a fasta file and produce the feature. I downloaded it two times, and I got different sizes of the tar files and the directories generated by the tars on each case. Maybe one of these tar files was corrupted. The issue is that the shorter of these tars produce the most extensive directory and vice versa. I can't figure what is the wrong file out. One of the tar has a size of 25 GB and produces a folder of 87 GB, and the other tar has a length of 24 GB and produces a directory of 165 GB. What is the right size of the tar file? Inside the uniclust30_2018_08 folder, What is supposed the size of uniclust30_2018_08_a3m_db and uniclust30_2018_08_hhm_db? Because I download again the tar file monitoring the process, and I got the tar that produces the shorter folder, and these two files have a size practically of a cero, then it is a little weird. I've been using the webpage provide in the setup HHBlits from the HH-suite3 section of the readme (wget http://wwwuser.gwdg.de/~compbiol/uniclust/2018_08/uniclust30_2018_08_hhsuite.tar.gz). Is there another trustable webpage to download the same dataset? I couldn't find one.
Thank you for your help.

I guess,
python alphafold.py -i test_data/T1019s2.pkl -o T1019s2_out -t D -t 0

should be
python alphafold.py -i test_data/T1019s2.pkl -o T1019s2_out -t D -r 0

• Do you not support other protein besides casp13 dataset?
• I want to use covid 19 protein(SARS-CoV-2) on alphafold_casp13 project. Does not support it?
• Please give me the solutions.

What do the resulting output (.distance, .sec) files mean? How should I interpret it?
What tool-kit could I use to view the .distance, .sec files?

I loaded the checkpoints and they have shapes like [1, 1837, 64, 64]
What kind of input does this represent?

I thought the input was supposed to be a list of amino acids.

Thank you for putting this repo together, it is really helpful in understanding and preparing the features for AlphaFold. I was wondering if it is possible to include non-covalently-linked ligands alongside the amino acids in the protein. For instance, could I include a small-molecule substrate for an enzyme? If this is possible, can you explain how I could do this?

Hi!

I am trying using your great implementation of alphafold in pytorch. I am new to pytorch as I mostly used tensorflow and I ran into this problem:

python alphafold.py -i test_data/T0953s2/T0953s2.tfrec -o T0953s2_out -m tf_model_path/
Input file: test_data/T0953s2/T0953s2.tfrec
Output dir: T0953s2_out/Distogram/0
Distogram model: tf_model_path/873731
Replica: 0
Device: cuda
Model parameters: 21182345
Data: T0953s2-l128_s32 128
Traceback (most recent call last):
File "alphafold.py", line 221, in
run_eval(TARGET_PATH, MODEL_PATH, REPLICA, OUT_DIR, DEVICE)
File "alphafold.py", line 68, in run_eval
out = model(x_2d, crop_x, crop_y)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/mnt/meganeura/domeemod/alphafold-pytroch/alphafold_pytorch/network.py", line 295, in forward
biases = self.build_crops_biases(config.position_specific_bias_size, self.position_specific_bias, crop_x, crop_y)
File "/mnt/meganeura/domeemod/alphafold-pytroch/alphafold_pytorch/network.py", line 269, in build_crops_biases
row_offsets = start_diag + increment
RuntimeError: expected device cuda:0 but got device cpu

I found during some search on the net that this bug is related to how I am specifying the parts of the code to the cpu and the gpu. I have installed everything properly and I am using your provided scripts as it was written in your readme file.

We are doing research related to phylogenetics here in Hungary and your implementation can provide us some really good insights.

Thank you for uploading this project to github, and hope you can help us move forward.

Best wishes!

Kind regard, Demeter Márton

In the 197th line of alphafold.py, the variable TARGET should be TARGET_PATH

Hi,
thanks for awesome repo. alphafold.sh script generates a bunch for files with multiple extensions but does not generate .rr file how can I get that?
Greetings!

Thank you for the quick update of the code, but still (at least for my configuration) it doesn't work with gpu, running with updated scripts with ./alphafold.sh:

Saving output to T1019s2_out/

Launching model: D 0
Launching model: B 0
Launching model: T 0
Launching model: D 1
Launching model: B 1
Launching model: D 2
Launching model: B 2
Launching model: D 3
Launching model: B 3
All models running, waiting for them to complete

Traceback (most recent call last):
Input file: test_data/T1019s2.pkl
Output dir: T1019s2_out/Background/2
Background model: tf_model_path/916425
Replica: 2
Device: cuda
Model parameters: 1091392
File "alphafold.py", line 223, in
run_eval(TARGET_PATH, MODEL_PATH, REPLICA, OUT_DIR, DEVICE)
File "alphafold.py", line 18, in run_eval
model.load_state_dict(torch.load(model_file, map_location=device))
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
return _load(f, map_location, pickle_module)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 469, in _load
result = unpickler.load()
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 437, in persistent_load
data_type(size), location)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 323, in restore_location
result = map_location(storage, location)
TypeError: 'torch.Device' object is not callable
Traceback (most recent call last):
Input file: test_data/T1019s2.pkl
Output dir: T1019s2_out/Background/1
Background model: tf_model_path/916425
Replica: 1
Device: cuda
Model parameters: 1815936
File "alphafold.py", line 223, in
run_eval(TARGET_PATH, MODEL_PATH, REPLICA, OUT_DIR, DEVICE)
File "alphafold.py", line 18, in run_eval
model.load_state_dict(torch.load(model_file, map_location=device))
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
return _load(f, map_location, pickle_module)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 469, in _load
result = unpickler.load()
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 437, in persistent_load
data_type(size), location)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 323, in restore_location
result = map_location(storage, location)
TypeError: 'torch.Device' object is not callable
Traceback (most recent call last):
Input file: test_data/T1019s2.pkl
Output dir: T1019s2_out/Background/3
Background model: tf_model_path/916425
Replica: 3
Device: cuda
Model parameters: 1924992
File "alphafold.py", line 223, in
run_eval(TARGET_PATH, MODEL_PATH, REPLICA, OUT_DIR, DEVICE)
File "alphafold.py", line 18, in run_eval
model.load_state_dict(torch.load(model_file, map_location=device))
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
return _load(f, map_location, pickle_module)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 469, in _load
result = unpickler.load()
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 437, in persistent_load
data_type(size), location)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 323, in restore_location
result = map_location(storage, location)
TypeError: 'torch.Device' object is not callable
Traceback (most recent call last):
Input file: test_data/T1019s2.pkl
Output dir: T1019s2_out/Background/0
Background model: tf_model_path/916425
Replica: 0
Device: cuda
Model parameters: 1029568
File "alphafold.py", line 223, in
run_eval(TARGET_PATH, MODEL_PATH, REPLICA, OUT_DIR, DEVICE)
File "alphafold.py", line 18, in run_eval
model.load_state_dict(torch.load(model_file, map_location=device))
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
return _load(f, map_location, pickle_module)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 469, in _load
result = unpickler.load()
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 437, in persistent_load
data_type(size), location)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 323, in restore_location
result = map_location(storage, location)
TypeError: 'torch.Device' object is not callable
Traceback (most recent call last):
File "alphafold.py", line 223, in
Input file: test_data/T1019s2.pkl
Output dir: T1019s2_out/Distogram/1
Distogram model: tf_model_path/873731
Replica: 1
Device: cuda
Model parameters: 21182345
run_eval(TARGET_PATH, MODEL_PATH, REPLICA, OUT_DIR, DEVICE)
File "alphafold.py", line 18, in run_eval
model.load_state_dict(torch.load(model_file, map_location=device))
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
return _load(f, map_location, pickle_module)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 469, in _load
result = unpickler.load()
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 437, in persistent_load
data_type(size), location)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 323, in restore_location
result = map_location(storage, location)
TypeError: 'torch.Device' object is not callable
Traceback (most recent call last):
Input file: test_data/T1019s2.pkl
Output dir: T1019s2_out/Distogram/0
Distogram model: tf_model_path/873731
Replica: 0
Device: cuda
Model parameters: 21182345
File "alphafold.py", line 223, in
run_eval(TARGET_PATH, MODEL_PATH, REPLICA, OUT_DIR, DEVICE)
File "alphafold.py", line 18, in run_eval
model.load_state_dict(torch.load(model_file, map_location=device))
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
return _load(f, map_location, pickle_module)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 469, in _load
result = unpickler.load()
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 437, in persistent_load
data_type(size), location)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 323, in restore_location
result = map_location(storage, location)
TypeError: 'torch.Device' object is not callable
Traceback (most recent call last):
File "alphafold.py", line 223, in
Input file: test_data/T1019s2.pkl
Output dir: T1019s2_out/Distogram/2
Distogram model: tf_model_path/873731
Replica: 2
Device: cuda
Model parameters: 22381993
run_eval(TARGET_PATH, MODEL_PATH, REPLICA, OUT_DIR, DEVICE)
File "alphafold.py", line 18, in run_eval
model.load_state_dict(torch.load(model_file, map_location=device))
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
return _load(f, map_location, pickle_module)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 469, in _load
result = unpickler.load()
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 437, in persistent_load
data_type(size), location)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 323, in restore_location
result = map_location(storage, location)
TypeError: 'torch.Device' object is not callable
Traceback (most recent call last):
Input file: test_data/T1019s2.pkl
Output dir: T1019s2_out/Distogram/3
Distogram model: tf_model_path/873731
Replica: 3
Device: cuda
Model parameters: 22381993
File "alphafold.py", line 223, in
run_eval(TARGET_PATH, MODEL_PATH, REPLICA, OUT_DIR, DEVICE)
File "alphafold.py", line 18, in run_eval
model.load_state_dict(torch.load(model_file, map_location=device))
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
return _load(f, map_location, pickle_module)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 469, in _load
result = unpickler.load()
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 437, in persistent_load
data_type(size), location)
File "/scratch/domeemod/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/torch/serialization.py", line 323, in restore_location
result = map_location(storage, location)
TypeError: 'torch.Device' object is not callable

My config: pytorch conda env with pytorch 1.4, tensorflow-gpu 2.1, Nvidia Geforce 1070GTX:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.26 Driver Version: 430.26 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+

how to get 3D structure of protein?
the output is distogram.

google-deepmind/deepmind-research#43

The 'exit' on the line 164 of feature.py produce a lot of problems. When there are more then 1 crop produce that the feature.sh doesn't save the numpy. The first time that use feature generation (second call of feature.py on feature.sh), it create the first .aln and then close the code. After that, It use the plmDCA to produce a .mat from the only .aln, but on the update read again the crops, and created the .aln of the second crops, and then exit, never saving the final npy. I believe that the best way to solved it is replacing the 'exit' for a 'continue', but this produce a aln for each crops and also a mat. With this change the second call of feature.py on feature.sh only produce the .aln of each crop, and the third call produce the entire numpy with the .mat of each crop on it. Is this correct? or Is supposed that the plmDCA be only apply over the .aln that correspond to the fasta of the original sequence/the full sequence?. If the last is the case it's necesary change the code to only produce one .mat, maybe retiring the 'exit' on feature.py but also replacing the second call of feature.py on feature.sh by another code that only produce the .aln of the main sequence. Please any comment is useful because I'm not totally sure of the right way to fix it but i understand the problem i maybe we can figure it out together.

aln, aln_id = read_aln(fas_file)
aln = aln[:, aln[0] != '-']
write_aln(aln, aln_id, aln_file)
continue

Hej, I tried to run plmDCA.m test_data/T1019s2.aln, but I got the following error:
octave: no graphical display found
octave: disabling GUI features

N = 88 B_with_id_seq = 393 B = 393 B_eff = 393.00 q = 22

Minimizing g_r for node r=1
warning: non-integer range used as index
warning: called from
plmDCA>g_r at line 212 column 8
plmDCA>g_r at line 215 column 23
minFunc at line 314 column 10
plmDCA>min_g_r at line 206 column 7
plmDCA at line 91 column 15
error: reshape: SIZE must be non-negative
error: execution exception in plmDCA.m

I don't use matlab, so can't really fix the problem. Can you please help me out? Thank you!
Maj