Hello,

I ran DeepMod on the region 5 000 000 - 10 000 000 of the Human chromosome 20. I did It on CPUs and it took me 1 hour. Then I tried the same run but on GPU and so with the same installation except for tensorflow-gpu. I was expecting to be faster on GPUs. Actually, it takes 2h30 on the GPUs. So my questions are; is DeepMod able to take advantage of the GPUs? Is it normal to have such result?
The command I used to run DeepMod:
`#!/bin/bash
#SBATCH --job-name=deepmod_MethCall
#SBATCH --gpus=2
#SBATCH --mem=45000
#SBATCH --cpus-per-gpu=20
#SBATCH --time=05:00:00
#SBATCH --partition=gpu
#SBATCH --output=/CECI/home/ulg/genhum/thissen/slurm_files/slurm-%j-deepmod.out

module load Python/3.7.4-GCCcore-8.3.0

srun singularity exec --bind /CECI/home/ulg/genhum/thissen --nv /CECI/home/ulg/genhum/thissen/methylation/deepmod/deepmod_gpu_nvidia-based-2020-07-23-4c83937316b7.simg
python /usr/local/bin/DeepMod.py detect --wrkBase fast5_files_albacore/workspace/pass/0 --Ref /CECI/home/ulg/genhum/thissen/data/brut/methylation/reference.fasta --FileID deepmod_methylation_call --modfile /tools/DeepMod/train_deepmod/rnn_conmodC_P100wd21_f7ne1u0_4/mod_train_conmodC_P100wd21_f3ne1u0 --threads 15 --outFolder .`

Thanks!

The nanopore technology is evolving fast. I wondered if the trained model is compatible with data produced by using different versions of sequencing chemistry.

Hi Liu:
I am working on the problem about 5mc/6ma base modification in human brain. I have tried the pipeline that you recommend on Na12878 dataset(https://github.com/WGLab/DeepMod/blob/master/docs/Reproducibility.md), but for the last step, I just see the guidance for data downloading, is there any scripts for doing the evaluation of Na12878 dataset？Can you write a brief guidance for it?
Many thanks.

The command python bin/DeepMod.py detect, mentioned under "Install DeepMod" throws a TypeError:

(mdeepmod) /.../DeepMod$ python bin/DeepMod.py detect
No mod file is provided. The default one is used
Traceback (most recent call last):
  File "bin/DeepMod.py", line 383, in <module>
    args.func(args);
  File "bin/DeepMod.py", line 140, in mDetect
    moptions['modfile'] = ('{}/lib/python{}.{}/site-packages/DeepMod/train_deepmod/rnn_P90wd%d_f53/mod_train_P90wd%d_f53' % (sys.prefix,sys.version_info.major,sys.version_info.minor, moptions['windowsize'], moptions['windowsize']))
TypeError: %d format: a number is required, not str

When I use this command：
python ~/biosoft/DeepMod/bin/DeepMod.py detect --wrkBase fast5_files/ --Ref reference/reference.fasta --outFolder out_folder --Base C --modfile ~/biosoft/DeepMod/train_mod/rnn_f7_wd21_chr1to10_4/mod_train_f7_wd21_chr1to10 --threads 4
Several errors were made,like:
File "/biosoft/DeepMod/bin/DeepMod.py", line 372, in args.func(args);
File "/biosoft/DeepMod/bin/DeepMod.py", line 170, in mDetect from scripts import myDetect
File "/biosoft/DeepMod/bin/scripts/myDetect.py", line 24, in import tensorflow as tf
Module Not Found Error: No module named 'tensorflow'
How can I fix these mistakes, thanks a lot!!

The installation guide page does not mention the version of tensorflow. I read the source code and find that Deepmod uses the tf.contrib api. It seems that tf.contrib has been deprecated since tensorflow 1.13, so i guess the proper version of tensorflow might be 1.13. Is that right?

Dear Prof. Liu,
Thank you for the great software for DNA methylation.

I found that your program will filter out sites that have no methylation-callings in the final results. See your code at (amod_dict[pk][1] is the methylated reads predicted by DeepMod): https://github.com/WGLab/DeepMod/blob/master/DeepMod_tools/sum_chr_mod.py#L57

In this case, all DeepMod outputs for final methylation results will lost too much CpG sites that have unmethylated predictions in all reads by DeepMod. In NA12878, there will lost CpG sites that DeepMod reports 0% methylation levels for performance evaluation and comparison with nanopolish. How do you ensure this kind of performance comparison in your paper is an objective/genome-wide evaluation? Thank you!

Below is your provided report of NA12878 on chr18, all your output results contains no lines with methylation coverage (last 2nd column)== 0:

head cpredecoli_org_clusterCpG.chr18.C.bed

chr18 10689 10690 C 8 +  10689 10690 0,0,0 8 25 2 68
chr18 10703 10704 C 8 +  10703 10704 0,0,0 8 100 8 91
chr18 10718 10719 C 8 +  10718 10719 0,0,0 8 75 6 89
chr18 10730 10731 C 8 +  10730 10731 0,0,0 8 25 2 95
chr18 10753 10754 C 8 +  10753 10754 0,0,0 8 100 8 94
chr18 10767 10768 C 2 +  10767 10768 0,0,0 2 100 2 94
chr18 10787 10788 C 8 +  10787 10788 0,0,0 8 25 2 87
chr18 10791 10792 C 8 +  10791 10792 0,0,0 8 75 6 92
chr18 10795 10796 C 8 +  10795 10796 0,0,0 8 75 6 94
chr18 10815 10816 C 8 +  10815 10816 0,0,0 8 100 8 98
=================================================^===

Hi,

I converted multiFast5 into single fast5 with ot_fast5_api. But deepmod still can not recognize my data and said 'no event data'. So I'm thinking re-basecalling with albacore. I searched online, but can't find where to download it. Do you know where I can download it? Thank you!

Hello, I have the same problem when I use it. This is my fast5 file, the result of viewing with h5ls - r
/ Group
/Analyses Group
/Analyses/Basecall_1D_000 Group
/Analyses/Basecall_1D_000/BaseCalled_template Group
/Analyses/Basecall_1D_000/BaseCalled_template/Events Dataset {1561}
/Analyses/Basecall_1D_000/BaseCalled_template/Fastq Dataset {SCALAR}
/Analyses/Basecall_1D_000/Configuration Group
/Analyses/Basecall_1D_000/Configuration/basecall_1d Group
/Analyses/Basecall_1D_000/Summary Group
/Analyses/Basecall_1D_000/Summary/basecall_1d_template Group
/Analyses/Calibration_Strand_Detection_000 Group
/Analyses/Calibration_Strand_Detection_000/Configuration Group
/Analyses/Calibration_Strand_Detection_000/Configuration/calib_detector Group
/Analyses/Calibration_Strand_Detection_000/Summary Group
/Analyses/Calibration_Strand_Detection_000/Summary/calibration_strand_template Group
/Analyses/Segmentation_000 Group
/Analyses/Segmentation_000/Configuration Group
/Analyses/Segmentation_000/Configuration/stall_removal Group
/Analyses/Segmentation_000/Summary Group
/Analyses/Segmentation_000/Summary/segmentation Group
/PreviousReadInfo Group
/Raw Group
/Raw/Reads Group
/Raw/Reads/Read_6 Group
/Raw/Reads/Read_6/Signal Dataset {23415/Inf}
/UniqueGlobalKey Group
/UniqueGlobalKey/channel_id Group
/UniqueGlobalKey/context_tags Group
/UniqueGlobalKey/tracking_id Group

What should I do? I look forward to your reply

Hi,
I checked the result explanation for the output file, but I still have questions on the column meanings. Can you tell more about the 7th-9th columns(148655 148656 0,0,0) especially the 9th column?

Thank you so much for your help!

@liuqianhn hello, i downloaded test data from http://s3.climb.ac.uk/nanopolish_tutorial/methylation_example.tar.gz, a subset of the NA12878 WGS Consortium data used in the tutorial of nanopolish calling methylation.
The command line below like this:

python DeepMod.py detect
--wrkBase ~/deepmod_test/fast5_files
--Ref ~/deepmod_test/reference/reference.fasta
--FileID test
--modfile ~/DeepMod/train_mod/rnn_conmodC_P100wd21_f7ne1u0_4/mod_train_conmodC_P100wd21_f3ne1u0
--threads 5 --outFolder myoutput/

note: directory ~/deepmod_test/fast5_files canotains signal-level FAST5 files unpacked from the downlaod data package.

the error information:
Nanopore sequencing data analysis is resourece-intensive and time consuming.
Some potential strong recommendations are below:
If your reference genome is large as human genome and your Nanopore data is huge,
It would be faster to run this program parallelly to speed up.
You might run different input folders of your fast5 files and
give different output names (--FileID) or folders (--outFolder)
A good way for this is to run different chromosome individually.

         Current directory: ~/software_test/deepmod_test
                  outLevel: 2
                   wrkBase: ~/deepmod_test/fast5_files
                    FileID: test
                 outFolder: myoutput/
                 recursive: 1
          files_per_thread: 1000
                   threads: 5
                windowsize: 21
                  alignStr: minimap2
               basecall_1d: Basecall_1D_000
          basecall_2strand: BaseCalled_template
                    ConUnk: True
               outputlayer: 
                      Base: C
               mod_cluster: 0
                   predDet: 1
                       Ref: ~/deepmod_test/reference/reference.fasta
                      fnum: 7
                    hidden: 100
                   modfile: ~/DeepMod/train_mod/rnn_conmodC_P100wd21_f7ne1u0_4/mod_train_conmodC_P100wd21_f3ne1u0
                    region: [[None, None, None]]

Total files=19275
Error!!! No Fastq data in ~/deepmod_test/fast5_files/nanopore2_20161128_FNFAB49712_MN17633_sequencing_run_20161128_Human_Qiagen_1D_R9_4_64849_ch388_read4650_strand.fast5
... ...

Beside this, i also used my own data to run DeepMod and same errors " No Fastq data in *.fast5" were produced.

Could you please help me with providing some solution ? Thanks !

Hello ,
In my output file resulting from DeepMod, the methylation coverages of many sites were 0. So the mehylation percentage of these corresponding sites were also equal to 0. Should I remove these sites in which methylation coverages and mehylation percentage were both 0 ?
In addition, it is necessary to filter real coverages and (or) methylation coverages to obtain more reliable methylated sites? If so, what are suitable cutoff values of real coverages and (or) methylation coverages that you would recommend ?
Thanks!

Hi, Liu
I'm very interested in the DeepMod, and want to use it to call the 5-mc methylation in the datasets provided by Simpon et.al, and the datasets is downloaded from the https://www.ebi.ac.uk/ena/browser/view/PRJEB13021.

Since I don't have my own GPU, so I try to test these data in a GPU server. I download the dataset named 'ecoli_er2925.pcr.timp.021216.tar.gz' (75.3 GB) and 'ecoli_er2925.pcr_MSssI.timp.021216.tar.gz' (59.5 GB). Unfortunately, these dataset can't be uploaded to GPU server because of the large size, so I extract a part of data corresponding to ch67 and zip them to upload to the GPU server.

When I try to use the model 'mod_train_sinmodC_P100wd21_f3ne1u0' to call the 5-mc methylation on the data about ch67 in 'ecoli_er2925.pcr_MSssl.timp.021216.tar.gz', I get the following error information:

Error!!! No Raw_reads/Signal data /Raw/Reads in data/meth10_lib3/ecoli_er2925.pcr_MSssI.timp.021216.fast5_small/fail/kelvin_021116_methecoli_4101_1_ch67_file7_strand.fast5

Then I use the following command to examine this file:

h5ls -r kelvin_021116_methecoli_4101_1_ch67_file101_strand.fast5

Indeed, this fast5 file doesn't have Raw_reads/Signals information. Then I check other fast5 files about ch67 in 'ecoli_er2925.pcr_MSssl.timp.021216.tar.gz', and other fast5 file also doesn't have raw_read/Signal information.

So, I'm wondering how do you train or test the DeepMod if you use the dataset provided by Simpon et.al. If the datasets provided by Simpon et.al is partially usable, could you tell me which part of the data you use to train and test the model. Additional, Is there any other public datasets I can use to run DeepMod?

I'm just new to using nanopores to detect methylation, so maybe some strange questions were asked, but I still hope and appreciate you can help me to deal with these problems.

Yours
Chen.

Hi teacher ! I'm trying to call DNA modifications by using deepmod.However,i get the problem 'Error information for different fast5 files: Not in alignment sam 1095'.Can you give me some advice about this error?
What's more ,what's the fast5 data you input?.The sequence of motif ?Why i input raw data , i get a few methylation coverage?

NC_008261.1 649 650 A 74 - 649 650 0,0,0 74 0 0
NC_008261.1 651 652 A 65 - 651 652 0,0,0 65 0 0
NC_008261.1 652 653 A 79 - 652 653 0,0,0 79 0 0
NC_008261.1 656 657 A 77 - 656 657 0,0,0 77 0 0
NC_008261.1 658 659 A 73 - 658 659 0,0,0 73 0 0
NC_008261.1 661 662 A 65 - 661 662 0,0,0 65 0 0
NC_008261.1 662 663 A 81 - 662 663 0,0,0 81 0 0
NC_008261.1 663 664 A 75 - 663 664 0,0,0 75 0 0
NC_008261.1 682 683 A 78 - 682 683 0,0,0 78 0 0
NC_008261.1 715 716 A 79 - 715 716 0,0,0 79 1 1
NC_008261.1 721 722 A 73 - 721 722 0,0,0 73 0 0
NC_008261.1 724 725 A 72 - 724 725 0,0,0 72 0 0
NC_008261.1 726 727 A 79 - 726 727 0,0,0 79 6 5
NC_008261.1 730 731 A 55 - 730 731 0,0,0 55 0 0
NC_008261.1 734 735 A 72 - 734 735 0,0,0 72 1 1
NC_008261.1 736 737 A 69 - 736 737 0,0,0 69 0 0
NC_008261.1 738 739 A 82 - 738 739 0,0,0 82 0 0
NC_008261.1 741 742 A 74 - 741 742 0,0,0 74 0 0
NC_008261.1 742 743 A 85 - 742 743 0,0,0 85 0 0
NC_008261.1 744 745 A 53 - 744 745 0,0,0 53 0 0
NC_008261.1 748 749 A 79 - 748 749 0,0,0 79 0 0
NC_008261.1 751 752 A 83 - 751 752 0,0,0 83 0 0
NC_008261.1 756 757 A 80 - 756 757 0,0,0 80 0 0
NC_008261.1 758 759 A 82 - 758 759 0,0,0 82 0 0
NC_008261.1 763 764 A 70 - 763 764 0,0,0 70 0 0
NC_008261.1 767 768 A 73 - 767 768 0,0,0 73 0 0
NC_008261.1 768 769 A 82 - 768 769 0,0,0 82 0 0
NC_008261.1 772 773 A 75 - 772 773 0,0,0 75 0 0
NC_008261.1 775 776 A 67 - 775 776 0,0,0 67 0 0

Hi @liuqianhn
I'm running DeepMod on several fast5 files, and I got the following error
Error!!! The index of the first base is less than -2(-12323486=0.000000*4000-12323486) in DEAMERNANOPORE_20161117_FNFAB43577_MN16450_sequencing_run_MA_821_R9_4_NA12878_11_17_16_88738_ch137_read888_strand.fast5
The data was downloaded from NA12878， and was basecalled by Albacore-2.3.1 with the following command
read_fast5_basecaller.py --input fast5 --worker 20 --save_path out --flowcell FLO-MIN106 --kit SQK-RAD002 -o fast5
The DeepMod command used is shown below
python DeepMod.py detect --wrkBase base_called_fast5 --Ref hg19.fa --FileID test --modfile train_mod/rnn_conmodC_P100wd21_f7ne1u0_4/mod_train_conmodC_P100wd21_f3ne1u0 --threads 15 --outFolder output
Could you please tell me where the problem is? Thanks so much!
Best.

Hi @liuqianhn
I realized that DeepMod supports move table. Does it consistent with your trained model?, as you trained them on events detected by Albacore
I assume that the segmentation and event detection algorithm of guppy might have some differences with albacore.

Thanks,
Vahid.

Hi @liuqianhn ,

Can you give more detail about the output format of detect function? I don't totally understand the meaning of all 12 columns of the BED file, especially the last 3 columns.

Thank you in advance.

Best,
Peng

Hello!@liuqianhn
We borrowed Chlamydomonas reinhardtii to carry out experiments . However, I can't get the label of C. reinhardtii .We don't know whether all A is 6mA in motif GATC.What's more ,how did you divide the negative data set and the positive data set.Can you give me some suggestions.Thank you!

As albacore will output two subfolder in the workspace, one named"fail", another one named "pass", should I put both fail and pass fast5 file into DeepMod to do the methylation analysis? If I should , should I set the "--recursive" of "DeepMod.py detect " to 2 ? as there are two-layer folder.

Hi,
Does DeepMod just accept basecalled fast5 files from albacore?
I based called my fast5 files using Guppy and used --fast5_out option that gives me basecalled fast5 file.
But when I used DeepMod with the following commands it gives me an error?
"python ../anaconda3/DeepMod/bin/DeepMod.py detect --wrkBase ../fast5_files/ --Ref ../reference.fasta --outFolder ../ --Base C --modfile ../anaconda3/DeepMod/train_mod/rnn_f7_wd21_chr1to10_4/mod_train_f7_wd21_chr1to10 --FileID User_Uniq_name --threads 16"
the error is:
Cannot open fast5 or other errors: ../DeepSignal/fast5_files/PLSP61583_20161015_FNFAB42316_MN17048_sequencing_run_Hum94_ss_jt_86199_ch172_read45_strand1.fast5

So it cannot open fast5 files.
What is the problem? what should I do?
Thanks,
Vahid.

#Hi, teacher

I'm trying to call DNA modifications by using deepmod.
First I am successful to create virtual environment and install deepmod step by step through following the command lines you provided in your Installation Guide.
Then I begin to call DNA modifications.

1. I first run the following command to convert the multi-read fast5 files to single-read by using ont_fast5_api.

                     multi_to_single_fast5__ -i  ~/ds/ONT-fast5/D18395  -s  ./  -t 30  --recursive

• The first 50 lines of multi-read fast5 files are as belows.

/ Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Analyses Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Analyses/Basecall_1D_000 Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Analyses/Basecall_1D_000/BaseCalled_template Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Analyses/Basecall_1D_000/BaseCalled_template/Fastq Dataset {SCALAR}
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Analyses/Basecall_1D_000/Summary Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Analyses/Basecall_1D_000/Summary/basecall_1d_template Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Analyses/Segmentation_000 Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Analyses/Segmentation_000/Summary Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Analyses/Segmentation_000/Summary/segmentation Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Raw Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/Raw/Signal Dataset {21089/Inf}
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/channel_id Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/context_tags Group
/read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/tracking_id Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0 Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Analyses Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Analyses/Basecall_1D_000 Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Analyses/Basecall_1D_000/BaseCalled_template Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Analyses/Basecall_1D_000/BaseCalled_template/Fastq Dataset {SCALAR}
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Analyses/Basecall_1D_000/Summary Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Analyses/Basecall_1D_000/Summary/basecall_1d_template Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Analyses/Segmentation_000 Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Analyses/Segmentation_000/Summary Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Analyses/Segmentation_000/Summary/segmentation Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Raw Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/Raw/Signal Dataset {8036/Inf}
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/channel_id Group
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/context_tags Group, same as /read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/context_tags
/read_002a0d4c-d81c-400e-aa3d-56f980a361d0/tracking_id Group, same as /read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/tracking_id
/read_003f4b93-0399-434a-8643-da1dae43ab48 Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/Analyses Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/Analyses/Basecall_1D_000 Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/Analyses/Basecall_1D_000/BaseCalled_template Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/Analyses/Basecall_1D_000/BaseCalled_template/Fastq Dataset {SCALAR}
/read_003f4b93-0399-434a-8643-da1dae43ab48/Analyses/Basecall_1D_000/Summary Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/Analyses/Basecall_1D_000/Summary/basecall_1d_template Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/Analyses/Segmentation_000 Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/Analyses/Segmentation_000/Summary Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/Analyses/Segmentation_000/Summary/segmentation Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/Raw Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/Raw/Signal Dataset {128721/Inf}
/read_003f4b93-0399-434a-8643-da1dae43ab48/channel_id Group
/read_003f4b93-0399-434a-8643-da1dae43ab48/context_tags Group, same as /read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/context_tags
/read_003f4b93-0399-434a-8643-da1dae43ab48/tracking_id Group, same as /read_0004d61f-0c92-4b10-ba33-5a5dac260f3a/tracking_id
/read_0046fafb-384b-4fd4-81a6-c778c9d5e6cd Group
/read_0046fafb-384b-4fd4-81a6-c778c9d5e6cd/Analyses Group
/read_0046fafb-384b-4fd4-81a6-c778c9d5e6cd/Analyses/Basecall_1D_000 Group
/read_0046fafb-384b-4fd4-81a6-c778c9d5e6cd/Analyses/Basecall_1D_000/BaseCalled_template Group

• The first 50 lines of one of the single-read fast5 files are as belows.

/ Group
/Analyses Group
/Analyses/Basecall_1D_000 Group
/Analyses/Basecall_1D_000/BaseCalled_template Group
/Analyses/Basecall_1D_000/BaseCalled_template/Fastq Dataset {SCALAR}
/Analyses/Basecall_1D_000/Summary Group
/Analyses/Basecall_1D_000/Summary/basecall_1d_template Group
/Analyses/RawGenomeCorrected_001 Group
/Analyses/RawGenomeCorrected_001/BaseCalled_template Group
/Analyses/RawGenomeCorrected_001/BaseCalled_template/Alignment Group
/Analyses/RawGenomeCorrected_001/BaseCalled_template/Events Dataset {29849}
/Analyses/Segmentation_000 Group
/Analyses/Segmentation_000/Summary Group
/Analyses/Segmentation_000/Summary/segmentation Group
/Raw Group
/Raw/Reads Group
/Raw/Reads/Read_36017 Group
/Raw/Reads/Read_36017/Signal Dataset {365072/Inf}
/UniqueGlobalKey Group
/UniqueGlobalKey/channel_id Group
/UniqueGlobalKey/context_tags Group
/UniqueGlobalKey/tracking_id Group

2. I index my reference genome by using bwa index.

                   ln -s path/to/genome.nextpolish.fasta  ./
                   baw index genome.nextpolish.fasta

3.I run the DeepMod.py detect

            source activate mdeepmod
            path=/home/cxs3_z4/software/DeepMod
            time python ${path}/bin/DeepMod.py detect --wrkBase /home/cxs3_z4/ds/deepsignal/ --Ref /home/cxs3_z4/ds/deepmod/D18395/genome.nextpolish.fasta --FileID D18395 --modfile ${path}/train_deepmod/rnn_sinmodC_P100wd21_f7ne1u0_4/mod_train_sinmodC_P100wd21_f3ne1u0 --threads 15 --outFolder ./D18395_deepmod

• The resluts are as follows.
  Nanopore sequencing data analysis is resourece-intensive and time consuming.
  Some potential strong recommendations are below:
  If your reference genome is large as human genome and your Nanopore data is huge,
  It would be faster to run this program parallelly to speed up.
  You might run different input folders of your fast5 files and
  give different output names (--FileID) or folders (--outFolder)
  A good way for this is to run different chromosome individually.

         Current directory: /home/cxs3_z4/ds/deepmod/D18395
                  outLevel: 2
                   wrkBase: /home/cxs3_z4/ds/deepsignal/
                    FileID: D18395
                 outFolder: ./D18395_deepmod/
                 recursive: 1
          files_per_thread: 1000
                   threads: 15
                windowsize: 21
                  alignStr: minimap2
               SignalGroup: simple
                      move: False
               basecall_1d: Basecall_1D_000
          basecall_2strand: BaseCalled_template
                    ConUnk: True
               outputlayer:
                      Base: C
               mod_cluster: 0
                   predDet: 1
                       Ref: /home/cxs3_z4/ds/deepmod/D18395/genome.nextpolish.fasta
                      fnum: 7
                    hidden: 100
                   modfile: /home/cxs3_z4/software/DeepMod/train_deepmod/rnn_sinmodC_P100wd21_f7ne1u0_4/mod_train_sinmodC_P100wd21_f3ne1u0
                    region: [[None, None, None]]
  

/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
ard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Total files=94416
WARNING:tensorflow:From /home/cxs3_z4/software/DeepMod/bin/DeepMod_scripts/myMultiBiRNN.py:30: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /home/cxs3_z4/software/DeepMod/bin/DeepMod_scripts/myMultiBiRNN.py:30: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/cxs3_z4/software/DeepMod/bin/DeepMod_scripts/myMultiBiRNN.py:47: static_bidirectional_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use keras.layers.Bidirectional(keras.layers.RNN(cell, unroll=True)), which is equivalent to this API
WARNING:tensorflow:Entity <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x7fed9f108c50>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x7fed9f108c50>>: AttributeError: module 'gast' has no attribute 'Index'
2021-02-22 21:13:31.726078: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:The saved meta_graph is possibly from an older release:
'metric_variables' collection should be of type 'byte_list', but instead is of type 'node_list'.
WARNING:tensorflow:From /home/cxs3_z4/software/miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Error!!! No events data in /home/cxs3_z4/ds/deepsignal/8/e617a13d-211d-4cac-a14f-c012fa49577f.fast5
Error!!! No events data in /home/cxs3_z4/ds/deepsignal/8/f0d90ef8-da77-4672-9798-f2ed6a46860a.fast5
Error!!! No events data in /home/cxs3_z4/ds/deepsignal/8/6537dde7-e8e2-4fb3-ac40-95897a2eeabb.fast5
Error!!! No events data in /home/cxs3_z4/ds/deepsignal/8/87ff5443-034a-42c6-8f59-a85134648d6c.fast5
Error!!! No events data in /home/cxs3_z4/ds/deepsignal/8/e8382d8f-7e94-4090-a8b9-10b2ee7cee72.fast5
2021-02-22 21:13:31.970272: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
Error!!! No events data in /home/cxs3_z4/ds/deepsignal/8/a86bbfe4-9c90-46cb-a62b-4f2b60394b46.fast5
2021-02-22 21:13:31.991899: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
Error!!! No events data in /home/cxs3_z4/ds/deepsignal/8/a8a9d5ce-88fb-4f32-9960-ea0182cd1fd4.fast5
.
.
.
[M::mm_idx_gen::0.3781.07] collected minimizers
[M::mm_idx_gen::0.5091.56] sorted minimizers
[M::main::0.5091.56] loaded/built the index for 5 target sequence(s)
[M::mm_mapopt_update::0.5431.53] mid_occ = 23
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 5
[M::mm_idx_stat::0.569*1.50] distinct minimizers: 1483561 (90.31% are singletons); average occurrences: 1.171; average spacing: 5.361
[M::main] Version: 2.17-r941
[M::main] CMD: minimap2 -ax map-ont /home/cxs3_z4/ds/deepmod/D18395/genome.nextpolish.fasta /tmp/tmpqziwwa6g.fa
[M::main] Real time: 0.585 sec; CPU: 0.871 sec; Peak RSS: 0.444 GB
Cur Prediction consuming time 29 for 0 93
Error information for different fast5 files:
No events data 93416
Per-read Prediction consuming time 372
Find: ./D18395_deepmod//D18395 0 rnn.pred.ind
[]
Genomic-position Detection consuming time 0

real 6m17.097s
user 23m9.292s
sys 2m8.143s

• I try to add the --move parameter but it doesn't work.One of the resluts is as follows.

Error!!! No move data in /home/cxs3_z4/ds/deepsignal/10/5f095916-4683-4e3b-9b5a-59feaac0c9eb.fast5

I don't know why it wrongs.

DeepMod gives out "Error!!! Channel information could not be found in UniqueGlobalKey/channel_id in ..." error when i use multi-fast5 files as input, but when i split them using "multi_to_single_fast5", it cannot detect events file the generated single fast5 files.
Should i redo basecalling after spliting multi-fast5? Or there are other potential issues?

Hello teacher, what should I do if there are too many methylation sites detected? Tens of thousands of sites have been detected on each chromosome.Thanks teacher

Chlamydomonas_ONT_6ma.zip

Hi, authors

Thanks for open-source the code. Just curious what is the GPU recommendation (e.g. minimal memory size) for running DeepMod?

Here says GPU can speed up detection.

But Here suggests GPU won't benefit if there is IO bottleneck. Can you please clarify how to avoid such bottleneck?

Thank you.

Issue #1 mentioned 6mA analysis in HX1 but manuscript mentioned performance prediction of 6mA in E. coli and C. reinhardtii.

Can you tell me the difference between "Capped coverage" and "Real coverage " ? Is there a quality control process to filter "Capped coverage" into "Real coverage " like nanopolish?

I would very much like to try DeepMod in my study in that it not only reports the genomic position of modifications with high accuracy but methylation percentage.

I think the major problem with modification detection from nanopore data is the lack of training data. As mentioned in the paper, prediction accuracy is sequence context dependent. Also, a commonly found modification in bacteria is 4mC, which is missing from DeepMod. So I wondered if you will further improve DeepMod for more sequence contexts and 4mC by producing more training data.

Hi Liu,
I am thinking about using DeepMod for our paper, but the specie I'm working with is different from Human and E.coli. Is it possible to insert a different genome dataset on DeepMod and analyse it for methylation patterns?

Thank you in advance!

Hi,

May I ask what is the meaning of the parameters --fulmod, --anymod, and --nomod for the getfeatures option? I am trying to train a custom model but am not sure what to pass in for these parameters. Also is there any possible option to specify an output for model metrics after training?

Thanks.

Hi,
I am trying to find a suitable software to conduct a research in the field of epigenetics based on my ONT data. I cannot find your paper (Detection of DNA base modifications by deep recurrent neural network on Oxford Nanopore sequencing data) about this software on scholar.google, can you tell me why? Can I have a look at that paper now?
Is the software presented here the latest version that used in your paper?
Best wishes,
David.

Hi,
Can the "move table" produced by Guppy (v2.3.1 and newer) be used to support the analysis of Deepmod as the same role of the old "event table" produced by albacore? As Oxford Nanopore Technologies said, " The move table contains just the 'Move' column for the old event table: it shows how much the basecall moved at a given block index." ( https://community.nanoporetech.com/posts/guppy-2-3-1-release)
Best wishes,
David.

In the usage page it is stated that FAST5 must be basecalled and events data must be available in them. However, it seems that the latest Guppy basecaller does not include any events data as Albacore used to do (see below). As mentioned in the readme, it is possible to convert multi-fast5 to single-fast5 using ont-fast5-api. However, I am not sure how Guppy can be asked to save events data in FAST5. Could you shed some light on this?

I met a question that "can not open fast5 or other errors". Could you please tell me what I should do to deal with this question? thank you!

Hello, teacher, how should this problem be solved？

Hello, I'm interested in your wonderful work and trying to use it, but I don't understand the difference between model files under DeepMod/train_mod. Could you please provide information for users to choose the most suitable model for their analyzing, or do you mean that the users should re-train the model for their work? Thank you!

If I understand correctly, Albacore basecaller v2.3.1 was used for the model building in your manuscript. If I run DeepMod using Nanopore data that was basecalled with latest version of Guppy, are the models even valid?

Would you suggest re-basecalling my data using the exact same settings as in your paper?

My dataset is FLO-MIN106, LSK-109 kit, Guppy v3.4.5, dna_r9.4.1_450bps_hac
In your paper: FLO-MIN106, LSK-108, Albacore v2.3.1, what basecalling model?

Thanks,
Jeff

Hi,

I got 2 error messages:

1 /miniconda3/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])

2 Error!!! No Channel Info in ./fast5_pass/PAD32019_248c10ef90f82bed1f365e2d7a0fff9f78c5cd1b_0.fast5
Error!!! Channel information could not be found in UniqueGlobalKey/channel_id in ./fast5_pass/PAD32019_248c10ef90f82bed1f365e2d7a0fff9f78c5cd1b_0.fast5
Error!!! No Channel Info in ./fast5_pass/PAD32019_248c10ef90f82bed1f365e2d7a0fff9f78c5cd1b_1.fast5

Do you know what happened? Thank you so much!

Hi! :) Nice method!

I would like to test DeepMod on the ecoli data set from Simpson et al.
I was wondering which one of the trained models you have in the repo would allow me to recover the results presented in the paper.
Would it be either rnn_sinmodC_P100wd21_f7ne1u0_4 or rnn_conmodC_P100wd21_f7ne1u0_4?
Could you highlight the difference between them?

Alternatively, could you give some guidance on how to train DeepMod on any data set.

Thank you in advance,

Pepe

I have convert multi-fast5 to single fast5 file,
but it has the the error
Error!!! No events data in
/mnt/CCX/User/liuqingshan/methylation/deepsignal/00.data/115.single_fast5_reads/109/c006ef2c-7bf4-432a-9e21-d1bc399c3766.fast5
can you help me?

Hi,

We were attempting to calculate model metrics using the NA12878 dataset as mentioned in https://github.com/WGLab/DeepMod/blob/master/docs/Reproducibility.md

Is there any script available for performance evaluation for this dataset like for the E. coli dataset? Otherwise, what script did you use for bisulfite data to evaluate the performance of your model on NA12878?

Thanks

Hi,
If I want to try your software on the research of some other organism, which modfile should I choose? rnn_f7_wd21_chr1to10_4 or rnn_conmodC_P100wd21_f7ne1u0_4?
Best wishes,
David.

Hello Liu:
I've been working on analyzing CpG methylation in human genome. I tried to run DeepMod on NA12878 and HX1 nanopore data by myself, but I can't get the expected results.
So is it convenient for you to send me the DeepMod results of NA12878 and HX1?