Pipeline fails with my data despite successful test run about atac-seq-pipeline HOT 27 CLOSED

Pipeline failed due to lack of a shared library.

bedClip: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

Please install it and try again

sudo apt-get install libssl1.0.0 libssl-dev

Or you can try with other methods (singularity, docker).

from atac-seq-pipeline.

So I can't install libssl1.0.0 or libel-dev as you describe - I don't have privileges. Lack of privileges
is also one of the reasons that I can't use singularity or docker.

I tried to get the libssl.so.1.0.0 by installing openssl with conda, but the pipeline fails with the same error that you found in the logs.

I've attached a file that shows the relevant parts of my environment variables before and after sourcing encode-atac-seq-pipeline or encode-atac-seq-pipeline-python3.

I also included a find command in my conda tree for the library that bedClip can't locate. The files is there in more than one place. It just seems not to be part of the appropriate library path.

environments.txt

from atac-seq-pipeline.

Please try this. This will downgrade UCSC tools.

VER=357

source activate encode-atac-seq-pipeline
conda install ucsc-fetchchromsizes==$VER -c bioconda
conda install ucsc-wigtobigwig==$VER -c bioconda
conda install ucsc-bedgraphtobigwig==$VER -c bioconda
conda install ucsc-bigwiginfo==$VER -c bioconda
conda install ucsc-bedclip==$VER -c bioconda
conda install ucsc-bedtobigbed==$VER -c bioconda
conda install ucsc-twobittofa==$VER -c bioconda
source deactivate

If VER=357 doesn't work then try with VER=332 instead.

BTW you don't need super user privileges to install singularity. It's different from docker.

from atac-seq-pipeline.

Unfortunately, I'm getting failures with both ucsc version 357 and 332, although perhaps with different specifics than the original problem. Output and error logs in the attached zipped archive.

I do see that to install singularity itself does not have a requirement for root, but it seems that its dependencies do.

Given some of what I'm seeing in the logs, I wonder if there's any issue because my organism for this run is Xenopus laevis. The genome is much less studied than many others, and the chromosome naming scheme is atypical. If it's helpful at all, I've attached the chromosome size table that was generated when I built the genome (sorted by chromosome size).

Thanks for your help with this!

ucsc_version_runs.zip

Xlae9_2.chrom.sizes.txt.zip

from atac-seq-pipeline.

Please try the following. This will make a soft link of that missing library file on pipeline's conda env's lib/.

cd /home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/lib/
ln -s /home/jtobias/software/miniconda3/lib/libssl.so.1.0.0

from atac-seq-pipeline.

Adding the symbolic link did not fix the library not found problem, so I tried to work around it by adding these lines to my run script:

source activate encode-atac-seq-pipeline
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline-python3/lib/:/home/jtobias/software/miniconda3/lib/
INPUT=/home/jtobias/pk201811/input/4_10.json
java -jar -Dconfig.file=backends/backend.conf cromwell-34.jar run atac.wdl -i ${INPUT}

This seems to enable the discovery of libssl.so.1.0.0, but the pipeline still fails. I'm not sure what library environment variables conda should be setting with the source activate encode-atac-seq-pipeline command, but I wonder if that's part of the problem.

I've attached the tarball of the failed run (with my modified $LD_LIBRARY_PATH, but I think the failure may be because this command:

cat /home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/1f97d7f8-1fbd-482e-b8db-e525e87ca209/call-macs2_pr2/shard-0/inputs/2090236487/Xlae9_2.chrom.sizes | grep -P 'chr[\dXY]+[ \t]'

has no output given how the chromosomes are named in the Xenopus laevis genome (none begin with "chr."

Maybe that's just a red herring, but I'm eager to hear what you think.

debug_normal_lib.tar.gz

from atac-seq-pipeline.

Instead of re-running pipeline, run the following link for testing. This will save much time.

source activate encode-atac-seq-pipeline

bedtools slop -i "4_10_dedup.tn5"_FE.bdg -g /home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/49255ba0-ee52-41c6-91d4-7caa15d7174c/call-macs2/shard-0/inputs/2090236487/Xlae9_2.chrom.sizes -b 0 | bedClip stdin /home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/49255ba0-ee52-41c6-91d4-7caa15d7174c/call-macs2/shard-0/inputs/2090236487/Xlae9_2.chrom.sizes 4_10_dedup.tn5.fc.signal.bedgraph

1. Your log says, Conda still cannot find libssl.

Traceback (most recent call last):
  File "/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_macs2_atac.py", line 215, in <module>
    main()
  File "/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_macs2_atac.py", line 188, in main
    args.out_dir)
  File "/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_macs2_atac.py", line 115, in macs2
    run_shell_cmd(cmd4)
  File "/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_common.py", line 263, in run_shell_cmd
    pid, pgid, cmd, ret))
Exception: Killed PID=7653, PGID=7653
CMD=bedtools slop -i "4_10_dedup.tn5"_FE.bdg -g /home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/49255ba0-ee52-41c6-91d4-7caa15d7174c/call-macs2/shard-0/inputs/2090236487/Xlae9_2.chrom.sizes -b 0 | bedClip stdin /home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/49255ba0-ee52-41c6-91d4-7caa15d7174c/call-macs2/shard-0/inputs/2090236487/Xlae9_2.chrom.sizes 4_10_dedup.tn5.fc.signal.bedgraph
STDOUT=bedClip: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

Is this a log file after exporting LD_LIBRARY_PATH?

Please run the following for debugging.

$ source activate encode-atac-seq-pipeline
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline-python3/lib/:/home/jtobias/software/miniconda3/lib/
$ which openssl
$ ldd $(which openssl)
        linux-vdso.so.1 =>  (0x00007ffc0d92b000)
        libssl.so.1.0.0 => /users/leepc12/miniconda3/envs/encode-atac-seq-pipeline/bin/../lib/libssl.so.1.0.0 (0x00007f945dbdd000)
        libcrypto.so.1.0.0 => /users/leepc12/miniconda3/envs/encode-atac-seq-pipeline/bin/../lib/libcrypto.so.1.0.0 (0x00007f945d79c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f945d3d2000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f945d1ce000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f945e0e5000)

2. Please activate this flag to prevent that chromosome name error.

    "atac.keep_irregular_chr_in_bfilt_peak" : true,

from atac-seq-pipeline.

Please see below for the output of the debugging commands and the results for the test command both before and after my adding to the command to update LD_LIBRARY_PATH:

$ source activate encode-atac-seq-pipeline
(encode-atac-seq-pipeline) [jtobias@node107 atac-seq-pipeline]$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline-python3/lib/:/home/jtobias/software/miniconda3/lib/
(encode-atac-seq-pipeline) [jtobias@node107 atac-seq-pipeline]$ which openssl
~/software/miniconda3/envs/encode-atac-seq-pipeline/bin/openssl
(encode-atac-seq-pipeline) [jtobias@node107 atac-seq-pipeline]$ ldd $(which openssl)
/bin/ps: /home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline-python3/lib/liblzma.so.5: no version information available (required by /lib64/libsystemd.so.0)
/bin/ps: /home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline-python3/lib/liblzma.so.5: no version information available (required by /lib64/libdw.so.1)
linux-vdso.so.1 => (0x00007ffe6e779000)
libssl.so.1.1 => /home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/../lib/libssl.so.1.1 (0x00002abfa3515000)
libcrypto.so.1.1 => /home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/../lib/libcrypto.so.1.1 (0x00002abfa365b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002abfa3954000)
libc.so.6 => /lib64/libc.so.6 (0x00002abfa3b70000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002abfa3f3d000)
/lib64/ld-linux-x86-64.so.2 (0x00002abfa3437000)

$ source activate encode-atac-seq-pipeline
$ bedtools slop -i "4_10_dedup.tn5"_FE.bdg -g /home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/1f97d7f8-1fbd-482e-b8db-e525e87ca209/call-macs2/shard-0/inputs/2090236487/Xlae9_2.chrom.sizes -b 0 | bedClip stdin /home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/1f97d7f8-1fbd-482e-b8db-e525e87ca209/call-macs2/shard-0/inputs/2090236487/Xlae9_2.chrom.sizes 4_10_dedup.tn5.fc.signal.bedgraph
bedClip: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory
Error: The requested file (4_10_dedup.tn5_FE.bdg) could not be opened. Error message: (No such file or directory). Exiting!

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline-python3/lib/:/home/jtobias/software/miniconda3/lib/
$ bedtools slop -i "4_10_dedup.tn5"_FE.bdg -g /home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/1f97d7f8-1fbd-482e-b8db-e525e87ca209/call-macs2/shard-0/inputs/2090236487/Xlae9_2.chrom.sizes -b 0 | bedClip stdin /home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/1f97d7f8-1fbd-482e-b8db-e525e87ca209/call-macs2/shard-0/inputs/2090236487/Xlae9_2.chrom.sizes 4_10_dedup.tn5.fc.signal.bedgraph
Error: The requested file (4_10_dedup.tn5_FE.bdg) could not be opened. Error message: (No such file or directory). Exiting!

from atac-seq-pipeline.

Please try with different openssl in Conda and run ldd $(which openssl) to check if libssl.so.1.0.0 exists in the output. Or ask your admin to install libssl. sudo apt-get install libssl1.0.0 libssl-dev.

$ conda search openssl
Loading channels: done
Name                       Version                   Build  Channel
openssl                    1.0.1c                        0  defaults
openssl                    1.0.1g                        0  defaults
openssl                    1.0.1h                        0  defaults
openssl                    1.0.1h                        1  defaults
openssl                    1.0.1j                        0  defaults
openssl                    1.0.1j                        1  defaults
openssl                    1.0.1j                        2  defaults
openssl                    1.0.1j                        3  defaults
openssl                    1.0.1j                        4  defaults
openssl                    1.0.1j                        5  defaults
openssl                    1.0.1k                        0  defaults
openssl                    1.0.1k                        1  defaults
openssl                    1.0.2d                        0  defaults
openssl                    1.0.2e                        0  defaults
openssl                    1.0.2f                        0  defaults
openssl                    1.0.2g                        0  defaults
openssl                    1.0.2h                        0  defaults
openssl                    1.0.2h                        1  defaults
openssl                    1.0.2i                        0  defaults
openssl                    1.0.2j                        0  defaults
openssl                    1.0.2k                        0  defaults
openssl                    1.0.2k                        1  defaults
openssl                    1.0.2k                        2  defaults
openssl                    1.0.2l                        0  defaults
openssl                    1.0.2l               h077ae2c_5  defaults
openssl                    1.0.2l               h9d1a558_3  defaults
openssl                    1.0.2l               hd940f6d_1  defaults
openssl                    1.0.2m               h26d622b_1  defaults
openssl                    1.0.2m               h8cfc7e7_0  defaults
openssl                    1.0.2n               hb7f436b_0  defaults
openssl                    1.0.2o               h14c3975_1  defaults
openssl                    1.0.2o               h20670df_0  defaults
openssl                    1.0.2p               h14c3975_0  defaults
openssl                    1.1.1a               h7b6447c_0  defaults

$ source activate encode-atac-seq-pipeline
$ conda installl openssl==VERSION

from atac-seq-pipeline.

I found initially that libssl.so.1.0.0 wasn't there after source activate encode.... so I tried:
conda install openssl==1.0.2p (after activating) and now:

(encode-atac-seq-pipeline) [jtobias@node062 execution]$ ldd $(which openssl) |grep libssl.so.1.0.0
libssl.so.1.0.0 => /home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/../lib/libssl.so.1.0.0 (0x00002b2b3d970000)
(encode-atac-seq-pipeline) [jtobias@node062 execution]$ conda list | grep openssl
openssl 1.0.2p h470a237_1 conda-forge

So with that result (and the test command you suggested above seeming to no longer give the library error, I reran my pipeline.

It seems to fail with a new error (full diagnostic tarball linked below):

(encode-atac-seq-pipeline) [jtobias@node062 execution]$ pwd
/home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/2b7201be-ea7d-4a43-b106-4fad7e5e3e36/call-macs2_pr2/shard-0/execution
(encode-atac-seq-pipeline) [jtobias@node062 execution]$ less stderr
(encode-atac-seq-pipeline) [jtobias@node062 execution]$ cat stderr
Traceback (most recent call last):
File "/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_macs2_atac.py", line 215, in
main()
File "/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_macs2_atac.py", line 198, in main
peak_to_bigbed(bfilt_npeak, 'narrowPeak', args.chrsz, args.keep_irregular_chr, args.out_dir)
File "/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_common_genomic.py", line 354, in peak_to_bigbed
run_shell_cmd(cmd4)
File "/home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_common.py", line 263, in run_shell_cmd
pid, pgid, cmd, ret))
Exception: Killed PID=25948, PGID=25948
CMD=bedToBigBed -type=bed6+4 -as=4_10_dedup.tn5.pr2.pval0.01.300K.bfilt.as 4_10_dedup.tn5.pr2.pval0.01.300K.bfilt.bb.tmp2 4_10_dedup.tn5.pr2.pval0.01.300K.bfilt.chrsz.tmp 4_10_dedup.tn5.pr2.pval0.01.300K.bfilt.narrowPeak.bb
STDOUT=4_10_dedup.tn5.pr2.pval0.01.300K.bfilt.bb.tmp2 is not case-sensitive sorted at line 263222. Please use "sort -k1,1 -k2,2n" with LC_COLLATE=C, or bedSort and try again.

Line 263222 in that file seems to be a break between chromosomes named chrXX and MT. This run was done with my json containing:

"atac.keep_irregular_chr_in_bfilt_peak" : true,

If my mitochondrial chromosome is named "MT" should I be setting:
atac.bam2ta.regex_grep_v_ta

If so, what's the exact syntax?

debug_issue62_libsslFixed_newError.tar.gz

from atac-seq-pipeline.

Please unzip this file and copy it into /home/jtobias/software/miniconda3/envs/encode-atac-seq-pipeline/bin/ and try again.
encode_common_genomic.zip

from atac-seq-pipeline.

Now things fail with a new error (tarball attached).

CMD=bedToBigBed -type=bed6+4 -as=4_10_dedup.tn5.pr2.pval0.01.300K.bfilt.as 4_10_dedup.tn5.pr2.pval0.01.300K.bfilt.bb.tmp2 4_10_dedup.tn5.pr2.pval0.01.300K.bfilt.chrsz.tmp 4_10_dedup.tn5.pr2.pval0.01.300K.bfilt.narrowPeak.bb
STDOUT=pass1 - making usageList (6075 chroms): 99 millis
Error line 1 of 4_10_dedup.tn5.pr2.pval0.01.300K.bfilt.bb.tmp2: score (22980) must be between 0 and 1000

Also, how should I specify parameters if my mitochondrial chromosome is MT (not chrM)?

debug_issue62_bigToBigBed.tar.gz

from atac-seq-pipeline.

This will be fixed in the next release v1.1.4.

If you want to remove mitochondrial reads then edit your input JSON. atac.regex_filter_reads is chrM by default if undefined.

{
   "atac.regex_filter_reads" : "MT"
}

But the pipeline will not give you correct mito reads statistics about in the ATAQC step (chrM is hard-coded there).

from atac-seq-pipeline.

I downloaded the build for 1.1.4 at ~8pm Eastern to give it a shot. Now I see this in the stderr in the console - but I don't know if the error caused the failure. Tarball attached.

CMD=bedGraphToBigWig 4_10_dedup.tn5.fc.signal.srt.bedgraph /home/jtobias/pk201811/atac-seq-pipeline/cromwell-executions/atac/c4a2f8df-88fc-4dca-89fb-62345e69a306/call-macs2/shard-0/inputs/2090236487/Xlae9_2.chrom.sizes 4_10_dedup.tn5.fc.signal.bigwig
STDOUT=Overlapping regions in bedGraph line 16506 of 4_10_dedup.tn5.fc.signal.srt.bedgraph

Apologies if I did this test before 1.1.4 was ready.

debug_issue62_bedGraphToBigWig.tar.gz

from atac-seq-pipeline.

Please check if 4_10_dedup.tn5.fc.signal.srt.bedgraph is sorted.

from atac-seq-pipeline.

Yes, that file seems to be sorted, but I see that the first chromosome listed is MT (mitochondrial in this genome) followed by 44K ScaffoldNNNN chromosome names (number length is variable) followed by 18 chrXXX. The regions within each chromosome are sorted by position. I assume that capitalization is the reason that chrXXX are not at the top of the file.

from atac-seq-pipeline.

What is in the line 16506 of 4_10_dedup.tn5.fc.signal.srt.bedgraph? Please zcat -f 4_10_dedup.tn5.fc.signal.srt.bedgraph | head -n16510 | gzip -nc > test.zip and send me test.zip.

from atac-seq-pipeline.

$ head -n16510 4_10_dedup.tn5.fc.signal.srt.bedgraph |gzip -nc > test.zip

test.zip

from atac-seq-pipeline.

Wanted to make sure you saw my comment above.

from atac-seq-pipeline.

Please edit src/encode_macs2_atac.py on your git directory. This hotfix will remove any overlapping regions in bedgraph files.

        cmd5 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} > {}'

with

        cmd5 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} | awk \'BEGIN{{OFS="\\t"}}{{if (NR>1 && prev_chr==$1 && prev_chr_e<=$2) {{print $0}}; prev_chr=$1; prev_chr_e=$3;}}\' > {}'

Replace line 152

        cmd9 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} > {}'

with

        cmd9 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} | awk \'BEGIN{{OFS="\\t"}}{{if (NR>1 && prev_chr==$1 && prev_chr_e<=$2) {{print $0}}; prev_chr=$1; prev_chr_e=$3;}}\' > {}'

Update pipeline's Conda environment and then try running a pipeline again.

$ bash conda/update_conda_env.sh

Please let me know if this works.

from atac-seq-pipeline.

I believe I've made the updated changes as described and updated the Conda environment. I seem to get two different error modes depending on whether I specify

"atac.keep_irregular_chr_in_bfilt_peak" : true,

or not. In all runs I use:

"atac.regex_filter_reads" : "MT",

Attached are the logs, stdout and stderr from both runs.

debug_keepIrrChr.tar.gz

debug_stdChr.tar.gz

stdChr.zip

keepIrr.zip

from atac-seq-pipeline.

This will be fixed in the next release (v1.1.5).

from atac-seq-pipeline.

I believe 1.1.5 was released so I reran my data. I used a different build of the Xenopus laevis genome with more conventional, but still non-standard chromosome names (although the mitochondrial chromosome is now chrM). I still used:

"atac.keep_irregular_chr_in_bfilt_peak" : true,

My error logs and other debugging info are attached.

debug_v115.tar.gz
4_10_irr.stdout.zip
4_10_irr.stderr.zip

from atac-seq-pipeline.

The hotfix seems to have solved the problem, at least for my test sample. I'm running 5 additional samples now. Will the fix be rolled into the master builds in the future? Thanks for your perseverance!

from atac-seq-pipeline.

@jwtobias: Sure, I just merged it to the master branch.

from atac-seq-pipeline.

My runs have completed successfully. Thanks again!

from atac-seq-pipeline.

@jwtobias: Sure, I just merged it to the master branch.

Please edit src/encode_macs2_atac.py on your git directory. This hotfix will remove any overlapping regions in bedgraph files.

atac-seq-pipeline/src/encode_macs2_atac.py

Line 117 in 163ecb6

cmd5 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} > {}'

L117
Replace line 117

        cmd5 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} > {}'

with

        cmd5 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} | awk \'BEGIN{{OFS="\\t"}}{{if (NR>1 && prev_chr==$1 && prev_chr_e<=$2) {{print $0}}; prev_chr=$1; prev_chr_e=$3;}}\' > {}'

atac-seq-pipeline/src/encode_macs2_atac.py

Line 152 in 163ecb6

cmd9 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} > {}'

L152
Replace line 152

        cmd9 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} > {}'

with

        cmd9 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} | awk \'BEGIN{{OFS="\\t"}}{{if (NR>1 && prev_chr==$1 && prev_chr_e<=$2) {{print $0}}; prev_chr=$1; prev_chr_e=$3;}}\' > {}'

Update pipeline's Conda environment and then try running a pipeline again.

$ bash conda/update_conda_env.sh

Please let me know if this works.
Hello, I am a freshman in analyzing the ATAC-seq data. I met a similar question and I am very happy to see the answer. However I am still confused about what "NR" and "prev" refer to in "awk 'BEGIN{{OFS="\t"}}{{if (NR>1 && prev_chr==$1 && prev_chr_e<=$2) {{print $0}}; prev_chr=$1; prev_chr_e=$3;}}'". Although I have red code in

atac-seq-pipeline/src/encode_macs2_atac.py

Line 161 in 163ecb6

chrsz,

. Could you please explain them to me at your convenience? thanks

from atac-seq-pipeline.