Good afternoon,
I downloaded the 358 input bam files for Drosophila (dm3). Next I downloaded the respective umap files. Then I ran the blacklisting tool. It did not give me the same results as found in dm3-blacklist.v2.bed.gz. Is there something I am missing?

When I run the demo, I get the expected results.

I hope to hear from you soon.
Thanks,
Sammy

@aboyle
I hade to use the following to build Blacklist:

git clone --recurse-submodules https://github.com/Boyle-Lab/Blacklist.git
cd Blacklist

Added CMAKE_INSTALL_PREFIX:PATH and make install for bamtools

# Build bamtools API
cd bamtools/
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX:PATH=$(cd ..; pwd)/install ..
make
make install
cd ../..

Change Makefile for Blacklist so it finds bamtools headers and libs.
The C++ code requires c++14 features or higher.

diff --git a/Makefile b/Makefile
index 4aae320..5f040a1 100644
--- a/Makefile
+++ b/Makefile
@@ -1,10 +1,14 @@
 prefix=$(shell pwd)
 
+BAMTOOLS_INCLUDE_DIR=$(prefix)/bamtools/install/include/bamtools
+BAMTOOLS_LIB_DIR=$(prefix)/bamtools/install/lib
+
+
 all:
-       g++ -o Blacklist blacklist.cpp -I $(prefix)/bamtools/include/ -L $(prefix)/bamtools/lib/ -lbamtools -lz -Wl,-rpath,$(prefix)/bamtools/lib/
+       g++ -std=c++14 -o Blacklist blacklist.cpp -I$(BAMTOOLS_INCLUDE_DIR) -L$(BAMTOOLS_LIB_DIR) -lbamtools -lz -Wl,-rpath,$(BAMTOOLS_LIB_DIR)
 
 debug:
-       g++ -g -o Blacklist blacklist.cpp -I $(prefix)/bamtools/include/ -L $(prefix)/bamtools/lib/ -lbamtools -lz -Wl,-rpath,$(prefix)/bamtools/lib/
+       g++ -std=c++14 -g -o Blacklist blacklist.cpp -I$(BAMTOOLS_INCLUDE_DIR) -L$(BAMTOOLS_LIB_DIR) -lbamtools -lz -Wl,-rpath,$(BAMTOOLS_LIB_DIR)
 
 blacklist:
        ./Blacklist chr1 > final.chr1.out

make

I tried to run the demo. I expected that I had to run it like this (with the name of the chromosome in the BAM file):

$ cd demo
$ ../Blacklist chrUn_GL456392
Unable to read mappability files!

But I only was able to get results when I ran it like this:

$ cd demo
$ ../Blacklist demo
demo	0	23600	1	227

Also it seems to require the mappabily file. I assumed it would create this file from the BAM file. How do I generate this file myself?

I have a question. I have a Whole Exome sequencing experiment with 161 bam files. I ran Blacklist and for some chromosomes there are many regions but for chr3, chr4, chr5, and chr6 the entire chromosomes are High Signal Region. What does this mean?

chr3    0       198295500       High Signal Region
chr4    0       190214500       High Signal Region
chr5    0       181538200       High Signal Region
chr6    0       170805900       High Signal Region

I have now compiled on Ubuntu and getting error:

./Blacklist: error while loading shared libraries: libbamtools.so.2.4.1: cannot open shared object file: No such file or directory

Dear Alan,

Thanks for the official release of Blacklist, I hope it will be useful for every genomic lab even those working on non-model organisms.

I try since several days to build Blacklist and the file that I get from the compiler is wrong. It seems something is broken inside the code of blacklist.cpp or in the Makefile. Or maybe I didn't use the proper version of g++?

Currently I try to build it on a ubuntu 18.04 LTS machine with the g++ version 7.4.0

Best,
Simon

Any chance to update the "blacklist" to "deny/exclude list" instead? I could make a pull request with the changes.
I think that those files are really important to have this name.

Hi dear Boyle
Thank you so much for developing this Blacklist.
Now I would like to use hg38-blacklist.v2.bed.gz to filter ATACseq data using -e , --blacklist in HMMRATAC,but there are some warnings and 1 error when I install in CentOS Linux 7 (Core).
[qiaowei@clust1 build]$ make Scanning dependencies of target SharedHeaders [ 1%] Exporting SharedHeaders [ 1%] Built target SharedHeaders Scanning dependencies of target APIHeaders [ 2%] Exporting APIHeaders [ 2%] Built target APIHeaders Scanning dependencies of target AlgorithmsHeaders [ 3%] Exporting AlgorithmsHeaders [ 3%] Built target AlgorithmsHeaders Scanning dependencies of target BamTools [ 4%] Building CXX object src/api/CMakeFiles/BamTools.dir/BamAlignment.cpp.o [ 5%] Building CXX object src/api/CMakeFiles/BamTools.dir/BamMultiReader.cpp.o [ 6%] Building CXX object src/api/CMakeFiles/BamTools.dir/BamReader.cpp.o [ 7%] Building CXX object src/api/CMakeFiles/BamTools.dir/BamWriter.cpp.o [ 8%] Building CXX object src/api/CMakeFiles/BamTools.dir/SamHeader.cpp.o [ 9%] Building CXX object src/api/CMakeFiles/BamTools.dir/SamProgram.cpp.o [ 10%] Building CXX object src/api/CMakeFiles/BamTools.dir/SamProgramChain.cpp.o [ 11%] Building CXX object src/api/CMakeFiles/BamTools.dir/SamReadGroup.cpp.o [ 12%] Building CXX object src/api/CMakeFiles/BamTools.dir/SamReadGroupDictionary.cpp.o [ 13%] Building CXX object src/api/CMakeFiles/BamTools.dir/SamSequence.cpp.o [ 14%] Building CXX object src/api/CMakeFiles/BamTools.dir/SamSequenceDictionary.cpp.o [ 15%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/bam/BamHeader_p.cpp.o [ 16%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/bam/BamMultiReader_p.cpp.o [ 17%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/bam/BamRandomAccessController_p.cpp.o [ 18%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/bam/BamReader_p.cpp.o [ 19%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/bam/BamWriter_p.cpp.o [ 20%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/index/BamIndexFactory_p.cpp.o [ 21%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/index/BamStandardIndex_p.cpp.o /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/index/BamStandardIndex_p.cpp: In member function ‘void BamTools::Internal::BamStandardIndex::WriteLinearOffsets(const int&, BamTools::Internal::BaiLinearOffsetVector&)’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/index/BamStandardIndex_p.cpp:958:89: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if ( numBytesWritten != (sizeof(offsetCount) + linearOffsets.size()*sizeof(uint64_t)) ) ^ [ 22%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/index/BamToolsIndex_p.cpp.o [ 23%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/BamDeviceFactory_p.cpp.o [ 24%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/BamFile_p.cpp.o [ 25%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/BamFtp_p.cpp.o [ 26%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/BamHttp_p.cpp.o /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/io/BamHttp_p.cpp: In member function ‘bool BamTools::Internal::BamHttp::SendGetRequest(size_t)’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/io/BamHttp_p.cpp:409:62: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if ( WriteToSocket(requestHeader.c_str(), headerSize) != headerSize ) { ^ /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/io/BamHttp_p.cpp: In member function ‘bool BamTools::Internal::BamHttp::SendHeadRequest()’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/io/BamHttp_p.cpp:501:62: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if ( WriteToSocket(requestHeader.c_str(), headerSize) != headerSize ) { ^ [ 27%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/BamPipe_p.cpp.o [ 28%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/BgzfStream_p.cpp.o [ 29%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/ByteArray_p.cpp.o [ 30%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/HostAddress_p.cpp.o [ 31%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/HostInfo_p.cpp.o [ 32%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/HttpHeader_p.cpp.o [ 33%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/ILocalIODevice_p.cpp.o [ 34%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/RollingBuffer_p.cpp.o [ 35%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/TcpSocket_p.cpp.o [ 36%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/TcpSocketEngine_p.cpp.o [ 37%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/io/TcpSocketEngine_unix_p.cpp.o [ 38%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/sam/SamFormatParser_p.cpp.o [ 39%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/sam/SamFormatPrinter_p.cpp.o [ 40%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/sam/SamHeaderValidator_p.cpp.o [ 41%] Building CXX object src/api/CMakeFiles/BamTools.dir/internal/utils/BamException_p.cpp.o Linking CXX shared library ../../../lib/libbamtools.so [ 41%] Built target BamTools Scanning dependencies of target BamTools-static [ 42%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/BamAlignment.cpp.o [ 43%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/BamMultiReader.cpp.o [ 44%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/BamReader.cpp.o [ 45%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/BamWriter.cpp.o [ 46%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/SamHeader.cpp.o [ 47%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/SamProgram.cpp.o [ 48%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/SamProgramChain.cpp.o [ 49%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/SamReadGroup.cpp.o [ 50%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/SamReadGroupDictionary.cpp.o [ 51%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/SamSequence.cpp.o [ 52%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/SamSequenceDictionary.cpp.o [ 53%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/bam/BamHeader_p.cpp.o [ 54%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/bam/BamMultiReader_p.cpp.o [ 55%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/bam/BamRandomAccessController_p.cpp.o [ 56%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/bam/BamReader_p.cpp.o [ 57%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/bam/BamWriter_p.cpp.o [ 58%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/index/BamIndexFactory_p.cpp.o [ 59%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/index/BamStandardIndex_p.cpp.o /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/index/BamStandardIndex_p.cpp: In member function ‘void BamTools::Internal::BamStandardIndex::WriteLinearOffsets(const int&, BamTools::Internal::BaiLinearOffsetVector&)’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/index/BamStandardIndex_p.cpp:958:89: **warning: comparison between signed and unsigned integer expressions [-Wsign-compare]** if ( numBytesWritten != (sizeof(offsetCount) + linearOffsets.size()*sizeof(uint64_t)) ) ^ [ 60%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/index/BamToolsIndex_p.cpp.o [ 61%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/BamDeviceFactory_p.cpp.o [ 62%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/BamFile_p.cpp.o [ 63%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/BamFtp_p.cpp.o [ 64%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/BamHttp_p.cpp.o /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/io/BamHttp_p.cpp: In member function ‘bool BamTools::Internal::BamHttp::SendGetRequest(size_t)’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/io/BamHttp_p.cpp:409:62: **warning: comparison between signed and unsigned integer expressions [-Wsign-compare]** if ( WriteToSocket(requestHeader.c_str(), headerSize) != headerSize ) { ^ /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/io/BamHttp_p.cpp: In member function ‘bool BamTools::Internal::BamHttp::SendHeadRequest()’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/internal/io/BamHttp_p.cpp:501:62: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if ( WriteToSocket(requestHeader.c_str(), headerSize) != headerSize ) { ^ [ 65%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/BamPipe_p.cpp.o [ 66%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/BgzfStream_p.cpp.o [ 67%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/ByteArray_p.cpp.o [ 68%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/HostAddress_p.cpp.o [ 69%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/HostInfo_p.cpp.o [ 70%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/HttpHeader_p.cpp.o [ 71%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/ILocalIODevice_p.cpp.o [ 72%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/RollingBuffer_p.cpp.o [ 73%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/TcpSocket_p.cpp.o [ 74%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/TcpSocketEngine_p.cpp.o [ 75%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/io/TcpSocketEngine_unix_p.cpp.o [ 76%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/sam/SamFormatParser_p.cpp.o [ 77%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/sam/SamFormatPrinter_p.cpp.o [ 78%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/sam/SamHeaderValidator_p.cpp.o [ 79%] Building CXX object src/api/CMakeFiles/BamTools-static.dir/internal/utils/BamException_p.cpp.o Linking CXX static library ../../../lib/libbamtools.a [ 79%] Built target BamTools-static Scanning dependencies of target jsoncpp [ 80%] Building CXX object src/third_party/jsoncpp/CMakeFiles/jsoncpp.dir/json_reader.cpp.o [ 81%] Building CXX object src/third_party/jsoncpp/CMakeFiles/jsoncpp.dir/json_value.cpp.o [ 82%] Building CXX object src/third_party/jsoncpp/CMakeFiles/jsoncpp.dir/json_writer.cpp.o Linking CXX static library ../../../../lib/libjsoncpp.a [ 82%] Built target jsoncpp Scanning dependencies of target BamTools-utils [ 83%] Building CXX object src/utils/CMakeFiles/BamTools-utils.dir/bamtools_fasta.cpp.o [ 84%] Building CXX object src/utils/CMakeFiles/BamTools-utils.dir/bamtools_options.cpp.o [ 85%] Building CXX object src/utils/CMakeFiles/BamTools-utils.dir/bamtools_pileup_engine.cpp.o [ 86%] Building CXX object src/utils/CMakeFiles/BamTools-utils.dir/bamtools_utilities.cpp.o Linking CXX static library ../../../lib/libbamtools-utils.a [ 86%] Built target BamTools-utils Scanning dependencies of target bamtools_cmd [ 87%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_convert.cpp.o [ 88%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_count.cpp.o [ 89%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_coverage.cpp.o [ 90%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_filter.cpp.o [ 91%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_header.cpp.o [ 92%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_index.cpp.o [ 93%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_merge.cpp.o [ 94%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_random.cpp.o [ 95%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_resolve.cpp.o [ 96%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_revert.cpp.o [ 97%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_sort.cpp.o [ 98%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_split.cpp.o /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp: In member function ‘bool BamTools::SplitTool::SplitToolPrivate::SplitListTagImpl(BamTools::BamAlignment&)’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:474:23: warning: typedef ‘TagElementType’ locally defined but not used [-Wunused-local-typedefs] typedef T TagElementType; ^ In file included from /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamReader.h:14:0, from /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:14: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h: In instantiation of ‘bool BamTools::BamAlignment::GetTag(const string&, std::vector<T>&) const [with T = signed char; std::string = std::basic_string<char>]’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:499:13: required from ‘bool BamTools::SplitTool::SplitToolPrivate::SplitListTagImpl(BamTools::BamAlignment&) [with T = signed char]’ /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:415:95: required from here /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h:581:9: warning: variable ‘elementLength’ set but not used [-Wunused-but-set-variable] int elementLength = 0; ^ /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h: In instantiation of ‘bool BamTools::BamAlignment::GetTag(const string&, std::vector<T>&) const [with T = short int; std::string = std::basic_string<char>]’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:499:13: required from ‘bool BamTools::SplitTool::SplitToolPrivate::SplitListTagImpl(BamTools::BamAlignment&) [with T = short int]’ /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:416:96: required from here /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h:581:9: **warning: variable ‘elementLength’ set but not used [-Wunused-but-set-variable]** /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h: In instantiation of ‘bool BamTools::BamAlignment::GetTag(const string&, std::vector<T>&) const [with T = int; std::string = std::basic_string<char>]’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:499:13: required from ‘bool BamTools::SplitTool::SplitToolPrivate::SplitListTagImpl(BamTools::BamAlignment&) [with T = int]’ /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:417:96: required from here /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h:581:9: **warning: variable ‘elementLength’ set but not used [-Wunused-but-set-variable]** /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h: In instantiation of ‘bool BamTools::BamAlignment::GetTag(const string&, std::vector<T>&) const [with T = unsigned char; std::string = std::basic_string<char>]’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:499:13: required from ‘bool BamTools::SplitTool::SplitToolPrivate::SplitListTagImpl(BamTools::BamAlignment&) [with T = unsigned char]’ /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:418:96: required from here /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h:581:9: **warning: variable ‘elementLength’ set but not used [-Wunused-but-set-variable]** /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h: In instantiation of ‘bool BamTools::BamAlignment::GetTag(const string&, std::vector<T>&) const [with T = short unsigned int; std::string = std::basic_string<char>]’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:499:13: required from ‘bool BamTools::SplitTool::SplitToolPrivate::SplitListTagImpl(BamTools::BamAlignment&) [with T = short unsigned int]’ /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:419:97: required from here /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h:581:9: **warning: variable ‘elementLength’ set but not used [-Wunused-but-set-variable]** /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h: In instantiation of ‘bool BamTools::BamAlignment::GetTag(const string&, std::vector<T>&) const [with T = unsigned int; std::string = std::basic_string<char>]’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:499:13: required from ‘bool BamTools::SplitTool::SplitToolPrivate::SplitListTagImpl(BamTools::BamAlignment&) [with T = unsigned int]’ /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:420:97: required from here /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h:581:9: **warning: variable ‘elementLength’ set but not used [-Wunused-but-set-variable]** /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h: In instantiation of ‘bool BamTools::BamAlignment::GetTag(const string&, std::vector<T>&) const [with T = float; std::string = std::basic_string<char>]’: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:499:13: required from ‘bool BamTools::SplitTool::SplitToolPrivate::SplitListTagImpl(BamTools::BamAlignment&) [with T = float]’ /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/toolkit/bamtools_split.cpp:421:94: required from here /mnt/disk1/qiaowei/reference/Blacklist/bamtools/src/api/BamAlignment.h:581:9: **warning: variable ‘elementLength’ set but not used [-Wunused-but-set-variable]** [ 99%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools_stats.cpp.o [100%] Building CXX object src/toolkit/CMakeFiles/bamtools_cmd.dir/bamtools.cpp.o Linking CXX executable ../../../bin/bamtools [100%] Built target bamtools_cmd [qiaowei@clust1 build]$ l total 40K drwxrwxr-x. 4 qiaowei qiaowei 4.0K Oct 5 22:48 CMakeFiles drwxrwxr-x. 7 qiaowei qiaowei 4.0K Oct 5 22:45 src -rw-rw-r--. 1 qiaowei qiaowei 1.9K Oct 5 22:45 cmake_install.cmake -rw-rw-r--. 1 qiaowei qiaowei 9.0K Oct 5 22:45 Makefile -rw-rw-r--. 1 qiaowei qiaowei 13K Oct 5 22:45 CMakeCache.txt [qiaowei@clust1 build]$ make install [ 1%] Exporting SharedHeaders [ 1%] Built target SharedHeaders [ 2%] Exporting APIHeaders [ 2%] Built target APIHeaders [ 3%] Exporting AlgorithmsHeaders [ 1%] Exporting SharedHeaders [ 1%] Built target SharedHeaders [ 2%] Exporting APIHeaders [ 2%] Built target APIHeaders [ 3%] Exporting AlgorithmsHeaders [ 3%] Built target AlgorithmsHeaders [ 41%] Built target BamTools [ 79%] Built target BamTools-static [ 82%] Built target jsoncpp [ 86%] Built target BamTools-utils [100%] Built target bamtools_cmd Install the project... -- Install configuration: "Release" -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/shared/bamtools_global.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/lib/bamtools/libbamtools.so.2.4.1 -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/lib/bamtools/libbamtools.so -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/lib/bamtools/libbamtools.a -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/api_global.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/BamAlgorithms.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/BamAlignment.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/BamAux.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/BamConstants.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/BamIndex.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/BamMultiReader.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/BamReader.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/BamWriter.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/IBamIODevice.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/SamConstants.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/SamHeader.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/SamProgram.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/SamProgramChain.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/SamReadGroup.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/SamReadGroupDictionary.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/SamSequence.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/SamSequenceDictionary.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools/api/algorithms/Sort.h -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/bin/bamtools-2.4.1 -- Installing: /mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/bin/bamtools -- Removed runtime path from "/mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/bin/bamtools-2.4.1" [qiaowei@clust1 build]$ l total 44K -rw-rw-r--. 1 qiaowei qiaowei 2.2K Oct 5 22:51 install_manifest.txt drwxrwxr-x. 4 qiaowei qiaowei 4.0K Oct 5 22:51 CMakeFiles drwxrwxr-x. 7 qiaowei qiaowei 4.0K Oct 5 22:45 src -rw-rw-r--. 1 qiaowei qiaowei 1.9K Oct 5 22:45 cmake_install.cmake -rw-rw-r--. 1 qiaowei qiaowei 9.0K Oct 5 22:45 Makefile -rw-rw-r--. 1 qiaowei qiaowei 13K Oct 5 22:45 CMakeCache.txt [qiaowei@clust1 build]$ cd ../.. [qiaowei@clust1 Blacklist]$ ; -bash: syntax error near unexpected token ;'
[qiaowei@clust1 Blacklist]$ l
total 68K
drwxrwxr-x. 9 qiaowei qiaowei 4.0K Oct 5 22:51 bamtools
drwxrwxr-x. 3 qiaowei qiaowei 4.0K Oct 5 22:40 lists
drwxrwxr-x. 4 qiaowei qiaowei 68 Oct 5 22:29 demo
-rw-rw-r--. 1 qiaowei qiaowei 14K Oct 5 22:29 blacklist.cpp
-rw-rw-r--. 1 qiaowei qiaowei 4.0K Oct 5 22:29 README.md
-rw-rw-r--. 1 qiaowei qiaowei 35K Oct 5 22:29 LICENSE
-rw-rw-r--. 1 qiaowei qiaowei 1.3K Oct 5 22:29 Makefile
[qiaowei@clust1 Blacklist]$ make
g++ -std=c++14 -o Blacklist blacklist.cpp -I/mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/include/bamtools -L/mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/lib/bamtools -lbamtools -lz -Wl,-rpath,/mnt/disk1/qiaowei/reference/Blacklist/bamtools/install/lib/bamtools
g++: error: unrecognized command line option ‘-std=c++14’
make: *** [all] Error 1
`
I know you said this installation needed Linux: Ubuntu 18.04, so that is to say I cannot use this Blacklist?or how can I fix my problem?
Thank you so much in advance.

I have a question.Can I build a blacklist for pigs?

We'd like to be able to install this software on our HPC infrastructure and make it available to all users but as it stands that's not possible, the two hurdles being:

1. The executable assumes that the two directories input/ and mappability/ are co-located with it, and writeable by the end user(s). This isn't compatible with a multi-user system where typically the executable will be in a read-only bin/ directory and input data location will vary by user.
2. There's no install target in the top-level makefile so there's no simple method to customize the location of the executable and its libraries, in particular the linked bamtools library

Thank you; our users would be appreciative!

Hi,

I'm investigating the building of a blacklist for pigs, and I wonder how many chip-seq tracks I need and which type of data (input or controls ?).

I'm also seeking the code necessary to map the fastq and split them by chromosomes.

Thanks for your helps,
Adrien Dufour.

Hello,

I am trying to run the demo files but instead of numbers I get "High Signal Region" in the 4th field of the bed file.
Such as:
Blacklist chrUn_GL456392
chrUn_GL456392 5200 23600 High Signal Region

What does "High Signal Region" mean?

Thank you.
Best,
Danvor

Hello,

I'm following the instruction of the installation. Everything is fine until I reach the final step to build Blacklist.

I followed the instructions to build the bamtools, and no errors occurred. When I tried to build Blacklist, this is the output of make:

g++ -std=c++14 -o Blacklist blacklist.cpp -I/work/bio-chenx/tools/Blacklist/bamtools/install/include/bamtools -L/work/bio-chenx/tools/Blacklist/bamtools/install/lib/bamtools -lbamtools -lz -Wl,-rpath,/work/bio-chenx/tools/Blacklist/bamtools/install/lib/bamtools
blacklist.cpp: In function ‘int getdir(std::__cxx11::string, std::vector<std::__cxx11::basic_string<char> >&, std::__cxx11::string)’:
blacklist.cpp:186:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
/share/base/binutils/2.31-gcc-4.8.5/bin/ld: /tmp/ccIW6QKO.o: in function `SequenceData::getInputBins(std::vector<unsigned char, std::allocator<unsigned char> >&, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)':
blacklist.cpp:(.text._ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x152): undefined reference to `BamTools::BamReader::Open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/share/base/binutils/2.31-gcc-4.8.5/bin/ld: blacklist.cpp:(.text._ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x16f): undefined reference to `BamTools::BamReader::OpenIndex(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/share/base/binutils/2.31-gcc-4.8.5/bin/ld: blacklist.cpp:(.text._ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x185): undefined reference to `BamTools::BamReader::GetReferenceID(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
/share/base/binutils/2.31-gcc-4.8.5/bin/ld: blacklist.cpp:(.text._ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x1bb): undefined reference to `BamTools::BamReader::GetReferenceID(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
/share/base/binutils/2.31-gcc-4.8.5/bin/ld: blacklist.cpp:(.text._ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x1db): undefined reference to `BamTools::BamReader::GetReferenceID(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
collect2: error: ld returned 1 exit status
make: *** [all] Error 1

Our system has gcc4.8.5 by default, and I have used module load gcc/8.2.0 to make gcc to be 8.2.0 before I build bamtools and Blacklist.

Any clue?

Thank you.

Regards,
Xi

hi,

i have checked your mouse mm10 blacklists against some of my atac datasets and found that the regions excluded are sometimes really large. much larger than the actual regions of low mappability/high signal. i think this is due to your combination of the hit regions inside a 20k window. this drops quite a few genes that are completely fine.

would it be possible to release a version of the v2 blacklists that does not combine neighbouring hits like this? just the "real" hits?

also right now you classify regions that are both, "High Signal" and "Low Mappability" as "High Signal". Can you completely separate those?

i think it would make sense to release a much more granular version of this. since merging flanking or intersecting regions is a one-liner using bedtools that runs in seconds and can be done by the user as necessary. but downloading the data and running your tool will take weeks for just a single organism.

best,
carsten

Hello,

I am getting an error when running
make
from inside Blacklist directory

2:25: warning: 
      implicit conversion from 'int' to 'const char' changes value from 255 to -1 [-Wconstant-conversion]
const char OS_UNKNOWN = 255;
           ~~~~~~~~~~   ^~~
blacklist.cpp:186:1: warning: control may reach end of non-void function [-Wreturn-type]
}
^
blacklist.cpp:284:7: error: no member named 'fmod' in namespace 'std'; did you mean simply 'fmod'?
                        if(std::fmod(bins[p],1) != 0) {
                           ^~~~~~~~~
                           fmod
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/math.h:917:1: note: 'fmod' declared here
fmod(_A1 __lcpp_x, _A2 __lcpp_y) _NOEXCEPT
^
3 warnings and 1 error generated.
make: *** [all] Error 1

Hi,

Can you give a list of blacklisted regions upon request? I am working with the bovine genome and would like to filter out the blacklisted regions. Could you tell me if you are able to produce this file? Thanks

Hi, I tried to download the blacklist hg19-blacklist.bed.gz. However, while gunzip the file, gzip complains:
"not in gzip format." I tried both v2 and v1. Can you check if this is true?

I seemed to install Bamtools without any errors. I am using GCC 8.2.0 and when I run make in the Blacklist list directory I get the following error log

blacklist.cpp: In function ‘int getdir(std::__cxx11::string, std::vector<std::__cxx11::basic_string<char> >&, std::__cxx11::string)’:
blacklist.cpp:186:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
/tmp/ccQnkBUG.o: In function `SequenceData::getInputBins(std::vector<unsigned char, std::allocator<unsigned char> >&, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)':
blacklist.cpp:(.text._ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x152): undefined reference to `BamTools::BamReader::Open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
blacklist.cpp:(.text._ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x16f): undefined reference to `BamTools::BamReader::OpenIndex(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
blacklist.cpp:(.text._ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x185): undefined reference to `BamTools::BamReader::GetReferenceID(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
blacklist.cpp:(.text._ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x1bb): undefined reference to `BamTools::BamReader::GetReferenceID(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
blacklist.cpp:(.text._ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN12SequenceData12getInputBinsERSt6vectorIhSaIhEEiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x1db): undefined reference to `BamTools::BamReader::GetReferenceID(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
collect2: error: ld returned 1 exit status
make: *** [all] Error 1

I followed your instruction in README and tried to build Blacklist. When I tried to build Blacklist, an error message pops up.

# make
g++ -std=c++14 -o Blacklist blacklist.cpp -I/opt/The_ENCODE_Blacklist/20210922/bamtools/install/include/bamtools -L/opt/The_ENCODE_Blacklist/20210922/bamtools/install/lib/bamtools -lbamtools -lz -Wl,-rpath,/opt/The_ENCODE_Blacklist/20210922/bamtools/install/lib/bamtools
g++: error: unrecognized command line option ‘-std=c++14’
make: *** [all] Error 1

Could you give me any instructions to solve this error? If you need more information about my environment, please leave what information you need in a comment. Then I will reply to it.

Thank you in advance!

Recently we have downloaded black list BED file of human genome which provided by your paper (the eNCoDe Blacklist: Identification of Problematic Regions of the Genome). We simply calculated the "Genomic coverage" using "each.region.length= END - START + 1", "SUM.black.list.region= SUM (each.region.length)", "Genomic.coverage=SUM.black.list.region/human.Genome.size". However, the Genomic.coverage from our calculation is over 7%, which is totally different as your published paper (1.5%). The same problem is the "Average size (kb)".

Is it possible to create such a blacklist for other organisms? In my case, chicken.
Thanks!

Hi, I am woking on a evodevo project and I need to compare mouse and pig ATAC-seq data. For the basic qc of the ATAC I wanted to remove blacklisted regions from my species; I know there is a blacklist for mouse mm10, but not one for pig susScr11.

To keep the analysis comparable and best-practice-compliant, I would like to generate a blacklist for susScr11, but I only have 12 input bam files, and I read in other discussions here that you recommend >100. So my first question would be if in your opinion it is worth generating this blacklist to use in qc, or if it would actually introduce more bias/artefacts? Would it then be better to just skip the blacklisting for both species before analysis?

If you do think it's worth generating the blacklist for susScr11, my second question is regarding what needs to be in the mappability/ directory. I have never worked with these umap files, are they the ones that can be generated using this tool?

Thank you in advance!
Anna

Is there anywhere I can find the UMAP files that were used to generate the blacklist for dm6?
I found this link: https://www.encodeproject.org/comparative/regulation/
This link contains the files for dm3 but not for dm6. Where can I find the files used for dm6?
Thank you so much for your time.
Sincerely,
Sammy

Hi,

I need help with building blacklist for my small plant genome (Marchantia polymorpha, ~280 Mbp total, 9 chromosoms between 30 and 7.5 Mbp).
I tried Blacklist on demo data and it runs ok. When I run it on any of my chromosomes, using its respective input bam file and umap mappability file, it outputs the whole chromosome (e.g. chr1 0 30584100 High Signal Region).
I know there are a few regions with excessive coverage in the genome, because I see them in genome browser in every CHIP-seq input file.
Do you have any idea? I'll be really grateful for any help!

Best,
Rotem

Hello,

Thanks for this resource. There seems to be a large discrepancy between the v2 blacklist for hg38 listed here and the ENCODE hg38 blacklist(https://www.encodeproject.org/annotations/ENCSR636HFF/). I was wondering why there is such a discrepancy and which version I should be using. Example:

In general it looks like the regions in the github version(top) are longer and overlap less with the ENCODE version in centromeric regions

Hi!

I am a bit confused about the file named hg19-blacklist.v2.bed.gz. It contains 834 Blacklisted regions while the google sheet Merged lists contain merely 412 regions. Which blacklist must be ultimately used. How do they differ? I couldn't find an answer to this in your paper as well.

I used 13 bam files for the input and the program called most of the genome to be part of the blacklist. Is there a minimum number of inputs that need to be used? If so, how many? If not, what could be the issue?

I get this error when trying to run ./Blacklist

dyld: Library not loaded: libbamtools.2.4.1.dylib
  Referenced from: /Users/brian/tools/Blacklist/./Blacklist
  Reason: image not found

Entire chr29 as High Signal Region

I have created mappability file for chr29 (smallest chromosome for quicker analysis) with kmer 100 and 150, and used final uint output file of combine_umaps
However, it spits out entire chromosome as high signal region, not sure if this is because of low number of inputs I have provided (5 inputs).

Blacklist was installed using https://anaconda.org/bioconda/encode-blacklist

$ Blacklist chr29
chr29   0       43051100        High Signal Region

Directory structure below

$ tree
.
├── input
│   ├── Ss1_final_chr29.bam
│   ├── Ss1_final_chr29.bam.bai
│   ├── Ss2_final_chr29.bam
│   ├── Ss2_final_chr29.bam.bai
│   ├── Ss3_final_chr29.bam
│   ├── Ss3_final_chr29.bam.bai
│   ├── Ss4_final_chr29.bam
│   ├── Ss4_final_chr29.bam.bai
│   ├── Ss5_final_chr29.bam
│   └── Ss5_final_chr29.bam.bai
└── mappability
    └── chr29.uint8.unique

Contents of uint file # combined kmer 100 and 150

$ od -t x1 mappability/chr29.uint8.unique | head
0000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
0000140 00 00 00 00 00 00 00 96 96 96 96 96 96 96 96 96
0000160 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96 96
*
0000220 96 96 96 96 96 96 96 96 96 64 64 64 64 64 64 64
0000240 64 64 64 64 64 64 64 64 64 64 64 64 64 64 64 64
*
0000740 64 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000760 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Please could you suggest why Blacklist output is the entire chromosome.
Thanks.

Hello,
I am assuming that the HG38 blacklist file is specific to the the genome version used in ENCODE (GRCh38_no_alt_analysis_set_GCA_000001405.15). Is this assumption correct?

I'm a little confused about the input/ and mappability/ directory. Is it possible to generate the .sam files if I only have a .fa file of a species like zebrafish? Or is there somewhere to download these files?

There are not mappabilty files of pig genome. We have generated uint8 Umap files based on different k-mers. However, the numbers and genome coverages of blacklist regions we identifed were much different between diffetent k-mers. So how can we choose the most suitable k-mers to identify blacklist ?

Thanks!

For example

chr16   34586660        34587100
chr16   34587060        34587660

chr20   31067930        31069060
chr20   31069000        31069280

This will cause same tools error, for example bamCoverage of deeptools.

Hi!

thanks for your resource.

I have a comment regarding the blacklists for hg38 which you describe at the end of the README.md file. I understand these are the "older" DAC blacklists (not the newer ones which you describe in the 2019 paper 10.1038/s41598-019-45839-z)

I have noticed that the hg38 blacklist which you link to in the README.md is not the same between the two links: the file in the first link is much shorter than the one in the ENCODE website.

• HUMAN (hg19/GRCh38): http://mitra.stanford.edu/kundaje/akundaje/release/blacklists/hg38-human/hg38.blacklist.bed.gz ENCODE portal link: https://www.encodeproject.org/annotations/ENCSR636HFF/ (Select GRCh38)

I checked the mouse files, for example, and they do point to the same files. So I don't know if for hg38 the two links are supposed to also have the same file.

• MOUSE (mm10): http://mitra.stanford.edu/kundaje/akundaje/release/blacklists/mm10-mouse/mm10.blacklist.bed.gz ENCODE portal link: https://www.encodeproject.org/annotations/ENCSR636HFF/ (Select mm10)

Finally, just to be safe that I'm not confused, if I want to use the newer (improved) blacklists, derived from your 2019 paper, these are the ones in the /lists folder containing the "v2" string in their names, right?

thanks for your help!

Hello,

I downloaded the mm10-blacklist.v2.bed.gz file and tried to use that in an argument for the Genrich peak caller (-E mm10-blacklist.v2.bed.gz). However I my script failed with the following message. "Error: Poorly Formatted BED record".

I do not get the error message if I remove the -E mm10-blacklist.v2.bed.gz argument.

Do I need to sort the blacklist BED file before using it?

Thanks.

I wonder why there are so few regions in hg38 v1 bed file ?