Comments (14)
Dear @pgrosu and @kishwarshafin,
Thank you very much for the added information. This certainly helps me along.
Kind regards,
from deepvariant.
We'd be happy to exclude commonly known non-analyzable contigs on b38. Do you have a proposed list of such contig names, along with why they should be excluded?
The MT: we don't expect, like most variant callers, to properly handle the structure of genetic variation within the mitochrondira. You can read a bit about the challenges of MT calling here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4201154/
from deepvariant.
Thanks! Attached is a list from ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome/GRCh38_full_analysis_set_plus_decoy_hla.fa
This list has all the GRCh38 contigs after dropping chr1-chr22, chrX, chrY and chrM.
GRCh38 has lots of additional contigs which are described here:
https://software.broadinstitute.org/gatk/documentation/article.php?id=7857
https://software.broadinstitute.org/gatk/blog?id=8180
from deepvariant.
Thanks @jjfarrell. We've added a bug and will fix for the next release.
from deepvariant.
Nice! This is fixed now and will go out with the next release of DeepVariant. Thanks for the great suggestion.
from deepvariant.
@rpoplin @depristo Is there a specific reason to exclude all of the super contigs in GRCh38 from deepvariant analysis? I agree with removing the HLA, EBV, M/MT and decoys, but removing the placed and unplaced contigs doesn't seem justified. Other callers do work on these regions so it doesn't seem like they are truly "Common problematic contigs on GRCh38" as stated in the release notes for v0.5.0.
Does it affect training or calling if they were included? I'd suggest the user should be able to provide the list of regions to exclude or include as its taken a bit of work to sort out why deepvariant is not making calls on all contigs in the provided BAM files like a number of other callers we are testing
from deepvariant.
Thanks @PedalheadPHX
Reopening this bug so it's easier to track that we have an unanswered question.
Also assigning to @AndrewCarroll
from deepvariant.
Hi @jjfarrell
I hadn't encountered users who meaningfully used the unplaced, but non-decoy contigs. The exclude list is current coded into DeepVariant and not modifiable. However, I agree that it would be reasonable to assess adding those in.
The main reason a decoy or unplaced contig should be excluded is if it aggregates a large amount of incorrectly mapped coverage. It makes sense for us to investigate how much the placed and unplaced contigs are affected by that and decide based on those results.
Training will not be affected, because these contigs do not have truth label variants that I am aware of. The model will be the same.
We'll consider this a potential change for the next release, which will hopefully be in the next month or two. I hope that timing will be adequate.
from deepvariant.
Dear @AndrewCarroll ,
I know this issue was closed, but I did not want to open a new one for a question this trivial. Do I understand correctly the team has now implemented some form of alt contig consideration, as the documentation now says: "Second, the BAM file provided to --reads should be aligned to a "compatible" version of the genome reference provided as the --ref. By compatible here we mean the BAM and FASTA share at least a reasonable set of common contigs, as DeepVariant will only process contigs shared by both the BAM and reference. "" [https://github.com/google/deepvariant/blob/r1.5/docs/deepvariant-details.md].
Thanks in advance
from deepvariant.
Dear @ESDeutekom,
There are a few things to unpack here. The goal is to make sure that the BAM file and reference FASTA file match in their contigs and bases to ensure correct variants get called between the two, and if local realignment is required that the regions match appropriately. Thus, a couple of steps get performed:
https://github.com/google/deepvariant/blob/r1.5/deepvariant/exclude_contigs.py
This is done in the reference first, because most likely these would not exist in most BAM files.
_ensure_consistent_contigs
function.
The two steps summarize the statement you noticed in the documentation.
Now the only excluding option I see is for exclude_regions
, though the rest of the above excluded contigs are hard-coded as exclude_contigs=exclude_contigs.EXCLUDED_HUMAN_CONTIGS
. To exclude a region, that would come as a space-separated list of regions, which can be region literals (e.g., chr20:10-20
) or paths to BED/BEDPE files.
Hope it helps,
Paul
from deepvariant.
Dear @AndrewCarroll ,
I know this issue was closed, but I did not want to open a new one for a question this trivial. Do I understand correctly the team has now implemented some form of alt contig consideration, as the documentation now says: "Second, the BAM file provided to --reads should be aligned to a "compatible" version of the genome reference provided as the --ref. By compatible here we mean the BAM and FASTA share at least a reasonable set of common contigs, as DeepVariant will only process contigs shared by both the BAM and reference. "" [https://github.com/google/deepvariant/blob/r1.5/docs/deepvariant-details.md].
Thanks in advance
Hi @ESDeutekom ,
Note that the documentation you quoted actually has been there for a while now, since 2018: https://github.com/google/deepvariant/blob/r0.6/docs/deepvariant-details.md
If this wording is confusing, we're happy to improve it. Please let us know which part is particularly confusing.
If your question is whether DeepVariant takes alt contigs into account, the answer also hasn't changed - not really. In general, when choosing between mapping to GRCh38 with alt contigs or without alt contigs, we empirically find that it's better to use the latter.
Hope this helps.
from deepvariant.
Thank you for your answers. This does help me further along. With regards to the following:
In general, when choosing between mapping to GRCh38 with alt contigs or without alt contigs, we empirically find that it's better to use the latter.
Is this stated somwhere, or published with comparative results? I am curious to get some insight and elaboration on why this would happen.
Kind regards,
Eva
from deepvariant.
@ESDeutekom , this article https://lh3.github.io/2017/11/13/which-human-reference-genome-to-use from Heng describes different caveats of using different reference genomes. As suggested in that article and from our previous experience we found GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz
to be the most suitable.
from deepvariant.
Hi Eva,
Usually such comparisons are rare, as Kishwar suggested through Heng's article. If you are curious about such a comparison, below is a table generated on GIAB samples through GATK's HaplotypeCaller on GRCh38 and GRCh38 with masked alt regions:
Hope it helps,
~p
from deepvariant.
Related Issues (20)
- ValueError: Reference contigs span 3137161264 bases but only 63025520 bases (2.01%) were found in common among our input files. Check that the sources were created on a common genome reference build. HOT 1
- I read DeepVariant essay and want to download some data. HOT 4
- Error while running tests on Calling variants in non-autosomal contigs HOT 4
- Memory issue while running deepvariant_1.6.0.sif with ONT_R104 HOT 7
- Program always run HOT 3
- DeepVariant running slow HOT 1
- AssertionError: Some objects had attributes which were not restored: HOT 4
- I have a question about training DeepVariant. HOT 2
- Source code compilation of DeepVariant failed HOT 3
- What is deepvariant trained on? HOT 3
- DeepVariant Channel clarification HOT 2
- Info Singularity HOT 1
- dealing with chimeric alignments HOT 3
- Question about RNA-seq HOT 5
- Running with Conda HOT 4
- Checkpoint "Model files do not exist" when testing custom model HOT 1
- make_examples_core.py gets stuck on STAR aligned rnaseq BAM (deepvariant 1.6.1) HOT 1
- No output files (VCFs) for all genome HOT 7
- No output vcf obtained from custom model HOT 1
- [libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepvariant.