objective

• would like to have a plot
  • x-axis time
  • y-axis total length of nucleotides (on chromosome 22)

method

• try to use poretools, porekit, or some other software
• if all of them fails, need to write a script to read fast5 file directly

Decided to keep track on the progress on the collaboration project with James's team on GitHub as well

meetings

2017/1/11

• Presented my project to their team
• https://github.com/rivas-lab/presentations/blob/master/yosuke/20170111_yosuke.pdf

2017/1/23

• Journal club & discussion
• I presented Genome Graph pre-print paper
  • https://github.com/rivas-lab/presentations/blob/master/yosuke/20170123_yosuke.pdf
• They presented Deep Variant

2017/1/30

• discussion
  • Haplotype inference (& phasing) with deep net ??

2017/2/6

• discussion
• we will initially focus on chromosome 20
• Yosuke have pointed out GIAB data set and provided download script for
  James's team so that they can prepare 'label' for their variant caller
• They also have bam file of selected long reads from Nanopore consortium
  consortium data, which will be used to construct pile-up image (input)
  for their deep network
• Yosuke will continue to work on my project to produce posterior
  probability of haplotype given reads

Oxford Nanopore MinION data from Wellcome Trust Centre for Human Genetics

logistics

• http://www.well.ox.ac.uk/ogc/nanopore-human-genome
• downloaded to /share/PI/mrivas/data/WTCHG/
• Learned that Globus is available on data transfer node
  • http://sherlock.stanford.edu/mediawiki/index.php/DTN

data description from README

• DNA was prepared according to the Damaged library improvement protocol
  with size selection using BluePippin 0.75% DF Marker S1 high-pass
  6-10kb v3 protocol with BPstart of 6kb.
• Libraries were generated with the SQK-LSK108 Ligation Sequencing Kit
  1D (R9.4) and sequenced using SpotON R9.4 flowcells on a MinION Mk 1B.
• In our analyses we declare PASS reads to be at least 1 k base in length and have and average quality of 14 or more.
• Reads were mapped to hs37d5lam.fasta including lambda phage control sequences using "bwa mem" with the following options: -x ont2d -t 12 -R "@RG\tID:1\tSM:NA12878" -M.
• Reads were mapped per flow-cell and then merged using samtools merge to produce the final output, comprising bam and bam index (.bai) files. PASS and FAIL reads are provided in separate files.

new data set

• https://github.com/nanopore-wgs-consortium/NA12878
• https://nanoporetech.com/human-genetics/results
• https://nanoporetech.com/about-us/news/update-day-1-nanopore-community-meeting-new-york
• https://nanoporetech.com/about-us/news/nanopore-community-meeting-day-2-writeup

• Need to formulate inference algorithm

• It would be great if we have a python binding of PINK2 library

• R9.4 is not fully supported by the following tools:
  • poretools
  • porekit

• Understand how to use Google Cloud on Sherlock Cluster

Write a script starting from FASTQ and outputs pgen

• current implementation in haplotype inference does not work properly with missing data
  • read_haplotype_dev in this following note has ad-hock solution for this problem
  • https://github.com/rivas-lab/nanopore/blob/2d62b7a0ab9dd6bc67249ace1d888ae245bf454a/dev/scripts/20170213/20170213_Bayesian_online_inference_of_haplotype.ipynb
• Basically, I need to re-write the whole pipeline in fully object-oriented style so that I can treat each read / mapped fragment independently.

• 1000 Genome reference panel has an advantage that they have phased information
• [TODO]
  • Prepare 1KG population reference (pgen)
  • Compute prior probability of haplotype (population frequency)
  • Compute posterior probability given data

objective

• Familiarize to work with nanopore data

procedure

• Follow Helio's pipeline

software installation

• poretools
• samtools
• LAST
• Picard

data

• reference genome (transcriptome)
• nanopore data (of course)

1. How many FAST5 files do we have?
2. Take several sample files, and convert them into FASTQ file
3. Map to the reference genome (hg19)
4. Bring in prior data (UKBB 500k individuals)
5. Take platinum whole genome VCF file to compute $\theta$

• https://www.ncbi.nlm.nih.gov/variation/docs/human_variation_vcf/
• ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606/VCF/

$ wget -O - ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606/VCF/00-All.vcf.gz | zcat - > /home/ytanigaw/PI_HOME/data/dbsnp/dbsnp_all_20160527.vcf
$ awk '(! /^#/){ print "chr"$0 }(/^#/){ print $0 }' dbsnp_all_20160527.vcf > bgzip -c - > dbsnp_all_20160527.vcf.gz

• Read Until haplotyping with sufficient confidence level

references

• work by Matthew Loose
  • http://www.nature.com/nmeth/journal/v13/n9/full/nmeth.3930.html
  • https://vimeo.com/175675134?from=outro-embed

• Install MinoTour on Google Could Engine

plink1.9 failed to convert /share/PI/mrivas/ukbb/download/chr20impv1.bgen with the following error massage:

Error: File read failure.

• script file:
  • https://github.com/rivas-lab/nanopore/blob/20170206_chr20impv1.bgen/dev/scripts/20170206/bgen2pgen.12572552.sh
• stdout file:
  • https://github.com/rivas-lab/nanopore/blob/20170206_chr20impv1.bgen/dev/scripts/20170206/bgen2pgen.12572552.out
• stderr file:
  • https://github.com/rivas-lab/nanopore/blob/20170206_chr20impv1.bgen/dev/scripts/20170206/bgen2pgen.12572552.err

The same script works for chromosome11, so it is likely this file is broken.

Would like to take a benchmark of different mapping algorithms

• bwa mem
• GraphMap
  • https://github.com/isovic/graphmap
• canu
  • https://github.com/marbl/canu
• miniAlign
  • https://github.com/ocxtal/minialign
• DALGNER
  • https://github.com/thegenemyers/DALIGNER
  • This is for de novo assembly, but list it here because it's famous

ToDo

• prepare benchmark data set (sample from Nanopore consortium data with read length filter)
• install softwares
• take benchmark

To run nanopolish, you need to have c++ dynamic libraries, which you can activate by loading gcc module on sherlock cluster.

[ytanigaw@sherlock-ln01 login_node ~/nanopolish]$ ml purge
[ytanigaw@sherlock-ln01 login_node ~/nanopolish]$ ./nanopolish_test 
./nanopolish_test: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.7' not found (required by ./nanopolish_test)
./nanopolish_test: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./nanopolish_test)
./nanopolish_test: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by ./nanopolish_test)
./nanopolish_test: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./nanopolish_test)
[ytanigaw@sherlock-ln01 login_node ~/nanopolish]$ ml load gcc
[ytanigaw@sherlock-ln01 login_node ~/nanopolish]$ ./nanopolish_test 
===============================================================================
All tests passed (427 assertions in 5 test cases)

Deepnano

• base caller with RNN
• references
  • https://arxiv.org/abs/1603.09195
  • http://compbio.fmph.uniba.sk/deepnano/
  • https://bitbucket.org/vboza/deepnano

• generate plots with poretools
  • https://github.com/arq5x/poretools

• Pacbio data
  • Download
  • select chr 11 and 20 with samtools
• chr11 >20kb
• chr20 >10kb
• start with chr11

• Understand the BGEN file format
• Be able to manipulate haplotype reference data stored in BGEN file.
  • extract the region of interest, etc..