tarah28 / kmerid Goto Github PK
View Code? Open in Web Editor NEWThis project forked from ukhsa-collaboration/kmerid
License: MIT License
This project forked from ukhsa-collaboration/kmerid
License: MIT License
Table of content ---------------- * Prerequisites * Installation * Set up reference genome groups * Running KmerID * KmerID output * Examples Prerequisites ------------- Python version >= 2.6.6 (not Python 3) The following Python libraries are required: sys, argparse, subprocess, os, operator, glob, ConfigParser, tempfile, scipy Installation ------------ 1.) Unpack the tar.gz archive that you downloaded into a folder of your choosing. This folder will be called $KMERROOT in the following. 2.) Navigate to $KMERROOT. You should see, among others, subfolders src, config and bin and two Python scripts. 3.) Compile the auxilliary executables by typing make all. make all This should create the files intersect_kmer_lists_filelist, kmer_jaccard_index, kmer_reads_process_stdin, and kmer_refset_process in the bin folder. You still need to prepare your reference genome sets before you can run the software. Set up reference genome groups ------------------------------ KmerID will compare your fastq reads against one or more sets of reference genomes. We have been using it in such a way each set of reference genomes contains only genomes of species from a single bacterial genus. KmerID will first compare your fastq reads again 3 representatives of each group. On the basis of this it will determine against which group(s) to further compare your reads. There are no limits how many groups and how many genomes in each group you can have, but it will take longer the more you have. Each group of reference genomes needs to be set up using the setup_refs.py utility: usage: setup_refs.py [-h] -f FILE -n FILE -c FILE version 0.1, date 12Feb2014, author [email protected] optional arguments: -h, --help show this help message and exit -f FILE, --folder FILE REQUIRED: Folder that contains a set of fasta files with one reference genome each. -n FILE, --name FILE REQUIRED: Unique name for this set of references. Case insensitive. [e.g. salmonella] -c FILE, --config FILE REQUIRED: Configuration file. Usually config/config.cnf. e.g. setup_refs.py -n salmonella -f ref/genus01 -c config/config.cnf Please note: The files in the references' folder need be in fasta format and have one of these file endings .fa, .fna, .fas, .fasta. They can be optionally gzipped (adding.gz) at the end of the file name. This step includes a hierarchical clustering step of all genomes in the group. This includes the creation of an all-by-all similarity matrix (stored under $KMERROOT/config/) for the genomes in the respective group. Therefore this step will take a significant amount of time for larger groups. Note: It is recommended to keep the number of genomes per group under 15. If larger groups are required, the all-by-all simmilarity matrix for this group should be computed in a parallel manner on a HPC infrastructure. Example groups of genomes for a variety of genera of pathogenic bacteria are part of this download.. After each group has been setup a config file in the config subfolder is updated. This file is a required input to the main programme. Running KmerID -------------- After setting up your reference groups, run Kmerid like this: usage: kmerid.py [-h] -f FILE -c FILE [-n] version 0.1, date 12Feb2014, author [email protected] optional arguments: -h, --help show this help message and exit -f FILE, --fastq FILE REQUIRED: Investigate this fastq file. -c FILE, --config FILE REQUIRED: Configuration file. Usually config/config.cnf. -n, --nomix Do not investigate sample for mixing. [default: Investigate. (Takes about 2 minutes.)] e.g. python kmerid.py -f reads.fastq --config=config/config.cnf KmerID output ------------- KmerID writes a table to stdout containing three columns, the similarity of the reads to the reference genome, the groups the genome is in, and the file name of the genome (without fasta file ending). The table contains one line per genome is the groups that are candidates. If one candidate group contains more than 40 genomes, 40 representatives from the group are chosen and shown. Candidate groups are selected by comparing 3 representatives of each group against the read set and selecting all groups accouring within the top 5 hits. Optional: The KmerID tool will write another table to stdout. This table can give an indication whether the possibility exists that the reads in the sample stem from two different organisms. This table contains the following four columns. 1.) absolute difference between two similarities: a) similarity between the reads and respective genome b) similarity between the top hit genome and the respective genome 2.) the non-absolute difference between the similarities (a-b) 3.) the groups the genome is in 4.) the file name of the genome (without fasta file ending) Examples -------- a) Finding the closest genome in a single group: * Make sure the reference genomes from the genus Legionella are present in the folder $KMERROOT/refs/Legionella * Prepare these reference genomes for KmerID by calling python setup_refs.py -n legionella -f $KMERROOT/ref/Legionella/ -c config/config.cnf * Process the sample data by calling: python kmerid.py -f samples/Legionella_pneumophila_example_reads.fastq.gz -c config/config.cnf -n Output: ------- #Kmer based similarities #similarity groups file 98.377968 legionella Legionella_pneumophila_str_Paris.fa 71.318146 legionella Legionella_pneumophila_subsp_pneumophila_NC_018140.fa 66.087273 legionella Legionella_pneumophila_subsp_pneumophila_str_Hextuple_3a.fa 66.087234 legionella Legionella_pneumophila_subsp_pneumophila_str_Hextuple_2q.fa 64.594894 legionella Legionella_pneumophila_subsp_pneumophila_NC_018139.fa 63.234699 legionella Legionella_pneumophila_str_Corby.fa 62.958763 legionella Legionella_pneumophila_2300_99_Alcoy.fa 62.147953 legionella Legionella_pneumophila_subsp_pneumophila_str_Philadelphia_1.fa 61.927429 legionella Legionella_pneumophila_subsp_pneumophila_ATCC_43290.fa 59.857174 legionella Legionella_pneumophila_subsp_pneumophila_str_Thunder_Bay.fa 56.115902 legionella Legionella_pneumophila_str_Lens.fa b) Identify unknown sample against multiple groups: * Make sure all reference genomes are present in subfolders of $KMERROOT/ref * Prepare all reference groups for KmerID by calling for s in `ls $KMERROOT/ref`;do python setup_refs.py -f $KMERROOT/ref/$s -c config/config.cnf -n $s; done * Process the sample data by calling: python kmerid.py -f $KMERROOT/samples/unknown_pathogen_example_reads.fastq.gz -c $KMERROOT/config/config.cnf -n Output: ------- #Kmer based similarities #similarity groups file 90.137718 pseudomonas Pseudomonas_aeruginosa_PAO1_uid57945.fa 5.215832 pseudomonas Pseudomonas_resinovorans_NBRC_106553_uid208671.fa 4.093119 pseudomonas Pseudomonas_fulva_12_X_uid67351.fa 3.923861 pseudomonas Pseudomonas_stutzeri_DSM_10701_uid170940.fa 3.875630 pseudomonas Pseudomonas_stutzeri_DSM_4166_uid162113.fa 3.717735 pseudomonas Pseudomonas_mendocina_NK_01_uid66299.fa 3.383106 pseudomonas Pseudomonas_fluorescens_CHA0_uid203393.fa 2.662693 pseudomonas Pseudomonas_putida_F1_uid58355.fa 2.155928 pseudomonas Pseudomonas_fluorescens_SBW25_uid158693.fa 1.383858 pseudomonas Pseudomonas_syringae_phaseolicola_1448A_uid58099.fa 0.938764 mycobacterium Mycobacterium_JDM601_uid67369.fa 0.874562 mycobacterium Mycobacterium_intracellulare_ATCC_13950_uid167994.fa 0.844299 mycobacterium Mycobacterium_MCS_uid58465.fa 0.749032 mycobacterium Mycobacterium_smegmatis_MC2_155_uid57701.fa 0.567610 mycobacterium Mycobacterium_tuberculosis_H37Rv_uid170532.fa 0.561620 mycobacterium Mycobacterium_liflandii_128FXT_uid59005.fa 0.550582 mycobacterium Mycobacterium_rhodesiae_NBB3_uid75107.fa 0.539385 mycobacterium Mycobacterium_smegmatis_JS623_uid184820.fa 0.451246 mycobacterium Mycobacterium_massiliense_GO_06_uid170732.fa 0.192369 mycobacterium Mycobacterium_leprae_Br4923_uid59293.fa c) Investigate contaminated sample: * Make sure references for all groups have been set up under example b) * Process the sample by calling python kmerid.py -f $KMERROOT/samples/contaminated_sample_example.fastq.gz -c $KMERROOT/config/config.cnf Output: ------- #Kmer based similarities #similarity groups file 84.077347 staphylococcus Staphylococcus_aureus_uid193761.fa 80.150398 enterococcus Enterococcus_faecalis_OG1RF_uid54927.fa 76.079369 enterococcus Enterococcus_faecalis_Symbioflor_1_uid183342.fa 72.809959 enterococcus Enterococcus_7L76_uid197170.fa 71.491592 enterococcus Enterococcus_faecalis_D32_uid171261.fa 70.721642 staphylococcus Staphylococcus_aureus_71193_uid162141.fa 67.780098 enterococcus Enterococcus_faecalis_V583_uid57669.fa 19.011084 staphylococcus Staphylococcus_aureus_MSHR1132_uid89393.fa 2.557667 staphylococcus Staphylococcus_warneri_SG1_uid187059.fa 2.151581 staphylococcus Staphylococcus_haemolyticus_JCSC1435_uid62919.fa 2.100819 staphylococcus Staphylococcus_epidermidis_RP62A_uid57663.fa 1.779690 staphylococcus Staphylococcus_lugdunensis_N920143_uid162143.fa 1.615321 staphylococcus Staphylococcus_saprophyticus_ATCC_15305_uid58411.fa 1.358824 staphylococcus Staphylococcus_carnosus_TM300_uid59401.fa 1.189253 staphylococcus Staphylococcus_pseudintermedius_ED99_uid162109.fa 0.951751 enterococcus Enterococcus_hirae_ATCC_9790_uid70619.fa 0.859111 enterococcus Enterococcus_faecium_NRRL_B_2354_uid188477.fa 0.832898 enterococcus Enterococcus_faecium_DO_uid55353.fa 0.792562 enterococcus Enterococcus_faecium_Aus0085_uid214432.fa 0.612133 enterococcus Enterococcus_casseliflavus_EC20_uid55693.fa #Mixing analysis: #Top hit - Group: staphylococcus File: Staphylococcus_aureus_uid193761.fa Similarity: 84.077347% #Comparison of results: #sim diff absolute sim(reads,thisfile)-sim(tophit,thisfile) group file 79.904867 79.904867 enterococcus Enterococcus_faecalis_OG1RF_uid54927.fa 75.834908 75.834908 enterococcus Enterococcus_faecalis_Symbioflor_1_uid183342.fa 72.577836 72.577836 enterococcus Enterococcus_7L76_uid197170.fa 71.258053 71.258053 enterococcus Enterococcus_faecalis_D32_uid171261.fa 67.514926 67.514926 enterococcus Enterococcus_faecalis_V583_uid57669.fa 0.738434 0.738434 enterococcus Enterococcus_hirae_ATCC_9790_uid70619.fa 0.643232 0.643232 enterococcus Enterococcus_faecium_DO_uid55353.fa 0.62082 0.62082 enterococcus Enterococcus_faecium_NRRL_B_2354_uid188477.fa 0.582432 -0.582432 staphylococcus Staphylococcus_epidermidis_RP62A_uid57663.fa 0.578488 0.578488 enterococcus Enterococcus_faecium_Aus0085_uid214432.fa 0.481429 0.481429 enterococcus Enterococcus_casseliflavus_EC20_uid55693.fa 0.409216 -0.409216 staphylococcus Staphylococcus_aureus_MSHR1132_uid89393.fa 0.302525 -0.302525 staphylococcus Staphylococcus_haemolyticus_JCSC1435_uid62919.fa 0.28907 -0.28907 staphylococcus Staphylococcus_aureus_71193_uid162141.fa 0.149772 0.149772 staphylococcus Staphylococcus_lugdunensis_N920143_uid162143.fa 0.132699 0.132699 staphylococcus Staphylococcus_carnosus_TM300_uid59401.fa 0.089541 0.089541 staphylococcus Staphylococcus_pseudintermedius_ED99_uid162109.fa 0.016231 -0.016231 staphylococcus Staphylococcus_warneri_SG1_uid187059.fa 0.011532 0.011532 staphylococcus Staphylococcus_saprophyticus_ATCC_15305_uid58411.fa
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.