should be able to filter on arbitrary list of identifiers (listed as the first space-separated field of lines in a text file), identifiers from a fasta file, identifiers from an existing tree, ... what ever else makes sense.

we've have several requests for this and other filtering scripts have been very useful (e.g., filter_fasta.py)

this will allow for better structuring of scripts in the interfaces and will include terms such as 'core' (i.e., the workflows), 'utility', ...

convenient for datebase and core analysis script

Currently, this script does not work on my computer or Greg's, so we need to fix the issue so it behaves like other matplotlib visualizations in QIIME. Also, need to update the output of the files to not use mapping file values.

needs to be updated to reflect recent changes

Should validate mapping files by default, and have options to not throw error on missing Barcode and LinkerPrimer sequences.

model_fp in qiime_config should be easy.

Function should have handle:

inflate_denoiser_output(centroid_seqs,singleton_seqs,denoiser_map)

and yield fasta records. Some notes from an email on this subject:

[The idea is that we want to] run denoiser in the normal way (following Jens's tutorial) to
just before the cat centroids/singletons step. Then, instead of
catting directly, write each centroid sequence n times, where n is the
number of reads in that cluster, and write each singleton once. While
writing these out you'd also map back to original sequence
identifiers. Then pass that directly to pick_otus.py -m uclust. That
wrapper now presorts by abundance and pre-collapses identical reads,
so I think it would take care of the sort_denoiser_output.py step, and
there'd be no merge_denoiser_output.py step required as you already
have the right number of reads with the right sequence identifiers.

The benefit of this is that we could make denoising a completely
separate step, which would allow us to use the
pick_otus_through_otu_table.py and core_analyses.py workflows in the
same way regardless of whether denoising is being performed. We could
also then write a separate denoiser workflow script to facilitate that
process. The downside is that we'd be expanding the denoiser fasta
output only to collapse it right back down again, but that is a really
minor issue compared to the amount of headaches this would save if it
worked. (We could also separate the processes, I think, by passing the
original fna and denoisier mapping to pick_otus_through_otu_table.py
and calling merge_denoiser_output.py only if those are present. If
both of these would work I might prefer the former which would have
slightly more compute time but less interface complexity.)

this is a generally useful script.

using the choice option type with the append action could
be useful for things like lists of metrics to beta_diversity.py, and
would let us generate multiselect fields for these types of values in
the web interface. See the --metrics option to this script and try
passing invalid metrics.

need to look into the feasibility of this, but would be a nice simplification of the interfaces.

This is hackish and seems relevant only to cluster environments (not e.g., ec2 or multicore), so should be wrapper in the cluster_jobs.py files rather than the jobs generation code itself. Some cluster_jobs scripts strip these from the commands before running them which is a really ugly/annoying way to handle the situation.

excluding these should be default as most people (Greg and Jesse at least) turn these off every time -- note that the pdfs still get created and are linked, they're just not displayed.

steps that require category will be suppressed

the cluster_quality.py results would make more sense as components of the bdiv output directory in the core_qiime_analyses.py script output

as this object is being used in more applications we need a single source of documentation for all currently used keys in the dictionary

waiting on python 2.7 for conditional unit tests.

can the make_prefs_file code take all taxonomic summaries and auto-generate a pref files that would cover all taxa?

should use the core_qiime_analyses.py workflow script and possibly incorporate some filtering of the data set, etc.

this will be useful for better documentation of scripts as well as easier identification of scripts of interest (for example by having a tag-based browsing section of the docs where users can search for scripts that generate OTU tables or which take OTU maps, ...)

move this functionality from beta_diversity_through_3d_plots.py to make_3d_plots.py so it's generally available. use the qiime.util.get_interesting_mapping_fields function for this.

currently a lot of code just uses the default (/tmp) which is not ideal, and screws up in some cluster environments where the user may not have write access to /tmp.

these will not go into QIIME, but will be for more robust testing on our end. these tests should use the core_qiime_analyses.py, and this could possibly be release as a more real-world tutorial data set.

1. this script does not allow for underscores in the column headers of the mapping file. 2) the output of the script uses values from the mapping file, which will create issues when there are large values in the mapping file. 3) Need to cleanup the output of the monte-carlo statistics 4) need to lower dpi and update html to work with large values in the legend 5) change output filename

Analogous to make_distance_histograms. Possible display items: distances within each group, distances between each pair of groups, overall within-group distances, overall between-group distances.

In order to support more intuitive handling of options in
web/cocoa/n3phele interfaces, we need to be able to differentiate
file/directory paths from normal string options so we can e.g. open a
browse dialog for users to input a file. After looking into this, I
think the way to go is to add a few new option types into PyCogent --
these will then be available in QIIME and other projects. The new
option types would be: "existing_filepath", "new_filepath",
"existing_dirpath", and "new_dirpath".

I've attached an example script where I implement the ones that I
think would be necessary to get us started. The plan would be to put
this in PyCogent and then do a mini code sprint to propagate the
changes through QIIME. All code would be backwards compatible (i.e.,
we're not getting rid of old option types, just making new ones
available). These also let us centralize some error checking on input
options like checking for an input file's existence (or an output
file's lack thereof).

gavin added an ability to group at an additional level (below required/optional) that may be useful for this (e.g., under optional arguments could have "rdp-specific", "blast-specific", and "general" options). would need some tweaking in parse_command_line_parameters to support this directly -- currently only used in PyCogent GUI interface.

something we talked about along time ago for general quality control of reads:

given an input fasta file in post-split_libraries format:

• if good blast hit, keep
• if no good blast hit, and seq shows up in multiple samples, keep
• if no good blast hit, and seq shows up in single samples, discard

will involve extensive testing of code to ensure that all code that parses OTU tables is calling parse_otu_table.

allow user to specify which of these are created:

• 3d plots
• 2d plots
• distance histograms
• cluster_quality (maybe? figure out how to make params optional)

currently these must be converted to the format initially received from illumina, and qiime doesn't do a good job of supporting that conversion.

additionally split the quality filtering and demultiplexing code at this stage.