Description

A file browser for choosing what files to load or save

Objectives

• Launch of the Modal from the main menu
• Modal view for selecting files on the users hard disk
• Side panel for navigating the directory
• Main table view for viewing file properties
• Filter for file extensions
• Search for file names

Tests

• Working example

Description

<thread> and <future> can implemented to allow for parallel processing of RawData in a sequence.

Objectives

• implement parallel processing of "sequence processing" and "sequence segment processing" pipelines
• add a std::cout redirection to the OpenMS log ouptuts
• add an option to set the number of threads per workflow

Validation

• Unit tests passing
• Check that cores are being invoked
• Check for data race conditions and memory leaks

I will collect info from multiple PRs into this issue:

• we are using #pragma once instead of define guards
• we keep using OpenMS:: namespace explicitly, avoiding using namespace OpenMS;
• using namespace std; is fine where it actually increases readability
• convert python to c++ first, and then take care of improving code
• rename "Targeted" to "TargetedExperiment" for identifiers and methods in code
• .cpp/.h C++ files should indented using two spaces
• .cmake (or other cmake related files), build scripts, and docker files should be indented using tab/4 spaces
• will need to correct EOF consistency

Description

A method to alert the user to errors found in the data integrity check. The method should also present to the user a review of what components and samples will be analyzed so that the user can go back and correct any mistakes before running the workflow.

Objectives

• list out components that will be analyzed
• list out samples that will be analyzed (including their type)
• list out samples and components with a calibration curve
• list out components with a quantitation method
• list out any discrepancies found in the Data integrity check

Validation

• passing unit tests with examples of each of the objectives

Description

Automated external dependencies build using CMake add_externalProject and "Superbuild"

Objectives

• imgui build
• eigen3 build
• boost.test build
• openms build

Generic documentation

Wikipedia: https://en.wikipedia.org/wiki/Tagged_union
cppreference.com (Union-like classes paragraph): https://en.cppreference.com/w/cpp/language/union

Issue

Suppose we have a union-like class defined as:

struct CastValue
{
  enum {
    FLOAT,
    INT,
    STRING,
  } tag;
  union
  {
    float f;
    int i;
    std::string s;
  };
};

If one tries to instantiate a variable of type CastValue as in:

CastValue c;

The compiler will complain:

error: use of deleted function ‘SmartPeak::Utilities::CastValue::CastValue()

and:

note: ‘SmartPeak::Utilities::CastValue::CastValue()’ is implicitly deleted because the default definition would be ill-formed

and more:

/home/pasdom/SmartPeak2/src/smartpeak/include/SmartPeak/core/Utilities.h:69:21: error: union member ‘SmartPeak::Utilities::CastValue::<unnamed union>::s’ with non-trivial ‘std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string() [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]’
         std::string s;

What is happening

Usually we can take advantage of default constructors as in CastValue() = default;. In this case the compiler needs some help understanding how to construct the object:

• what value tag should have
• which union member should be initialized

I don't know if the issue is simply caused by one of the union's members having non-trivial constructors (std::string).

But what happens is that the compiler deletes the implicit constructors (and destructor) and forces us to define them.

Solution

Simply define the default constructor and destructor.

CastValue() : tag(UNKNOWN), s() {}
~CastValue() {}

Bonus

In case one tries to push_back() (vector) or insert() (map) a variable of this type into a container, a new long error will pop up:

In file included from /home/pasdom/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/c++allocator.h:33:0,
                 from /home/pasdom/local/include/c++/7.3.0/bits/allocator.h:46,
                 from /home/pasdom/local/include/c++/7.3.0/string:41,
                 from /home/pasdom/OpenMS/src/openms/include/OpenMS/DATASTRUCTURES/String.h:41,
                 from /home/pasdom/OpenMS/src/openms/include/OpenMS/ANALYSIS/QUANTITATION/AbsoluteQuantitationMethod.h:38,
                 from /home/pasdom/SmartPeak2/src/smartpeak/source/io/OpenMSFile.cpp:3:
/home/pasdom/local/include/c++/7.3.0/ext/new_allocator.h: In instantiation of ‘void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [wit
h _Up = std::pair<const std::__cxx11::basic_string<char>, SmartPeak::Utilities::CastValue>; _Args = {const std::__cxx11::basic_string<char, std::cha
r_traits<char>, std::allocator<char> >&, SmartPeak::Utilities::CastValue&}; _Tp = std::_Rb_tree_node<std::pair<const std::__cxx11::basic_string<char
>, SmartPeak::Utilities::CastValue> >]’:
/home/pasdom/local/include/c++/7.3.0/bits/alloc_traits.h:475:4:   required from ‘static void std::allocator_traits<std::allocator<_Tp> >::construct(
std::allocator_traits<std::allocator<_Tp> >::allocator_type&, _Up*, _Args&& ...) [with _Up = std::pair<const std::__cxx11::basic_string<char>, Smart
Peak::Utilities::CastValue>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, SmartPeak::Utilities::
CastValue&}; _Tp = std::_Rb_tree_node<std::pair<const std::__cxx11::basic_string<char>, SmartPeak::Utilities::CastValue> >; std::allocator_traits<st
d::allocator<_Tp> >::allocator_type = std::allocator<std::_Rb_tree_node<std::pair<const std::__cxx11::basic_string<char>, SmartPeak::Utilities::Cast
Value> > >]’
/home/pasdom/local/include/c++/7.3.0/bits/stl_tree.h:626:32:   required from ‘void std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_M_cons
truct_node(std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_Link_type, _Args&& ...) [with _Args = {const std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >&, SmartPeak::Utilities::CastValue&}; _Key = std::__cxx11::basic_string<char>; _Val = std::pair<const 
std::__cxx11::basic_string<char>, SmartPeak::Utilities::CastValue>; _KeyOfValue = std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, 
SmartPeak::Utilities::CastValue> >; _Compare = std::less<std::__cxx11::basic_string<char> >; _Alloc = std::allocator<std::pair<const std::__cxx11::b
asic_string<char>, SmartPeak::Utilities::CastValue> >; std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_Link_type = std::_Rb_tree_node<std
::pair<const std::__cxx11::basic_string<char>, SmartPeak::Utilities::CastValue> >*]’
/home/pasdom/local/include/c++/7.3.0/bits/stl_tree.h:643:21:   required from ‘std::_Rb_tree_node<_Val>* std::_Rb_tree<_Key, _Val, _KeyOfValue, _Comp
are, _Alloc>::_M_create_node(_Args&& ...) [with _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, Sma
rtPeak::Utilities::CastValue&}; _Key = std::__cxx11::basic_string<char>; _Val = std::pair<const std::__cxx11::basic_string<char>, SmartPeak::Utiliti
es::CastValue>; _KeyOfValue = std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, SmartPeak::Utilities::CastValue> >; _Compare = std::
less<std::__cxx11::basic_string<char> >; _Alloc = std::allocator<std::pair<const std::__cxx11::basic_string<char>, SmartPeak::Utilities::CastValue> 
>; std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_Link_type = std::_Rb_tree_node<std::pair<const std::__cxx11::basic_string<char>, Smart
Peak::Utilities::CastValue> >*]’

(cutting the error here; it's pages long)

Solution

A copy constructor is needed.

CastValue(const CastValue& other) {
  tag = other.tag;
  switch (other.tag) {
    case UNKNOWN:
    case STRING:
      s = other.s;
      break;
    case FLOAT:
      f = other.f;
      break;
    case INT:
      i = other.i;
      break;
  }
}

Description

Conversion of MRMFeatureValidation from .py to .cpp. In addition, load_validationData from FileReaderOpenMS class will need to be implemented to read validation data from .csv. Please note that the functionality of load_validationData corresponding to accessing data from a database does not need to be implemented (i.e., db_json_i parameter can be ignored).

Objectives

• re-implementation of all methods from MRMFeatureValidation
• re-implementation of load_validationData without DB access functionality

Validation

• converted and passing unit tests

Objectives

• Continuous integration using Jenkins and Dart
• Unit testing using Boost.Test library
• Continuous documentation building using Doxygen

Description

Table to manage the injection order and injection properties

Objectives

• Ability to quickly view by sample, sample group, or sequence group
• Ability to add or remove samples to or from sample groups or sample sequences
• Ability to modify sample metadata (e.g., a radio button to change the sample_type attribute)
• Drop down to add tags
• Colored buttons to indicate labels and groupings with “x” to remove tags
• Unit tests for internal logic

Validation

• Passing unit tests
• Working example

Objectives

• conversion of methods from .py to .cpp
• conversion of tests

Objectives

• conversion of methods from .py to .cpp
• conversion of tests

Description

As a developer, I would like to have checks for each pull request that SmartPeak builds on unix systems (linux/macOS), and also passes all unit tests

Objectives

• CircleCI integration and notification on each PR
• OpenMS and SmartPeak2 build integration
• Unit test integration
• Example test integration (?)

Validation

• CircleCI check notification on each PR
• Working build and run on develop

Description

A method that inspects each of the user input data files for typos, naming inconsistencies, or missing files.

Objectives

• ensure sample names are consistent across sequence and standardsConcentrations files
• ensure component names are consistent across TraML ("transition_name"), featureFilter, featuerQC, quantitationMethod, and standardConcentrations files
• ensure component name groups are consistent across TraML ("transition_group_id"), featureFilter, and featureQC files
• ensure all filenames were found
• ensure Heavy components match IS in standardConcentrations ("IS_component_name") and quantitationMethods ("IS_name") files ( FROM quant TO standards)

Validation

• passing unit tests with examples for each of the objectives

Description

Seamless integration with pwiz msconvert to convert proprietary data files to .mzML.

Objectives

• Nice to have Automated raw data upload from the instrument to a data store server (on site or in the cloud)
• integration with msconvert from pwiz to convert files to .mzML

Objectives

• conversion of methods from .py to .cpp
• conversion of tests

I'm having facing issues with SequenceProcessor tests, and I noticed that SequenceHandler tests are not implemented.

Better take care of this before continuing with SequenceProcessor.

@dmccloskey
What are these for?

OpenMS::MSExperiment msExperiment_;
OpenMS::MSExperiment chromatogram_map_;
OpenMS::TransformationDescription trafo_;
OpenMS::MSExperiment swath_;
std::map<std::string, float> validation_metrics_;

I noticed I'm not using them in the current implementation of the class.

Description

Optionality for re-zeroing the baseline for HPLC data should be added to loadMzML method. If deemed necessary, the loadMzML method can also be refactored into smaller sub methods for easier testing.

Objectives

• option to call the re-zeroing baseline methods for HPLC data
• refactor of loadMzML into smaller sub-methods (only if necessary)

Validation

• passing unit tests

Description

Initial .cpp repository for smartPeak with CMake and other build files

Objectives

• initial repository with folder structure
• CMake files

Description

A UI for configuring a report of the processed data using pre-built templates or a user specified template

Objectives

• Selection of sample types to include in the report (as checkbox)
• Selection of metadata to include in the report (as checkbox)
• Option to export as a database table (FeatureSummary) or as a pivot table (SequenceSummary)
• Unit tests for internal data handling methods

Tests

• Passing unit tests
• Validation

• load input from .csv into DB
• save output into DB
• plot from DB

Description

As a user, I would like to have support on MacOS, Windows, and Linux

Objectives

• Dedicated server from DTU and web domain
• Scripts for CI with Jenkins for MacOS, Windows, and Linux

Description

A list of refactors and optimizations that can be made to the converted C++ smartPeak code base.

Optimizations

• OpenMSFile::loadMSExperiment() needs to be discussed and eventually split into smaller functions that try solving fewer tasks.

  • Break into internal submethods loadMZML_, ExtractChromatograms_, MapChromatograms_

• Refactor of CastValue code and generic structures

  • Move Utilities::CastValue to its own class files called DataStructures (i.e. CastValue instead of being in Utilities::CastValue). Implement extensive tests on it.
  • implement typedef for very-long types (e.g., parameters, etc)

Description

Port of python implemented classes and workflows into C++ for seamless native integration and deployment

Objectives

• port existing .py classes to C++ (modules for io, core, tests, and examples)
• port .py algorithms to C++ (waiting on Svetlana)

Validation

• working tests ported from .py
• working examples ported from .py

Description

Port of all python-based IO functions to cpp

Objects

• conversion of FileReaderOpenMS and FileWriterOpenMS
• sequenceReader and sequenceWriter
• conversion of FileReader and FileWriter classes
• conversion of tests

Description

The table widget provides users an in UI table editor used for updating parameters used for filtering, quantification, etc., on the fly without have to re-load .csv files

Objectives

• Ability to sort by columns
• Ability to search elements in a column
• Ability to filter elements in a column
• Ability to modify row text
• Ability to add or delete a row
• Unit testing of internal data handling methods

Feature list

• Table layout of the data
• Column filter/search/sort a la Excel
• Popup that allows for element editing
• Add/delete row button

Validation

• Passing unit tests
• Working example

Description

The graphics view allows users a dynamic plot to visualize the data

Objectives

• Main frame for visualizing data
• Search and filter elements to filter text based metadata
• Sliders to filter continuous data
• Multiple plot types
• Zoom and pan
• Tooltip of the plot data
• Save and restore plot views
• FilterManager class to handle saving and restoring data filtered states
• Unit tests of internal data handling methods

Validation

• Passing unit tests
• Working example

Re-write MRMFeatureSelector in C++ in OpenMS.

Description

Sequence explorer view provides a tree structure that allows users to navigate through the sequence and explore properties of each injection. The explorer also allows users to monitor and manage the data processing queue.

Objectives

• Ability to view by sample, sample group, or sequence segment
• Color code and/or graphic to indicate sample type
• Color and progress bar to indicate sample, sequence segment, or sample group progress and status respectively
• Tooltip for detailed information
• Check box to include or exclude a sample, sequence segment, or sample group from the processing

Validation

• Unit tests of internal methods
• Working example

Description

Conversion of the python dictionary representation of metadata into a full featured classes with attribute checks, etc.

Objectives

• conversion from python dictionary to C++ class
• transfer of unit tests from sequenceHandler class

And update SampleHandler::clear() removing the TODO inside it.

Description

Formal record of all application events, messages, and warnings into a time-stamped document that can be used for debugging and audit trail purposes

Objectives

• decide upon third party integration or creating a minimal version
• add external logging to external build
• add in application logging
  - [ ] add in features that comply with regulatory Audit Trail requirements

Requirements for the Event object

1. Realtime output to the stream (control of the verbosity level)
2. Realtime output to a text file (control of the text output level)
3. Record of events in a vector that can then be searched or pushed to a database
4. time stamps
5. class that generated the "event"
6. thread that generated the event

Requirements for the Logger object

1. can log to file or output to screen at specified time intervals
2. has multiple logging levels and a way to specify which levels to log to

References

• Reviews of C++11 header only loggers
  • https://softwarerecs.stackexchange.com/questions/31775/header-based-c-logging-library
  • https://stackoverflow.com/questions/5028302/small-logger-class
  • easylogging and spdlog appear the most promising for 3rd party incorporation
• Recommended C++11 header only loggers that fulfill the above requirements
  • easylogging: https://github.com/zuhd-org/easyloggingpp
  • spdlog: https://github.com/gabime/spdlog/blob/v1.x/include/spdlog/logger.h
  • cpplog: https://github.com/andrew-d/cpplog
  • https://github.com/fjanisze/logger
  • Dr. Dobbs 2013: http://www.drdobbs.com/cpp/a-lightweight-logger-for-c/240147505?pgno=3
  • Dr. Dobbs 2017: http://www.drdobbs.com/cpp/logging-in-c/201804215
• Observer pattern: https://thispointer.com/designing-a-configurable-logging-framework-using-observer-design-pattern/

Description

Users currently experience constant problems checking that filenames are correct, the input .csv files are typo free (e.g., misspelling of sample or component names), and that sample or component names are consistent across all of the input .csv files. In addition, it is problematic and time-consuming to identify what files have a misspelling in them or what filenames are named incorrectly. A workflow "wizard" that reads in all files, checks for data integrity, and allows the user to review the found data before executing the workflow should resolve a majority of these problems.

Features

• Data integrity check method
• Data review method

Description

Native installation of smartPeak

Objectives

• standalone .exe installation launcher for native platforms
• standalone .exe that starts and runs SmartPeak for native platforms

Description

As a user, I would like to consolidate all .csv input and output files into a binary file format (session object) that can be loaded, saved, and queried. The session object should be able to be shared by users across multiple computers (with potentially different OSs). Note that the sesson object will not store the raw .mzML files, but instead will point to them using the Filenames class. The user will need to update the path of the raw .mzML file if they e.g., copy and paste the session to a new directory or computer. The session should allow for "undo" and "redo" functionality.

Requirements

• Single binary file format
• Multi-thread transaction support
• transaction speed performance: TBD
• disk IO: TBD
• memory size limitations: TBD

Objectives

• save application data to persistent storage (e.g., sqlite)
• convert OpenMS structures to/from table structures
• load and save sessions
• session audit tracking
• undo and redo session commands

Description

A UI that allows the user to specify the order and steps for processing samples in a sequence

Objectives

• drop down that allows for selection of pre-configured processing steps
• drop down that allows for adding RawDataProcessing steps
• drop down that allows for adding SequenceSegmentProcessing steps
• list that allows users to order the processing steps
• button on the list items that allows the user to delete processing steps
• unit tests of adding and removing workflow steps

Save for a later version

• ability to save and load previous workflows
• "workflow management" methods in SequenceProcessor to handle saving and loading workflows
• "workflow management" methods in SequenceHandler to handle saving and loading workflows
• unit tests for saving and loading workflows

Validation

• passing unit tests
• working example

Problem

The current python implementation duplicates a large amount of data between ...Handler classes. For example, quantitation_methods in SequenceSegmentHandler is duplicated across all RawDataHandler classes that belong to that Sequence segment. Consequently, it is easy for the RawDataHandlers to become "out of sync" with the SequenceSegmentHandlers and vice versa.

Potential strategies depending on the appropriate case

std::shared_ptrs could be used to share the address of the data between handlers. In this case, if one Handler were to update the data, the updated version would be accessible to all other handlers who are referenced to the same memory address.

Objectives

• implement std::shared_ptr for member data that are referenced across multiple ...Handler classes
• Apply to quantitation_method
• Apply to metadata

Validation

• all current unit tests pass
• check that when one Handler is updated, another handler referencing the same data is also updated.

Description

As a user, I would like to have a Graphical user interface for SmartPeak that allows one to navigate the file system when importing files, view chromatograms and spectrum from data files, edit and modify settings and parameters, and view and edit data in an in-app table editor.

Workflow steps

1. Import raw data and data upload validation
2. Workflow/algorithm parameterization
3. Workflow/algorithm execution, monitoring, and error reporting
4. Data review
5. Data reporting

Objectives

• File IO routines
• new session wizard
• algorithm parameters, filters, and settings tabular editors
• tabular data editor and viewer
• graphical window for peak plots, QC plots, and analysis plots
• sequence tag manager
• sequence progress bar manager