neurodatawithoutborders / nwb-overview Goto Github PK

OptiNiSt - Calcium Imaging Pipeline Tool

OptiNiSt(Optical Neuroimage Studio) helps researchers try multiple data analysis methods, visualize the results, and construct the data analysis pipelines easily and quickly. OptiNiSt’s data-saving format follows NWB standards.

It looks like it can read HDF5/NWB and outputs in NWB.

Documentation: https://optinist.readthedocs.io/en/latest/index.html
Code: https://github.com/oist/optinist/
Tutorial at CNS 2023: https://www.dropbox.com/sh/z0awh38hpwrjchs/AABjmXTvNjrHnsO9lh-jh9fma?dl=0

Describe the orgnization of and where folks can find different documentation for their needs.

As part of the "Converting data to NWB" section we should include a brief discussion about how to find extensions in the catalog and when users should consider creating extensions (e.g., vs. when to use others ways to add custom data, e.g. adding columns to existing tables).

Describe e.g, the relationship between PyNWB and HDMF and other repos

Neo support import/export of data in NWB via:

https://neo.readthedocs.io/en/stable/io.html#neo.io.NWBIO

The main Neo docs are at:

https://neo.readthedocs.io/

To further enhance the extension tutorial it will be useful to create end-to-end examples for how to create extension for specific use cases. One such use case is extending theLabMetaData type to add lab-specific metadata to a file.

DataJoint: a scientific workflow data management platform
https://datajoint.com/

Docs: https://datajoint.com/docs/

Three approaches to integration:

1. Export from DataJoint Elements to NWB (need docs)
2. Store (refer to) NWB files in tables, store object_ids in a column, use the object_id to fetch NWB objects (ala Spyglass)
3. Store NWB objects in tables. Notebooks/code demonstrating this method (dj.AttributeAdapter method):
   • https://github.com/ttngu207/db-programming-with-datajoint/blob/master/notebooks/NWB-Adapter.ipynb
   • https://github.com/ttngu207/db-programming-with-datajoint/blob/master/Integrated_NWB_adaptive_attributes/pipeline/nwb_adapter.py

Add short section on NWB best practices and include link to the NWB Inspector Best Practices page in the overview docs

To further enhance the extension tutorial it will be useful to create end-to-end examples for how to create extension for specific use cases. One such use case is extending the DynamicTable type to collect results from a new analysis algorithm (e.g., for feature detection) or to extend an existing type, e.g., TimeIntervals to define custom interval definitions.

See also #59 for an example

FieldTrip is the MATLAB software toolbox for MEG, EEG and iEEG analysis

Documentation: https://www.fieldtriptoolbox.org/getting_started/nwb/

Initial integration PR: https://github.com/fieldtrip/fieldtrip/pull/1419/files

It looks like there is support for reading LFP and spiking data as long as they are named a certain way. I have not tested this myself.

As part of the tutorials at COSYNE 2023 the IBL team presented how to reuse their Brainwide Map via NWB (and the IBL's ONE API). This could make for a nice example to add to the community gallery page.

From the tutorial website:

The IBL recently released a Brainwide Map of neural activity during decision-making! It consists of 547 Neuropixel recordings of 32784 neurons across 194 regions of the mouse brain, and we want to make it easy for you to use it to test your own hypotheses. At this tutorial we’ll teach you how to use our API, ONE, to search and download datasets, and will show you how to do basic analysis such as raster plots, single cell PSTHs, and brain region averages.

Online notebooks:

• Introduction to IBL and Brain-wide map dataset
• Using IBL data with ONE - Cosyne 2023 tutorial
• Using IBL data with NWB - Cosyne 2023 tutorial

Sphinx ships with a standard link checker that we can run simply by calling make linkcheck. It would be great if we could run the link checker as a standard test on PRs to make sure links are being fixed before merging PRs.

DeepCINAC: A Deep-Learning-Based Python Toolbox for Inferring Calcium Imaging Neuronal Activity Based on Movie Visualization

DeepCINAC supports read from NWB (see tutorial below)

Code: https://gitlab.com/cossartlab/deepcinac
Paper: https://www.eneuro.org/content/7/4/ENEURO.0038-20.2020
Tutorial: https://deepcinac.readthedocs.io/en/latest/tutorial_gui.html
Demo Video: https://www.youtube.com/watch?v=rdgTCdeVyNw

I have not tested this tool.

See:

• https://open-ephys.github.io/gui-docs/User-Manual/Recording-data/NWB-format.html
• https://github.com/open-ephys-plugins/nwb-format

list OpenEphys and NeuroNexus

Add new page showing conversion pipelines actually in practice used by labs, including

NeuroConv Catalogue: https://neuroconv.readthedocs.io/en/main/catalogue/catalogue.html

and things like

Ecephys in practice with MatNWB and PyNWB: https://dervinism.github.io/bristol-neuroscience-data-guide/tutorials/Bristol%20GIN%20for%20Silicon%20Probe%20Data.html

Ophys in practice with MatNWB and PyNWB: https://dervinism.github.io/bristol-neuroscience-data-guide/tutorials/Bristol%20GIN%20for%20Calcium%20Imaging%20Data.html

and the others from that site for imaging and icephys.

Those community examples still need to be vetted for Best Practices and proper choice/use of neurodata types, but it would still be very useful to acknowledge and point to somewhere in the NWB docs

cc: @dervinism for all the hard work putting the tutorials together

See discussion in NeurodataWithoutBorders/pynwb#1736 (comment)

Update https://nwb-overview.readthedocs.io/en/latest/tools/hdfview/hdfview.html to provide a list general HDF5 tools that can be useful or inspecting NWB HDF5 file. E.g,:

• HDFView
• HDF5 Command Line Tools, e.g., h5dump, h5ls etc. see https://portal.hdfgroup.org/display/HDF5/HDF5+Command-line+Tools
  • At least some of those tools also support S3 h5ls --vfd=ros3 -r <s3path> or on Windows h5ls --vfd=ros3 --s3-cred="(,,)" <s3path>
• In VS Code, you can browse NWB files in VS Code using this extension https://marketplace.visualstudio.com/items?itemName=h5web.vscode-h5web and adding an editor association for *.nwb
• In Jupyter the following may be useful
  * https://github.com/jupyterlab/jupyterlab-hdf5
  * https://github.com/European-XFEL/h5glance
  * https://github.com/silx-kit/jupyterlab-h5web

https://github.com/PierreLeMerre/NWB_generator_example

Move the general parts of https://pynwb.readthedocs.io/en/stable/contributing.html here

To further enhance the extension tutorial it will be useful to create end-to-end examples for how to create extension for specific use cases. One such use case is extending the TimeSeries type (or a subtype of it) to integrate a new data modality (or refine an existing modality).

See also #59 for an example

In the extension tutorial we are currently discussing how to create APIs, but we are not discussing how to use extension, i.e., how to read and write data with extensions. How to install extensions. How to read data with extensions if the extensions is not installed etc.. Overall I think doesn't need to be a very long document, but it would be useful to show how this works.

naplib-python: Neural acoustic data processing and analysis tools in python

Paper: https://www.sciencedirect.com/science/article/pii/S2665963823000787
The paper says "Convenient data loading functions are available for common raw data recording structures including Neurodata Without Borders format (NWB) [33]"

Code repo: https://github.com/naplab/naplib-python

The NWB support is limited to opening an NWB file in Python (pynwb>=2.3.0) and reading an object named "ieeg" with data & rate (presumably an ElectricalSeries) and an "audio" object with data and rate (presumably a TimeSeries) from nwbfile.acquisition. This is not particularly robust. There is also no documentation on how to read data from an NWB file or structure the file to be readable. pynwb is also not installed during the pip install naplib process.
See https://github.com/naplab/naplib-python/blob/main/naplib/io/load_nwb.py

Participants at the 2022 NWB User Days suggested that it would be useful to create a "Best-practices guide on how to design an extension", which would cover topics such as:

• When to use Groups, Datasets, Attributes, Link etc.
• Best practices for data type, quantity, assignment of neurodata_types etc.
• Extension design principles
• Core data modeling principles
• How to best pick neurodata_type to build from. Also include common example use cases. E.g., example for extending LabMetaData (Here is an example for adding cell_id. NOTE: cell_id has been added in NWB so we should use a different example)

I would suggest adding a new section at the end of the extension tutorial on "Best practices for designing extensions" . Alternatively this could also be part of the PyNWB or HDMF documentation, but I think it may fit better here.

We have the development plan here https://www.nwb.org/development-plan/ We should look into how to maintain this page and possibly port it to here or manage this in some other form in GitHub

https://github.com/emeyers/Brain-Observatory-Toolbox

While working on #59 I noticed that we don't have documentation on how to create unit tests for an extension. The NDX template sets up a class for this, but it would be useful to have some discussion on how to write unit tests as well.

hdmf-zarr has been released and should be added to the list of core-tools as part of the HDMF section

Consider adding to the community gallery. Possibly as a subsection for DANDI-related reuse examples by other that are not part of the DANDI notebooks:

Notebook: https://github.com/neurovium/Neuromatch-AJILE12/blob/master/Notebook/exploreAJILE12.ipynb
Source: https://github.com/neurovium/Neuromatch-AJILE12
Data descriptor paper: https://www.nature.com/articles/s41597-022-01280-y

Consider adding GraFT as an optical physiology analysis tool

Repo: https://github.com/adamshch/GraFT-analysis
NWB examples: https://github.com/adamshch/GraFT-analysis/tree/main/code/nwb_tutorials

Once NeurodataWithoutBorders/nwbinspector#338 has been merged we should add a link to the extension best practices in an easy-to-find place in the extension docs.

https://github.com/INCF/example-live-scripts

It would be useful to also list relevant generic HDF5 tools (e.g., HDFView, h5dump, h5ls etc.) that can be useful to inspect NWB files to the tools page. We may also include generic Python tools here, e.g., the Jupyter HDF5 view plugins.

"Radiens provides a uniform interface, control, workflow, and terminology for data acquisition, early-stage analysis and visualization, and transition to study-specific analysis and data-presentation scripts."

Website, including download and purchase link: https://www.neuronexus.com/products/software/radiens/

Code appears to be closed source.

Data file format page says

Additionally, Radiens, supports many file types common in Ephys, including both proprietary (NeuroExplorer, Plexon, and open (Neurodata Without Borders, Kilosort2, etc.)

In response to my email inquiry, their support person wrote:

Thank you for reaching out to us regarding the compatibility of Radiens with NWB file format. Our platform handles files in version2.

Radiens Software (Allego, Curate, and Videre Apps) fully supports NWB files and provides comprehensive functionalities for both reading and writing.

We offer the Curate App, which allows you to conveniently convert your data to the NWB format. Additionally, our Videre app enables you to efficiently read and interpret NWB files, providing you with the necessary tools to explore and analyze the data effectively.

I have not yet tested the software because it requires a license, but I reached out to see if I can get a demo license.

https://github.com/flatironinstitute/neurosift

In the same vein as PyNWB #1647, this question has occurred before (mostly within the team or with other dev teams such as SLEAP/DANDI) but always given as a verbal response.

Would it be possible to have a concise & precise FAQ description of why it is discouraged for users to write videos of natural behavior from lossy formats (mpg, mp4) to internal NWB Datasets?

And why we 'cannot simple integrate' the codecs directly into NWB (which as I understand is more legal than technical)?

@oruebel @bendichter

Came up in meeting with @rly today

Thomas pointed out this tool for us https://neuroimage.usc.edu/brainstorm/Introduction

Apparently it can read NWB files!

https://github.com/hdmf-dev/hdmf-schema-language

Add DANDI to the list of tools: https://nwb-overview.readthedocs.io/en/latest/tools/tools_home.html

See reference in elife publication: https://elifesciences.org/articles/78116#s4-5-6
Documentation: https://pycicada.readthedocs.io/en/latest/tutorial.html
Source: https://gitlab.com/cossartlab/cicada/

Port the “Frequently Asked Questions” page from NWB.org at https://www.nwb.org/faq/ to the overview docs here

With #17 we have a general section for reading NWB files. I've written the MatNWB side but the PyNWB side is currently empty.

It looks like ArControl https://github.com/chenxinfeng4/ArControl provides at least a converter to NWB https://github.com/chenxinfeng4/ArControl-convert2-nwb. Could be relevant as a data acquisition system.

For some reason the list on the SpikeInterface tools page doesn't render correctly in ReadTheDocs. The image on the left shows how the page renders incorrectly in ReadTheDocs and on the right it shows how it renders (correctly) when I build the docs on my computer. The relevant file is docs/source/tools/spikeinterface/spikeinterface.rst. Any idea what might be the problem?

The page should contain items such as:

• Links to good first issues and help-wanted issues pages for the different repos
• All NWB projects and everyone participating in them are governed by our Code of Conduct guidelines. By participating, you are expected to uphold this code.
  Reporting issues
• Link to the Contributing Guidelines.
• Move the Contributing Guidelines as much as possible to the nwb-overview page and link to the individual pages for details if necessary.
• Link to the core tools page for links to sources and docs

nwbview: Rust powered NWB viewer

Code: https://github.com/brainhack-ch/nwbview
Requires HDF5 1.12 or earlier. Cannot be built with HDF5 1.14

It looks like an HDF5 viewer with some added features

Screenshot from their README

In development. It took a little effort to get this to work. My setup (I don't have rust installed globally) is:

conda create --name nwbview --yes
conda activate nwbview
mamba install "hdf5<1.14" rust --yes
HDF5_DIR="/Users/rly/mambaforge/envs/nwbview/" RUSTFLAGS="-C link-args=-Wl,-rpath,$HDF5_DIR/lib" cargo install nwbview
cd /Users/rly/.cargo/bin
./nwbview