Giter VIP home page Giter VIP logo

hdf5_vibez's Introduction

HDF5 Plugin for ImageJ and Fiji

This plugin is an HDF5 reader/writer plugin for ImageJ and Fiji. It is a continuation of the original code by Olaf Ronneberger and others.

HDF5 is a data format for storing extremely large and complex data collections. For more information see the official website. The plugin uses the jhdf5 library from ETH SIS for reading and writing HDF5 files.

Features

The HDF5 plugin for ImageJ and Fiji provides the following features:

  • Loading 2D - 5D datasets
  • Loading and combining mulitple 2d/3D datasets to 3D/4D/5D Hyperstacks
  • Writing Hyperstacks to multiple 3D datasets
  • scriptable load and save commands

Example Data Sets

  • pollen.h5 (3D confocal data of a pollen grain: 8 bit gray, approx. 16MB

Load data sets

  1. Select "File -- Import -- HDF5...". The file selector will pop up. Double click the file you want to load.
  2. The "Select data sets" dialog will open:
  3. select one or more datasets. Multiple selections can be done by
    • mouseclick on first and Shift+mouseclick on last item
    • mousedown on first and drag to last item
    • CTRL+mouseclick to select / deselect individual items
    • CTRL+A selects all items
  4. chose how they should be loaded or combined to a hyperstack.
    • Load as ... individual stacks will create an individual window for each selected data set
    • Load as ... individual hyperstacks (custom layout) will create a new hyperstack for each selected dataset. The data set layout has to be specified in the textfield below. HDF5 uses C-style / Java-style indexing of the array, i.e., the slowest changing dimension comes first (see size in the table). Typical storage orders are:
      • "yx": 2D image
      • "zyx": 3D image
      • "tyx": 2D movie
      • "tzyx": 3D movie
      • "cyx": 2D multi-channel image
      • "tczyx": 3D multi-channel move
      • ... Of course, any other permutation of the letters y,x,z,t,c is allowed.
    • Combine to ... hyperstack (multichannel) loads the selected 2D/3D data sets and combines them to a multi-channel hyperstack
    • Combine to ... hyperstack (time series) loads the selected 2D/3D data sets and combines them to a time-series hyperstack
    • Combine to ... hyperstack (multichannel time series) loads the selected 2D/3D data sets and combines them to a multichannel time-series hyperstack. You have to specify the Number of channels of the resulting hyperstack. The number of time points is then determined from the number of selected data sets divided by the number of channels

Save data sets

  1. Select "File -- Save As -- HDF5 (new or replace)..." to create a new HDF5 file or "File -- Save As -- HDF5 (append)..." to append the dataset(s) to an existing HDF5 file. The file selector will pop up. Select the file name.
  2. The Save Dialog will open to select the data set layout
  3. Compression Level allow to select the compression of the data set. The compression is lossless, i.e. it works like a zip-archive. Possible compression levels are
    • no compression,
    • 1 (fastest, larger file)
    • 2
    • ...
    • 9 (slowest, smallest file)
  4. Presets: allows to select presets for the data set layout. There is no official standard, how to name the datasets. For general purpose data we usually name it as "/t0/channel0", "t0/channel1", ... which is the "Standard" Preset.
  5. Dataset Names Template specifies the template string for the data set names. The placeholders {t} and {c} will be replaced for each timepoint/channel combination with the strings specified in the following two textfields.
  6. Replace {t} with: and Replace {c} with: specifies the enconding of time points and channels in the filename. Possible entries area printf-style format string or a list of strings (one entry per line), e.g.,
    • %d for number style like 1,2,3,...
    • %.03d for zero-padded-numbers with 3 digits: 001, 002, 003, ...
    • nuclei cellborder pattern for named channels
  7. The Update Preview button shows the Resulting Mapping: of the hyperstack time points and channels to the HDF5 data set names

Internals

The HDF5 plugin saves and loads the pixel/voxel size in micrometer of the image in the attribute "element_size_um". It has always 3 components in the order z,y,x (accordingly to the c-style indexing). Other meta data is not saved/loaded

Wish list for next version

  • Support for single data sets with more than 2GB size (will require a slice-wise or block-wise loadin/saving)
  • disable the Log Window
  • load a sub cube of the data set (e.g. for large 5D arrays stored in a single dataset)

hdf5_vibez's People

Contributors

ctrueden avatar gfleishman avatar kmdouglass avatar matthiasch avatar rejsmont avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hdf5_vibez's Issues

int16 data sets not read properly

I work for a large microscopy center which is starting to generate data written in HDF5 format (FEI's EMD format). Your HDF5 vibez plugin for Fiji is extremely useful for users who are not able to read the data using any other GUI based software. However, signed integer 16-bit data is not being properly read. It looks like there is an issue between converting from HDF5 int16 and ImageJ's int16 format.

I had a similar issue writing a Jython plugin for Fiji. The solution was to read unsigned and signed data as signed. Then everything worked when creating a ImagePlus.

The issue is probably in this part of the Vibez plugin

As per information from my issue posted to the ImageJ Forum linked above:

short in Java is always signed (-32,768 to 32,767)
a ShortProcessor in ImageJ1 is always unsigned

I am not a Java developer and not setup to compile/test this in ImageJ. Do you have anytime to look into this issue?

I can send a small test H5 file with int16 and uint16 datasets to show the problem. It's easy to create this using the following Python code

import h5py
import numpy as np

int1 = np.linspace(-32768,32767,100,dtype=np.int16)
uint1 = np.linspace(0,65535,100,dtype=np.uint16)

YY,XX = np.meshgrid(int1,int1)
YYu,XXu = np.meshgrid(uint1,uint1)

with h5py.File('integers.h5','w') as f1:
    f1.create_dataset('int16',data=XX.astype(np.int16),compression='gzip')
    f1.create_dataset('uint16',data=XXu.astype(np.uint16),compression='gzip')
    f1.create_dataset('XX',data=XX.astype(np.float32),compression='gzip')
    f1.create_dataset('XXu',data=XXu.astype(np.float32),compression='gzip')

Here is the data read into Fiji using your plugin:
uint16
image

int16
image

PS Thank you very much for all your work on this plugin. Its extremely useful and will see a lot more use as HDF5 data sets become more ubiquitous.

float(), uint8(), ... and object()

Hello,

I'm about to set up a project in Eclipse, that uses scifio and HDF5_Vibez.

It's fine so far, all imports are resolved, the only issues are with float(), uint8(), ... and object() in HDF5_Reader_Vibez, HDF5ImageJ and Vibez_Validate (see attached screenshot of HDF5_Reader_Vibez).

What jar or java files are missing to solve these compilation problems? What needs to be done?

Thanks

Juergen
hdf5_reader_vibez errors

Support for reading from external links

As far as I can tell, HDF5_Vibez does not support reading from external links. I would like to add this functionality. I've forked the repo, will do my best, and submit a pull request when/if I come up with something that works. Any pointers from existing contributors on this matter would be a big help (e.g. which file, which section of which file, would be most appropriate to begin looking at). Some documentation of the issue:

Suppose I have a .h5 file which contains only external links; when I attempt to open such a file in Fiji with File --> import --> hdf5, I see the following:

screen shot 2018-04-11 at 3 05 57 pm

So, HDF5_Vibez knows the external links are there, but does not make them available for reading.
On the other hand, HDF5 compass can traverse the links in the same file no problem:

screen shot 2018-04-11 at 3 07 28 pm

So, I think the file is ok. FYI, I created the file using h5py:

`#!/usr/bin/env python3

-- coding: utf-8 --

import os
import h5py

get all paths, working directory and hdf5 file names

cwd = os.getcwd()
h5_files = os.listdir(cwd)
h5_files.sort()
h5_files = [s for s in h5_files if s.endswith('.h5')]

create container object to write external links to

container_name = 'container.h5'
container = h5py.File(container_name, 'w')

loop through h5py files and add them as external links

for i, h5_file in enumerate(h5_files):
key = 'timepoint{}_extlink.h5'.format(i)
container[key] = h5py.ExternalLink(cwd+'/'+h5_file, '/default')

`

Use case:
I have thousands of separate .h5 files, each a separate time point from the same time series experiment. I don't want to build a new .h5 file, rewriting the separate time points as groups, as this will take forever and the file will be huge. So, creating container files with external links to just the subset of time points that I want, and then using that to access those time points, seems like a crafty solution... but I need to be able to load and view the results.

Apple M1 Support

Hi,

I've been trying to get this plugin working with Fiji on an M1 MacBook. While Fiji is running natively it seems that there are no native HDF5 libraries available for this plugin. Is there any chance that this will be updated in the future?

Open API

Hi guys,

I know that this project needs serious reworking to become Scifio compatible, but could you please make reading and writing methods from HDF5ImageJ public? Using IJ.run() always shows the image which is pretty annoying in scripts...

Cheers,

Radek

Make it work with IJ2 data structures

It would be great to implement HDF5 as IJ2 plugin so one could work with Datasets directly. This could be implemented quite quickly... Of course SCIFIO would be the next step, but this would require some architecture decisions.

For example, IOService.open(String source) only takes one argument, yet HDF5 requires both filename and dataset name. One could return some kind of lazily-loaded Dataset pointing to separate HDF5 datasets, but yeah... This really needs architecture decisions...

@ctrueden your thoughts on this?

Load hdf5 dataset > 2GB

Dear team,

Could I ask for the support to load dataset larger than 2GB? Because I think that hdf5 is chosen for much larger data, which has performances better than stack tiff.

Thanks!!!

zooming to selection fails

We are working with highly asymmetric datasets (100 px x 40,000 px). Here an example. Zooming to an arbitrary selection fails.

Image --> Zoom --> To Selection

The action zooms out totally, instead of to the selected region (yellow box in screenshot). Could you please explain how to zoom asymmetrically i.e. with different factors in x and y direction? Thanks.
Screenshot 2019-06-26 at 09 32 09

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.