niaid / tomojs-pytools Goto Github PK

Python 81.44% Shell 1.58% Jupyter Notebook 16.97%

tomojs-pytools's Introduction

This package contains python based command line imaging utilities for the Hedwig project.

The Sphinx documentation can be found here: https://niaid.github.io/tomojs-pytools

Installation

This pytools is a proper Python package, and can be installed from published wheels, tarballs or from the git repository.

The wheels are packaged as part of the Github Actions and can be manually downloaded for the action interface. They are also published to the bcbb-pypi hosted by NIAID actifactory. This extra repository can be configured with the --extra-index option on the pip install command line. However, the argument needs to include the username and password. The specifics can be found the the NIAID artifactory.

Use pip to install the tomojs_pytools package:

python3 -m pip install -extra-index https://USERNAME:[email protected]/artifactory/api/pypi/bcbb-pypi/simple tomojs_pytools

The requirements are specified conventionally in requirements.txt and setup.py, so they will be enforced at installation time. Additionally, the installation process installs the scripts as executables.

Alternatively, the package can be installed from the git repository:

python3 -m pip install git+https://github.com/niaid/[email protected]

Note for Mac M1 development: The imagecodec package does not provide a Mac OSX ARM binary at this time. The current workaround is to use an x86_64 python version on the M1 Mac in order to use the exiting binaries on PyPi.

tomojs-pytools's People

Contributors

Stargazers

Watchers

Forkers

philipmac mbopfnih blowekamp annshress

tomojs-pytools's Issues

TypeError: dtype: >u2 is not supported as an array.

Log file (needs vpn connection): https://prefect1.hedwig-workflow-api.niaiddev.net/default/task-run/d5a09c0b-d9f9-4f15-b682-723b6fb2694b?logs

Traceback (most recent call last):
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/SimpleITK/extra.py", line 245, in _get_sitk_vector_pixelid
    return _np_sitk[numpy_array_type.dtype]
KeyError: dtype('>u2')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "<string>", line 103, in generate_czi_imageset
  File "<string>", line 77, in gen_imageSet
  File "<string>", line 36, in gen_thumb
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/pytools/HedwigZarrImage.py", line 170, in extract_2d
    img = sitk.GetImageFromArray(arr.compute(), isVector=is_vector)
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/SimpleITK/extra.py", line 311, in GetImageFromArray
    id = _get_sitk_vector_pixelid(z)
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/SimpleITK/extra.py", line 247, in _get_sitk_vector_pixelid
    raise TypeError('dtype: {0} is not supported as an array.'.format(numpy_array_type.dtype))
TypeError: dtype: >u2 is not supported as an array.

.zarr file in consideration: The zarr file: /mnt/ai-fas12/RMLEMHedwigDev/Assets/TEST/shress/test-2023-09-29-TK/CZI-stuffs/DD/KC_M3_S2_ReducedImageSubset1/KC_M3_S2_ReducedImageSubset1.zarr

All .zarray in the zarr file show >u2 as their data types.

Allow mrc_visual_min_max be called as a python function

and not mandate the use of the CLI to use.

Create a function, eg mrc_visual_min_max(f_in: Path) -> dict which does the same thing as mrc_visual_min_max, except does not mandate the use of the CLI, and allows workflows to call that function directly.

Unexpected behaviour while attempting to check mrc file.

$ mrc_visual_min_max HUA13_00014_Jan11_patch_aligned_doseweighted.mrc
INFO: Reading "/mnt/ai-fas12/RMLEMHedwigQA/TestData/2DmrcFiles/HUA13_00014_Jan11_patch_aligned_doseweighted.mrc" image information...
WARNING: In /tmp/SimpleITK-build/ITK/Modules/IO/MRC/src/itkMRCHeaderObject.cxx, line 159
MRCHeaderObject (0x55604433bf70): extendedHeaderBytes is: 0If you see this log please contact https://github.com/InsightSoftwareConsortium/ITK/pull/2778

INFO: 	Pixel Type: 32-bit float
INFO: 	Size: (5760, 4092, 1)
INFO: 	Spacing: (0.8636999130249023, 0.8636999726295471, 0.8636999726295471)
INFO: 	Origin:  (0.0, 0.0, 0.0)
INFO: Building histogram for "/mnt/ai-fas12/RMLEMHedwigQA/TestData/2DmrcFiles/HUA13_00014_Jan11_patch_aligned_doseweighted.mrc"...
WARNING: In /tmp/SimpleITK-build/ITK/Modules/IO/MRC/src/itkMRCHeaderObject.cxx, line 159
MRCHeaderObject (0x55604433af30): extendedHeaderBytes is: 0If you see this log please contact https://github.com/InsightSoftwareConsortium/ITK/pull/2778

WARNING: In /tmp/SimpleITK-build/ITK/Modules/IO/MRC/src/itkMRCHeaderObject.cxx, line 159
MRCHeaderObject (0x55604433b370): extendedHeaderBytes is: 0If you see this log please contact https://github.com/InsightSoftwareConsortium/ITK/pull/2778

WARNING: In /tmp/SimpleITK-build/ITK/Modules/IO/MRC/src/itkMRCHeaderObject.cxx, line 159
MRCHeaderObject (0x55604401dbd0): extendedHeaderBytes is: 0If you see this log please contact https://github.com/InsightSoftwareConsortium/ITK/pull/2778

Traceback (most recent call last):
  File "/gs1/home/macmenaminpe/code/hedwig/dev/bin/mrc_visual_min_max", line 8, in <module>
    sys.exit(main())
  File "/gs1/home/macmenaminpe/code/hedwig/dev/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/gs1/home/macmenaminpe/code/hedwig/dev/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/gs1/home/macmenaminpe/code/hedwig/dev/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/gs1/home/macmenaminpe/code/hedwig/dev/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/gs1/home/macmenaminpe/code/hedwig/dev/lib/python3.9/site-packages/pytools/ng/build_histogram.py", line 228, in main
    h, bins = stream_build_histogram(input_image, histogram_bin_edges=None)
  File "/gs1/home/macmenaminpe/code/hedwig/dev/lib/python3.9/site-packages/pytools/ng/build_histogram.py", line 109, in stream_build_histogram
    histogram_bin_edges = np.arange(np.iinfo(img_dtype).min - 0.5, np.iinfo(img_dtype).max + 1.5)
  File "/gs1/home/macmenaminpe/code/hedwig/dev/lib/python3.9/site-packages/numpy/core/getlimits.py", line 657, in __init__
    raise ValueError("Invalid integer data type %r." % (self.kind,))
ValueError: Invalid integer data type 'f'.

Usage of Pixels XML element will become legacy at some point.

Source: http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome.html
Given this is required, but set to be removed this might be quite far in the future - still might be useful to note however.

 Pixels is going to be removed in the future, but it is still required.

        This is just notice that the contents of Pixels will be
        moved up to Image in a future release. This is because there
        has only been 1 Pixels object in each Image for some time.
        The concept of multiple Pixels sets for one Image failed to
        take off. It is therefore redundant.

        The Image will be unreadable if any of the required Pixel attributes are missing.

        The Pixels themselves can be stored within the OME-XML compressed by plane, and encoded
        in Base64.
        Or the Pixels may be stored in TIFF format.

Code: https://github.com/niaid/tomojs-pytools/pull/65/files#diff-296186ce9db389fa6d7da54343f2eb8b97b6831d5feab3f32bc313ca4398ebe9R37

Outputting a pyramid as a no-compression zip archive

I would like to have an option that the neuroglancer pyramid comes out ~~of volume-to-precomputed-pyramid~~ as a ZIP file with no compression, instead of a folder.

This will simplify things downstream considerably, when it comes to managing generated assets. "Every asset is a file" is a good invariant to have, and the neuroglancer pyramid is the odd one out. Wire level compression will be maintained by having the chunks gzipped, but since they are, there is no sense in zip-compressing them all over again. Transparent retrieval from the archive on reading is already in place. And with the recent revelation that the assets will have to be moved around on certain Hedwig operations, I'd much rather have one archive file than recurse through chunk directories.

The structure of the archive should mimic the structure of the pyramid folder - info on the root level, scale subdirectories under the root, gzipped chunk files in those. Please give the archive has a "zip" extension.

The asset path on the callback would be the path to the zip.

Thank you.

Add extra outputs to mrc_visual_min_max

Add: Floor and Limit

Also, prefix all keys with string "neuroglancerPrecomputed"

Leaving:

    neuroglancerPrecomputedFloor - the physical minimum pixel value in the pyramid - (int)
    neuroglancerPrecomputedLimit - the physical maximum pixel value in the pyramid - (int)
    neuroglancerPrecomputedMin - the default lower limit for the Neuroglancer intensity scale - (int)
    neuroglancerPrecomputedMax - the default upper limit for the Neuroglancer intensity scale - (int)

mrc_visual_min_max usage query on zarr

given:

$ ls -l  /tmp/tmpn1_aohf2/2013-1220-dA30_5-BSC-1_10.zarr
total 20
drwxrwxr-x 3 macmenaminpe macmenaminpe 4096 Apr 11 22:17 0
drwxrwxr-x 3 macmenaminpe macmenaminpe 4096 Apr 11 22:17 1
drwxrwxr-x 3 macmenaminpe macmenaminpe 4096 Apr 11 22:18 2
drwxrwxr-x 3 macmenaminpe macmenaminpe 4096 Apr 11 22:18 3
drwxrwxr-x 2 macmenaminpe macmenaminpe 4096 Apr 11 22:17 OME

what would be usage?

$ mrc_visual_min_max /tmp/tmpn1_aohf2/2013-1220-dA30_5-BSC-1_10.zarr
HDF5-DIAG: Error detected in HDF5 (1.12.1) thread 0:
  #000: /tmp/SimpleITK-build/ITK/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5Fdeprec.c line 156 in itk_H5Fis_hdf5(): unable to determine if file is accessible as HDF5
    major: File accessibility
    minor: Not an HDF5 file
  #001: /tmp/SimpleITK-build/ITK/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5VLcallback.c line 3769 in itk_H5VL_file_specific(): file specific failed
    major: Virtual Object Layer
    minor: Can't operate on object
  #002: /tmp/SimpleITK-build/ITK/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5VLcallback.c line 3699 in H5VL__file_specific(): file specific failed
    major: Virtual Object Layer
    minor: Can't operate on object
  #003: /tmp/SimpleITK-build/ITK/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5VLnative_file.c line 384 in itk_H5VL__native_file_specific(): error in HDF5 file check
    major: File accessibility
    minor: Unable to initialize object
  #004: /tmp/SimpleITK-build/ITK/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5Fint.c line 1086 in itk_H5F__is_hdf5(): error while trying to locate file signature
    major: File accessibility
    minor: Not an HDF5 file
  #005: /tmp/SimpleITK-build/ITK/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FDint.c line 113 in itk_H5FD_locate_signature(): unable to read file signature
    major: Low-level I/O
    minor: Unable to initialize object
  #006: /tmp/SimpleITK-build/ITK/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FDint.c line 189 in itk_H5FD_read(): driver read request failed
    major: Virtual File Layer
    minor: Read failed
  #007: /tmp/SimpleITK-build/ITK/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FDsec2.c line 755 in H5FD__sec2_read(): file read failed: time = Wed Apr 12 21:41:02 2023
, filename = '/tmp/tmpn1_aohf2/2013-1220-dA30_5-BSC-1_10.zarr', file descriptor = 3, errno = 21, error message = 'Is a directory', buf = 0x7fff7fe7af48, total read size = 8, bytes this sub-read = 8, bytes actually read = 18446744073709551615, offset = 0
    major: Low-level I/O
    minor: Read failed
Traceback (most recent call last):
  File "/home/macmenaminpe/code/hedwig/dev/bin/mrc_visual_min_max", line 8, in <module>
    sys.exit(main())
  File "/home/macmenaminpe/code/hedwig/dev/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/macmenaminpe/code/hedwig/dev/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/macmenaminpe/code/hedwig/dev/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/macmenaminpe/code/hedwig/dev/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/macmenaminpe/code/hedwig/dev/lib/python3.8/site-packages/pytools/ng/build_histogram.py", line 370, in main
    img = sitk.ReadImage(input_image)
  File "/home/macmenaminpe/code/hedwig/dev/lib/python3.8/site-packages/SimpleITK/extra.py", line 368, in ReadImage
    return reader.Execute()
  File "/home/macmenaminpe/code/hedwig/dev/lib/python3.8/site-packages/SimpleITK/SimpleITK.py", line 8846, in Execute
    return _SimpleITK.ImageFileReader_Execute(self)
RuntimeError: Exception thrown in SimpleITK ImageFileReader_Execute: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:105:
sitk::ERROR: Unable to determine ImageIO reader for "/tmp/tmpn1_aohf2/2013-1220-dA30_5-BSC-1_10.zarr"

assuming I'm using incorrectly

Neuroglancer metadata datatype

Looks like the tomojs-generated Neuroglancer metadata strings contain their values as string formatted integers (as opposed to numbers), like this:

{"neuroglancerPrecomputedFloor":"31403", "neuroglancerPrecomputedLimit":"34295", "neuroglancerPrecomputedMax":"33467", "neuroglancerPrecomputedMin":"32252"}

May we have compatible metadata, please? No fractions, and stored as strings. Thank you.

imagecodecs error while building tomojs-pytools

While attempting to build tomojs-pytools on an M1 Mac running Monterey Version 12.6.2, I get the following error:

imagecodecs/_aec.c:756:10: fatal error: 'libaec.h' file not found
      #include "libaec.h"
               ^~~~~~~~~~
      1 warning and 1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> imagecodecs

Same error with both conda Python 3.9.15 and Homebrew Python 3.10.10

Hardware Overview:

 Model Name:	MacBook Pro
  Model Identifier:	MacBookPro18,1
  Chip:	Apple M1 Pro
  Total Number of Cores:	10 (8 performance and 2 efficiency)
  Memory:	16 GB
  System Firmware Version:	8419.60.44
  OS Loader Version:	7459.141.1
  Serial Number (system):	TNV664X05Y
  Hardware UUID:	B543DCDB-F3E3-51D8-88B2-6860F79AAF3A
  Provisioning UDID:	00006000-000C04C11A02401E
  Activation Lock Status:	Disabled

See attached file for complete run.
imagecodes_error.txt

niaid / tomojs-pytools Goto Github PK

tomojs-pytools's Introduction

Installation

tomojs-pytools's People

Contributors

Stargazers

Watchers

Forkers

tomojs-pytools's Issues

TypeError: dtype: >u2 is not supported as an array.

Allow mrc_visual_min_max be called as a python function

Unexpected behaviour while attempting to check mrc file.

Usage of Pixels XML element will become legacy at some point.

Outputting a pyramid as a no-compression zip archive

Add extra outputs to mrc_visual_min_max

mrc_visual_min_max usage query on zarr

Neuroglancer metadata datatype

imagecodecs error while building tomojs-pytools

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent