royerlab / dexp Goto Github PK

Few commands in dexp.cli.dexp_commands have circular import between themselves and dexp_main.py . @royerloic maybe you want to take this or explain us what is the desired behavior to us so we can try to address.

This function makes several GPU call but with very little computation.

Currently, dexp commands such as stablize and copy only takes datasets with the same exact shape (except for time dimension).
It would be nice to support datasets with different shapes.

We need to be able to just deskew the raw data and save the results, without fusion.
This is useful when we only acquire single view data.

We have raw DaXi data in TIFF format, saved by MicroManager.
Currently I am running a python script to read the data and save to Zarr.
It makes sense to incorporate this into Dexp so that our collaborators don't need to run the script independently.

The tests of warp_2d.py and warp_3d.py of interpolation and registration might be redundant and need checking.

Right now there are two sub-packages with optimization functions, dexp.processing.denoising and dexp.processing.optimization. This can be improved, I prefer how the latter is implemented, but the butterworth_calibrator has much more functionality and we need to think more about the best way to refactor this.

The parent argument would allow propagating the metadata to children datasets. For example, some image labels could inherit the scale from the original dataset.

Additionally, this would be useful to track the dataset history.

Dexp view should also support tiff dataset viewing.

It should validate with the number of selected channels and not the actual number of channels.

We are planning to add metadata information to our zarr dataset to store and avoid hard-coding different parameters for each different task.

For the parameters my suggestion is to have three levels, where the latter has the highest priority and overwrite the previous ones:

1. Our default values, the user would never touch these parameters.
2. Dataset-wise, these parameters would be applied to every channel (whole dataset).
3. Channel-wise, these parameters are only applied to the specified channel.

Currently, I think we should support:

• Different scales for each axis, detonated by (dt, dz, dy, dx).
• Different translations for each axis.

We could try not to diverge too much from ome-zarr

Any suggestions @royerloic, @BinYangOptics, @merlinzebra, or anyone else?

Hi @JoOkuma ,

I'm just trying to install dexp on my Windows machine into a clean conda environment. These were the commands I ran:

conda create --name dexp python=3.9 napari=0.4.15 -c conda-forge
conda activate dexp
pip install dexp[color,cuda112,napari]

And that's the error I'm receiving:

ERROR: Could not find a version that satisfies the requirement dexp[color,cuda112,napari] (from versions: none)
ERROR: No matching distribution found for dexp[color,cuda112,napari]

Let me know if I can help tracing down the issue.

Best,
Robert

We have a custom built lightsheet microscope. If someone could detail the expected Zarr dataset structure, I could transform our data into it.

pip install dexp seems to not work on OSX. Will address in a PR. Getting the following error for those wonder:

ERROR: Command errored out with exit status 1:
   command: /Users/ahmetcan.solak/.conda/envs/dexp/bin/python /Users/ahmetcan.solak/.conda/envs/dexp/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /var/folders/4_/7bzvr3mx1_qb91k468xgk6h40000gp/T/tmpzbf817ia
       cwd: /private/var/folders/4_/7bzvr3mx1_qb91k468xgk6h40000gp/T/pip-install-sqj5wgse/pycairo_1e5102aa29fe4d168690c84e748abfb5
  Complete output (12 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.9-x86_64-3.8
  creating build/lib.macosx-10.9-x86_64-3.8/cairo
  copying cairo/__init__.py -> build/lib.macosx-10.9-x86_64-3.8/cairo
  copying cairo/__init__.pyi -> build/lib.macosx-10.9-x86_64-3.8/cairo
  copying cairo/py.typed -> build/lib.macosx-10.9-x86_64-3.8/cairo
  running build_ext
  'pkg-config' not found.
  Command ['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']
  ----------------------------------------
  ERROR: Failed building wheel for pycairo
Successfully built dask-image gpustat nvidia-ml-py3 gputil pyrsistent docstring-parser
Failed to build pycairo
ERROR: Could not build wheels for pycairo which use PEP 517 and cannot be installed directly

This issue seems to be related to Dask, it could be implemented without it.

We do not have csbdeep as dependency, inevitably IsoNet module cannot import csbdeep. Will make a PR. Also we do not have any test for IsoNet submodule. Would be nice to have some tests.

Sorry to bother you, can you provide us with the complete commands when running the program? For example, which is the main program we should run? When we try to add an image, we cannot execute the add() function by running the following command:
(dexp) g: \ learning \ 3D deconvolution \ dexp master \ dexp master \ dexp \ cli > Python dexp_ main. py add data_ fused. zarr ./ output
We hope you can give us a screenshot or specific description of the running program. Thank you very much!!!

1. It would be great if tensorflow-gpu is not a required installation and all the associated gpu tools. I am working with the commit a978169 that was actually stable and worked symmetrically on fry2 and locally.

2. The recent package installations in dexp using gputools etc have raised different opencl errors

clGetPlatformIDs failed: PLATFORM_NOT_FOUND_KHR on fry2 in converting to tiff files. I had to revert few commits and stop importing the sharpen module and fusion that interact with gputools to make dexp work for me.

As given in the issue title we need to put some work on docstrings in dexp.processing.backends. It is lower priority but still something to not forget, so dropping this issue.

• Make CI linting compatible with pre-commit.
• Checkout --max-complexity flag, it seems interesting.

Deskewing function classic_deskew not working properly, classic_deskew_dimensionless is working correctly on the demo/test file, but not the higher-level API.

Basically similar to #8 , we are missing gputools as a dependency and volumerender submodule cannot import gputools. Will address in a separate PR.

Parameters such as device, input and output dataset should be modified from click, ref.

Functions like fusion or deconvolution where parameters are different for different methods could use click to invoke other commands, ref.

Currently, it is quite verbose to import the DEXP's classes. For example:

from dexp.processing.backends.backend import Backend
from dexp.processing.backends.best_backend import BestBackend
from dexp.processing.backends.numpy_backend import NumpyBackend

We could update the dexp/processing/backends/__init__.py so the imports could be:

from dexp.processing.backends import Backend, BestBackend, NumpyBackend

This would not require us to change the current file structure.

The same could be done for the Datasets classes and maybe other classes.

@royerloic what do you think?