deepregnet / deepreg Goto Github PK
View Code? Open in Web Editor NEWMedical image registration using deep learning
License: Apache License 2.0
Medical image registration using deep learning
License: Apache License 2.0
The options for sampling groups, image pairs and labels
Please delete options that are not relevant.
clearly described method that us consistent with data loaders
Inter-subject neural image registration
data: https://drive.google.com/drive/folders/1Plum9c41tHPmA84-L0K0RrIztictFuNg
This demo tests:
1 - unsupervised inter-subject
2 - weakly supervised inter-subject
3 - unsupervised inter-subject with weak supervision
Each demo will have an independent folder directly under the 'Demos'. For simplicity, avoiding sub-folders and additional functions/classes.
Each demo should have a 'data.py' script
Preferably, data are hosted in a reliable and efficient (not store in this repo please).
Each demo should have a 'train.py' script
This is accompanied by a config file in the same folder;
Each demo should have a 'predict.py' script
Ideally,
Briefly describe the clinical application and the need for registration
Acknowledge data source.
For now, all the configurations are put into the same file.
We should split it into data and model config, and when launching experiments, we can pass multiple configs and merge them into one
It's good practice to follow some code formating standards. A good means of doing so is to use an automated code formatter such as black.
See e.g. how it's done in MONAI: https://github.com/Project-MONAI/MONAI/blob/master/CONTRIBUTING.md#automatic-code-formatting
CI times out at 30 minutes which causes CI to fail for h5 data loaders. The total expected time it should take to finish all tests is around 40-50 minutes so an extension to the timeout is likely to fix the problem
Additional support to enable training data can be sampled from multiple folders.
Please delete options that are not relevant.
training
, validation
, test
dataset foldersThe detailed steps how to add a new loss
add add_loss.md in /tutorials
add link to the wiki Tutorial Index
Currently, the package depends on .yaml files / config paths to run registrations. It would be useful to create a guide and CLI app for users to generate their own configuration files to customise trainings.
## Type of issue
Please delete options that are not relevant.
N/A
Users currently have to design yaml files manually by inspecting yaml files provided in deepreg/config. It's not entirely clear what these parameters are or why they are necessary, what expected range should be required or how wrong values could break the training - I would assume the guesswork involved could prove quite frustrating at the end point.
Please delete options that are not relevant.
Click on the links "How to configure deepreg options" leads to a 404.
Clicking on the broken links on README on master leads to a 404 error.
Abdominal Organ CT Registration
data: https://learn2reg.grand-challenge.org/Dataset/
https://drive.google.com/file/d/1aWyS_mQ5n7X2bTk9etHrn5di2-EZEzyO/view
This demo can test:
1 - unpaired inter-subject reg with/without labels
2 - weakly supervised
https://joss.readthedocs.io/en/latest/review_checklist.html
Community guidelines
There should be clear guidelines for third-parties wishing to:
Overview on demos
add links to individual demos
add demo.md in /tutorials
add link to the wiki Tutorial Index
Describe the basics in training registration:
with data loader and links to demos
Discuss the feasibility and pros/cons for using the higher-order and potentially faster and memory-saving resampler:
https://github.com/NifTK/NiftyNet/tree/dev/niftynet/contrib/niftyreg_image_resampling
Please comment below.
Please delete options that are not relevant.
N/A
N/A
deepreg/model/optimizer.py
MR-to-ultrasound registration
data and previous demo: https://github.com/YipengHu/label-reg
This demo tests:
1 - weakly supervision
2 - conditional segmentation
and possibly
3 - different data formats?
We do not have any unit tests for layer utils. These are base functions that are relied on throughout the package. Good place to start for unit tests as we will rely on these functions working as expected for further testing down the line.
## Type of issue
Please delete options that are not relevant.
N/A
Unit tests which prove the functionality of the following functions:
Unit test(s), pytest style, for each function that assert inputs, outputs for a couple of cases each.
CT Lung Registration
data: https://zenodo.org/record/3835682#.XuVJ-0BFzx8
https://learn2reg.grand-challenge.org/Dataset/
This demo can test:
1 - unpaired inter-subject reg with/without labels
2 - unpaired intra-subject reg with/without labels
and potentially
3 - paired intra-subject reg with/without labels
Such a feature can be a nice simple integration test to ensure for example that the gradients are useable.
A basic toy example could involve translating an image and registering it with itself.
If useful, this quick and dirty notebook could be a source of inspiration:
https://colab.research.google.com/drive/1OmU-jTvWmRRVerc23nsyNz_p8jU3S7wX?usp=sharing
The detailed information on dependency, installation etc
add setup.md in /tutorials
add link to the wiki Tutorial Index
The detailed information how to use predefined data loaders
add predefined_loader.md in /tutorials
add link to the wiki Tutorial Index
The current format is summarised as follows:
To facilitate the user experience, we plan to prepare some default data loaders for different use scenarios. Currently, Nifti and H5 formats are supported. For different types of use cases and image formats, a customised data loader is needed (add a link to the tutorial).
There are some prerequisites on the data:
(width, height, depth)
; label has shape (width, height, depth)
or (width, height, depth, num_labels)
.In the following, we take train directory as an example to list how the files should be stored.
Assuming each .nii.gz
file contains only one tensor, which is either image or label.
This is the simplest case. Data are assumed to be stored under train/images
and train/labels
directories.
We only have images without any labels and all images are considered to be independent samples. So all data should be stored under train/images
, e.g.:
(It is also ok if the data are further grouped into different directories under images
as we will directly scan all nifti files under train/images
.)
In this case, we have both images and labels. So all images should be stored under train/images
and all labels should be stored under train/labels
. The corresponding image file name and label file name should be exactly the same, e.g.:
We have images without any labels, but images are grouped under different subjects/groups, e.g. time-series observations for each subject/group. For instance, the data set can be the CT scans of multiple patients (subjects/groups) where each patient has multiple scans acquired at different time points. So all data should be stored under train/images
and the leaf directories (directories that do not have sub-directories) must represent different subjects/groups, e.g.:
(It is also ok if the data are grouped into different directories, but the leaf directories will be considered as different subjects/groups.)
We have both images and labels. So all images should be stored under train/images
and all labels should be stored under train/labels
. The leaf directories will be considered as different subjects/groups and the corresponding image file name and label file name should be exactly the same, e.g.:
In this case, images are paired, for example, to represent a multimodal moving and fixed image pairs to register. Data are assumed to be stored under train/moving_images
, train/fixed_images
, train/moving_labels
, and train/fixed_labels
directories.
We only have paired images without any labels. So all data should be stored under train/moving_images
, train/fixed_images
and the images corresponding to the same subject should have exactly the same name, e.g.:
(It is ok if the data are further grouped into different directories under train/moving_images
and train/fixed_images
as we will directly scan all nifti files under them.)
We have both images and labels. So all data should be stored under train/moving_images
, train/fixed_images
, train/moving_labels
, and train/fixed_labels
. The images and labels corresponding to the same subjects/groups should have exactly the same names, e.g.:
Each .h5
file is similar to a dictionary, having multiple key-value pairs. Hierarchical multi-level h5 indexing is not used. Each value is either image or label.
Each key corresponds to one image, e.g. {"subject1": data1, "subject2": data1, ...}
. All data should be stored under train/images
, it can be a single h5 file or multiple h5 files e.g.:
Each key corresponds to one subject. Data can be stored in two single h5 files (one for image and one for label), the keys in the files should be the same.
Similar to case 1-1 above, but the keys, in this case, have to share the same format like subject%d-%d
where %d
represents a number. For instance, subject3-2
corresponds to the second observation for the subjects. Otherwise, the file structure is the same as case 1-1, e.g.
Similar to case 1-2 and 2-1 above, the keys have to share the same format like subject%d-%d
and the keys for images and labels should be consistent.
In this case, data are paired. Data are assumed to be stored under train/moving_images
, train/fixed_images
, train/moving_labels
, and train/fixed_labels
directories.
We only have paired images without any labels. So all data should be stored under train/moving_images
, train/fixed_images
and the keys corresponding to the same subject should be the same, e.g.:
We have both images and labels. So all data should be stored under train/moving_images
, train/fixed_images
, train/moving_labels
, and train/fixed_labels
. The keys corresponding to the same subject should be the same, e.g.:
It's good practice to explain in the README what the code is about before going into how to use it
Editing repo without changing functionality to set up a consistent coding style.
Related issue #10
We can add a pre-commit check which verifies the code format before committing.
To facilitate the user experience, we plan to prepare some default data loaders for different use scenarios. Currently, Nifti and H5 formats are supported. For different types of use cases and image formats, a customised data loader is needed (add a link to the tutorial).
There are some prerequisites on the data:
(width, height, depth)
; label has shape (width, height, depth)
or (width, height, depth, num_labels)
.In any case when corresponding labels are available and there are multiple types of labels, e.g. the segmentation of different organs in a CT image, two options are available:
label_idx
are the corresponding types in (width, height, depth, label_idx)
- the same type of landmark or ROI - between all labelsWhen multiple subjects each with multiple images are available, multiple different sampling methods are supported:
For the first two options, the intra-subject images will be ascending-sorted by name to represent ordered sequential images, such as time-series data
*Multiple label sampling is also supported once image pair is sampled; In case there are no consistent label types defined between subjects, an option is available to turned off label contribution to the loss for those inter-subject image pairs.
In the following, we take train directory as an example to list how the files should be stored.
Assuming each .nii.gz
file contains only one tensor, which is either image or label.
This is the simplest case. Data are assumed to be stored under train/images
and train/labels
directories.
We only have images without any labels and all images are considered to be independent samples. So all data should be stored under train/images
, e.g.:
(It is also ok if the data are further grouped into different directories under images
as we will directly scan all nifti files under train/images
.)
In this case, we have both images and labels. So all images should be stored under train/images
and all labels should be stored under train/labels
. The corresponding image file name and label file name should be exactly the same, e.g.:
We have images without any labels, but images are grouped under different subjects/groups, e.g. time-series observations for each subject/group. For instance, the data set can be the CT scans of multiple patients (subjects/groups) where each patient has multiple scans acquired at different time points. So all data should be stored under train/images
and the leaf directories (directories that do not have sub-directories) must represent different subjects/groups, e.g.:
(It is also ok if the data are grouped into different directories, but the leaf directories will be considered as different subjects/groups.)
We have both images and labels. So all images should be stored under train/images
and all labels should be stored under train/labels
. The leaf directories will be considered as different subjects/groups and the corresponding image file name and label file name should be exactly the same, e.g.:
In this case, images are paired, for example, to represent a multimodal moving and fixed image pairs to register. Data are assumed to be stored under train/moving_images
, train/fixed_images
, train/moving_labels
, and train/fixed_labels
directories.
We only have paired images without any labels. So all data should be stored under train/moving_images
, train/fixed_images
and the images corresponding to the same subject should have exactly the same name, e.g.:
(It is ok if the data are further grouped into different directories under train/moving_images
and train/fixed_images
as we will directly scan all nifti files under them.)
We have both images and labels. So all data should be stored under train/moving_images
, train/fixed_images
, train/moving_labels
, and train/fixed_labels
. The images and labels corresponding to the same subjects/groups should have exactly the same names, e.g.:
Each .h5
file is similar to a dictionary, having multiple key-value pairs. Hierarchical multi-level h5 indexing is not used. Each value is either image or label.
Each key corresponds to one image, e.g. {"subject1": data1, "subject2": data1, ...}
. All data should be stored under train/images
, it can be a single h5 file or multiple h5 files e.g.:
Each key corresponds to one subject. Data can be stored in two single h5 files (one for image and one for label), the keys in the files should be the same.
Similar to case 1-1 above, but the keys, in this case, have to share the same format like subject%d-%d
where %d
represents a number. For instance, subject3-2
corresponds to the second observation for the subjects. Otherwise, the file structure is the same as case 1-1, e.g.
Similar to case 1-2 and 2-1 above, the keys have to share the same format like subject%d-%d
and the keys for images and labels should be consistent.
In this case, data are paired. Data are assumed to be stored under train/moving_images
, train/fixed_images
, train/moving_labels
, and train/fixed_labels
directories.
We only have paired images without any labels. So all data should be stored under train/moving_images
, train/fixed_images
and the keys corresponding to the same subject should be the same, e.g.:
We have both images and labels. So all data should be stored under train/moving_images
, train/fixed_images
, train/moving_labels
, and train/fixed_labels
. The keys corresponding to the same subject should be the same, e.g.:
This is related to #28 and #26
It seems like autodiff and linear interpolation don't play very nicely together (at least for classical registration approaches). It would be interesting to evaluate how regularised gradients can help and how they would impact learning-based registration as well.
Making even a toy example using autodiff and linear interpolation converge seems indeed complicated:
https://colab.research.google.com/drive/1OmU-jTvWmRRVerc23nsyNz_p8jU3S7wX?usp=sharing
The toy example simply involves registering two slightly offset crops of a 2D image. This is using MSE, translation and BFGS optimiser.
The reason for the convergence issue I think is related to resampling artifacts. deepreg/pytorch/jax currently offers only linear of nearest neighbour resampling, e.g.:
https://pytorch.org/docs/master/nn.functional.html#grid-sample
One option discussed in #26 could be to port the niftyreg-based resampling that had been integrated in niftynet:
https://github.com/NifTK/NiftyNet/tree/dev/niftynet/contrib/niftyreg_image_resampling
Another option is to continue use linear interpolation but implement gradient regularisation. When autograd is used with SSD and linear resampling, you indeed get non negligible local peaks in the cost function and is gradient when the translation parameters are integers.
Obviously, it's not ideal that autograd apparently can't be used directly for classical registration.
In ITK and other classical registration toolkits that do not use autodiff, we always relied on a chain rule for the continuous formulation that eventually implied using the gradient of the moving image (e.g. -2(F-MoT)∇MoT), thereby approximating the gradient of the actual cost function + resampling scheme. This actually has a smoothing/regularisation effect on the gradient which helps avoid local minima at pixel locations.
Note that further regularisation can be obtained by computing ∇MoT using a convolution with the gardient of a Gaussian kernel (see e.g. https://itk.org/Doxygen/html/classitk_1_1DiscreteGaussianDerivativeImageFilter.html)
One could think of implementing a custom interpolation layer that would return such a regularised gradient.
I guess one other option is to change the cost function to use a proper integral rather than a simple resampling the MSE on pixels but that doesn't sound super convenient either. This is briefly discussed in Modersitzki's classical text book (e.g. chapter 6.2):
https://archive.siam.org/books/fa06/Modersitzki_FAIR_2009_FA06.pdf
Adding a bit of noise on the coordinates of the grid points (i.e. some form of dithering) is another form of regularisation wich seems to help.
In terms of further references on this issue. The litterature seems sparse. I saw the paper below earlier but it does not discuss the gradient question:
https://doi.org/10.1109/TIP.2012.2224356
deepreg/model/network.py
Missing module docstring
Missing function docstring: build_inputs
Missing function docstring: build_model
Expand docstring for build_backbone
Expand docstring for build_ddf_model
- Expand dosctring for forward
- Expand docstring for add_loss_metric
- Expand dosctring for forward
- Expand docstring for add_loss_metric
Adding (normalised) mutual information as an image dissimilarity loss
Current globalnet with affine transformation has a format not consistent with other ddf or dvf pr conditional_registration.
## Type of issue
Please delete options that are not relevant.
refactored code
## Type of issue
Please delete options that are not relevant.
Should be a relatively easy fix by removing the line "show_default = True" in the argument --gpu_allow_growth for both train and predict scripts (line 145 in train, line 232 in predict).
tbc
The detailed information on config files and their key-value pairs
add configurations.md in /tutorials
add link to the wiki Tutorial Index
MR-to-ultrasound registration
data: https://learn2reg.grand-challenge.org/Dataset/
https://archive.sigma2.no/pages/public/datasetDetail.jsf?id=10.11582/2020.00025
This demo tests:
1 - weakly supervision
2 - conditional segmentation
3 - unsupervised intra-subject
and potentially
4 - demonstrating new loss
Remove nilearn dependency
## Type of issue
Improvement of code
with a paper format
https://joss.theoj.org/
Currently all the click commands exist outside of the main() functions as global vars which are then used in the command line interface, which makes them not PEP8 compliant and probably a bit nightmarish to test further along the line.
Refactor (/remove) reliance on click by converting any CLI interaction to argparse module.
Scripts which use click:
Examples of random-split and cross-validation
add experiment.md in /tutorials
add link to the wiki Tutorial Index
The detailed steps how to add a new (backbone) network
add add_network.md in /tutorials
add link to the wiki Tutorial Index
see also #6
comment what needs to be included
mirror the two "get started"
links to tutorials
links to demos
Apache 2 recommended.
In deepreg/predict, there is no reliance on a config file - instead, it is picked up from the ckpt path and assumes a similar file structure like:
This seems like a fragile patch that might benefit from modifying such that users can pass their own config file? Or is there some reason I don't understand for the design choice?
First I would like to understand the above, and if needs corrected we can assign/create a better way to access config?
## Type of issue
Please delete options that are not relevant.
N/A
Users pass config files related to their checkpoints manually such that the config file isn't taken from ckpt path.
Currently, the tests for this pass when running predict and train but I'm not entirely sure about the functionality of taking the path for config from ckpt string (it seems a bit strict).
deepreg/model/metric.py
For more information, see #5
The detailed steps how to add a new data loader
add add_loader.md in /tutorials
add link to the wiki Tutorial Index
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.