Giter VIP home page Giter VIP logo

Comments (8)

celstark avatar celstark commented on July 29, 2024

If it helps, there are some outputs that are perfectly fine and others that are borked:

In BallStick output
Same: AIC, AICc, BIC, Ball, LogLikelihood, OffsetGaussian.sigma, ReturnCodes, Stick0*, UsedMask, w_ball.w.std
Different: FS, S0.s0, w_ball.w, w_stick0.w

In NODDI output
Same: Ball.d, NODDI_?C.d, NODDI_?C.phi, NODDI_?C.theta, NODDI_?C.vec0, OffsetGaussian.sigma, UsedMask, w_{csf,ec,ic}.w.std
Different: AIC, AICc, BIC, LogLikelihood, NDI, NODDI_?C.dperp0, NODDI_?C.kappa, ODI, ReturnCodes, S0.s0, w_csf.w, w_stat

from mdt.

robbert-harms avatar robbert-harms commented on July 29, 2024

Hi Celstark,

Thanks for reporting in such detail. From what I understand from all your experiments is that MDT works fine as long as you run it on bare-metal, but as soon as you run it within Singularity, results default to 0.5?

I am also confused on why this would happen. Possible reasons I can think of are:

  • it is not using the right device within Singularity
  • you use a different set of models within Singularity?

The fact that it returns 0.5 is not a coincidence, this is the default starting value and if no computations are performed, this is typically what you would get as a result.

I am not very experienced with Singularity and as such I am afraid I can't be much of an help. Perhaps you could run the shell command "clinfo" within the singularity container and list the devices found in the container?

Best,

Robbert

from mdt.

celstark avatar celstark commented on July 29, 2024

Reviving this old issue as I've returned to hit the problem ;)

Unfortunately, neither of these ideas seem to be it. I've got trivially-tweaked Singularity images based off of the github code provided recipes that build both Intel and NVIDIA based Singularity images. I'm processing the same exact data with the same model in both.

For example:

singularity shell --nv -B /mnt/hippocampus/starkdata1 /mnt/extradata/singularity/MDT_cuda1a.sif 

Singularity> mdt-list-devices 
Device 0:
GPU - NVIDIA GeForce RTX 3070 (NVIDIA CUDA)
Device 1:
CPU - pthread-AMD Ryzen 7 5800X 8-Core Processor (Portable Computing Language)

Singularity> time mdt-model-fit NODDI_ExVivo sub-MCV747_dwi.nii sub-MCV747_dwi.prtcl sub-MCV747_dwi_mask.nii --cl-device-ind 0 -o output_MDT_cuda1a
[2023-01-03 17:45:32,087] [INFO] [mdt.lib.processing.model_fitting] [get_model_fit] - Starting intermediate optimization for generating initialization point.
[2023-01-03 17:45:32,130] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Using MDT version 1.2.6
[2023-01-03 17:45:32,130] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Preparing for model BallStick_r1
[2023-01-03 17:45:32,155] [INFO] [mdt.models.composite] [_prepare_input_data] - No volume options to apply, using all 92 volumes.
[2023-01-03 17:45:32,155] [INFO] [mdt.utils] [estimate_noise_std] - Trying to estimate a noise std.
[2023-01-03 17:45:32,157] [INFO] [mdt.utils] [estimate_noise_std] - Estimated global noise std 271.5927429199219.
[2023-01-03 17:45:32,157] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitting BallStick_r1 model
[2023-01-03 17:45:32,157] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - The 4 parameters we will fit are: ['S0.s0', 'w_stick0.w', 'Stick0.theta', 'Stick0.phi']
[2023-01-03 17:45:32,157] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Saving temporary results in output_MDT_cuda1a/BallStick_r1/tmp_results.
[2023-01-03 17:45:32,200] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 0.00%, processing next 44158 voxels (44158 voxels in total, 0 processed). Time spent: 0:00:00:00, time left: ? (d:h:m:s).
[2023-01-03 17:45:32,200] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting optimization
[2023-01-03 17:45:32,200] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using MOT version 0.11.3
[2023-01-03 17:45:32,200] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use a single precision float type for the calculations.
[2023-01-03 17:45:32,201] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using device 'GPU - NVIDIA GeForce RTX 3070 (NVIDIA CUDA)'.
[2023-01-03 17:45:32,201] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using compile flags: ('-cl-denorms-are-zero', '-cl-mad-enable', '-cl-no-signed-zeros')
[2023-01-03 17:45:32,201] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use the optimizer Powell with default settings.
[2023-01-03 17:45:33,282] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished optimization
[2023-01-03 17:45:33,282] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting post-processing
[2023-01-03 17:45:33,347] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished post-processing
[2023-01-03 17:45:33,435] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 100%
[2023-01-03 17:45:33,435] [INFO] [mdt.lib.processing.processing_strategies] [process] - Computed all voxels, now creating nifti's
[2023-01-03 17:45:33,597] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitted BallStick_r1 model with runtime 0:00:00:01 (d:h:m:s).
[2023-01-03 17:45:33,609] [INFO] [mdt.lib.processing.model_fitting] [get_model_fit] - Finished intermediate optimization for generating initialization point.
[2023-01-03 17:45:33,644] [INFO] [mdt] [fit_model] - Preparing NODDI_ExVivo with the cascaded initializations.
[2023-01-03 17:45:33,651] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Using MDT version 1.2.6
[2023-01-03 17:45:33,651] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Preparing for model NODDI_ExVivo
[2023-01-03 17:45:33,674] [INFO] [mdt.models.composite] [_prepare_input_data] - No volume options to apply, using all 92 volumes.
[2023-01-03 17:45:33,675] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitting NODDI_ExVivo model
[2023-01-03 17:45:33,675] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - The 7 parameters we will fit are: ['S0.s0', 'w_stat.w', 'w_ic.w', 'NODDI_IC.theta', 'NODDI_IC.phi', 'NODDI_IC.kappa', 'w_ec.w']
[2023-01-03 17:45:33,675] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Saving temporary results in output_MDT_cuda1a/NODDI_ExVivo/tmp_results.
[2023-01-03 17:45:33,716] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 0.00%, processing next 44158 voxels (44158 voxels in total, 0 processed). Time spent: 0:00:00:00, time left: ? (d:h:m:s).
[2023-01-03 17:45:33,716] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting optimization
[2023-01-03 17:45:33,716] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using MOT version 0.11.3
[2023-01-03 17:45:33,716] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use a single precision float type for the calculations.
[2023-01-03 17:45:33,716] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using device 'GPU - NVIDIA GeForce RTX 3070 (NVIDIA CUDA)'.
[2023-01-03 17:45:33,716] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using compile flags: ('-cl-denorms-are-zero', '-cl-mad-enable', '-cl-no-signed-zeros')
[2023-01-03 17:45:33,716] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use the optimizer Powell with default settings.
[2023-01-03 17:46:21,152] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished optimization
[2023-01-03 17:46:21,152] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting post-processing
[2023-01-03 17:46:21,249] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished post-processing
[2023-01-03 17:46:21,374] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 100%
[2023-01-03 17:46:21,374] [INFO] [mdt.lib.processing.processing_strategies] [process] - Computed all voxels, now creating nifti's
[2023-01-03 17:46:21,636] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitted NODDI_ExVivo model with runtime 0:00:00:47 (d:h:m:s).

real	0m51.014s
user	0m50.069s
sys	0m0.649s

I've run this a few times, even using different Singularity images built slightly differently and the results are always consistent. Runs inside Singularity on the GPU - check.

Now the Intel build:

(base) stark@titan:/mnt/hippocampus/starkdata1/Limoli/exvivo_flash/derivatives/mrtrix/mdt/sub-MCV747$ singularity shell -B /mnt/hippocampus/starkdata1 /mnt/extradata/singularity/MDT_intel1a.sif 
Singularity> mdt-list-devices 
Device 0:
CPU - AMD Ryzen 7 5800X 8-Core Processor              (Intel(R) OpenCL)
Singularity> time mdt-model-fit NODDI_ExVivo sub-MCV747_dwi.nii sub-MCV747_dwi.prtcl sub-MCV747_dwi_mask.nii --cl-device-ind 0 -o output_MDT_intel_titan
[2023-01-03 22:13:58,968] [INFO] [mdt.lib.processing.model_fitting] [get_model_fit] - Starting intermediate optimization for generating initialization point.
[2023-01-03 22:13:59,028] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Using MDT version 1.2.6
[2023-01-03 22:13:59,028] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Preparing for model BallStick_r1
[2023-01-03 22:13:59,071] [INFO] [mdt.models.composite] [_prepare_input_data] - No volume options to apply, using all 92 volumes.
[2023-01-03 22:13:59,071] [INFO] [mdt.utils] [estimate_noise_std] - Trying to estimate a noise std.
[2023-01-03 22:13:59,075] [INFO] [mdt.utils] [estimate_noise_std] - Estimated global noise std 271.5927429199219.
[2023-01-03 22:13:59,075] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitting BallStick_r1 model
[2023-01-03 22:13:59,075] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - The 4 parameters we will fit are: ['S0.s0', 'w_stick0.w', 'Stick0.theta', 'Stick0.phi']
[2023-01-03 22:13:59,075] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Saving temporary results in output_MDT_intel_titan/BallStick_r1/tmp_results.
[2023-01-03 22:13:59,130] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 0.00%, processing next 44158 voxels (44158 voxels in total, 0 processed). Time spent: 0:00:00:00, time left: ? (d:h:m:s).
[2023-01-03 22:13:59,130] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting optimization
[2023-01-03 22:13:59,130] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using MOT version 0.11.3
[2023-01-03 22:13:59,130] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use a single precision float type for the calculations.
[2023-01-03 22:13:59,130] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using device 'CPU - AMD Ryzen 7 5800X 8-Core Processor              (Intel(R) OpenCL)'.
[2023-01-03 22:13:59,130] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using compile flags: ('-cl-denorms-are-zero', '-cl-mad-enable', '-cl-no-signed-zeros')
[2023-01-03 22:13:59,130] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use the optimizer Powell with default settings.
/usr/lib/python3/dist-packages/pyopencl/__init__.py:63: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
  "to see more.", CompilerWarning)
[2023-01-03 22:14:05,226] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished optimization
[2023-01-03 22:14:05,226] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting post-processing
[2023-01-03 22:14:05,450] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished post-processing
[2023-01-03 22:14:05,556] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 100%
[2023-01-03 22:14:05,556] [INFO] [mdt.lib.processing.processing_strategies] [process] - Computed all voxels, now creating nifti's
[2023-01-03 22:14:05,720] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitted BallStick_r1 model with runtime 0:00:00:06 (d:h:m:s).
[2023-01-03 22:14:05,728] [INFO] [mdt.lib.processing.model_fitting] [get_model_fit] - Finished intermediate optimization for generating initialization point.
[2023-01-03 22:14:05,767] [INFO] [mdt] [fit_model] - Preparing NODDI_ExVivo with the cascaded initializations.
[2023-01-03 22:14:05,774] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Using MDT version 1.2.6
[2023-01-03 22:14:05,774] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Preparing for model NODDI_ExVivo
[2023-01-03 22:14:05,808] [INFO] [mdt.models.composite] [_prepare_input_data] - No volume options to apply, using all 92 volumes.
[2023-01-03 22:14:05,809] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitting NODDI_ExVivo model
[2023-01-03 22:14:05,809] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - The 7 parameters we will fit are: ['S0.s0', 'w_stat.w', 'w_ic.w', 'NODDI_IC.theta', 'NODDI_IC.phi', 'NODDI_IC.kappa', 'w_ec.w']
[2023-01-03 22:14:05,809] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Saving temporary results in output_MDT_intel_titan/NODDI_ExVivo/tmp_results.
[2023-01-03 22:14:05,862] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 0.00%, processing next 44158 voxels (44158 voxels in total, 0 processed). Time spent: 0:00:00:00, time left: ? (d:h:m:s).
[2023-01-03 22:14:05,862] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting optimization
[2023-01-03 22:14:05,862] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using MOT version 0.11.3
[2023-01-03 22:14:05,862] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use a single precision float type for the calculations.
[2023-01-03 22:14:05,862] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using device 'CPU - AMD Ryzen 7 5800X 8-Core Processor              (Intel(R) OpenCL)'.
[2023-01-03 22:14:05,862] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using compile flags: ('-cl-denorms-are-zero', '-cl-mad-enable', '-cl-no-signed-zeros')
[2023-01-03 22:14:05,862] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use the optimizer Powell with default settings.
[2023-01-03 22:15:32,990] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished optimization
[2023-01-03 22:15:32,991] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting post-processing
[2023-01-03 22:15:33,432] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished post-processing
[2023-01-03 22:15:33,554] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 100%
[2023-01-03 22:15:33,554] [INFO] [mdt.lib.processing.processing_strategies] [process] - Computed all voxels, now creating nifti's
[2023-01-03 22:15:33,789] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitted NODDI_ExVivo model with runtime 0:00:01:27 (d:h:m:s).

real	1m35.980s
user	24m6.364s
sys	0m0.943s

We run and we get output, but it's not quite right and differs from run to run as if some matrix actually isn't getting initialized. Yes, that's an AMD processor, so here's a 144-core Intel one:

singularity shell -B /mnt/hippocampus/starkdata1 /mnt/hippocampus/starkdata1/Mouse_DWI_tutorial/code/MDT_intel1a.sif
Singularity> mdt-list-devices 
Device 0:
CPU - Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz (Intel(R) OpenCL)
Singularity> time mdt-model-fit NODDI_ExVivo sub-MCV747_dwi.nii sub-MCV747_dwi.prtcl sub-MCV747_dwi_mask.nii --cl-device-ind 0 -o output_MDT_intel_wario
[2023-01-03 22:30:54,282] [INFO] [mdt.lib.processing.model_fitting] [get_model_fit] - Starting intermediate optimization for generating initialization point.
[2023-01-03 22:30:54,409] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Using MDT version 1.2.6
[2023-01-03 22:30:54,409] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Preparing for model BallStick_r1
[2023-01-03 22:30:54,499] [INFO] [mdt.models.composite] [_prepare_input_data] - No volume options to apply, using all 92 volumes.
[2023-01-03 22:30:54,500] [INFO] [mdt.utils] [estimate_noise_std] - Trying to estimate a noise std.
[2023-01-03 22:30:54,505] [INFO] [mdt.utils] [estimate_noise_std] - Estimated global noise std 271.5927429199219.
[2023-01-03 22:30:54,505] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitting BallStick_r1 model
[2023-01-03 22:30:54,505] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - The 4 parameters we will fit are: ['S0.s0', 'w_stick0.w', 'Stick0.theta', 'Stick0.phi']
[2023-01-03 22:30:54,505] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Saving temporary results in output_MDT_intel_wario/BallStick_r1/tmp_results.
[2023-01-03 22:30:54,604] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 0.00%, processing next 44158 voxels (44158 voxels in total, 0 processed). Time spent: 0:00:00:00, time left: ? (d:h:m:s).
[2023-01-03 22:30:54,604] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting optimization
[2023-01-03 22:30:54,604] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using MOT version 0.11.3
[2023-01-03 22:30:54,605] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use a single precision float type for the calculations.
[2023-01-03 22:30:54,605] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using device 'CPU - Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz (Intel(R) OpenCL)'.
[2023-01-03 22:30:54,605] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using compile flags: ('-cl-denorms-are-zero', '-cl-mad-enable', '-cl-no-signed-zeros')
[2023-01-03 22:30:54,605] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use the optimizer Powell with default settings.
/usr/lib/python3/dist-packages/pyopencl/__init__.py:63: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
  "to see more.", CompilerWarning)
[2023-01-03 22:30:57,504] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished optimization
[2023-01-03 22:30:57,506] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting post-processing
[2023-01-03 22:30:58,073] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished post-processing
[2023-01-03 22:30:58,220] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 100%
[2023-01-03 22:30:58,220] [INFO] [mdt.lib.processing.processing_strategies] [process] - Computed all voxels, now creating nifti's
[2023-01-03 22:30:58,494] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitted BallStick_r1 model with runtime 0:00:00:03 (d:h:m:s).
[2023-01-03 22:30:58,514] [INFO] [mdt.lib.processing.model_fitting] [get_model_fit] - Finished intermediate optimization for generating initialization point.
[2023-01-03 22:30:58,580] [INFO] [mdt] [fit_model] - Preparing NODDI_ExVivo with the cascaded initializations.
[2023-01-03 22:30:58,591] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Using MDT version 1.2.6
[2023-01-03 22:30:58,591] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Preparing for model NODDI_ExVivo
[2023-01-03 22:30:58,656] [INFO] [mdt.models.composite] [_prepare_input_data] - No volume options to apply, using all 92 volumes.
[2023-01-03 22:30:58,657] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitting NODDI_ExVivo model
[2023-01-03 22:30:58,657] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - The 7 parameters we will fit are: ['S0.s0', 'w_stat.w', 'w_ic.w', 'NODDI_IC.theta', 'NODDI_IC.phi', 'NODDI_IC.kappa', 'w_ec.w']
[2023-01-03 22:30:58,657] [INFO] [mdt.lib.processing.model_fitting] [fit_composite_model] - Saving temporary results in output_MDT_intel_wario/NODDI_ExVivo/tmp_results.
[2023-01-03 22:30:58,753] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 0.00%, processing next 44158 voxels (44158 voxels in total, 0 processed). Time spent: 0:00:00:00, time left: ? (d:h:m:s).
[2023-01-03 22:30:58,753] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting optimization
[2023-01-03 22:30:58,753] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using MOT version 0.11.3
[2023-01-03 22:30:58,753] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use a single precision float type for the calculations.
[2023-01-03 22:30:58,753] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using device 'CPU - Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz (Intel(R) OpenCL)'.
[2023-01-03 22:30:58,753] [INFO] [mdt.lib.processing.model_fitting] [_process] - Using compile flags: ('-cl-denorms-are-zero', '-cl-mad-enable', '-cl-no-signed-zeros')
[2023-01-03 22:30:58,753] [INFO] [mdt.lib.processing.model_fitting] [_process] - We will use the optimizer Powell with default settings.
[2023-01-03 22:31:19,729] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished optimization
[2023-01-03 22:31:19,730] [INFO] [mdt.lib.processing.model_fitting] [_process] - Starting post-processing
[2023-01-03 22:31:20,698] [INFO] [mdt.lib.processing.model_fitting] [_process] - Finished post-processing
[2023-01-03 22:31:20,891] [INFO] [mdt.lib.processing.processing_strategies] [_process_chunk] - Computations are at 100%
[2023-01-03 22:31:20,892] [INFO] [mdt.lib.processing.processing_strategies] [process] - Computed all voxels, now creating nifti's
[2023-01-03 22:31:21,353] [INFO] [mdt.lib.processing.model_fitting] [_model_fit_logging] - Fitted NODDI_ExVivo model with runtime 0:00:00:22 (d:h:m:s).

real	0m29.667s
user	38m5.241s
sys	0m3.145s

The attached screenshot shows 2 runs in the Singularity NVIDIA on the top (identical) and 3 runs (2 on the "titan" AMD machine and 1 on the "wario" Intel machine). The bands in one are at 0.5 exactly.

Screenshot 2023-01-03 225049

from mdt.

robbert-harms avatar robbert-harms commented on July 29, 2024

Hi celstrark,

I am sorry to hear that MDT gave you so much problems. The problem is that OpenCL is so ill-supported by different vendors.

At this point I don't know how to help you. Family and work are taking 120% of my time already.

Are you interested in developing MDT further?

Best,

Robbert

from mdt.

celstark avatar celstark commented on July 29, 2024

I understand the constraints and am under similar ones here. On our end, MDT works wonderfully so long as we run it in a more typical setup. So, for this step in the processing, we will just run things manually. I poked around for a bit to see if I could spot something to no avail, but I'll keep this in the back of my mind.

Thanks for all you've done with the package!

Craig

from mdt.

fkm24 avatar fkm24 commented on July 29, 2024

Hi Craig,

Many thanks for sharing the Singularity recipe. I have implemented your recipe on our HPC and came across the same issue, with ODI values fixed at 0.5.

It sounds like you have a working solution to run MDT on your workstation though, would you be happy to share more about your setup so we can continue to implement MDT? That will be much appreciated!

Best Wishes,
Eli

from mdt.

celstark avatar celstark commented on July 29, 2024

On the workstation, it's running via CUDA. Here's the definition file I used:

Bootstrap: docker
From: nvidia/opencl

%setup
        mkdir -p $SINGULARITY_ROOTFS/src
        cp containers/silent.cfg $SINGULARITY_ROOTFS/src

%post
        # install dependencies
        apt-get update && apt-get install -y lsb-core wget locales

        # install mdt
        export DEBIAN_FRONTEND=noninteractive
        apt-get update && apt-get install -y software-properties-common && add-apt-repository ppa:robbert-harms/cbclab
        apt-get update && apt-get install -y python3-mdt python3-pip
        pip3 install tatsu==4.2.6

As you can see, not much to it -- really just your Dockerfile.nvidia converted over. I never did get a working setup to run on the CPU in a container.

from mdt.

fkm24 avatar fkm24 commented on July 29, 2024

Thanks very much for sharing this, Craig. I have implemented this on our HPC and the script was able to detect the GPUs.

Device 1:
GPU - NVIDIA A100-SXM4-80GB (NVIDIA CUDA)
Device 2:
GPU - NVIDIA A100-SXM4-80GB (NVIDIA CUDA)
Device 3:
GPU - NVIDIA A100-SXM4-80GB (NVIDIA CUDA)

However, the script is still getting stuck at the “We will use the optimiser Powell with default settings” (#58).

I think it might be related to my OpenCL too. When I ran clinfo, it says:

 NOTE:   your OpenCL library only supports OpenCL 2.2,
                but some installed platforms support OpenCL 3.0.
                Programs using 3.0 features may crash
                or behave unexepectedly

I will ask the system administrators to update these drivers and see if that helps.

Best Wishes,
Eli

from mdt.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.