file needs to be put in the new_patient_pipeline folder

error: MRISread(/.../meld_classifier_v1.1.0/data/fs_outputs/MELD_< ID >/surf/rh.pial): could not open file [ ok ]
error: No such file or directory
error: mris_curvature_stats: could not read surface file /.../meld_classifier_v1.1.0/data/fs_outputs/MELD_< ID >/surf/rh.pial

Observation:

running prediction for single subject works fine, but process is killed for 100+ subjects because of memory overflow

e.g. data_preprocessing.py line 160:

hdf5_file_context = h5py.File(hdf5_file, "r")

with hdf5_file_context as f:

is erroneous code, the file is opened before 'with'

run_script_preprocessing.py takes quite a while and sometimes has run already, but there is no option to skip it (or skip automatically)

It would be convenient to have a container, such as a docker container, that includes all of the software and data needed to apply the MELD classifier to a subject.

An example of a Dockerfile that would build such an environment is here:

FROM ubuntu:jammy AS FCDDetection
ENV DEBIAN_FRONTEND="noninteractive"

Update the ubuntu.

RUN apt-get -y update &&
apt-get -y upgrade

Install the prerequisite software

RUN apt-get install -y build-essential
apt-utils
vim
nano
curl
wget
pip
python3
git

Install freesurfer in /opt/freesurfer.

RUN mkdir -p /freesurfertar
&& curl -SL https://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/7.3.2/freesurfer-linux-ubuntu22_amd64-7.3.2.tar.gz
| tar -xzC /freesurfertar
&& mv /freesurfertar/freesurfer /opt/
&& rmdir /freesurfertar

This will also need to be configured in the running docker container.

This modifies only the build environment.

ENV PATH=/opt/freesurfer/bin:$PATH
RUN echo "PATH=/opt/freesurfer/bin:$PATH" >> ~/.bashrc
ENV FREESURFER_HOME=/opt/freesurfer
RUN echo "FREESURFER_HOME=/opt/freesurfer" >> ~/.bashrc

Install miniconda

ENV CONDA_DIR /opt/conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh &&
/bin/bash ~/miniconda.sh -b -p /opt/conda

Put conda in path so we can use conda activate

ENV PATH=$CONDA_DIR/bin:$PATH

Update to the newest version of conda:

RUN conda update -n base -c defaults conda

RUN conda init bash

checkout and install the github repo

RUN cd / && git clone https://github.com/MELDProject/meld_classifier.git

enter the meld_classifier directory

create the meld classifier environment with all the dependencies

SHELL ["/bin/bash", "-c"]
RUN cd /meld_classifier && conda env create -f environment.yml
RUN cd /meld_classifier && conda run -n meld_classifier /bin/bash -c "pip install -e ."

Use the bash shell this way because conda cant activate.

#SHELL ["conda", "run", "-n", "meld_classifier", "/bin/bash", "-c"]

Configure MELD for this docker image:

COPY ./meld_config.ini /meld_classifier/meld_config.ini
RUN mkdir /MELD-data

RUN conda run --no-capture-output -n meld_classifier /bin/bash -c " python3 /meld_classifier/scripts/prepare_classifier.py --skip-config"

COPY user-freesurfer-license.txt ${FREESURFER_HOME}/.license

RUN cd /meld_classifier && conda run --no-capture-output -n meld_classifier /bin/bash -c " pytest "

Option A:

provide script to transform BIDS conform data to the MELD format

Option B [preferred ?]:

include bids as a dependency and make all scripts expect BIDS conform data

Generating predictions via run_script_prediction.py crashes consistently after 30 subjects (for 128 GB RAM).

This still happens when using the --split option.

especially recon-surf instead of recon-all with similar performance at much shorter processing times?

--> would probably require validation first ?

requires update from python 3.7

Need to remove features_smooth as not used in the code after

due to:

proc = Popen(command, shell=True, )
proc.wait()  # <---------------

in the run_command function

relates to #14
but also e.g. new_pt_pipeline_script2.py does nothing except call run_data_processing_new_subjects.py

1. python can be called within python and not via subprocess.
2. freesurfer commands are called from within python for new_pt_pipeline_script1.py e.g. but via bash for create_xhemi.sh.

Dear all,

If I understood correctly, when using different MRI sequences images such as MP2RAGE, which were not used in the original training, then a new training should be done and the already existing trained model cannot be used?

Best

Need to clarify file naming for the 1.5T data is 15T.

Currently it just checks if the subject folder exist in the fs_outputs directory

(in new_pt_pipeline_script1.py)
if os.path.isdir(os.path.join(fs_folder, subject)):

If not all outputs have been generated, the rest of the pipeline fails.
Rather check if all necessary feature files exist and create missing ones specifically.

reformat code according to autopep8 and/or specify code formatting

It seems I cannot run the prediction on my full cohort at once because it uses 120+ GB of ram for 170 subjects

It seems to build up a large "exp" MeldCohort class, which might be unnecessary?
At least it should internally split the pipeline or run subjects sequentially.

Hello MELD team,

The project seems very cool, however it seems impossible to play wit its own MRI data without having a model dataset === having an access to lots of EEG records other patients to build a model.

In my opinion, no law would be broken in case you publish the model build on a patient's data publicly in this or in the separate repository so all people could try the project without contacting or being locked on a medical universities. Probably several models should be published per specific MRI scanner I guess. As an example OpenAI company - they harvested data from all over the world and everything is fine so far. Could we give a possibility to every individual to run the project it would help for the project development overall.

new_site_harmonisation_script.py accepts a relative path for -ids but must be an absolute path for -demos

File "/.../meld_classifier_v1.1.0/scripts/manage_results/plot_prediction_report.py", line 180, in get_cluster_location
ind = pred_rois[np.where(pred_rois == pred_rois[:,1].max())[0]][0][0]
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

Currently, processing skips a step if the main directory, which is created in the beginning, exists.

It would be better to check for files indicating a step of the pipeline has finished successfully.

The gziped predictions require ~O(2) less space!

fs_outputs folder contains non fs_outputs such as the prediction.mgh and everything under 'surf_meld'

Make new_pt_pipeline.py --ids input (e.g.) accept both:

--ids < some file under data/ > or < path to some file >

A lot of times if files are not found or paths are incorrect the pipeline just crashes without a specific error message.

Checks if a previous pipeline step was successful (i.e. all necessary files were produced) would be preferable, along with messages if that is not the case.

should 'required_subject_features' not be linked with 'subject_features_to_include' ?

same in