jstout211 / enigma_meg Goto Github PK

Need to make a QA folder
Need to make a logfile in the QA folder

In the final process_meg function:
Write raw-cov.fif
Write eroom-cov.fif
Write -trans.fif
Write downsampled...fif

from mne import Report

#Run the following to collect and write out the QA files
report = Report(image_format='png', subjects_dir=subjects_dir,
subject='APBWVFAR_fs_ortho', info_fname='APBWVFAR_300fs_meg.fif',
raw_psd=False) # use False for speed here

meg_path = '/home/stoutjd/data/DEMO_ENIGMA/outputs/APBWVFAR_fs_ortho'
report.parse_folder(meg_path, on_error='ignore', mri_decim=10)
report.save('outfile.html')

Better organize argparse so that required and common options are listed together

There are a bunch of output directories that default to the local dir as an output for MEGnet. Internally enigma redirects these to the bids derivatives folder, leaving an empty directory.

Something in here is causing errors that prevent the code from completing

        QAdir = op.join(self.bids_root,'derivatives/ENIGMA_MEG_QA')
        QAsubjdir = QAdir + '/sub-' + self.subject + '/ses-' + self.session
        figname_icaoverlay = QAsubjdir + '/sub-' + self.subject + '_ses-' + self.session + '_run-' + self.run + '_cleaned.png'
        ica=mne.preprocessing.read_ica(op.join(self.fnames.ica))
        ica.exclude = self.ica_comps_toremove #meg_rest_raw.icacomps
        self.load_data()
        fig=ica.plot_overlay(self.raw_rest, exclude=self.ica_comps_toremove)
        fig.savefig(figname_icaoverlay)

Remove model dependency on tensorflow addons (currently required because of the f1_score). Tensorflow addons is currently required to load the model. The source code support will be deprecated 2024.

Solution - compile model without f1_score metric or use f1_score metric from another source -- see below announcement. Sci-kit learn also has an f1 score metric.

planned end of life in May 2024. Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP)

For more information see: https://github.com/tensorflow/addons/issues/2807

use
import warnings
warnings.filter( )

• Add output file to proc.fnames
• Add test to check for csv file output

In commit fbdf2e7, the below change was made. @nugenta - confirm this is correct.

•                inversion='matrix', weight_norm='unit-noise-gain', rank=noise_rank)
  

•                inversion='matrix', weight_norm='unit-noise-gain', rank=epo_rank)
  

f-string incorporates session - which errors out when optional session is None - line 827 in process_meg

01/10/23
MOUS data processed using: 913824e
NIH_mdd : 913824e (no ER noise used)
CAMCAN: 913824e

01/11/23
Omega data processed using: b4b5e43
Only the first rest run in the series was used - found using extras/OMEGA/make_swarm_file.py
(no computational diffs from 913824e)

This has been done in beamformer simulation code, needs to be updated here

There is a lag between "page" turns / next because the images have to be resized and converted to base64.

This should be done before the GUI component is run to prevent lags.

    proc.do_proc_allsteps()
  File "/vf/users/MEGmodules/modules/enigma_meg0.2_extras/enigma_MEG/enigmeg/process_meg.py", line 504, in do_proc_allsteps
    self.do_beamformer()
  File "/vf/users/MEGmodules/modules/enigma_meg0.2_extras/enigma_MEG/enigmeg/process_meg.py", line 374, in do_beamformer
    if epo_rank < noise_rank:
TypeError: '<' not supported between instances of 'dict' and 'dict'

Solution - set the rank according to sensor type

            if 'mag' in epo_rank:
                if epo_rank['mag'] < noise_rank['mag']:
                    noise_rank['mag']=epo_rank['mag']
            if 'grad' in epo_rank:
                if epo_rank['grad'] < noise_rank['grad']:
                    noise_rank['grad']=epo_rank['grad']

This is probably an option in PIL - maybe PIL.ImageFont (?)

The make QAfsrecon - is now a function and is not called by default.

Maybe move this to another file

Before computing anythiny, just check that all the required input datasets are present.

Use .csv to link the converted meg and mris

Build routine to run after parsing commandline inputs, but before calling main function.

If input is null >> check config file and populate

Use - github actions yaml from nih2mne code for data pull

The mri_segstats is part of the parameterize spectrum code - so if it errors, the logging will report out incorrectly

Move the function initialize from csv to outside of the process object. This can just be a function that loads the process object.

def initialize_fromcsv(self, csv_info)

Remove datasets that have been generated previously except the freesurfer stuff

Logfile additions

• Add total time
• Good epochs
• Bad epochs

ERROR:23490_ses_None:vendor_prep :: 'NoneType' object has no attribute 'compensation_grade'
Traceback (most recent call last):
File "/vf/users/MEGmodules/modules/enigma_meg0.5_extras/enigma_MEG/enigmeg/process_meg.py", line 78, in wrapper
output = function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vf/users/MEGmodules/modules/enigma_meg0.5_extras/enigma_MEG/enigmeg/process_meg.py", line 447, in vendor_prep
if self.raw_eroom.compensation_grade != 3:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'compensation_grade'
Traceback (most recent call last):
File "/vf/users/MEGmodules/modules/enigma_meg0.5/bin/process_meg.py", line 7, in
exec(compile(f.read(), file, 'exec'))
File "/vf/users/MEGmodules/modules/enigma_meg0.5_extras/enigma_MEG/enigmeg/process_meg.py", line 1403, in
process_subject(args.subject, args) # process the single specified subject
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vf/users/MEGmodules/modules/enigma_meg0.5_extras/enigma_MEG/enigmeg/process_meg.py", line 1134, in process_subject
proc.do_proc_allsteps()
File "/vf/users/MEGmodules/modules/enigma_meg0.5_extras/enigma_MEG/enigmeg/process_meg.py", line 836, in do_proc_allsteps
self.vendor_prep()
File "/vf/users/MEGmodules/modules/enigma_meg0.5_extras/enigma_MEG/enigmeg/process_meg.py", line 78, in wrapper
output = function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vf/users/MEGmodules/modules/enigma_meg0.5_extras/enigma_MEG/enigmeg/process_meg.py", line 447, in vendor_prep
if self.raw_eroom.compensation_grade != 3:

Code doesn't seem to work for the CAMCAN or MOUS datasets

Read previous logfile
Parse the last time
Readout all of the subject status items into a dictionary
During subject initialization - reference the history dictionary to pull status and initialize button color

Users can specify a freesurfer directory not in BIDS_ROOT/derivatives/freesurfer/subjects if their subjects have already been processed using FreeSurfer. In this case, it is quite likely that the BIDS subject ID (which is always appended with sub-, as in sub-Sub01) is different from the Freesurfer subject ID (which is most likely Sub01, using the prior example). We should make this combination of options work. At the moment, it doesn't. This would involve changes to both process_meg.py and enigma_prep_QA.py

In group processing add log to perform set difference on demographics and enigma output dataframes

Need to loop over entire tree rather than call for individual subjects. Should this be in a multi processing pool? And there will need to be handling of BIDS trees that have more than one run/session per subject, or meg and anat in different sessions. Write separate function to handle different scenarios.

In compile_alpha_hist_plots (to be renamed) - provide a parc file from fsaverage. Remove any labels in dataframe that are not part of the current parc (eg. aparc vs aparc_sub).

Currently the button is defined by the image - therefore having the item change with a button press requires some more thought.
Possibly of use: PySimpleGUI/PySimpleGUI#5420
PySimpleGUI/PySimpleGUI#4909

Use pandas to pull headers from csv file and submit entries for analysis.

Check DEC120 session 2 run 01 - ambiguity in bids path

Merge test code in the engimeg/group folder into the calc_display_stats.py file. Add this to the group analysis pipeline.

The preprocessing steps can differ between megnet and enigma_meg - leading to a mismatch between the number of channels in the ICA apply operation.

Currently the datatype and processing pull the suffix to identify the data type.

If the ENIGMA_MEG_QA is not created - mkdir it.

(enigma_meg_test) jstout@kani:/fast/ENIGMA_OHBM$ ~/src/enigma_MEG/enigmeg/QA/Run_enigma_QA_GUI.py -bids_root $(pwd) -QAtype 'bem'
Traceback (most recent call last):
  File "/home/jstout/src/enigma_MEG/enigmeg/QA/Run_enigma_QA_GUI.py", line 82, in <module>
    history_log = initialize(bids_root, QAtype)        
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jstout/src/enigma_MEG/enigmeg/QA/enigma_QA_GUI_functions.py", line 170, in initialize
    logging.basicConfig(filename=logfile, encoding='utf-8', level=logging.INFO, 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jstout/miniconda3/envs/enigma_meg_test/lib/python3.11/logging/__init__.py", line 2050, in basicConfig
    h = FileHandler(filename, mode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jstout/miniconda3/envs/enigma_meg_test/lib/python3.11/logging/__init__.py", line 1181, in __init__
    StreamHandler.__init__(self, self._open())
                                 ^^^^^^^^^^^^
  File "/home/jstout/miniconda3/envs/enigma_meg_test/lib/python3.11/logging/__init__.py", line 1213, in _open
    return open_func(self.baseFilename, self.mode,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/fast/ENIGMA_OHBM/derivatives/ENIGMA_MEG_QA/bem_QA_logfile.txt'

filename was added to assess ICA to help with pytest code.

• QA list will be empty
• Upon review - the classification will be unchecked
• After the next/final button is selected - change to good/bad
• Write good/bad to QA text file