mmagnuski / eegdb Goto Github PK

General goals

1. Redesign the way recovered EEG is held in winreject
2. Redesign the checks performed to see if we need to recover again (separate function!)

Discussion

So far the ICAw version that is being recovered was placed in EEG.etc.recov and this was later used to see if it is still relevant.

Consider

❓ the EEG kept in memory could be more raw than what is displayed. Even if bad channels are defined interpolation can be done on the fly. The same with epoch rejection ( all or at least 'post' epochs are held in memory and are ony selected according to rejections when needed)

Related:
github-linguist/linguist#1456

For some reason it seems that choosing two mark types at once to apply as rejections applies only one of these.

Dropbox artifacts are littering project repository. We need to compare the original files with conflicting files and then if nothing important present in the conflicting ones - remove them.

This is a rather long-term plan but it would be nice to decide/find out what would be the best way to organize our GUI behavior (best in terms of convenience and function - like synchronizing windows).

Blocking GUIs are easy because they can have simple input-output pipelinie without worrying about how to synchronize with base workspace etc.
But - they are blocking - command line input is not possible, which is rather sad 😢
Going the non-blocking way is neither easy nor elegant on the other hand - usually requires either awful lot of guidata() usage and assignin, evalin kind of stuff or using global variables which are evil 🔥

We will try to work out some standard here. This is a goal far beyond v 0.1 - but we should keep this in mind.

some files may be unnecessary - these should be removed.
(files that are in progress etc. will be moved to relevant branches)

Discussion

One problem I am currently having is how eegDb.epoch should work.
It should have short, decriptive fields that work for onesecepoch and/or classical epoching.

currently the assumptions are:

1. locked informs whether onesecepoch is used. When locked is false it means we use epoching that is not event-locked. If no other fields (winlen, distance, etc.) are present this is the only field that differentiatiates whether onesec epoching is used.
   • If locked is false and there are no other fields, default onesecepoch options are assumed
   • if fields are present and do not agree with locked - the user is asked for it OR the locked is changed to match field values

|
|

❓ Default onesec options could be filled by some function (or option in buildbase?) for example consec_windows(eegDb)
❓ Use subfields epoch.locked and epoch.consec

In menu use consecutive windows instead of onesecepoch - this would be more meaningful.
(or some other name - any suggestions?)

Tasks

• check how recent ICAw_checkepoch (or something like this) checks epoching
• write some code that takes care of keeping epoch field in line with assumptions

It would be really nice if users could assign multiple marks to a given epoch / signal segments.
Currently clicking on a signal segment marks (if no marks are present) or unmarks (if at least one is present) the segment.
But - before implementing multi-selection we should figure out how it should work.

🚧 IN PROGRESS

Problems

There are some problems with the current rejections system:

• rejections are managed inconcistently
• there is no single, unified function for adding/removing rejections that would take all the mess into account
• moreover - there seems to be no consensus as to how some things should be handled - most non-obvious cases were just temporary fixes/solutions

Discussion

Handling reject.pre before and after onesec as well as when epoch.distance changes

When onesec has already filled reject.pre:

• clear distance (and fill reject.pre) to signal that it has been performed
  but this is problematic because we loose info on what epoch.distance was used
  epoch.distance can be moved to datainfo but this also generates some inconsistencies: datainfo with epoch field should mean that epoching has been already performed (and is present in the file) and how

  % distance option, epoching should be checking
  % pre - if it is filled, no need to check
  % distance again. If we change distance -->
  % some reworking is needed to save post
  % but clear pre. May be problematic if
  % pre is empty after distance was applied
  % use .distUsed or internal.distUsed ??

Use NOHELPINFO tag-word to mark these functions

1. avoid using GUIDE

2. write GUIs as objects - it eases up many things like graphical updates, syncing etc.

3. Externalize local functions that make sense to be used publicly

   for example - move winreject_refresh(h) out of winreject
   or - even better make refresh a pulic method of winreject object

4. Use linkfun_ prefix for functions that interface between GUIs (link functions).

   for example - a function that opens up multiedit_gui, sets relevant callbacks, validates output and updates main GUI would be a link function

5. have a clear structure in guidata (winreject has its guidata quite messy - this will be changed)

6. guidata should only store handles to figure objects, all other data should be setappdata and getappdata

Just like when ICAw_cleanline is run - files are saved and the database is reformatted (some info goes to datainfo etc.).

Although filtering and epoching etc. does not take long - in some cases it can take 5-10 seconds while loading a filtered and epoched file would be much faster. Also - we should remember than it would be definitely longer with 1000 Hz file for example or with much more epochs than 400.

• It should be optional to save files at some point and continue on the saved files.
• operations performed so far on the files should be moved to datainfo
• If files are saved as epoched and filtered - changing these fields should not make any effect.
• Going back to the original file with changes performed (like - marks, rejections) should be possible.

epoch_events are added directly in ICAw
(should be: events in ICAw.epoch)

Tasks/issues etc. should be all here.
xls files are not good for tracking progress ⛄

There are many files in the tests folder. These should be looked through and then either:

• deleted
• moved to a branch (or temporary should be in a branch folder)

or

• moved to core files (future master) and documented

Adding mark in cooleegplot:

Error using vertcat
Dimensions of matrices being concatenated are not consistent.

Error in eegplot2>add_rejcol_callb (line 2196)
g.labcol = [g.labcol; c];

Error using waitfor
Error while evaluating uicontrol Callback

Currently winreject has following fields in guidata:

figure1: 175.0266
opts: 182.0266
title_text: 181.0266
savingstruct: 180.0266
save2file: 179.0266
text6: 178.0266
versions_pop: 177.0266
vers: 16.0267
runICA: 15.0267
applyrej: 14.0267
multisel: 13.0267
EEGreco: 12.0271
slider: 11.0271
prev_butt: 10.0271
addit_text: 9.0271
next_butt: 8.0271
badel_butt: 7.0271
text2: 6.0271
notes_win: 5.0271
CL_checkbox: 4.0271
info_text: 3.0271
done_butt: 2.0276
dataplot_butt: 176.0266
coolplo_opt: 184.0266
recover_opts: 183.0266
manage_versions: 19.0267
clearica: 18.0267
movevers: 17.0267
output: 175.0266
UniOrig: 'pixels'
ecol: 'cosmic bubblegum'
ICAw: [1x61 struct]
ICAw_start: [1x61 struct]
r: 1
EEG: []
rEEG: []
figure2: []
selected: []
CloseReq: 0
multisel_col: [0.9412 0.9412 0.9412]
structpath: 0
recovopts: {[]}
cooleegopts: {'ecol' 'off' 'winlen' [3] 'badplot' 'grey' 'lsmo' 'on'}
last_recovered_opts: {[]}
recov: [1x61 logical]

Add full keyboard support to navigation in eegplot, similar to the way navigation is done in vim editor.
For example - h j k and l used for moving left, down, up, right with typing
11l moving eleven epochs/seconds right

Other things to attach keyboard shortcuts to:

• Selecting epochs
• choosing marks
• moving around with respect to marks
• etc.

This will need some designing instead of blindly implementing just so it would work as soon as possible (as I was usually doing in early days of eggDb 😭)

⚠️ This will require substantial reworking of eegplot (see #12)
💎 but will also be increadibly cool if implemented

Check out this error:

Attempt to reference field of non-structure array.

Error in winreject>dataplot_butt_Callback (line 198)
    currf = handles.ICAw(handles.r).versions.current;

Error in gui_mainfcn (line 96)
        feval(varargin{:});

Error in winreject (line 55)
    gui_mainfcn(gui_State, varargin{:});

Error in
@(hObject,eventdata)winreject('dataplot_butt_Callback',hObject,eventdata,guidata(hObject))


Error using waitfor
Error while evaluating uicontrol Callback

Error using fread
Invalid file identifier.  Use fopen to generate a valid file identifier.

Error in eegDb_fastread (line 14)
EEG.data = fread(fid, [EEG.nbchan, EEG.pnts * EEG.trials], 'float32');

Error in recoverEEG (line 287)
    EEG = eegDb_fastread(pth, ICAw(r).filename);

Error in winreject>dataplot_butt_Callback (line 215)
        handles.EEG = recoverEEG(handles.ICAw, handles.r, 'local', handles.recovopts{:});

Error in gui_mainfcn (line 96)
        feval(varargin{:});

Error in winreject (line 55)
    gui_mainfcn(gui_State, varargin{:});

Error in
@(hObject,eventdata)winreject('dataplot_butt_Callback',hObject,eventdata,guidata(hObject))


Error using waitfor
Error while evaluating uicontrol Callback

We may add automatic tests to ... well - test various aspects of the plugin.
This can be done with matlab unit test framework.
This is worth considering if we discover that having automatic tests would help (vastly).

Automatic GUI testing would be really nice, but I don't see how we could do this apart from programming an AI to steer the mouse and assess the graphical results 😃

Decide on a unified, coherent way of documenting the project.

Internal (within-function aka docstrings):
See example of Markdown-turned matlab helpdoc.

• scan the functions for those lacking documentation (moved to a separate issue)
• check the rest for out-of-date help docs
• decide on a single help schema (eeglab style?) that makes generating .rst from help easier
• add documentation to files tagged with NOHELPINFO

'External' docs

We are conisdering following documentation tools:

• readthedocs
• GitHub wiki pages

❗ It is possible to use .rst format in GitHub wiki pages (example).

|
|

Why GitHub wikis

We should start with github wikis, here's why:

• We may start the documentation on the wiki (it's easier than sphinx + read the docs) and later move elsewhere. (but we do not have to move)
• Using GitHub flavored markdown instead of .rst is tempting, but .rst will be easier to migrate to read the docs (if we decide to migrate).
• Staying with GitHub wiki has also the benefit that it is a github repo so you can:
  git clone https://github.com/mmagnuski/eegDb.wiki.git and then edit it with version control
• another benefit: ability to generate pdf's from the docs using relatively simple makefile or simple python scripts 🍰 (see materials section)

|
|

.rst vs .md

If we stay, we should decide between GitHub markdown and ReStructuredText. Answering following questions should help:

• which one makes pdf-generation easier in general?
• are there features in .rst that we will miss in .md?
• if we are going to use GitHub little icons in the docs ( these: 📖, 💩 ) - are their easier to pdf-ize with .rst or .md?

Taks

• add a few test pages on GitHub wiki
• test generating .rst or .md from matlab help

|
|

Materials

See:

• about GitHub wikis
• ReStructuredText primer
• quick rst reference
• rst specification
• readthedocs
• tutorial on ReadTheDocs + Sphinx
• how to write a documentation
• pandoc - convert files between markup formats
• python script using pandoc for markdown --> pdf generation
• makefile for converting markdown to pdf
• github markdown library
• MardkdownPad

Go to next mark of given type (keyboard shortcut or GUI button) - centers the plot preview on next currently non-visible epoch marked with currently selected mark type.

Option checks should be short and simple

• current options should be a structure
• the structure should be then unrolled and applied with:

opts_u = struct_unroll(opt);
eegplot(EEG, opts_u{:});

Think about how to best handle profile:

• profile should be a structure with fields containing structures regarding specific options like topopts, plotopts etc.
• ❓ option to save to a known location with a name (+default profile)?
• ❓ save along with ICAw (loaded to workspace)

This is almost settled down: see the wiki page tracking decisions made about this issue.

We should decide:

• what fields are necessary
  filename, filepath, epoch etc.
• what fields in datainfo (this will be a separate issue)
• what fields in epoch
• what fields can be added later and treated as optional
• what fields should be checked to evaluate whether a given structure is a eegDb database

Working on this issue should be coupled with documenting it in the wiki pages

This is currently under way here: #43

🚧 This is currently a draft wishlist from an old file listing things that would be nice to have.
The wishlist will mature and some tasks will get their own issues.
🇺🇸 Most of these issues are currently still in Polish, if you are very eager to know what they're all about - google translate may help you 😄

WINREJECT:

• interfejs do porównaniela ICA między wersjami / rejestrami ?
• opcje ICA - pop up menu na jakich wersjach robić itp. Na razie przycisk ICA włącza default: ICA na wszystkich wybranych rejestrach, dla wszystkich ich wersji, które nie mają jeszcze wag ICA. Mało roboty.

COOLEEGPLOT

• wyświetlanie procentu zaznaczeń dla typów plus unique suma procent epok zaznaczonych.
• opcja shift-click (selects all from the last of current type to current)?
• prawoklikowy topoplot dla zakresów czasowych (zaznaczane) i częstotliwości? 
• etykiety zaznaczeń na marginesie górnym oprócz kolorów
• plotowanie sygnału z komponentów z dodatkową opcją wyświetlania tylko N elektrod o największej wadze.

COMP_EXPLORE

• erpimage2 coś ma nie tak z plotowaniem numerków (i przy skali i przy osiach), robią się podwójnie.
• rozpbudowanie varscatter - axcopy, hovering labels, red-marks, SD, mean/median
  consider/later:
• możliwość podejrzenia jak przykladowo moga wyglądac oczy, mięśnie, serce itp. mogłoby być przydatne jeśli
• integracja z innymi pluginami: opcjonalnie można dodać opcję puszczenia i wyswietlenia wyników adjusta, ale mnie nie powalił wiec nie wiem czy warto.
• corrmap
• w którymś może dodać opcję labelu 'out' dla całej osoby? choć to chyba nie przy compexplorerze tylko później(?)

Zupełnie inne:

• rozróżnienie bad channels przed i po ICA. tzn. czasami kompozycja jest kiepska, są pojedyńcze elektordy rozkomponentowane i wtedy można chcieć zamiast ich usuwania interpolowac eleksy. Cenne jest info czy się chciało te elektordy wykluczyc przed ICA (bo np. plywaja i rozwalily by sygnal) czy po (bo ICA ich nie zbawia).

LATER

• przymierzyć się do automatyzacji czyszczenia
• rozważyć przyszłość - automatyzacja oznaczania szła by pewnie lepiej na sygnale ciągłym - trzeba się zastanowić w ogóle czy dodawać kiedyś obsługę ciągłego sygnału. To nie byłoby specjalnie trudne by wycinać ciągły sygnał tak jak robiona jest preselekcja okienek.

This is not a hard requirement for v0.1 (this can be done through command-line and menu-driven would require lots of checks) - but should be available soon.
More generally - there is no GUI for changing database properties.

How best to manage versions?
Currently only non-empty fileds are copied to the .versions but this can result in problems when switching between versions. I do not remember if I solved this.
Anyway - version management should be planned better and only then implemented.
(currently most of eegDb features were implemented 'on the fly' and therefore some of the code is weird, suboptimal and hard-for-the-eyes)

scripts in howto should be tutorial-like and not just some scripts we used.
Each howto should show one or more (but not to many) steps to get something done with eegDb.

shift + click to mark all epochs/windows since last mark of the current type

Often multiple consecutive epochs/windows need to be marked and clicking it by hand is a mundane solution. shift + click would be a useful feature.

BUT:
if we add shift + click following should be done:

• tracking last-clicked of all mark types
• ctrl + z to reverse changes

ICAw_checkfields is rather slow.
It has multiple options most of which are not used.
It should be either rewritten, abandoned or left only where no easy substitute can be placed.

1. The extent of reliance on this function should be checked
   • what other functions use it
   • which are easy to fix (do not rely on 'more advanced' ICAw_checkfields options)
2. easy situations should be fixed (substitute checkfields with something simpler)

http://choosealicense.com/

The problem

As we know MathWorks is ugly and stupid and don't have argument parsing naturally embeded in the matlab language like python or R have.

Some of the functions we have in eegDb have lots of possible arguments.
Instead of writing a dumb argument loop in each of them it would be much more elegant and consise to use some argument parser.

Mathworks has a class for that inputParser but it is not available on all matlab versions, and even on those it is available - the behavior changes from version to version 😠 _{(I still remember working with VideoReader in matlab that had a completely different name in previous matlab version - for open source this would not be a problem - installing the latest version could be a requirement, but not in proprietary dungeons...)}

What to do?

1. Use inputParser anyway
   • check out inputParser across different matlab versions
   • write a simple wrapper to inputPraser to reduce typing (or write a snippet to recycle) and increase productivity, peace and happiness in the Universe
2. Look for some user-defined argument parser on Matlab Central etc. This is such a common problem in matlab-oriented-world that everybody seems to be writing argument parsers! 😜
   • check out this
   • check out that
   • or this - seems to use evalin('caller' .. ) so it would be easy to switch to - the variables would be created in function calling this parser.
   • another parser to check
   • and another one
3. Write our own (would have to be named parseArse !!)

Requirements for input parser

Keywords can be 'shortened':
lineWidth for example could be input as:

• lin or lw (this is obligatory for the parser of choice from my point of view - the first way is available in inputParser for sure, the second one - maybe)
• or linw etc. (that would be 🆒 but not necessary)

Values can be checked easily:
by keywords like 'num' as well as function handles like @num or @mycrazychecker

🚧 issue in construction 🚧

❓ change name from comp_explore to compexp
compexp could be an object - this may be easier for syncing etc.
compexp_broadcast (function for broadcasting changes introduced in compexp) could be an object (too?)
OR a method of compexp
brodcasting should be well thought and planned - so that later some functions / objects could be reused

global way of working could still be possible:
compexp_broadcast can have 'setup fun' and 'setup base' option where it would set up caller or workspace namespaces for example to use global variables. In such case compexp_broadcast would not brodcast anything when called later (but could if someone wants to sync with workspace AND some gui)

eegDb_sync

Sinleton object, handle subclass. Responsible for syncing eegDb guis and workspace.
More details soon...

According to mathworks guidata is best for small pieces of data like gui handles.
For larger parts of data getappdata and setappdata should be used. This should speed up gui execution.

OVERVIEW

pop_selectcomps_new was written with compatibility with pop_selectcomps in mind.
These restrictions turned out to be too limiting so the new version = compsgui works a little different and has a short and easy to type name.
You can use it without db this way: compsgui(EEG)

This issue tracks ideas and considerations about how the function should work and how it could be enhanced.

SPECIFIC TASKS

• tagging components
• left and right arrows work as left and right gui buttons
• explore impact of component removal on signal with (some) other components removed
• do not save topos in the database
• right-click - change status
• position in the centre of the screen
• size options
• add page info
• change visual state only if not the same as in info (in sync)
• change status colors to those from screenshot
• grid options (to specify the dimensions of the topo grid)

LATER:

I will think about adding things like:

• change names to compsgui
• recover EEG with all the components even if some were marked
• options for compo plotting
• getting plotting options from eegDb gui (other GUI)
• check for figure existence
• if one resizes the figure the buttons do should not get scaled so much - some limits should be set
• buttons are sometimes white - gray button backgroundcolor (taken from info) instead of white

CONSIDERATIONS

❓ maybe allow plotting maps without EEG data (maybe ...)

The error is:

Error using vertcat
Dimensions of matrices being concatenated are not consistent.

Error in winreject>applyrej_Callback (line 604)
    seltypes = [seltypes; 'clear rejections'];

Error in gui_mainfcn (line 96)
        feval(varargin{:});

Error in winreject (line 55)
    gui_mainfcn(gui_State, varargin{:});

Error in
@(hObject,eventdata)winreject('applyrej_Callback',hObject,eventdata,guidata(hObject))


Error using waitfor
Error while evaluating uicontrol Callback

Files (especially lower-level or less used) should be moved to specific folders. Folder names should be short but informative for example "marks" should contain functions used for managing marks...

• first cleanup
• more clean-up (on the current folder structure)
• clear do we need these directory
• decide on a v 0.1 folder structure

pop_eegfiltnew is necessary for filtering but is not being checked for correctly.
This should be done in eeg_path or some new function for checking dependencies...
❓

Componets figures are saved and do not need to be computed again.
This is a feature that will enhance normal eeglab component plotting and interacting with comp_explore.
This has passed the first tests but needs more work - and a separate branch.
I am trying to build pop_selcomps2 in a way that it can be used from comp_explore but also from eeglab without the need of running comp_explore

Special functions for safely changing eegDb structure.
🚧

eegplot comes from an old version of eeglab and hasn't been updated much since.
This makes changing it to match our functionality quite cumbersome.
We should consider to rewrite eegplot to make it more modern and easier to work with.
This would take some time to accomplish but may be worth the time cost.

Add docs about:

Docs are now held at http://eegdb.readthedocs.org/en/latest/

Text:

• move to readthedocs
• installation
• creating a database
• manipulating the database and recovering

Video:

• simple db_create
• basic use of db_gui

(more complex cocumentation topics (lots of gui options) can be first covered in video, only then turned into text guide)
(scripting topics will be covered in text)

eegDb can be run separately from eeglab (paths to eeglab functions are added if necessary) but releasing it as a eeglab plugin is desired:

• it does in fact extend eeglab's functionality
• we can add more options for integration
• cool to be present in the EEGlab interface 😄
• etc.

May require some work if we plan substantial additional integration features... 😓

Previous unofficial work-in-progress name was ICAw but it does not reflect the current state of the project (in the early beginnings the 'ICAw' databse was mainly used for storing and moving ICA weights between same-subject-but-a-little-different-preprocessing files).
I have changed the name to eegDb - meaning "eeg database", but this is not necessarily a final name.

UPDATE: After a few days I'm starting to think eegDb is quite nice. I could stick with this name, but am open to suggestions.

Tasks:

• list proposed names
• choose a name
• Once a name is decided upon - the function prefixes should be changed (for example to eegdb_ or db_ if the name stays eegDb).

Although marks are now organized neatly in eegDb struct still some historical problems remain.
Marks are managed awkwardly and inconsistently - this will have to be fixed.

• it seems that marks are copied to EEG.reject (EEG.reject.userreject)
• marks are also present in (copied to) EEG.reject.ICAw or EEG.etc.ICAw (have to check)
• marks are read from eegplot2 through workspace (ugh...)
• marks are read form eegplo2 by color so if some marks happen to have same color they are confused
• so on...

All non-toolbox dependencies (our own functions, not part of eegDb but used in it) should be present in dependencies folder (at least for now).

Following checks should be performed

• whether these files are up to date
• whether all dependencies are present

prep_list is old and heavy - better use something lighter like dir() itself.
For example a snippet like this:

fls = dir([ PTH, filesep, '*.set' ]);
fls = {fls.name};

could be used in the code directly or through a wrapper that sets filesep to be a persistent
variable, so its not checked every time the wrapper is called.

When 'linesmoothing' is turned on eegplot2 seems to refresh a couple of times when scrolling instead of only once. It is acceptable if linesmoothing is slower - but it should not behave like this.
It works this way on my windows computer at least - should be checked on other machines.
eegplot2 should be also checked for drawnow and equivalent statements