Make EEGClassifier have very nice interface about braindecode HOT 6 CLOSED

I just took a short look at current examples to see what can be improved in the EEGClassifier interface. I think that the most unfriendly part is the callbacks part which takes a lot of lines and does really similar thing for CroppedDecoding and TrialDecoding but differs in details. My idea is to create two list of standard callbacks in the callbacks.py file so we can just import cropped_callbacks or trial_callbacks, sth like here sliwy@bbdf77e. This way we will remove callback definition from examples. Do you think that we should hide more from the user? @robintibor I think that other things should be easily accessible by a user (like batch_size, optimizer, lr, train_test_split).

from braindecode.

I feel this may be a little too opaque for the user, how about we allow to supply a list of scikit-learn scoring functions like ['accuracy', 'f1_score'] either as cropped_scorings or as trial_scorings and rest of code is created internally? Then you can still decide which scorings you want but they are automatically correctly created and by default applied to both train and valid... ? How you think @sliwy

from braindecode.

Seems legit to me and won't need a lot of modification. I'll try to implement.

from braindecode.

@robintibor Do you know any way to infer if the callbacks should be Cropped or Trial in the EEGClassifier? I started and my first option was as below:

callbacks = ['accuracy', 'f1', 'roc_auc']
EEGClassifier(...., callbacks=callbacks)

Few questions/problems I see we should discuss before implementing:

1. This form does not contain information about lower_is_better which is quite important. The only possible way to figure out is to use hacks from sklearn:
   get_scorer('accuracy')._score_func.__name__ should give us name (accuracy_score) of scoring function and based on it according to sklearn docs:

functions ending with _score return a value to maximize, the higher the better.

functions ending with _error or _loss return a value to minimize, the lower the better. When converting into a scorer object using make_scorer, set the greater_is_better parameter to False (True by default; see the parameter description below).

2. We will give the user a possibility to mix our way of defining callbacks with strings, only for scorings and skorch callbacks for anything else. Easy if in the callback parser should be sufficient for this.
3. String way of defining callbacks does not give a possibility to specify whether BatchScoring or EpochScoring should be used. I will start with EpochScoring as it should be a more general way (except we have too little memory).
4. Still a problem with CroppedScoring and TrialScoring, I don't see an easy way to infer the needed callback type.

from braindecode.

@robintibor Take a look at it: sliwy@82f2126 it does not support CroppedDecoding for now. First version for TrialDecoding, should be better tested, now just to show the conception.

from braindecode.

Ya this seems really nice sliwy@82f2126. regarding knowing cropped or not, I think this can be a constructor_parameter cropped=True/False. I think it's not bad if this is written explicitly by user

from braindecode.