neuro-ml / deep_pipe Goto Github PK

This test randomly fail but ideally it shouldn't

from dpipe.batch_iter import Loky
from dpipe.batch_iter.pipeline import wrap_pipeline

size = 100
    for i, item in enumerate(wrap_pipeline(range(size), Loky(lambda x: x ** 2, n_workers=2))):
        assert item == i ** 2

Previously written batch predictors were written to support models with multiple inputs in deepmedic-like manner, but now we use crops inside of the model instead. Maybe refactor them?

I can choose arbitrary save_func, but the extension remains the same.

For example, I may prefer to save predictions for the whole image in json, if my slice-wise predictions have different length.

Files in checkpoint folder have no extension

• The validator doesn't know which epoch it is. It will always start over from 0.
• Some objects (like policies) might be simply copied in the backup class, this means that reloading them from disk will change them only inside the backup class.

Consider reorganizing library structure and usages.

• move user-related things to user's repositories
• keep only interfaces and really valuable items in the library

Check if tensorboard lib is good enough for direct logging:

• It can be installed
• It allows user to write logs
• User can look at these logs
• No memory leak

It is a complex issue:

• Registration system and resource manager refactoring
• Config files syntax refactoring
• Separation of resource manager subsystem into separate repository

Are exceptionally complex and need refactoring

Merge batch iter branch

Consider:

• each batch iter has to be valuable without additional infrastructure
• complex batch iterators for the performance

import dpipe

dpipe.version
'0.2.7'

import dpipe.medim
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'dpipe.medim'

It's used for other purposes too.
Perhaps Counter is a better name.

Get information about:

• user pipelines with the library
• the biggest problem

We need a way to combine batch predictors, because many of them are repetitive.

Consider creating a small grammar for that.
Instead of writing a custom runner, it might be feasible to generate the snakefile/makefile on the fly.

model_core currently requires prediction method, which increases coupling. Object prediction logic has to be extracted from the model core to separate module.

Users should be able to create training functions easily to allow more customization. Decrease amount of code, required to write new training process. #20

Namely ['PatientID', 'SeriesInstanceUID', 'StudyID', 'SequenceName']

Current model satisfy only the simplest requirements with one predictor and one loss.
Refactor model as a set of utility functions and a bunch of prebuilt models for typical use

It would be better for the generated config to be named something like resources.config instead of config.

• coregistration
• affine transforms

Too many levels in the PyramidalPooling layer may cause job silent termination without any reports in the experiment's log file

If all gradients of the model are None-s, then scaler.step(optimizer) fails with AssertionError: No inf checks were recorded for this optimizer.

If mask in dpipe.torch.functional.masked_loss is empty, it creates and returns 0 wrapped in a torch tensor. This tensor is "detached" from model graph, so during backward no gradients flow into model parameters gradients, they are all None-s and scaler.step fails.

This can be simply fixed by not returning new 0 tensor in masked_loss, but multiplying prediction by 0, so there will be not None but 0 gradients in model parameters.

Find solution that is good enough for the logging:

• It can be installed
• It allows user to write logs
• User can look at these logs with good interface
• No memory leak

#15 #14

Model refactoring is required for writing custom user logs during training. Maybe remove model_controller?

Move logging to training process. #20

Change interfaces, to use both pytorch and tf

Consider usage of separate validation functions that encapsulate details about what metrics are used on validation set. #20 #21

Reorganize training process allowing more customization and logging