A Pytorch framework for Out-of-Distibution Detection. The technical report can be found at https://arxiv.org/abs/2208.00963.

If you use this framework in one of your projects please consider to cite

@misc{FrOoDo2022,
      title={FrOoDo: Framework for Out-of-Distribution Detection}, 
      author={Stieber, Jonathan and Fuchs, Moritz and Mukhopadhyay, Anirban},
      year={2022},
      eprint={2208.00963},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

This framework is made to tackle the problems of out of distrubtion data in digital pathology. During the data aquisition process different types of out of distribution data can occur e.g. artifacts like fat drops or unfocssed parts of the whole slide image. These ood data lower the segmentation metrics like the dice score in a critical way and it is therefore necessary to identify and remove ood data for the automatic segmentation with neural networks.

Currently the framework supports the augmentations below. The artifact images are taken from [1]. As you can see the severity of the augmentation can be chosen:

The framework is designed to be as flexible as possible to fit different types of experiments. As described in the framework overview it is possible to quickly add new models, new ood methods, new datasets and new metrics. Therefore you can focus on your specific research question and let the framework do the rest.

git clone https://github.com/MECLabTUDA/FrOoDo
cd froodo
pip install -r requirements.txt

This demo shows a typical ood evaluation scenario. A pathology dataset (in this case the BCSS dataset [2] ) is augmented with dark black spots which are marked as ood pixels. The images are then fed into a neual network and and ood score is comoputed with well known post-hoc ood methods (Odin [3] and Energy-Based [4] ).

These scores are used by the AuRoc Metrics to calculate the seperability of in and ood data. Because of the augmentation there is information about the percentage of ood pixels in an image and the scores can therefore be calculated for different numbers of percentages.

You can find this demo here.

A guide on how to install the BCSS dataset for FrOoDo can be found here.

from froodo import *

# init network
net = SegmentationModel().load()

# create BCSS dataset adapter (see BCSS docu)
adapter = GeneralDatasetAdapter(BCSS_Adapted_Cropped_Resized_Datasets().test)

# choose metrics
metrics = [OODAuRoC()]

# choose post-hoc OOD methods
methods = [MaxClassBaseline(), ODIN(), EnergyBased()]

# create experiment component
experiment = AugmentationOODEvaluationComponent(
    data_adapter=adapter,
    augmentation=SampledAugmentation(DarkSpotsAugmentation()),
    model=net,
    metrics=metrics,
    methods=methods,
    seed=4321,
    task_type=TaskType.SEGMENTATION
)

# run experiment
experiment()

The output of en experiment will be a sample of the evaluation data so see the correct augmentaion and the chosen visualized metrics.

The main advantage of froodo is that it can be extend easily because of clear defined interfaces. The manual pages describe how to e.g. add an augmentation or new ood method.

• Experiment tracking
• More augmentations, more models, more methods, more ...

[1] Schömig-Markiefka, B., Pryalukhin, A., Hulla, W., Bychkov, A., Fukuoka, J., Mad- abhushi, A., Achter, V., Nieroda, L., Büttner, R., Quaas, A., et al.: Quality control stress test for deep learning-based diagnostic model in digital pathology. Modern Pathology (2021)

[2] Amgad, M., Elfandy, H., Hussein, H., Atteya, L.A., Elsebaie, M.A., Abo Elnasr, L.S., Sakr, R.A., Salem, H.S., Ismail, A.F., Saad, A.M., et al.: Structured crowd- sourcing enables convolutional segmentation of histology images. Bioinformatics (2019)

[3] Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. In: International Conference on Learning Represen- tations (2018)

[4] Liu, W., Wang, X., Owens, J., Li, Y.: Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems (2020)