This is the source code accompanying the paper VOS: Learning What You Don’t Know by Virtual Outlier Synthesis by Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li

The codebase is heavily based on ProbDet and Detectron2.

Checkout our CVPR'22 work STUD on object detection in video datasets, NeurIPS'22 work SIREN on OOD detection for detection transformers if you are interested!

02/05/2023---we have uploaded the code here for reproducing the figure 1 of our paper.

05/08/2022---We have updated the openreview with the new results of using a nonlinear MLP for binary classification instead of the logistic regression, which is slightly better. Please check the code and models here.

pip install -r requirements.txt

In addition, install detectron2 following here.

PASCAL VOC

Download the processed VOC 2007 and 2012 dataset from here.

The VOC dataset folder should have the following structure:

 └── VOC_DATASET_ROOT
     |
     ├── JPEGImages
     ├── voc0712_train_all.json
     └── val_coco_format.json

COCO

Download COCO2017 dataset from the official website.

Download the OOD dataset (json file) when the in-distribution dataset is Pascal VOC from here.

Download the OOD dataset (json file) when the in-distribution dataset is BDD-100k from here.

Put the two processed OOD json files to ./anntoations

The COCO dataset folder should have the following structure:

 └── COCO_DATASET_ROOT
     |
     ├── annotations
        ├── xxx (the original json files)
        ├── instances_val2017_ood_wrt_bdd_rm_overlap.json
        └── instances_val2017_ood_rm_overlap.json
     ├── train2017
     └── val2017

BDD-100k

Donwload the BDD-100k images from the official website.

Download the processed BDD-100k json files from here and here.

The BDD dataset folder should have the following structure:

 └── BDD_DATASET_ROOT
     |
     ├── images
     ├── val_bdd_converted.json
     └── train_bdd_converted.json

OpenImages

Download our OpenImages validation splits here. We created a tarball that contains the out-of-distribution data splits used in our paper for hyperparameter tuning. Do not modify or rename the internal folders as those paths are hard coded in the dataset reader. The OpenImages dataset is created in a similar way following this paper.

The OpenImages dataset folder should have the following structure:

 └── OEPNIMAGES_DATASET_ROOT
     |
     ├── coco_classes
     └── ood_classes_rm_overlap

Visualization of the OOD datasets

The OOD images with respect to different in-distribution datasets can be downloaded from ID-VOC-OOD-COCO, ID-VOC-OOD-openimages, ID-BDD-OOD-COCO, ID-BDD-OOD-openimages.

Firstly, enter the detection folder by running

cd detection

Before training, modify the dataset address by changing "dataset-dir" according to your local dataset address.

Vanilla Faster-RCNN with VOC as the in-distribution dataset

python train_net.py
--dataset-dir path/to/dataset/dir
--num-gpus 8
--config-file VOC-Detection/faster-rcnn/vanilla.yaml 
--random-seed 0 
--resume

Vanilla Faster-RCNN with BDD as the in-distribution dataset

python train_net.py 
--dataset-dir path/to/dataset/dir
--num-gpus 8 
--config-file BDD-Detection/faster-rcnn/vanilla.yaml 
--random-seed 0 
--resume

VOS on ResNet

python train_net_gmm.py 
--dataset-dir path/to/dataset/dir
--num-gpus 8 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--random-seed 0 
--resume

VOS on RegNet

Before training using the RegNet as the backbone, download the pretrained RegNet backbone from here.

python train_net_gmm.py 
--dataset-dir path/to/dataset/dir
--num-gpus 8 
--config-file VOC-Detection/faster-rcnn/regnetx.yaml 
--random-seed 0 
--resume

Before training on VOS, change "VOS.STARTING_ITER" and "VOS.SAMPLE_NUMBER" in the config file to the desired numbers in paper.

Evaluation with the in-distribution dataset to be VOC

Firstly run on the in-distribution dataset:

python apply_net.py 
--dataset-dir path/to/dataset/dir
--test-dataset voc_custom_val 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Then run on the OOD dataset:

python apply_net.py
--dataset-dir path/to/dataset/dir
--test-dataset coco_ood_val 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Obtain the metrics using:

python voc_coco_plot.py 
--name vos 
--thres xxx 
--energy 1 
--seed 0

Here the threshold is determined according to ProbDet. It will be displayed in the screen as you finish evaluating on the in-distribution dataset.

Evaluation with the in-distribution dataset to be BDD

Firstly run on the in-distribution dataset:

python apply_net.py 
--dataset-dir path/to/dataset/dir
--test-dataset bdd_custom_val 
--config-file BDD-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Then run on the OOD dataset:

python apply_net.py 
--dataset-dir path/to/dataset/dir
--test-dataset coco_ood_val_bdd 
--config-file BDD-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Obtain the metrics using:

python bdd_coco_plot.py
--name vos 
--thres xxx 
--energy 1 
--seed 0

Pretrained models

The pretrained models for Pascal-VOC can be downloaded from vanilla and VOS-ResNet and VOS-RegNet.

The pretrained models for BDD-100k can be downloaded from vanilla and VOS-ResNet and VOS-RegNet.

Train on WideResNet

cd classification/CIFAR/ & 
python train_virtual.py 
--start_epoch 40 
--sample_number 1000 
--sample_from 10000 
--select 1 
--loss_weight 0.1 

where "start_epoch" denotes the starting epoch of the uncertainty regularization branch.

"sample_number" denotes the size of the in-distribution queue.

"sample_from" and "select" are used to approximate the likelihood threshold during virtual outlier synthesis.

"loss_weight" denotes the weight of the regularization loss.

Please see Section 3 and Section 4.1 in the paper for details.

Train on DenseNet

cd classification/CIFAR/ &
python train_virtual_dense.py 
--start_epoch 40 
--sample_number 1000 
--sample_from 10000 
--select 1 
--loss_weight 0.1 

Evaluation on different classifiers

cd classification/CIFAR/ & 
python test.py 
--model_name xx 
--method_name xx 
--score energy 
--num_to_avg 10

where "model_name" denotes the model architectures. ("res" denotes the WideResNet and "dense" denotes the DenseNet.)

"method_name" denotes the checkpoint name you are loading.

Pretrained models

We provide the pretrained models using WideResNet and DenseNet with the in-distribution dataset to be CIFAR-10.

If you found any part of this code is useful in your research, please consider citing our paper:

 @article{du2022vos,
      title={VOS: Learning What You Don’t Know by Virtual Outlier Synthesis}, 
      author={Du, Xuefeng and Wang, Zhaoning and Cai, Mu and Li, Yixuan},
      journal={Proceedings of the International Conference on Learning Representations},
      year={2022}
}