sushirice-dg / pixelformerplus-semanticbins Goto Github PK

This is the official PyTorch implementation of model PixelFormerPlus and SemanticBins
The optimization for WACV 2023 paper 'Attention Attention Everywhere: Monocular Depth Prediction with Skip Attention'.

Paper

1. Installation
2. Datasets
3. Training
4. Evaluation
5. Testing
6. Models
7. Results
8. Acknowledgements

Os ubuntu18.04

conda create -n pixelformer python=3.8
conda activate pixelformer
conda install pytorch=1.10.0 torchvision cudatoolkit=11.1
pip install matplotlib tqdm tensorboardX timm mmcv eiops

You can download the annotated depth maps data set of KITTI from here and download the raw data of KITTI form here
You can download the annotated depth maps data set of NYUv2 from here the raw data of NYUv2 from here
The original dataset and annotated data downloaded from the official website are provided in the .mat file format.
Therefore, you need to convert the .mat files into the .png format that can be recognized as input. Please refer to the open source code from here
You can download the annotated depth maps data set and the raw data of HabitatDyn from here

First download the pretrained encoder backbone Swin-L 22K model from here, and then modify the pretrain path in the config files.

Training the PixelFormer model on NYUv2:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files

python PixelFormerPlus/train.py arguments_train_nyu.txt

Training the PixelFormer model on KITTI:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files

python PixelFormerPlus/train.py arguments_train_kittieigen.txt

Training the PixelFormerPlus model on NYUv2:
Rename the network folder of PixelFormerPlus model to networks
Modify the datapaths of ground truth and raw data in the config files

python PixelFormerPlus/train.py arguments_train_nyu.txt

Training the PixelFormerPlus model on KITTI:
Rename the network folder of PixelFormerPlus model to networks
Modify the datapaths of ground truth and raw data in the config files

python PixelFormerPlus/train.py arguments_train_kittieigen.txt

Training the PixelFormer model on HabitatDyn Validation set 1:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val0102scenes.txt

python PixelFormerPlus/train.py arguments_train_habitat.txt

Training the PixelFormer model on HabitatDyn Validation set 2:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val16181619clips.txt

python PixelFormerPlus/train.py arguments_train_habitat.txt

Training the PixelFormerPlus model on HabitatDyn Validation set 1:
Rename the network folder of PixelFormerPlus model to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val0102scenes.txt

python PixelFormerPlus/train.py arguments_train_habitat.txt

Training the PixelFormerPlus model on HabitatDyn Validation set 2:
Rename the network folder of PixelFormerPlus model to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val16181619clips.txt

python PixelFormerPlus/train.py arguments_train_habitat.txt

Training the SemanticBins model on HabitatDyn Validation set 1:
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val0102scenes.txt

python PixelFormerPlus/train_semantic.py arguments_train_habitat.txt

Training the SemanticBins model on HabitatDyn Validation set 2:
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val16181619clips.txt

python PixelFormerPlus/train_semantic.py arguments_train_habitat.txt

Evaluate the PixelFormer model on NYUv2:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter checkpoint_path to pretrained/PixelFormer_NYU

python PixelFormerPlus/eval.py arguments_eval_nyu.txt

Evaluate the PixelFormer model on KITTI:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter checkpoint_path to pretrained/PixelFormer_KITTI

python PixelFormerPlus/eval.py arguments_eval_kitti.txt

Evaluate the PixelFormerPlus model on NYUv2:
Rename the network folder of PixelFormerPlus model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter checkpoint_path to pretrained/NewModel_NYU

python PixelFormerPlus/eval.py arguments_eval_nyu.txt

Evaluate the PixelFormerPlus model on KITTI:
Rename the network folder of PixelFormerPlus model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter checkpoint_path to pretrained/NewModel_KITTI

python PixelFormerPlus/eval.py arguments_eval_kitti.txt

Evaluate the PixelFormer model on HabitatDyn Validation set 1:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val0102scenes.txt
Set the parameter checkpoint_path to pretrained/PixelFormer_Validation1

python PixelFormerPlus/eval.py arguments_eval_habitat.txt

Evaluate the PixelFormer model on HabitatDyn Validation set 2:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val16181619clips.txt
Set the parameter checkpoint_path to pretrained/PixelFormer_Validation2

python PixelFormerPlus/eval.py arguments_eval_habitat.txt

Evaluate the PixelFormerPlus model on HabitatDyn Validation set 1:
Rename the network folder of PixelFormerPlus model to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val0102scenes.txt
Set the parameter checkpoint_path to pretrained/NewModel_Validation1

python PixelFormerPlus/eval.py arguments_eval_habitat.txt

Evaluate the PixelFormerPlus model on HabitatDyn Validation set 2:
Rename the network folder of PixelFormerPlus model to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val16181619clips.txt
Set the parameter checkpoint_path to pretrained/NewModel_Validation2

python PixelFormerPlus/eval.py arguments_eval_habitat.txt

Evaluate the SemanticBins model on HabitatDyn Validation set 1:
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val0102scenes.txt
Set the parameter checkpoint_path to pretrained/SemanticBins_Validation1

python PixelFormerPlus/eval_semantic.py arguments_eval_habitat.txt

Evaluate the SemanticBins model on HabitatDyn Validation set 2:
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val16181619clips.txt
Set the parameter checkpoint_path to pretrained/SemanticBins_Validation2

python PixelFormerPlus/eval_semantic.py arguments_eval_habitat.txt

Test the PixelFormer model and get depth maps on NYUv2:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter checkpoint_path to pretrained/PixelFormer_NYU

python PixelFormerPlus/test.py arguments_test_nyu.txt

Test the PixelFormer model and get depth maps on KITTI:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter checkpoint_path to pretrained/PixelFormer_KITTI

python PixelFormerPlus/text.py arguments_test_kitti.txt

Test the PixelFormerPlus model and get depth maps on NYUv2:
Rename the network folder of PixelFormerPlus model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter checkpoint_path to pretrained/NewModel_NYU

python PixelFormerPlus/test.py arguments_eval_test.txt

Test the PixelFormerPlus model and get depth maps on KITTI:
Rename the network folder of PixelFormerPlus model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter checkpoint_path to pretrained/NewModel_KITTI

python PixelFormerPlus/test.py arguments_test_kitti.txt

Test the PixelFormer model and get depth maps on HabitatDyn Validation set 1:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val0102scenes.txt
Set the parameter checkpoint_path to pretrained/PixelFormer_Validation1

python PixelFormerPlus/test.py arguments_test_habitat.txt

Test the PixelFormer model and get depth maps on HabitatDyn Validation set 2:
Rename the network folder of PixelFormer model from networkssource to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val16181619clips.txt
Set the parameter checkpoint_path to pretrained/PixelFormer_Validation2

python PixelFormerPlus/test.py arguments_test_habitat.txt

Test the PixelFormerPlus model on HabitatDyn Validation set 1:
Rename the network folder of PixelFormerPlus model to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val0102scenes.txt
Set the parameter checkpoint_path to pretrained/NewModel_Validation1

python PixelFormerPlus/test.py arguments_test_habitat.txt

Test the PixelFormerPlus model and get depth maps on HabitatDyn Validation set 2:
Rename the network folder of PixelFormerPlus model to networks
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val16181619clips.txt
Set the parameter checkpoint_path to pretrained/NewModel_Validation2

python PixelFormerPlus/test.py arguments_test_habitat.txt

Test the SemanticBins model and get depth maps on HabitatDyn Validation set 1:
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val0102scenes.txt
Set the parameter checkpoint_path to pretrained/SemanticBins_Validation1

python PixelFormerPlus/test_semantic.py arguments_test_habitat.txt

Test the SemanticBins model and get depth maps on HabitatDyn Validation set 2:
Modify the datapaths of ground truth and raw data in the config files
Set the parameter filenames_file to data_splits/val1.txt
Set the parameter filenames_file_eval data_splits/val16181619clips.txt
Set the parameter checkpoint_path to pretrained/SemanticBins_Validation2

python PixelFormerPlus/test_semantic.py arguments_test_habitat.txt

• You can download the pretrained models from here

Most of the code has been adpated from WACV 2023 paper PixelFormer and CVPR 2022 paper NewCRFS. We thank them.