This repository maintains our solution to 3D AI Challenge Instance Segmentation Track, which ranks 1st in both trackA (validation set)] and trackB (test set). Our solution is a weighted ensemble of several PointRend models. For more details please refer to our technical report:Towards Fine-grained Large Object Segmentation: 1st Place Solution to 3D AI Challenge 2020 - Instance Segmentation Track. This repository implements PointRend and train/evaluates it on the 3DFuture dataset, based on the open projects mmdetection and detectron2 PointRend. We also provide ensemble code on our trained PointRend models.

CUDA 10.0

Download the dataset from official challenge site, and then put it under the folder mmdet_furniture/data/, the dataset folder structure in our experiments looks like this:

mmdet_furniture
|-- data
     |-- future
           |-- annotations
                 |-- test_set.json 
                 |-- train_set.json  
                 |-- val_set.json
           |-- images
                 |-- train
                 |-- val
                 |-- test

We provide script which allows you to get our final submission without installing any packages:

export GPUS=8         # the exact GPU number you have
source eval.sh

The final submission file will be located at code_furniture/segmentation_results.json.zip

** If you want to execute step by step, please follow the following steps **

We implements PointRend under mmdetection framework, note that the mmdetection version here we used is 1.1.0. We follow the official mmdetection installation, and list our experiment environments in requirements.txt.

To train a PointRend model on the 3DFuture dataset, run:

# train with 8 gpus using one machine:
./tools/dist_train.sh furniture_config/pointrend/CONFIG_NAME.py 8 

# We list 5 config files with different training settings under folder furniture_config/pointrend/, 
# which are used for this competition. For instance, you can run one of the configs like this:
./tools/dist_train.sh furniture_config/pointrend/pointrend_x101_64x4d_dcn_fpn_fp16_p2p6.py 8 

After training finished, the trained model and training log will be saved under the folder work_dirs/CONFIG_NAME. We provide five pretrained PointRend models which can be used for test generation. Download the pretrained weights from here, and put it under folder work_dirs.

Two steps for generating final submission file format on test dataset. First generate infer results in the pkl format based on the trained model. Then convert data from pkl to json format which is then zipped and used for submit the test server.

# Firstly, evaluate on test set and generate infer results with pkl format:
./tools/dist_test.sh furniture_config/pointrend/CONFIG_NAME.py work_dirs/CONFIG_NAME/epoch_44.pth 8 --out out_results/CONFIG_NAME/segmentation_resutls.pkl
# Note we train PointRend for 44 epochs and defaultly pretrained weights of the last epoch are used for infer.

Above scripts will generate a segmentation_resutls.pkl file under the folder out_results/CONFIG_NAME.

# Then, convert from pkl to json format:
python tools/submit_test.py --root ROOT_PATH --config_name CONFIG_NAME  # ROOT_PATH: absolute file path for project mmdet_furniture 
# For example:
python tools/submit_test.py --root /mnt/truenas/scratch/lqf/code/github/delete_test/3DFuture_ins_seg/mmdet_furniture/ --config_name pointrend_x101_64x4d_dcn_fpn_fp16_p2p6

Above scripts will generate a segmentation_resutls.json under the folder out_results/CONFIG_NAME

# Finally, zip the json file and submit to test server
zip -q segmentation_resutls.zip segmentation_resutls.json

Above scripts will generate a segmentation_resutls.zip under the folder out_results/CONFIG_NAME, which can be used for submitting test server.

We report PointRend performance under the test dataset with multi-scale train/test. All the models are trained using 8X 2080Ti.

Before you run ensemble code, you need to install Soft-NMS implementation borrowed from SimpleDet.

cd code/cython_nms
python setup.py install

We adopted two different ways for model reweight. A detail description please refer to our technical report.