• [News]

  • Sep, 2021: We are now busy adding QueryInst to mmdetection library (open-mmlab/mmdetection#6050, open-mmlab/mmdetection#6000). The model will be augmented, please stay tuned. We will also extend QueryInst based on mmdetection library to other tasks, e.g., panoptic segmentation.

• TL;DR: QueryInst (Instances as Queries) is a simple and effective query based instance segmentation method driven by parallel supervision on dynamic mask heads, which outperforms previous arts in terms of both accuracy and speed.

• Our QueryTrack (i.e., Tracking Instances as Queries, tech report) based on QueryInst won the 2nd place (AP = 52.3 @ test set, AP = 54.3 @ val set) in video instance segmentation (VIS) track with single online end-to-end model, single scale testing & without using extra video training data in the 3rd Large-scale Video Object Segmentation Challenge, CVPR 2021.

• For the first time, we demonstrate that an end-to-end query based framework driven by parallel supervision is competitive with well-established and highly-optimized methods in a wide range of instance-level recognition tasks (object detection, instance segmentation and video instance segmentation).

Instances as Queries

by Yuxin Fang*, Shusheng Yang*, Xinggang Wang†, Yu Li, Chen Fang, Ying Shan, Bin Feng, Wenyu Liu.

(*) equal contribution, (†) corresponding author.

arXiv technical report (arXiv 2105.01928)

• This repo serves as the official implementation for QueryInst, based on mmdetection and built upon Sparse R-CNN & DETR. Implantations based on Detectron2 will be released in the near future.

• This project is under active development, we will extend QueryInst to a wide range of instance-level recognition tasks.

Notes:

• Accesscode for baidu is QIst.

• Our project is mainly developed on mmdetection toolbox (931d96), please refer to the mmdetection official installation.
• Install QueryInst by:

python setup.py develop

• Prepare datasets:

mkdir data && cd data
ln -s /path/to/coco coco

• Training QueryInst with single GPU:

python tools/train.py configs/queryinst/queryinst_r50_fpn_1x_coco.py

• Training QueryInst with multi GPUs:

./tools/dist_train.sh configs/queryinst/queryinst_r50_fpn_1x_coco.py 8

• Test QueryInst on COCO val set with single GPU:

python tools/test.py configs/queryinst/queryinst_r50_fpn_1x_coco.py PATH/TO/CKPT.pth --eval bbox segm

• Test QueryInst on COCO val set with multi GPUs:

./tools/dist_test.sh configs/queryinst/queryinst_r50_fpn_1x_coco.py PATH/TO/CKPT.pth 8 --eval bbox segm

If you find our paper and code useful in your research, please consider giving a star ⭐ and citation 📝 :

@article{QueryInst,
  title={Instances as Queries},
  author={Fang, Yuxin and Yang, Shusheng and Wang, Xinggang and Li, Yu and Fang, Chen and Shan, Ying and Feng, Bin and Liu, Wenyu},
  journal={arXiv preprint arXiv:2105.01928},
  year={2021}
}

@article{QueryTrack,
  title={Tracking Instances as Queries},
  author={Yang, Shusheng and Fang, Yuxin and Wang, Xinggang and Li, Yu and Shan, Ying and Feng, Bin and Liu, Wenyu},
  journal={arXiv preprint arXiv:2106.11963},
  year={2021}
}

• QueryInst training and inference code.
• QueryInst with Swin-Transformer and Test-Time-Augmentation.
• QueryInst based on Detectron2 toolbox will be released in the near future.
• QueryInst configurations for Cityscapes and YouTube-VIS.
• QueryInst pretrain weights.