Giter VIP home page Giter VIP logo

mimdet's Introduction

MIMDet ๐ŸŽญ

Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection

Yuxin Fang1 *, Shusheng Yang1 *, Shijie Wang1 *, Yixiao Ge2, Ying Shan2, Xinggang Wang1 ๐Ÿ“ง,

1 School of EIC, HUST, 2 ARC Lab, Tencent PCG.

(*) equal contribution, (๐Ÿ“ง) corresponding author.

ArXiv Preprint (arXiv 2204.02964)

News

  • 19 May, 2022: We update our preprint with stronger results and more analysis. Code & models are also updated in the main branch. For our previous results (code & models), please refer to the v1.0.0 branch.

  • 6 Apr, 2022: Code & models are released!

Introduction

This repo provides code and pretrained models for MIMDet (Masked Image Modeling for Detection).

  • MIMDet is a simple framekwork that enables a MIM pretrained vanilla ViT to perform high-performance object-level understanding, e.g, object detection and instance segmentation.
  • In MIMDet, a MIM pre-trained vanilla ViT encoder can work surprisingly well in the challenging object-level recognition scenario even with randomly sampled partial observations, e.g., only 25%~50% of the input embeddings.
  • In order to construct multi-scale representations for object detection, a randomly initialized compact convolutional stem supplants the pre-trained large kernel patchify stem, and its intermediate features can naturally serve as the higher resolution inputs of a feature pyramid without upsampling. While the pre-trained ViT is only regarded as the third-stage of our detector's backbone instead of the whole feature extractor, resulting in a ConvNet-ViT hybrid architecture.
  • MIMDet w/ ViT-Base & Mask R-CNN FPN obtains 51.7 box AP and 46.2 mask AP on COCO. With ViT-L, MIMDet achieves 54.3 box AP and 48.2 mask AP.
  • We also provide an unofficial implementation of Benchmarking Detection Transfer Learning with Vision Transformers that successfully reproduces its reported results.

Models and Main Results

Mask R-CNN

Model Sample Ratio Schedule Aug Box AP Mask AP #params config model / log
MIMDet-ViT-B 0.5 3x [480-800, 1333] w/crop 51.7 46.2 127.96M config model / log
MIMDet-ViT-L 0.5 3x [480-800, 1333] w/crop 54.3 48.2 349.33M config model / log
Benchmarking-ViT-B - 25ep [1024, 1024] LSJ(0.1-2) 48.0 43.0 118.67M config model / log
Benchmarking-ViT-B - 50ep [1024, 1024] LSJ(0.1-2) 50.2 44.9 118.67M config model / log
Benchmarking-ViT-B - 100ep [1024, 1024] LSJ(0.1-2) 50.4 44.9 118.67M config model / log

Notes:

  • The Box AP & Mask AP in the table above is obtained w/ sample ratio = 1.0, which is higher than the training sample ratio (0.25 or 0.5). Our MIMDet can benefit from lower sample ratio during training for better efficiency, as well as higher sample ratio during inference for better accuracy. Please refer to our paper for detailed analysis.
  • Benchmarking-ViT-B is an unofficial implementation of Benchmarking Detection Transfer Learning with Vision Transformers.

Installation

Prerequisites

  • Linux
  • Python 3.7+
  • CUDA 10.2+
  • GCC 5+

Prepare

  • Clone
git clone https://github.com/hustvl/MIMDet.git
cd MIMDet
  • Create a conda virtual environment and activate it:
conda create -n mimdet python=3.9
conda activate mimdet

Dataset

MIMDet is built upon detectron2, so please organize dataset directory in detectron2's manner. We refer users to detectron2 for detailed instructions. The overall hierachical structure is illustrated as following:

MIMDet
โ”œโ”€โ”€ datasets
โ”‚   โ”œโ”€โ”€ coco
โ”‚   โ”‚   โ”œโ”€โ”€ annotations
โ”‚   โ”‚   โ”œโ”€โ”€ train2017
โ”‚   โ”‚   โ”œโ”€โ”€ val2017
โ”‚   โ”‚   โ”œโ”€โ”€ test2017
โ”‚   โ”œโ”€โ”€ ...
โ”œโ”€โ”€ ...

Training

Download the full MAE pretrained (including the decoder) ViT-B Model and ViT-L Model checkpoint. See MAE repo-issues-8.

# single-machine training
python lazyconfig_train_net.py --config-file <CONFIG_FILE> --num-gpus <GPU_NUM> mae_checkpoint.path=<MAE_MODEL_PATH>

# multi-machine training
python lazyconfig_train_net.py --config-file <CONFIG_FILE> --num-gpus <GPU_NUM> --num-machines <MACHINE_NUM> --master_addr <MASTER_ADDR> --master_port <MASTER_PORT> mae_checkpoint.path=<MAE_MODEL_PATH>

Inference

# inference
python lazyconfig_train_net.py --config-file <CONFIG_FILE> --num-gpus <GPU_NUM> --eval-only train.init_checkpoint=<MODEL_PATH>

# inference with 100% sample ratio (please refer to our paper for detailed analysis)
python lazyconfig_train_net.py --config-file <CONFIG_FILE> --num-gpus <GPU_NUM> --eval-only train.init_checkpoint=<MODEL_PATH> model.backbone.bottom_up.sample_ratio=1.0

Acknowledgement

This project is based on MAE, Detectron2 and timm. Thanks for their wonderful works.

License

MIMDet is released under the MIT License.

Citation

If you find our paper and code useful in your research, please consider giving a star โญ and citation ๐Ÿ“ :)

@article{MIMDet,
  title={Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection},
  author={Fang, Yuxin and Yang, Shusheng and Wang, Shijie and Ge, Yixiao and Shan, Ying and Wang, Xinggang},
  journal={arXiv preprint arXiv:2204.02964},
  year={2022}
}

mimdet's People

Contributors

simonjjj avatar yuxin-cv avatar vealocia avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.