zhaozeno / zerovl Goto Github PK

This repository contains source code necessary to reproduce the results presented in the paper ZeroVL: A Strong Baseline for Aligning Vision-Language Representations with Limited Resources.

Pioneering dual-encoder pre-training works (e.g., CLIP and ALIGN) require a tremendous amount of data and computational resources (e.g., billion-level web data and hundreds of GPUs), which prevent researchers with limited resources from reproduction and further exploration. To this end, we provide a comprehensive training guidance, which allows us to conduct dual-encoder multi-modal representation alignment with limited resources. Meanwhile, we provide a reproducible strong baseline of competitive results, namely ZeroVL, with publicly accessible academic datasets and a popular experimental environment.

Image-text retreival RSUM scores on MSCOCO and Flickr30K datasets:

zs.: zero-shot setting, ft.: fine-tuned setting.

ImageNet-1K linear probing results:

Requirements:

• Python 3.7
• Pytorch 1.8.1
• torchvision 0.9.1
• cuda 11.1

Install requirements:

pip3 install -r requirements.txt

Check PRETRAINING.md for codebase usage.

ZeroVL 14M weights: Google Drive, Baidu Pan

ZeroVL 100M weights: Google Drive, Baidu Pan

Check EVALUATION.md for codebase usage.

Check LINEAR.md for codebase usage.

If you use ZeroVL in your research or wish to refer to the baseline results, please use the following BibTeX entry.

@article{cui2022zerovl,
  title={Contrastive Vision-Language Pre-training with Limited Resources},
  author={Cui, Quan and Zhou, Boyan and Guo, Yu and Yin, Weidong and Wu, Hao and Yoshie, Osamu and Chen, Yubo},
  journal={ECCV},
  year={2022}
}

ZeroVL is released under the MIT license. See LICENSE for details.