Implementation of "Data-efficient Large Vision Models through Sequential Autoregression" [ICML 2024].
Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens. In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models.
- Code about training models.
- Code about inferencing models.
- Huggingface & InternLM ckpts.
- Code about data generation.
based on InternLM-v0.2.1dev20231121
Install: https://github.com/InternLM/InternLM/blob/v0.2.1dev20231121/doc/en/install.md
Put your training data to /path/to/data/vision.
Training command:
torchrun --nproc_per_node 8 train.py --config ./configs/pretrain_300m.py --launcher torch
Training via KD command:
torchrun --nproc_per_node 8 train.py --config ./configs/kd_1b_to_300m.py --launcher torch
Convert model and inference example: ./tools
The corresponding huggingface ckpt can be downloaded at LLaMA-1b-hf Onedrive / LLaMA-1b-hf Baidu Disk and LLaMA-300m-hf.
Please refer to data_generation/README.md.
If you find this project useful in your research, please consider cite:
@article{guo2024dataefficient,
title={Data-efficient Large Vision Models through Sequential Autoregression},
author={Guo, Jianyuan and Hao, Zhiwei and Wang, Chengcheng and Tang, Yehui and Wu, Han and Hu, Han and Han, Kai and Xu, Chang},
journal={arXiv preprint arXiv:2402.04841},
year={2024}
}We maily follow the directon of project LVM. And this repo is based on InternLM, huggingface.co/transformers, and huggingface.co/openMUSE.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent