This is the official PyTorch implementation of MixGen, which is a joint data augmentation technique for vision-language representation learning to improve data efficiency.
Here are some image-text pairs generated by MixGen,
MixGen is an input-level data augmentation technique, which can be plugged-and-played into existing vision-language learning methods with minimal code change.
Here we adopt ALBEF, NeurIPS'21 as an illustrating example. We only need to add one line between dataloader and model forward here.
That is, change from
for i, (image, text) in enumerate(metric_logger.log_every(data_loader, print_freq, header)):
optimizer.zero_grad()
to
import mixgen as mg
for i, (image, text) in enumerate(metric_logger.log_every(data_loader, print_freq, header)):
image, text = mg.mixgen(image, text, num=16)
optimizer.zero_grad()
And that's it!!! No more changes needed to be made. You can simply kicoff training just like ALBEF does,
python -m torch.distributed.launch --nproc_per_node=8 --use_env Pretrain.py
If you find MixGen useful in your research, please kindly consider to cite the following paper.
@InProceedings{Hao_2023_WACV,
author = {Hao, Xiaoshuai and Zhu, Yi and Appalaraju, Srikar and Zhang, Aston and Zhang, Wanqian and Li, Bo and Li, Mu},
title = {MixGen: A New Multi-Modal Data Augmentation},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops},
month = {January},
year = {2023},
pages = {379-389}
}
See CONTRIBUTING for more information.
This project is licensed under the Apache-2.0 License.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent