Paper Link, Model Link, Dataset Link

• Cosmo, a fully open-source and comprehensive interleaved pre-training framework, is meticulously crafted for image and video processing.

Its primary focus lies on In-context Learning.

• 2/Jan/2024. We provide the preprocess scripts to prepare the pre-training/downstream dataset.
• 2/Jan/2024. We release Howto-Interlink7M dataset. See Huggingface List View for details.

• Provides an Interleaved Image/Video Dataloader.
• Integration with Webdataset.
• Utilizes Huggingface Transformers along with Deepspeed for training.
• Incorporates contrastive loss.
• Enables few-shot evaluation.
• Supports instruction tuning.

See MODEL_CARD.md for more details.

See HowToInterlink7M.md.

See INSTALL.md.

See DATASET.md.

See PRETRAIN.md.

This code support 44 downstream datasets. Include but not limited to COCO, FLICKR30K, OK-VQA, TextVQA, VizWiz, VQAV2, Hatefulmemes, Vatex, TGIF, MSVD, MSRVTT.

See Evaluation.md

See TUNING.md

If you find our work helps, please consider cite the following work

@article{wang2024cosmo,
  title={COSMO: Contrastive Streamlined Multimodal Model with Interleaved Pre-Training},
  author={Wang, Alex Jinpeng and Li, Linjie and Lin, Kevin Qinghong and Wang Jianfeng and Lin, Kevin and Yang, Zhengyuan  and Wang, Lijuan and Shou, Mike Zheng},
  journal={arXiv preprint arXiv:2401.00849},
  year={2024}
}

Email: awinyimgprocess at gmail dot com

Our work are mainly based on the following works:

MMC4,Open-flamingo, Open-CLIP, Huggingface Transformer and WebDataset.