This repo contains a PyTorch implementation of the paper Multimodal Federated Learning via Contrastive Representation Ensemble (ICLR 2023).

Note: This repository will be updated in the next few days for improved readability, easier environment setup, and datasets management. Please stay tuned!

The required packages of the environment we used to conduct experiments are listed in requirements.txt.

Please note that you should install apex by following the instructions from https://github.com/NVIDIA/apex#installation, instead of directly running pip install apex.

For datasets, please download the MSCOCO, Flicker30K, CIFAR-100, and AG_NEWS datasets, and arrange their directories as follows:

os.environ['HOME'] + 'data/'
├── AG_NEWS
├── cifar100
│   └── cifar-100-python
├── flickr30k
│   └── flickr30k-images
├── mmdata
│   ├── MSCOCO
│   │   └── 2014
│   │       ├── allimages
│   │       ├── annotations
│   │       ├── train2014
│   │       └── val2014

To reproduce CreamFL with BERT and ResNet101 as server models, run the following shell command:

python src/main.py --name CreamFL --server_lr 1e-5 --agg_method con_w --contrast_local_inter --contrast_local_intra --interintra_weight 0.5

If you find the paper provides some insights into multimodal FL or our code useful 🤗, please consider citing:

@article{yu2023multimodal,
  title={Multimodal Federated Learning via Contrastive Representation Ensemble},
  author={Yu, Qiying and Liu, Yang and Wang, Yimu and Xu, Ke and Liu, Jingjing},
  journal={arXiv preprint arXiv:2302.08888},
  year={2023}
}

We would like to thank for the code from PCME and MOON repositories.