BiomedGPT is developed based on OFA but pre-trained and fine-tuned with multi-modal & multi-task biomedical datasets. Details are shown in datasets.md. If you have any questions, feel free to contact us or post issues.

Breaking News! 💥 :

We have updated the fine-tuning receipts to to match or surpass the performance of prior state-of-the-art models, including Med-PaLM M (12B) and GPT-4V. For more details, please refer to the corresponding preprint version was updated. Below is a snapshot comparing these performances.

• [] We're updating our codebase and will soon release the latest SOTA checkpoints for various downstream tasks.
• [] Efforts are underway to translate our code from fairseq to Hugging Face, simplifying usage for users.
• We used instruction following data (10k) from LLaVA-Med to tune our pre-trained checkpoints, and showed much better zero-shot performance on the VQA-RAD test set.

We provid pretrained checkpoints of BiomedGPT (Dropbox), which can be put in the scripts/ folder for further development. For finetuned checkpoints, please refer to checkpoints.md.

We emphasize that BiomedGPT, including its files, code, and checkpoints, is strictly for academic research purposes. Commercial and clinical uses are strictly prohibited for three key reasons: First, BiomedGPT is based on the OFA framework, which carries a non-commercial license that we have inherited. Second, our model is not licensed for use in healthcare settings. Finally, we have not implemented sufficient security measures, and the current model cannot guarantee the accuracy required for medical diagnoses.

git clone https://github.com/taokz/BiomedGPT
conda env create -f biomedgpt.yml
python -m pip install pip==21.2.4
pip install fairseq

We provide the preprocessing, pretraining, finetuning and inference scripts in the scripts/ folder. You can follow the directory setting below:

BiomedGPT/
├── checkpoints/
├── datasets/
│   ├── pretraining/
│   ├── finetuning/
│   └── ...
├── scripts/
│   ├── preprocess/
│   │   ├── pretraining/
│   │   └── finetuning/
│   ├── pretrain/
│   ├── vqa/
│   └── ...
└── ...

Please follow datasets.md to prepare pretraining datasets, which includes 4 TSV files: vision_language.tsv, text.tsv, image.tsv and detection.tsv in the directory of ./datasets/pretraining/.

cd scripts/pretrain
bash pretrain_tiny.sh

Feel free to modify the hyperparameters in the bash script for your requirements or ablation study.

We provide the run scripts of fine-tuning and inference. There will be log files during execution. Before fine-tuning or inference, please refer to

cd scripts/vqa
# for fine-tuning
bash train_vqa_rad_beam.sh
# for inference
bash evaluate_vqa_rad_beam.sh

cd scripts/caption
# for fine-tuning
bash train_peir_gross.sh
# for inference
bash evaluate_peir_gross.sh

cd scripts/text_sum
# for fine-tuning
bash train_meqsum.sh
# for inference
bash evaluate_meqsum.sh

cd scripts/mednli
# for fine-tuning
bash train_mednli.sh
# for inference
bash evaluate_mednli.sh

cd scripts/image_cls
# for fine-tuning: I provide a template, please set different hyparameters for each dataset in MedMNIST if required.
bash train_medmnist.sh 
# for inference: a template
bash evaluate_medmnist.sh

• OFA
• Fairseq
• taming-transformers
• self-critical.pytorch
• LLaVA-Med
  
  

If you use BiomedGPT model or our code for publications, please cite 🤗:

@misc{zhang2023biomedgpt,
      title={BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks}, 
      author={Kai Zhang and Jun Yu and Zhiling Yan and Yixin Liu and Eashan Adhikarla and Sunyang Fu and Xun Chen and Chen Chen and Yuyin Zhou and Xiang Li and Lifang He and Brian D. Davison and Quanzheng Li and Yong Chen and Hongfang Liu and Lichao Sun},
      year={2023},
      eprint={2305.17100},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}