We make the first attempt towards unsupervised preference alignment in Large Vision-Language Models.

The whole pipeline do not need any GPT-4 or humman annotated labels in preference alignment. Paper

This package is developed by Ke Zhu (http://www.lamda.nju.edu.cn/zhuk/) and is free for academic usage. You can run it at your own risk. For other purposes, please contact Xiangyu Zhang (mail to [email protected]).

• Install
• Dataset
• SeVa pipeline
• Model
• Training
• Evaluation

First, clone this repo and navigate to SeVa folder:

git clone https://github.com/Kevinz-code/SeVa.git
cd SeVa

Then create a conda environment and install packages

conda create -n seva python==3.9 -y
conda activate seva
pip install torch==2.0.1 torchvision==0.15.2
pip install -e .

We expect the image dataset to have the following structure:

data/
|-- texvqa/
|---- train_images/
......
|-- ocrvqa/
|---- images/
......
|-- coco2014/
|---- val2014/

In which texvqa and ocrvqa are used for DPO data generation, coco2014 are optionally required for quick evalutation in POPE benchmark. You can download textvqa and coco2014 images in their official website. For ocrvqa images, download them using ocrvqa scripts (following LLaVA).

We have included a detailed DPO data construction pipeline in `data/' folder, with step1, step2 and step3. Refer to README

data/
|-- step1/
|-- step2/
|-- step3/
|-- README.md

Below are three SeVa models using 3 different DPO data source (Diffusion-steps500, Diffusion-steps800 and MOCO augmentaions). Standard deviation exists in some benchmarks (e.g., MM-Vet and LLaVA-Bench).

You need to first download weights of LLaVA-v1.5-7b.

For running with DPO data sourced from Diffusion noise (step=500):

sh run/llava1.5_lora_our_ocr8k_diffu500_text8k_diffu500.sh

For running with DPO data sourced from Diffusion noise (step=800):

sh run/llava1.5_lora_our_ocr8k_diffu800_text8k_diffu800.sh

Here we provide an evaluation on POPE benchmark, to help you get a quick evaluation of your models.

For models trained with diffusion steps500 DPO data, run

sh run/eval_pope_diffu500.sh

For models trained with diffusion steps800 DPO data, run

sh run/eval_pope_diffu800.sh

note: our evaluation are based on llava1.1.x version of its code repository

Refer to LLaVa-1.5 for a comprehension evaluation of multiple Benchmarks.

If you find our paper or codebase useful, please consider cite

@misc{zhu2024selfsupervised,
    title={Self-Supervised Visual Preference Alignment},
    author={Ke Zhu and Liang Zhao and Zheng Ge and Xiangyu Zhang},
    year={2024},
    eprint={2404.10501},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

This repo is based on HA-DPO and LLaVA. We thank their efforts in building their codebase.