Can not reproduce the results on LLaVA-1.5 about opera HOT 3 CLOSED

Hi, Could you please provide some details about your result reproducing? For example, which result.

from opera.

Hi, Could you please provide some details about your result reproducing? For example, which result.

The image above displays the results from your paper. However, following your instructions to modify transformers-4.31.0, I obtained the following values: CHAIRs=48.6 and CHAIRi=14.2. The parameters used were num_beams=5, max_new_tokens=512, and do_sample=false

from opera.

Hi, thanks for your interest!

To reproduce the CHAIR results, you might need to:

1. Make sure set the correct key_position and the parameters in the generate function, as noted in TL;DR:

opera_decoding=True,
key_position=key_position,
scale_factor=50,
threshold=15,
num_attn_candidates=5,
penalty_weights=1,

2. Use our provided codebase and transformers version (4.29.2). Different transformers versions may affects the final results.
3. Most crucially, use our provided 500 sample list, given in log/chair_eval_results. Different sample selection greatly influence the CHAIR results.

BTW: We also provide our generated results in log/chair_eval_results. You can directly evaluate them with CHAIR evaluation tool. You can also do the experiments in your own codebase with a new sample selection, but the results may be different with the OPERA results in our paper.

from opera.