Can you provide some details of the evaluation code for the reported results? about m3ke HOT 1 OPEN

Thank you for your interest in the M3KE dataset. As you mentioned, different evaluation methods can lead to significant differences in experimental results. We attempted to select the label with the highest probability from four choices as the final answer. However, we found that LLMs such as BLOOM-7b1 typically only choose one label, even when the question differs in zero and few-shot scenarios. As a result, we decided to extract ABCD from the model generation and limit the maximum generation length as short as possible. If more than one label is present in the generation, we consider the model's answer to be incorrect.

We plan to make the questions of the M3KE dataset publicly available before the end of June. We would be greatly appreciate if you use the M3KE dataset.

from m3ke.