eval opt-175B about bigscience HOT 1 OPEN

Hi, we used ranking evaluation for opt-175b, ie getting the loglikelihood of examples. Typically a classification task would look like:

sample1: "Review: {REVIEW}\nIs this review positive or negative? positive
sample2: "Review: {REVIEW}\nIs this review positive or negative? negative

We compute the likelihood of both samples, and rank them to get accuracy/recall/f1 for example.

Concerning the way we converted Megatron-DeepSpeed checkpoint, we essentially computed the opposite of https://github.com/huggingface/transformers/blob/main/src/transformers/models/bloom/convert_bloom_original_checkpoint_to_pytorch.py typically mapped each parameters to their stages (in the DeepSpeed sense), you can find:

• conversion script: #51
• modification to Meg-DS: bigscience-workshop/Megatron-DeepSpeed#301

from bigscience.