This repo contains the code for our ACL 2023 paper Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach.

The results on different datasets (using 128 labels as the budget) for fine-tuning are summarized as follows:

For prompt-based learning, we use the same pipeline as the LM-BFF. The result with 128 labels is shown as follows.

python 3.8
transformers==4.2.0
pytorch==1.8.0
scikit-learn
faiss-cpu==1.6.4
sentencepiece==0.1.96
tqdm>=4.62.2
tensorboardX
nltk
openprompt

We use the following four datasets for the main experiments.

The processed data can be found at this link. The folder to put these datasets will be discribed in the following parts.

Run the following commands

python gen_embedding_simcse.py --dataset [the dataset you use] --gpuid [the id of gpu you use] --batchsize [the number of data processed in one time]

We provide the pseudo prediction obtained via prompts in the above link for datasets. Please refer to the original papers for details.

Run the following commands (example on AG News dataset)

python patron_sample.py --dataset agnews --k 50 --rho 0.01 --gamma 0.5 --beta 0.5

Some important hyperparameters:

• rho: the parameter used for uncertainty propagation in Eq. 6 of the paper
• beta: the regularization of distance in Eq. 8 of the paper
• gamma: the weight of the regularization term in Eq. 10 of the paper

See finetune folder for detailed instructions.

See prompt_learning folder for detailed instructions.

See this link as the pipeline for generating the prompt-based predictions. Note that you need to customize your prompt verbalizers and templates.

To generate the document embeddings, you can follow the above commands by using SimCSE.

Once you generate the index for the selected data, then you can use the pipelines in Running Fine-tuning Experiments and Running Prompt-based Learning Experiments for the few-shot fine-tuning and prompt-based learning experiments.

Please kindly cite the following paper if you find this repo helpful for your research. Thanks in advance!

@article{yu2022patron,
  title={Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach
},
  author={Yue Yu and Rongzhi Zhang and Ran Xu and Jieyu Zhang and Jiaming Shen and Chao Zhang},
  journal={arXiv preprint arXiv:2209.06995},
  year={2022}
}

We would like to thank the authors from the repo SimCSE and OpenPrompt for the well-organized code.