Collection of papers and related works for Large Language Models (ChatGPT, GPT-3, Codex etc.).

This repository is contributed by the following contributors.

• Organizers: Guilin Qi (漆桂林), Xiaofang Qi (戚晓芳)
• Paper Collectors: Zafar Ali, Sheng Bi (毕胜), Yongrui Chen (陈永锐), Zizhuo Chen (陈孜卓), Xinbang Dai (戴鑫邦), Huan Gao (高桓), Nan Hu (胡楠), Shilong Hu (胡世龙), Jingqi Kang (康婧淇), Jiaqi Li (李嘉琦), Dehai Min (闵德海), Guilin Qi (漆桂林), Yiming Tan (谭亦鸣), Tongtong Wu (吴桐桐), Songlin Zhai (翟松林), Shenyu Zhang (张沈昱), Yuxin Zhang (张裕欣)
• Maintainers: Runzhe Wang (王润哲), Shenyu Zhang (张沈昱)

The automation script of this repo is powered by Auto-Bibfile. If you'd like to commit to this repo, please modify bibtex.bib or related_works.json and re-generate README.md using python scripts/run.py.

• [Overview] -- Homepage
• -- Summary
• -- Author
• -- Techniques
• -- Published Time
• -- Published Venue

• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity,
  by Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji et al.
  本文提出了一个使用公开数据集定量评估交互式LLM（如ChatGPT）的框架。我们使用涵盖8个不同的常见NLP应用任务的21个数据集对ChatGPT进行了广泛的技术评估。我们基于这些数据集和一个新设计的多模态数据集评估了ChatGPT的多任务、多语言和多模态方面。
  
  
• Is ChatGPT a General-Purpose Natural Language Processing Task Solver?,
  by Qin, Chengwei, Zhang, Aston, Zhang, Zhuosheng, Chen, Jiaao, Yasunaga, Michihiro and Yang, Diyi
  
  
• ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots,
  by Reham Omar, Omij Mangukiya, Panos Kalnis and Essam Mansour
  
  
• Mathematical Capabilities of ChatGPT,
  by Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier and Julius Berner
  
  
• Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization,
  by Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen and Wei Cheng
  
  
• On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective,
  by Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang et al.
  
  
• ChatGPT is not all you need. A State of the Art Review of large Generative AI models,
  by Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merch'an
  
  
• Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT,
  by Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du and Dacheng Tao
  
  
• Evaluation of ChatGPT as a Question Answering System for Answering Complex Questions,
  by Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen and Guilin Qi
  
  
• ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models,
  by Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu and Ben He
  
  
• Holistic Evaluation of Language Models,
  by Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan et al.
  
  
• Evaluating the Text-to-SQL Capabilities of Large Language Models,
  by Nitarshan Rajkumar, Raymond Li and Dzmitry Bahdanau
  
  
• Are Visual-Linguistic Models Commonsense Knowledge Bases?,
  by Hsiu-Yu Yang and Carina Silberer
  
  
• Is GPT-3 a Psychopath? Evaluating Large Language Models from a Psychological Perspective,
  by Xingxuan Li, Yutong Li, Linlin Liu, Lidong Bing and Shafiq R. Joty
  
  
• GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models,
  by Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li and Kai-Wei Chang
  
  
• RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners,
  by Soumya Sanyal, Zeyi Liao and Xiang Ren
  
  
• A Systematic Evaluation of Large Language Models of Code,
  by Frank F. Xu, Uri Alon, Graham Neubig and Vincent J. Hellendoorn
  
  
• Evaluating Large Language Models Trained on Code,
  by Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond'e de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda et al.
  
  
• GLGE: A New General Language Generation Evaluation Benchmark,
  by Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu et al.
  
  
• Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews,
  by Mohammad Abdul Hadi and Fatemeh H. Fard
  
  
• Do Language Models Perform Generalizable Commonsense Inference?,
  by Peifeng Wang, Filip Ilievski, Muhao Chen and Xiang Ren
  
  
• RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms,
  by Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara and Xiang Ren
  
  
• Evaluation of Text Generation: A Survey,
  by Asli Celikyilmaz, Elizabeth Clark and Jianfeng Gao
  
  
• Neural Language Generation: Formulation, Methods, and Evaluation,
  by Cristina Garbacea and Qiaozhu Mei
  
  
• BERTScore: Evaluating Text Generation with BERT,
  by Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger and Yoav Artzi
  
  

• A Survey for In-context Learning,
  by Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu et al.
  This paper surveys and summarizes the progress and challenges of ICL, including ICL's formal definition, correlation to related studies, advanced techniques (training strategies, related analysis) and potential directions.
  
  
• A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT,
  by Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji et al.
  
  
• Complex QA and language models hybrid architectures, Survey,
  by Xavier Daull, Patrice Bellot, Emmanuel Bruno, Vincent Martin and Elisabeth Murisasco
  
  
• Augmented Language Models: a Survey,
  by Gr'egoire Mialon, Roberto Dess`\i, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozi`ere, Timo Schick et al.
  
  
• The Life Cycle of Knowledge in Big Language Models: A Survey,
  by Boxi Cao, Hongyu Lin, Xianpei Han and Le Sun
  
  
• When Neural Model Meets NL2Code: A Survey,
  by Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang and Jian-Guang Lou
  
  
• A Survey on Knowledge-Enhanced Pre-trained Language Models,
  by Chaoqi Zhen, Yanlei Shang, Xiangyu Liu, Yifei Li, Yong Chen and Dell Zhang
  
  
• A Continual Learning Survey: Defying Forgetting in Classification Tasks,
  by Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory G. Slabaugh and Tinne Tuytelaars
  
  
• The survey: Text generation models in deep learning,
  by Touseef Iqbal and Shaima Qureshi
  
  
• From distributed machine learning to federated learning: a survey,
  by Ji Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong and Dejing Dou
  
  
• Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code,
  by Changan Niu, Chuanyi Li, Bin Luo and Vincent Ng
  
  
• A Survey on Knowledge Graph-Based Recommender Systems,
  by Qingyu Guo, Fuzhen Zhuang, Chuan Qin, Hengshu Zhu, Xing Xie, Hui Xiong and Qing He
  
  
• Mind the Knowledge Gap: A Survey of Knowledge-enhanced Dialogue Systems,
  by Sagi Shaier, Lawrence Hunter and Katharina Kann
  
  
• Towards Reasoning in Large Language Models: A Survey,
  by Jie Huang and Kevin Chen-Chuan Chang
  
  
• Reasoning with Language Model Prompting: A Survey,
  by Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang et al.
  
  
• A Survey on Retrieval-Augmented Text Generation,
  by Huayang Li, Yixuan Su, Deng Cai, Yan Wang and Lemao Liu
  
  
• Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey,
  by Prajjwal Bhargava and Vincent Ng
  
  
• A Review on Language Models as Knowledge Bases,
  by Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona T. Diab and Marjan Ghazvininejad
  
  
• Advances and Challenges in Conversational Recommender Systems: A Survey,
  by Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke and Tat-Seng Chua
  
  
• Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey,
  by Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, Vinay Adiga and Erik Cambria
  
  
• Relational World Knowledge Representation in Contextual Language Models: A Review,
  by Tara Safavi and Danai Koutra
  
  
• Evaluation of Text Generation: A Survey,
  by Asli Celikyilmaz, Elizabeth Clark and Jianfeng Gao
  
  

• A Survey for In-context Learning,
  by Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu et al.
  This paper surveys and summarizes the progress and challenges of ICL, including ICL's formal definition, correlation to related studies, advanced techniques (training strategies, related analysis) and potential directions.
  
  
• Explanation Selection Using Unlabeled Data for In-Context Learning,
  by Xi Ye and Greg Durrett
  
  
• In-Context Learning with Many Demonstration Examples,
  by Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu and Lingpeng Kong
  This paper proposes a LM named EvaLM to scale up the sequence length (trained with 8k tokens per batch line). Experiments based on EvaLM prove that in-context learning can achieve higher performance with more demonstrations under many-shot instruction tuning (8k) and further extending the length of instructions (16k) can further improve the upper bound of scaling in-context learning.
  
  
• Large Language Models Are Implicitly Topic Models: Explaining and Finding Good Demonstrations for In-Context Learning,
  by Xinyi Wang, Wanrong Zhu and William Yang Wang
  
  
• Finding Supporting Examples for In-Context Learning,
  by Xiaonan Li and Xipeng Qiu
  
  
• The Learnability of In-Context Learning,
  by Noam Wies, Yoav Levine and Amnon Shashua
  
  
• Meta-learning via Language Model In-context Tuning,
  by Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis and He He
  This paper proposes in-context tuning, which recasts task adaptation and prediction as a simple sequence prediction problem: to form the input sequence, concatenate the task instruction, labeled in-context examples, and the target input to predict; to meta train the model to learn from in-context examples, finetune a PLM to predict the target label given the input sequence on a collection of tasks (very similar to MetaICL). On LAMA and BinaryClfs, the proposed method outperforms MAML.
  
  
• MetaICL: Learning to Learn In Context,
  by Sewon Min, Mike Lewis, Luke Zettlemoyer and Hannaneh Hajishirzi
  MetaICL proposes a supervised meta-training framework to enable LMs to more effectively learn a new task in context. In MetaICL, each meta-training example includes several training examples from one task that will be presented together as a single sequence to the LM, and the prediction of the final example is used to calculate the loss.
  
  
• Selective Annotation Makes Language Models Better Few-Shot Learners,
  by Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf et al.
  This paper proposes a graph-based selective annotation method named vote-k to
(1) select a pool of examples to annotate from unlabeled data,
(2) retrieve prompts (contexts) from the annotated data pool for in-context learning.
Specifically, the selection method first selects a small set of unlabeled examples iteratively and then labels them to serve as contexts for LLMs to predict the labels of the rest unlabeled data. The method selects the predictions with highest confidence (log probability of generation output) to fill up the selective annotation pool.
  
  
• Improving In-Context Few-Shot Learning via Self-Supervised Training,
  by Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov and Zornitsa Kozareva
  This paper proposes to use self-supervision (MLM, NSP, CL, etc.) between pre-training and downstream usage to teach the LM to perform in-context learning. Analysis reveals that:
(1) benefits of self-supervised depends on the amount of training data,
(2) semantic similarity between training and evaluation tasks matters,
(3) adding training objectives without diversity does not help,
(4) model performance improves when choosing similar templates for both self-supervised and downstream tasks,
(5) self-supervised tasks and human-annotated datasets are complementary,
(6) self-supervised-trained models are better at following task instructions.
  
  
• Instruction Induction: From Few Examples to Natural Language Task Descriptions,
  by Or Honovich, Uri Shaham, Samuel R. Bowman and Omer Levy
  (1) 探索了利用LLM在几个样本的情况下归纳出任务指令的能力；
(2) 测量两个指标：1. 模型归纳指令与人类归纳的指令对比，2. 利用模型归纳的指令作为prompt进行预测的执行准确率；
(3) 相比于GPT-3，InstructGPT效果更好，理所当然。
  
  
• Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity,
  by Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel and Pontus Stenetorp
  (1) This work demonstrates that few-shot prompts suffer from order sensitivity, in that for the same prompt the order in which samples are provided can make a difference to model performance.
(2) This work introduces a probing method which constructs an artificial development set by language models themselves to alleviate the order sensitivity problem.
  
  
• Learning To Retrieve Prompts for In-Context Learning,
  by Ohad Rubin, Jonathan Herzig and Jonathan Berant
  This paper proposes a method to retrieve good contexts for in-context learning. Specifically, the method
(1) uses an unsupervised retriever (BM25/SBERT) to obtain a set of context candidates,
(2) passes the candidates to a scoring model (GPT-Neo/GPT-J/GPT-3/Codex) and select the top/bottom k as positive/negative examples,
(3) uses the examples to train a dense retriever (BERT-based).
  
  
• Active Example Selection for In-Context Learning,
  by Yiming Zhang, Shi Feng and Chenhao Tan
  (1) This paper revisits the effect of example selection (re-ordering & calibration) for ICL, observing that a large variance across set of demonstration examples still exists.
(2) This paper applies reinforcement learning (Q-Learning) to optimize example selection by formulating this task as sequential decision-making problem, which is appropriate for example selection from unlabeled datasets.
  
  
• Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator,
  by Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo and Sang-goo Lee
  
  
• Measuring Convergence Inertia: Online Learning in Self-adaptive Systems with Context Shifts,
  by Elvin Alberts and Ilias Gerostathopoulos
  
  
• An Explanation of In-context Learning as Implicit Bayesian Inference,
  by Sang Michael Xie, Aditi Raghunathan, Percy Liang and Tengyu Ma
  
  
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?,
  by Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi and Luke Zettlemoyer
  
  
• The Impact of Symbolic Representations on In-context Learning for Few-shot Reasoning,
  by Hanlin Zhang, Yi-Fan Zhang, Li Erran Li and Eric P. Xing
  
  
• What Makes Good In-Context Examples for GPT-3?,
  by Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin and Weizhu Chen
  (1) 探索了在in-context learning中什么样的demonstration example可以对GPT-3的效果取得帮助；
(2) 利用roberta对样本进行编码，并计算demonstration与test example的向量距离（欧氏距离），最终发现与test example越相近的demonstration越能取得较好的效果。
  
  
• Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again,
  by Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun and Yu Su
  
  
• Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers,
  by Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui and Furu Wei
  (1) 与The Dual Form of Neural Networks Revisited结合一起看，可以进一步理解in-context learning，通过与NN线性层对偶形式的类比，可以将ICL流程描述为：1. 基于Transformer的预训练语言模型作为元优化器；2. 通过正向计算，根据示范例子产生元梯度；3. 通过关注，将元梯度应用于原始语言模型，建立一个ICL模型；
(2)与Fine-tune类似，ICL也是在zero-shot learning参数的基础上，提供了一个更新量。
  
  
• The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention,
  by Kazuki Irie, R'obert Csord'as and J"urgen Schmidhuber
  (1) 很有意思的一篇，回顾神经网络（NN）线性层Y=WX（省略偏置b）的原始形式与对偶形式，两种形式完全等价；
(2) 从对偶形式中可以发现，通过反向传播训练的NN线性层的输出主要是该层在训练期间的训练误差信号et的线性组合，其中权重是通过比较测试查询x和每个训练输入计算出来的；进一步可以得出，如果测试时输入的x和训练时的输入是正交的，那么梯度下降所得到的参数更新对于该样本x完全没有影响。
  
  
• Self-adaptive In-context Learning,
  by Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye and Lingpeng Kong
  
  
• Careful Data Curation Stabilizes In-context Learning,
  by Ting-Yun Chang and Robin Jia
  
  
• Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale,
  by Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff and Dan Roth
  
  

• Large Language Models Can Be Easily Distracted by Irrelevant Context,
  by Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Sch"arli and Denny Zhou
  
  
• The Capacity for Moral Self-Correction in Large Language Models,
  by Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamile Lukosiute, Anna Chen, Anna Goldie, Azalia Mirhoseini et al.
  
  
• Exploring the Benefits of Training Expert Language Models over Instruction Tuning,
  by Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee and Minjoon Seo
  
  
• Chinese Open Instruction Generalist: A Preliminary Release,
  by Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li et al.
  
  
• Finetuned Language Models are Zero-Shot Learners,
  by Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai et al.
  
  
• LaMDA: Language Models for Dialog Applications,
  by Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos et al.
  
  
• Scaling Instruction-Finetuned Language Models,
  by Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang et al.
  
  
• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks,
  by Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran et al.
  
  
• Self-Instruct: Aligning Language Model with Self Generated Instructions,
  by Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi and Hannaneh Hajishirzi
  
  
• How Many Data Samples is an Additional Instruction Worth?,
  by Ravsehaj Singh Puri, Swaroop Mishra, Mihir Parmar and Chitta Baral
  
  

• The Capacity for Moral Self-Correction in Large Language Models,
  by Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamile Lukosiute, Anna Chen, Anna Goldie, Azalia Mirhoseini et al.
  
  
• Aligning Text-to-Image Models using Human Feedback,
  by Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh et al.
  
  
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback,
  by Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort et al.
  
  
• Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization,
  by Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kiant'e Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi and Yejin Choi
  
  
• Teaching language models to support answers with verified quotes,
  by Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, H. Francis Song, Martin Chadwick, Mia Glaese, Susannah Young et al.
  
  
• Improving alignment of dialogue agents via targeted human judgements,
  by Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger et al.
  
  
• Scaling Laws for Reward Model Overoptimization,
  by Gao, Leo, Schulman, John and Hilton, Jacob
  
  
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned,
  by Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez et al.
  
  
• Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning,
  by Deborah Cohen, Moonkyung Ryu, Yinlam Chow, Orgad Keller, Ido Greenberg, Avinatan Hassidim, Michael Fink, Yossi Matias et al.
  
  
• Training language models to follow instructions with human feedback,
  by Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal et al.
  
  
• Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences,
  by Erdem Biyik, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk and Dorsa Sadigh
  
  
• WebGPT: Browser-assisted question-answering with human feedback,
  by Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain et al.
  
  
• Recursively Summarizing Books with Human Feedback,
  by Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike and Paul F. Christiano
  
  
• Learning to summarize with human feedback,
  by Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei et al.
  
  
• Dialogue Response Ranking Training with Large-Scale Human Feedback Data,
  by Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett and Bill Dolan
  
  
• Fine-Tuning Language Models from Human Preferences,
  by Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano and Geoffrey Irving
  
  
• Deep Reinforcement Learning from Human Preferences,
  by Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg and Dario Amodei
  
  

• GPT-4 Technical Report,
  by OpenAI
  
  
• GPT-4 System Card,
  by OpenAI
  
  
• OPT: Open Pre-trained Transformer Language Models,
  by Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab et al.
  
  
• WeLM: A Well-Read Pre-trained Language Model for Chinese,
  by Hui Su, Xiao Zhou, Houjin Yu, Yuwen Chen, Zilin Zhu, Yang Yu and Jie Zhou
  
  
• Language Models are Few-Shot Learners,
  by Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam et al.
  
  
• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators,
  by Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning
  
  
• Revisiting Pre-Trained Models for Chinese Natural Language Processing,
  by Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang and Guoping Hu
  
  
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention,
  by Pengcheng He, Xiaodong Liu, Jianfeng Gao and Weizhu Chen
  
  
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,
  by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li et al.
  
  
• A Primer in BERTology: What We Know About How BERT Works,
  by Anna Rogers, Olga Kovaleva and Anna Rumshisky
  
  
• Language Models are Unsupervised Multitask Learners,
  by Radford, Alec, Wu, Jeffrey, Child, Rewon, Luan, David, Amodei, Dario and Sutskever, Ilya
  
  
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,
  by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova
  
  
• RoBERTa: A Robustly Optimized BERT Pretraining Approach,
  by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis et al.
  
  
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,
  by Nils Reimers and Iryna Gurevych
  
  
• Improving language understanding by generative pre-training,
  by Radford, Alec, Narasimhan, Karthik, Salimans, Tim, Sutskever, Ilya and others
  
  

• Efficient Large Scale Language Modeling with Mixtures of Experts,
  by Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du et al.
  
  
• MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation,
  by Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He, Tuo Zhao and Weizhu Chen
  
  
• Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints,
  by Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani et al.
  
  
• Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts,
  by Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu and Jin Tang
  
  

• Knowledge-enhanced Neural Machine Reasoning: A Review,
  by Tanmoy Chowdhury, Chen Ling, Xuchao Zhang, Xujiang Zhao, Guangji Bai, Jian Pei, Haifeng Chen and Liang Zhao
  
  
• Deep Bidirectional Language-Knowledge Graph Pretraining,
  by Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy Liang and Jure Leskovec
  
  
• A Survey on Knowledge-Enhanced Pre-trained Language Models,
  by Chaoqi Zhen, Yanlei Shang, Xiangyu Liu, Yifei Li, Yong Chen and Dell Zhang
  
  
• Review of Knowledge-Enhanced Pre-trained Language Models,
  by Yi, HAN, Linbo, QIAO, Dongsheng, LI and Xiangke, LIAO
  
  
• Mind the Knowledge Gap: A Survey of Knowledge-enhanced Dialogue Systems,
  by Sagi Shaier, Lawrence Hunter and Katharina Kann
  
  
• A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products,
  by Kesong Liu, Jianhui Jiang and Feifei Lyu
  
  
• Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding,
  by Jianing Wang, Wenkang Huang, Minghui Qiu, Qiuhui Shi, Hongbin Wang, Xiang Li and Ming Gao
  
  
• Dict-BERT: Enhancing Language Model Pre-training with Dictionary,
  by Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng and Meng Jiang
  
  
• GreaseLM: Graph REASoning Enhanced Language Models,
  by Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning and Jure Leskovec
  
  
• SPOT: Knowledge-Enhanced Language Representations for Information Extraction,
  by Jiacheng Li, Yannis Katsis, Tyler Baldwin, Ho-Cheol Kim, Andrew Bartko, Julian J. McAuley and Chun-Nan Hsu
  
  
• Relation-aware Language-Graph Transformer for Question Answering,
  by Jinyoung Park, Hyeong Kyu Choi, Juyeon Ko, Hyeon-Jin Park, Ji-Hoon Kim, Jisu Jeong, Kyung-Min Kim and Hyunwoo J. Kim
  
  
• Knowledge-based Review Generation by Coherence Enhanced Text Planning,
  by Junyi Li, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan and Ji-Rong Wen
  
  
• A Three-Stage Learning Framework for Low-Resource Knowledge-Grounded Dialogue Generation,
  by Shilei Liu, Xiaofeng Zhao, Bochao Li, Feiliang Ren, Longhui Zhang and Shujuan Yin
  
  
• Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledge,
  by Bodhisattwa Prasad Majumder, Sudha Rao, Michel Galley and Julian J. McAuley
  
  
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation,
  by Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen et al.
  
  
• Improving Biomedical Pretrained Language Models with Knowledge,
  by Zheng Yuan, Yijia Liu, Chuanqi Tan, Songfang Huang and Fei Huang
  
  
• KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation,
  by Wenhu Chen, Yu Su, Xifeng Yan and William Yang Wang
  
  
• A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation,
  by Jian Guan, Fei Huang, Minlie Huang, Zhihao Zhao and Xiaoyan Zhu
  
  
• Knowledge-Enhanced Personalized Review Generation with Capsule Graph Neural Network,
  by Junyi Li, Siqing Li, Wayne Xin Zhao, Gaole He, Zhicheng Wei, Nicholas Jing Yuan and Ji-Rong Wen
  
  
• MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models,
  by Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar and Bryan Catanzaro
  
  
• Zero-shot Learning with Semantic Output Codes,
  by Mark Palatucci, Dean Pomerleau, Geoffrey E. Hinton and Tom M. Mitchell
  
  

• Compressing Pre-trained Models of Code into 3 MB,
  by Jieke Shi, Zhou Yang, Bowen Xu, Hong Jin Kang and David Lo
  
  
• Less is More: Task-aware Layer-wise Distillation for Language Model Compression,
  by Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen and Tuo Zhao
  
  
• Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts,
  by Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu and Jin Tang
  
  
• CN-AutoMIC: Distilling Chinese Commonsense Knowledge from Pretrained Language Models,
  by Chenhao Wang, Jiachun Li, Yubo Chen, Kang Liu and Jun Zhao
  
  
• <img src=https://img.shields.io/badge/the_60th_Annual_Meeting_of_the_Association_for_Computational Linguistics_(Volume_1:Long_Papers),{ACL}_2022,_Dublin,_Ireland, May_22--27,_2022-2022-blue alt="img" style="zoom:100%; vertical-align: middle" /> Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary Distillation,
  by Dongha Choi, Hongseok Choi and Hyunju Lee
  
  
• Preparing lessons: Improve knowledge distillation with better supervision,
  by Tiancheng Wen, Shenqi Lai and Xueming Qian
  
  
• Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains,
  by Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong and Furu Wei
  
  
• Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation,
  by Shizhe Diao, Ruijia Xu, Hongjin Su, Yilei Jiang, Yan Song and Tong Zhang
  
  
• Distilling Knowledge Learned in BERT for Text Generation,
  by Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu and Jingjing Liu
  
  
• Improved Knowledge Distillation via Teacher Assistant,
  by Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa and Hassan Ghasemzadeh
  
  
• Regularizing Class-Wise Predictions via Self-Knowledge Distillation,
  by Sukmin Yun, Jongjin Park, Kimin Lee and Jinwoo Shin
  
  
• Relational Knowledge Distillation,
  by Wonpyo Park, Dongju Kim, Yan Lu and Minsu Cho
  
  
• Revisit Knowledge Distillation: a Teacher-free Framework,
  by Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang and Jiashi Feng
  
  
• Knowledge Distillation via Route Constrained Optimization,
  by Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan and Xiaolin Hu
  
  
• Improving Generalization and Robustness with Noisy Collaboration in Knowledge Distillation,
  by Elahe Arani, Fahad Sarfraz and Bahram Zonooz
  
  
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks,
  by Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova and Jimmy Lin
  
  
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks,
  by Jonathan Frankle and Michael Carbin
  
  
• Born-Again Neural Networks,
  by Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti and Anima Anandkumar
  
  
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer,
  by Sergey Zagoruyko and Nikos Komodakis
  
  
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,
  by Antti Tarvainen and Harri Valpola
  
  
• Deep Mutual Learning,
  by Ying Zhang, Tao Xiang, Timothy M. Hospedales and Huchuan Lu
  
  
• Deep Model Compression: Distilling Knowledge from Noisy Teachers,
  by Bharat Bhusan Sau and Vineeth N. Balasubramanian
  
  
• Distilling the Knowledge in a Neural Network,
  by Geoffrey E. Hinton, Oriol Vinyals and Jeffrey Dean
  
  

• Crawling the Internal Knowledge-Base of Language Models,
  by Roi Cohen, Mor Geva, Jonathan Berant and Amir Globerson
  本文提出一种从语言模型中提取结构化知识图谱的方法；使用专门设计的提示来控制提取过程中的精度和召回率；在GPT-3上进行了评估，显示了高精确度的结果。
  
  
• Understanding Finetuning for Factual Knowledge Extraction from Language Models,
  by Mehran Kazemi, Sid Mittal and Deepak Ramachandran
  
  
• Generative Knowledge Graph Construction: A Review,
  by Hongbin Ye, Ningyu Zhang, Hui Chen and Huajun Chen
  
  
• Calibrating Factual Knowledge in Pretrained Language Models,
  by Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui and Lei Li
  
  
• P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts,
  by Benjamin Newman, Prafulla Kumar Choubey and Nazneen Rajani
  
  
• Generated Knowledge Prompting for Commonsense Reasoning,
  by Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi and Hannaneh Hajishirzi
  
  
• Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering,
  by Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi and Yejin Choi
  
  
• <img src=https://img.shields.io/badge/the_2022_Conference_of_the_North_American_Chapter_of the_Association_for_Computational_Linguistics:_Human_Language_Technologies, {NAACL}_2022,_Seattle,_WA,_United_States,_July_10--15,_2022-2022-blue alt="img" style="zoom:100%; vertical-align: middle" /> Symbolic Knowledge Distillation: from General Language Models to Commonsense Models,
  by Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck et al.
  
  
• I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation,
  by Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West et al.
  
  

• Robustness of edited neural networks,
  by Davis Brown, Charles Godfrey, Cody Nizinski, Jonathan Tu and Henry Kvinge
  
  
• Transformer-Patcher: One Mistake worth One Neuron,
  by Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong and Zhang Xiong
  
  
• Fast Model Editing at Scale,
  by Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn and Christopher D. Manning
  
  
• Memory-Based Model Editing at Scale,
  by Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning and Chelsea Finn
  
  
• Locating and editing factual associations in gpt,
  by Meng, Kevin, Bau, David, Andonian, Alex J and Belinkov, Yonatan
  
  
• Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors,
  by Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim and Marzyeh Ghassemi
  
  
• Editing Factual Knowledge in Language Models,
  by Nicola De Cao, Wilker Aziz and Ivan Titov
  
  

• ThoughtSource: A central hub for large language model reasoning data,
  by Simon Ott, Konstantin Hebenstreit, Valentin Li'evin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther et al.
  
  
• Knowledge-enhanced Neural Machine Reasoning: A Review,
  by Tanmoy Chowdhury, Chen Ling, Xuchao Zhang, Xujiang Zhao, Guangji Bai, Jian Pei, Haifeng Chen and Liang Zhao
  
  
• Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning,
  by Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang and Yongbin Li
  
  
• Specializing Smaller Language Models towards Multi-Step Reasoning,
  by Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal and Tushar Khot
  
  
• MathPrompter: Mathematical Reasoning using Large Language Models,
  by Imani, Shima, Du, Liang and Shrivastava, Harsh
  
  
• Using Language Models For Knowledge Acquisition in Natural Language Reasoning Problems,
  by Fangzhen Lin, Ziyi Shou and Chengcai chen
  
  
• Improved logical reasoning of language models via differentiable symbolic programming,
  by Zhang, Hanlin, Li, Ziyang, Huang, Jiani, Naik, Mayur and Xing, Eric
  
  
• LILA: A Unified Benchmark for Mathematical Reasoning,
  by Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord et al.
  
  
• Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations,
  by Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras and Yejin Choi
  
  
• MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation,
  by Swarnadeep Saha, Xinyan Velocity Yu, Mohit Bansal, Ramakanth Pasunuru and Asli Celikyilmaz
  
  
• Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks,
  by Wenhu Chen, Xueguang Ma, Xinyi Wang and William W. Cohen
  
  
• The Impact of Symbolic Representations on In-context Learning for Few-shot Reasoning,
  by Hanlin Zhang, Yi-Fan Zhang, Li Erran Li and Eric P. Xing
  
  
• Towards Reasoning in Large Language Models: A Survey,
  by Jie Huang and Kevin Chen-Chuan Chang
  
  
• UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression,
  by Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen and Xiaodan Liang
  
  
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models,
  by Denny Zhou, Nathanael Sch"arli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet et al.
  (1) 两阶段的prompt，第一阶段问题分解（通过in-context learning实现，context中包含了其他问题的分解示例），对于每个问题，分解出回答该问题需要先回答什么子问题；
(2) 在第二阶段中，从后往前依次解决子问题，同样通过in-context learing得到，每次LLM的回答会参与组成下一个问题的prompt。
  
  
• Rationale-Augmented Ensembles in Language Models,
  by Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi and Denny Zhou
  
  
• The unreliability of explanations in few-shot prompting for textual reasoning,
  by Ye, Xi and Durrett, Greg
  
  
• JiuZhang: A Chinese Pre-trained Language Model for Mathematical Problem Understanding,
  by Wayne Xin Zhao, Kun Zhou, Zheng Gong, Beichen Zhang, Yuanhang Zhou, Jing Sha, Zhigang Chen, Shijin Wang et al.
  
  
• ThinkSum: Probabilistic reasoning over sets using large language models,
  by Batu Ozturkler, Nikolay Malkin, Zhen Wang and Nebojsa Jojic
  
  
• RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners,
  by Soumya Sanyal, Zeyi Liao and Xiang Ren
  
  
• Solving Quantitative Reasoning Problems with Language Models,
  by Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil et al.
  
  
• Thinking Like a Skeptic: Defeasible Inference in Natural Language,
  by Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith and Yejin Choi
  
  

• Multimodal Chain-of-Thought Reasoning in Language Models,
  by Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis and Alex Smola
  
  
• Rethinking with Retrieval: Faithful Large Language Model Inference,
  by Hangfeng He, Hongming Zhang and Dan Roth
  本文通过用GPT-3在三个复杂的推理任务：常识推理，时间推理和表格推理上进行大量实验来评估RR的有效性。结果表明，RR可以产生更忠实的解释，并提高LLM的性能。
  
  
• Active Prompting with Chain-of-Thought for Large Language Models,
  by Shizhe Diao, Pengcheng Wang, Yong Lin and Tong Zhang
  
  
• Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data,
  by Kashun Shum, Shizhe Diao and Tong Zhang
  
  
• Instruction Induction: From Few Examples to Natural Language Task Descriptions,
  by Or Honovich, Uri Shaham, Samuel R. Bowman and Omer Levy
  (1) 探索了利用LLM在几个样本的情况下归纳出任务指令的能力；
(2) 测量两个指标：1. 模型归纳指令与人类归纳的指令对比，2. 利用模型归纳的指令作为prompt进行预测的执行准确率；
(3) 相比于GPT-3，InstructGPT效果更好，理所当然。
  
  
• Iteratively Prompt Pre-trained Language Models for Chain of Thought,
  by Boshi Wang, Xiang Deng and Huan Sun
  (1) 提出了一种迭代式的prompt-tuning方法，他们认为soft prompt应该带有语境，即在自回归解码时不同时刻应该有不同的prompt向量；
(2) 利用BERT为encoder-decoder架构的PLM生成prompt，在每个解码时刻BERT都会根据先前时刻的上下文生成一组新的prompt向量，提供给PLM生成新的上下文，迭代往复。
  
  
• Complexity-Based Prompting for Multi-Step Reasoning,
  by Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark and Tushar Khot
  
  
• Measuring and Narrowing the Compositionality Gap in Language Models,
  by Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith and Mike Lewis
  
  
• Automatic Chain of Thought Prompting in Large Language Models,
  by Zhuosheng Zhang, Aston Zhang, Mu Li and Alex Smola
  
  
• Chain of Thought Prompting Elicits Reasoning in Large Language Models,
  by Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le and Denny Zhou
  
  
• Self-Consistency Improves Chain of Thought Reasoning in Language Models,
  by Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi and Denny Zhou
  
  
• Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango,
  by Aman Madaan and Amir Yazdanbakhsh
  
  
• Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters,
  by Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer and Huan Sun
  
  
• PaLM: Scaling Language Modeling with Pathways,
  by Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung et al.
  
  
• LAMBADA: Backward Chaining for Automated Reasoning in Natural Language,
  by Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu and Deepak Ramachandran
  
  
• Star: Self-taught reasoner bootstrapping reasoning with reasoning,
  by Zelikman, Eric, Mu, Jesse, Goodman, Noah D and Wu, Yuhuai Tony
  
  
• Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought,
  by Abulhair Saparov and He He
  
  
• Large Language Models are Zero-Shot Reasoners,
  by Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo and Yusuke Iwasawa
  
  
• Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning,
  by Antonia Creswell, Murray Shanahan and Irina Higgins
  
  
• Emergent Abilities of Large Language Models,
  by Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma et al.
  
  
• JiuZhang: A Chinese Pre-trained Language Model for Mathematical Problem Understanding,
  by Wayne Xin Zhao, Kun Zhou, Zheng Gong, Beichen Zhang, Yuanhang Zhou, Jing Sha, Zhigang Chen, Shijin Wang et al.
  
  
• Large Language Models Are Reasoning Teachers,
  by Namgyu Ho, Laura Schmid and Se-Young Yun
  
  
• Large Language Models are reasoners with Self-Verification,
  by Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu and Jun Zhao
  
  
• Reasoning with Language Model Prompting: A Survey,
  by Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang et al.
  
  
• PAL: Program-aided Language Models,
  by Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan and Graham Neubig
  
  
• Large Language Models are few(1)-shot Table Reasoners,
  by Wenhu Chen
  
  
• Large Language Models Can Self-Improve,
  by Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu and Jiawei Han
  
  

• Specializing Smaller Language Models towards Multi-Step Reasoning,
  by Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal and Tushar Khot
  
  
• MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation,
  by Swarnadeep Saha, Xinyan Velocity Yu, Mohit Bansal, Ramakanth Pasunuru and Asli Celikyilmaz
  
  
• Rationale-Augmented Ensembles in Language Models,
  by Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi and Denny Zhou
  
  

• LILA: A Unified Benchmark for Mathematical Reasoning,
  by Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord et al.
  
  
• Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks,
  by Wenhu Chen, Xueguang Ma, Xinyi Wang and William W. Cohen
  
  
• JiuZhang: A Chinese Pre-trained Language Model for Mathematical Problem Understanding,
  by Wayne Xin Zhao, Kun Zhou, Zheng Gong, Beichen Zhang, Yuanhang Zhou, Jing Sha, Zhigang Chen, Shijin Wang et al.
  
  
• Solving Quantitative Reasoning Problems with Language Models,
  by Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil et al.
  
  

• Improved logical reasoning of language models via differentiable symbolic programming,
  by Zhang, Hanlin, Li, Ziyang, Huang, Jiani, Naik, Mayur and Xing, Eric
  
  
• Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations,
  by Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras and Yejin Choi
  
  
• The Impact of Symbolic Representations on In-context Learning for Few-shot Reasoning,
  by Hanlin Zhang, Yi-Fan Zhang, Li Erran Li and Eric P. Xing
  
  
• UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression,
  by Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen and Xiaodan Liang
  
  
• Thinking Like a Skeptic: Defeasible Inference in Natural Language,
  by Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith and Yejin Choi
  
  

• Fairness and accuracy in horizontal federated learning,
  by Wei Huang, Tianrui Li, Dexian Wang, Shengdong Du, Junbo Zhang and Tianqiang Huang
  
  
• Federated Learning Meets Multi-Objective Optimization,
  by Zeou Hu, Kiarash Shaloudegi, Guojun Zhang and Yaoliang Yu
  
  
• From distributed machine learning to federated learning: a survey,
  by Ji Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong and Dejing Dou
  
  
• Meta-Learning Based Knowledge Extrapolation for Knowledge Graphs in the Federated Setting,
  by Mingyang Chen, Wen Zhang, Zhen Yao, Xiangnan Chen, Mengxiao Ding, Fei Huang and Huajun Chen
  
  
• Mitigating Biases in Student Performance Prediction via Attention-Based Personalized Federated Learning,
  by Yun-Wei Chu, Seyyedali Hosseinalipour, Elizabeth Tenorio, Laura M. Cruz Castro, Kerrie A. Douglas, Andrew Lan and Christopher G. Brinton
  
  
• Pretrained Models for Multilingual Federated Learning,
  by Orion Weller, Marc Marone, Vladimir Braverman, Dawn J. Lawrie and Benjamin Van Durme
  
  
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning,
  by Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli, Li Fei-Fei and Daniel L. Rubin
  
  
• FedBERT: When Federated Learning Meets Pre-training,
  by Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao, Hai Jin and Lichao Sun
  
  
• Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning,
  by John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi and Michael Rabbat
  
  
• Ditto: Fair and Robust Federated Learning Through Personalization,
  by Tian Li, Shengyuan Hu, Ahmad Beirami and Virginia Smith
  
  
• Fine-tuning is Fine in Federated Learning,
  by Gary Cheng, Karan N. Chadha and John C. Duchi
  
  
• Federated Learning with Fair Averaging,
  by Zheng Wang, Xiaoliang Fan, Jianzhong Qi, Chenglu Wen, Cheng Wang and Rongshan Yu
  
  
• Collaborative Fairness in Federated Learning,
  by Lingjuan Lyu, Xinyi Xu, Qian Wang and Han Yu
  
  
• Federated Visual Classification with Real-World Data Distribution,
  by Tzu-Ming Harry Hsu, Hang Qi and Matthew Brown
  
  

• Distributed Training of Knowledge Graph Embedding Models using Ray,
  by Nasrullah Sheikh, Xiao Qin, Yaniv Gur and Berthold Reinwald
  
  
• Distributed Learning With Sparsified Gradient Differences,
  by Yicheng Chen, Rick S. Blum, Martin Tak'ac and Brian M. Sadler
  
  
• Graph Attention Neural Network Distributed Model Training,
  by Esmaeilzadeh, Armin, Zadeh Nojoo Kambar, Mina Esmail and Heidari, Maryam
  
  
• Elastic Deep Learning Using Knowledge Distillation with Heterogeneous Computing Resources,
  by Daxiang Dong, Ji Liu, Xi Wang, Weibao Gong, An Qin, Xingjian Li, Dianhai Yu, Patrick Valduriez et al.
  
  
• GRACE: A Compressed Communication Framework for Distributed Machine Learning,
  by Hang Xu, Chen-Yu Ho, Ahmed M. Abdelmoniem, Aritra Dutta, El Houcine Bergou, Konstantinos Karatsenidis, Marco Canini and Panos Kalnis
  
  
• Load Balancing Optimization for Transformer in Distributed Environment,
  by Delu Ma, Zhou Lei, Shengbo Chen and Peng Wang
  
  
• DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs,
  by Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang et al.
  
  
• PyTorch Distributed: Experiences on Accelerating Data Parallel Training,
  by Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith et al.
  
  
• Ray: A Distributed Framework for Emerging AI Applications,
  by Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang et al.
  
  

• Selective Annotation Makes Language Models Better Few-Shot Learners,
  by Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf et al.
  This paper proposes a graph-based selective annotation method named vote-k to
(1) select a pool of examples to annotate from unlabeled data,
(2) retrieve prompts (contexts) from the annotated data pool for in-context learning.
Specifically, the selection method first selects a small set of unlabeled examples iteratively and then labels them to serve as contexts for LLMs to predict the labels of the rest unlabeled data. The method selects the predictions with highest confidence (log probability of generation output) to fill up the selective annotation pool.
  
  
• Selective Data Acquisition in the Wild for Model Charging,
  by Chengliang Chai, Jiabin Liu, Nan Tang, Guoliang Li and Yuyu Luo
  
  

• On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex,
  by Terry Yue Zhuo, Zhuang Li, Yujin Huang, Yuan-Fang Li, Weiqing Wang, Gholamreza Haffari and Fatemeh Shiri
  
  
• CodeT5Mix: A Pretrained Mixture of Encoder-decoder Transformers for Code Understanding and Generation,
  by Wang, Yue, Le, Hung, Gotmare, Akhilesh Deepak, Li, Junnan and Hoi, Steven
  
  
• CodeBERTScore: Evaluating Program&Code Generation with Pretrained Models of Code,
  by Zhou, Shuyan, Alon, Uri, Agarwal, Sumit and Neubig, Graham
  
  
• Code4Struct: Program&Code Generation for Few-Shot Structured Prediction from Natural Language,
  by Xingyao Wang, Sha Li and Heng Ji
  
  
• Language Models of Code are Few-Shot Commonsense Learners,
  by Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang and Graham Neubig
  
  
• When Neural Model Meets NL2Code: A Survey,
  by Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang and Jian-Guang Lou
  
  
• Evaluating the Text-to-SQL Capabilities of Large Language Models,
  by Nitarshan Rajkumar, Raymond Li and Dzmitry Bahdanau
  
  
• An extensive study on pre-trained models for program understanding and generation,
  by Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang and Lingming Zhang
  
  
• CodeRL: Mastering Program&Code Generation through Pretrained Models and Deep Reinforcement Learning,
  by Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese and Steven C. H. Hoi
  
  
• CoditT5: Pretraining for Source Code and Natural Language Editing,
  by Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li and Milos Gligoric
  
  
• Compressing Pre-trained Models of Code into 3 MB,
  by Jieke Shi, Zhou Yang, Bowen Xu, Hong Jin Kang and David Lo
  
  
• Diet code is healthy: simplifying programs for pre-trained models of code,
  by Zhaowei Zhang, Hongyu Zhang, Beijun Shen and Xiaodong Gu
  
  
• NatGen: generative pre-training by "naturalizing" source code,
  by Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar T. Devanbu and Baishakhi Ray
  
  
• Jigsaw: Large Language Models meet Program Synthesis,
  by Naman Jain, Skanda Vaidyanath, Arun Shankar Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram K. Rajamani and Rahul Sharma
  
  
• Natural Attack for Pre-trained Models of Code,
  by Zhou Yang, Jieke Shi, Junda He and David Lo
  
  
• Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models,
  by Sami Sarsa, Paul Denny, Arto Hellas and Juho Leinonen
  
  
• A Systematic Evaluation of Large Language Models of Code,
  by Frank F. Xu, Uri Alon, Graham Neubig and Vincent J. Hellendoorn
  
  
• Evaluating Large Language Models Trained on Code,
  by Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond'e de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda et al.
  
  
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation,
  by Yue Wang, Weishi Wang, Shafiq R. Joty and Steven C. H. Hoi
  
  
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation,
  by Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain et al.
  
  
• Unified Pre-training for Program Understanding and Generation,
  by Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray and Kai-Wei Chang
  
  
• Traceability Transformed: Generating more Accurate Links with Pre-Trained BERT Models,
  by Jinfeng Lin, Yalin Liu, Qingkai Zeng, Meng Jiang and Jane Cleland-Huang
  
  
• IntelliCode compose: Program&Code Generation using transformer,
  by Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu and Neel Sundaresan
  
  
• Multi-task Learning based Pre-trained Language Model for Code Completion,
  by Fang Liu, Ge Li, Yunfei Zhao and Zhi Jin
  
  

• CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training,
  by Xin Wang, Yasheng Wang, Yao Wan, Jiawei Wang, Pingyi Zhou, Li Li, Hao Wu and Jin Liu
  
  
• UniXcoder: Unified Cross-Modal Pre-training for Code Representation,
  by Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou and Jian Yin
  
  
• AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models,
  by Jos'e Antonio Hern'andez L'opez, Martin Weyssow, Jes'us S'anchez Cuadrado and Houari A. Sahraoui
  
  
• GraphCodeBERT: Pre-training Code Representations with Data Flow,
  by Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan et al.
  
  
• CLSEBERT: Contrastive Learning for Syntax Enhanced Code Pre-Trained Model,
  by Xin Wang, Yasheng Wang, Pingyi Zhou, Fei Mi, Meng Xiao, Yadao Wang, Li Li, Xiao Liu et al.
  
  

• CIRCLE: continual repair across programming languages,
  by Wei Yuan, Quanjun Zhang, Tieke He, Chunrong Fang, Nguyen Quoc Viet Hung, Xiaodong Hao and Hongzhi Yin
  
  
• Detect-Localize-Repair: A Unified Framework for Learning to Debug with CodeT5,
  by Nghi Bui, Yue Wang and Steven C. H. Hoi
  
  
• Multi-view Pre-trained Model for Code Vulnerability Identification,
  by Xuxiang Jiang, Yinhao Xiao, Jun Wang and Wei Zhang
  
  
• Towards JavaScript program repair with Generative Pre-trained Transformer (GPT-2),
  by M'ark Lajk'o, Viktor Csuvik and L'aszl'o Vid'acs
  
  
• Fast Changeset-based Bug Localization with BERT,
  by Agnieszka Ciborowska and Kostadin Damevski
  
  
• Applying CodeBERT for Automated Program Repair of Java Simple Bugs,
  by Ehsan Mashhadi and Hadi Hemmati
  
  
• A model with iterative trials for correcting logic errors in source code,
  by Matsumoto, Taku, Watanobe, Yutaka and Nakamura, Keita
  
  
• DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons,
  by Dawn Drain, Colin B. Clement, Guillermo Serrato and Neel Sundaresan
  
  

• AUGER: automatically generating review comments with pre-training models,
  by Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang and Chun Zuo
  
  
• Automating code review activities by large-scale pre-training,
  by Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green et al.
  
  
• Bridging Pre-trained Models and Downstream Tasks for Source Code Understanding,
  by Deze Wang, Zhouyang Jia, Shanshan Li, Yue Yu, Yun Xiong, Wei Dong and Xiangke Liao
  
  
• Using Pre-Trained Models to Boost Code Review Automation,
  by Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk and Gabriele Bavota
  
  
• What Do They Capture? - A Structural Analysis of Pre-Trained Language Models for Source Code,
  by Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guandong Xu and Hai Jin
  
  

• Learning to Program with Natural Language,
  by Yiduo Guo, Yaobo Liang, Chenfei Wu, Wenshan Wu, Dongyan Zhao and Nan Duan
  
  
• Tool Learning with Foundation Models,
  by Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang et al.
  
  
• Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models,
  by Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu and Jianfeng Gao
  
  

• Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code,
  by Changan Niu, Chuanyi Li, Bin Luo and Vincent Ng
  
  
• Do Pre-trained Language Models Indeed Understand Software Engineering Tasks?,
  by Yao Li, Tao Zhang, Xiapu Luo, Haipeng Cai, Sen Fang and Dawei Yuan
  
  
• Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews,
  by Mohammad Abdul Hadi and Fatemeh H. Fard
  
  
• What do pre-trained code models know about code?,
  by Anjan Karmakar and Romain Robbes
  
  
• Sentiment analysis for software engineering: How far can pre-trained transformer models go?,
  by Zhang, Ting, Xu, Bowen, Thung, Ferdian, Haryono, Stefanus Agus, Lo, David and Jiang, Lingxiao
  
  

• GPT-4 Technical Report,
  by OpenAI
  
  
• GPT-4 System Card,
  by OpenAI
  
  
• Mixture of Soft Prompts for Controllable Data Generation,
  by Derek Chen, Celine Lee, Yunan Lu, Domenic Rosati and Zhou Yu
  
  
• News Summarization and Evaluation in the Era of GPT-3,
  by Tanya Goyal, Junyi Jessy Li and Greg Durrett
  
  
• Fine-Grained Controllable Text Generation Using Non-Residual Prompting,
  by Fredrik Carlsson, Joey "Ohman, Fangyu Liu, Severine Verlinden, Joakim Nivre and Magnus Sahlgren
  
  
• The survey: Text generation models in deep learning,
  by Touseef Iqbal and Shaima Qureshi
  
  
• Factuality Enhanced Language Models for Open-Ended Text Generation,
  by Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Mohammad Shoeybi and Bryan Catanzaro
  
  
• FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models,
  by Rakesh Chada and Pradeep Natarajan
  
  
• All NLP Tasks Are Generation Tasks: A General Pretraining Framework,
  by Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang and Jie Tang
  
  
• Smelting Gold and Silver for Improved Multilingual AMR-to-Text Generation,
  by Leonardo F. R. Ribeiro, Jonas Pfeiffer, Yue Zhang and Iryna Gurevych
  
  
• PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation,
  by Jing Gu, Qingyang Wu, Chongruo Wu, Weiyan Shi and Zhou Yu
  
  
• DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances,
  by Xiaodong Gu, Kang Min Yoo and Jung-Woo Ha
  
  
• DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generation,
  by Xinyu Hua, Ashwin Sreevatsa and Lu Wang
  
  
• Latent Reasoning for Low-Resource Question Generation,
  by Xinting Huang, Jianzhong Qi, Yu Sun and Rui Zhang
  
  
• JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs,
  by Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu and Minlie Huang
  
  
• TextBox: A Unified, Modularized, and Extensible Framework for Text Generation,
  by Junyi Li, Tianyi Tang, Gaole He, Jinhao Jiang, Xiaoxuan Hu, Puzhao Xie, Zhipeng Chen, Zhuohao Yu et al.
  
  
• Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models,
  by Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan and Ji-Rong Wen
  
  
• Knowledge-based Review Generation by Coherence Enhanced Text Planning,
  by Junyi Li, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan and Ji-Rong Wen
  
  
• Prefix-Tuning: Optimizing Continuous Prompts for Generation,
  by Xiang Lisa Li and Percy Liang
  
  
• GLGE: A New General Language Generation Evaluation Benchmark,
  by Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu et al.
  
  
• A Three-Stage Learning Framework for Low-Resource Knowledge-Grounded Dialogue Generation,
  by Shilei Liu, Xiaofeng Zhao, Bochao Li, Feiliang Ren, Longhui Zhang and Shujuan Yin
  
  
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation,
  by Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang and Luo Si
  
  
• Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledge,
  by Bodhisattwa Prasad Majumder, Sudha Rao, Michel Galley and Julian J. McAuley
  
  
• ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation,
  by Kaushal Kumar Maurya, Maunendra Sankar Desarkar, Yoshinobu Kano and Kumari Deepshikha
  
  
• Structural Adapters in Pretrained Language Models for AMR-to-Text Generation,
  by Leonardo F. R. Ribeiro, Yue Zhang and Iryna Gurevych
  
  
• Towards Table-to-Text Generation with Numerical Reasoning,
  by Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura and Hiroya Takamura
  
  
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation,
  by Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen et al.
  
  
• Progressive Generation of Long Text with Pretrained Language Models,
  by Bowen Tan, Zichao Yang, Maruan Al-Shedivat, Eric P. Xing and Zhiting Hu
  
  
• Consistency and Coherency Enhanced Story Generation,
  by Wei Wang, Piji Li and Hai-Tao Zheng
  
  
• Structure-Aware Pre-Training for Table-to-Text Generation,
  by Xinyu Xing and Xiaojun Wan
  
  
• AugNLG: Few-shot Natural Language Generation using Self-trained Data Augmentation,
  by Xinnuo Xu, Guoyin Wang, Young-Bum Kim and Sungjin Lee
  
  
• DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling,
  by Lanqing Xue, Kaitao Song, Duocai Wu, Xu Tan, Nevin L. Zhang, Tao Qin, Wei-Qiang Zhang and Tie-Yan Liu
  
  
• FastSeq: Make Sequence Generation Faster,
  by Yu Yan, Fei Hu, Jiusheng Chen, Nikhil Bhendawade, Ting Ye, Yeyun Gong, Nan Duan, Desheng Cui et al.
  
  
• A Simple and Efficient Multi-Task Learning Approach for Conditioned Dialogue Generation,
  by Yan Zeng and Jian-Yun Nie
  
  
• DSGPT: Domain-Specific Generative Pre-Training of Transformers for Text Generation in E-commerce Title and Review Summarization,
  by Xueying Zhang, Yunjiang Jiang, Yue Shang, Zhaomeng Cheng, Chi Zhang, Xiaochuan Fan, Yun Xiao and Bo Long
  
  
• Language Models are Few-Shot Learners,
  by Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam et al.
  
  
• PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable,
  by Siqi Bao, Huang He, Fan Wang, Hua Wu and Haifeng Wang
  
  
• Evaluation of Text Generation: A Survey,
  by Asli Celikyilmaz, Elizabeth Clark and Jianfeng Gao
  
  
• KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation,
  by Wenhu Chen, Yu Su, Xifeng Yan and William Yang Wang
  
  
• Distilling Knowledge Learned in BERT for Text Generation,
  by Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu and Jingjing Liu
  
  
• Logic2Text: High-Fidelity Natural Language Generation from Logical Forms,
  by Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan and William Yang Wang
  
  
• Cross-Lingual Natural Language Generation via Pre-Training,
  by Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao and Heyan Huang
  
  
• Neural Language Generation: Formulation, Methods, and Evaluation,
  by Cristina Garbacea and Qiaozhu Mei
  
  
• TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching,
  by Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu and Ting Liu
  
  
• A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation,
  by Jian Guan, Fei Huang, Minlie Huang, Zhihao Zhao and Xiaoyan Zhu
  
  
• Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity,
  by Hamza Harkous, Isabel Groves and Amir Saffari
  
  
• Reformulating Unsupervised Style Transfer as Paraphrase Generation,
  by Kalpesh Krishna, John Wieting and Mohit Iyyer
  
  
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,
  by Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov and Luke Zettlemoyer
  
  
• Knowledge-Enhanced Personalized Review Generation with Capsule Graph Neural Network,
  by Junyi Li, Siqing Li, Wayne Xin Zhao, Gaole He, Zhicheng Wei, Nicholas Jing Yuan and Ji-Rong Wen
  
  
• Rigid Formats Controlled Text Generation,
  by Piji Li, Haisong Zhang, Xiaojiang Liu and Shuming Shi
  
  
• UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation,
  by Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen and Ming Zhou
  
  
• GPT-too: A Language-Model-First Approach for AMR-to-Text Generation,
  by Manuel Mager, Ram'on Fernandez Astudillo, Tahira Naseem, Md. Arafat Sultan, Young-Suk Lee, Radu Florian and Salim Roukos
  
  
• Few-shot Natural Language Generation for Task-Oriented Dialog,
  by Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng and Jianfeng Gao
  
  
• PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking,
  by Hannah Rashkin, Asli Celikyilmaz, Yejin Choi and Jianfeng Gao
  
  
• Investigating Pretrained Language Models for Graph-to-Text Generation,
  by Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Sch"utze and Iryna Gurevych
  
  
• Leveraging Pre-trained Checkpoints for Sequence Generation Tasks,
  by Sascha Rothe, Shashi Narayan and Aliaksei Severyn
  
  
• T3: Tree-Autoencoder Constrained Adversarial Text Generation for Targeted Attack,
  by Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang and Bo Li
  
  
• MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models,
  by Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar and Bryan Catanzaro
  
  
• StyleDGPT: Stylized Response Generation with Pre-trained Language Models,
  by Ze Yang, Wei Wu, Can Xu, Xinnian Liang, Jiaqi Bai, Liran Wang, Wei Wang and Zhoujun Li
  
  
• Generalized Conditioned Dialogue Generation Based on Pre-trained Language Model,
  by Yan Zeng and Jian-Yun Nie
  
  
• BERTScore: Evaluating Text Generation with BERT,
  by Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger and Yoav Artzi
  
  
• DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation,
  by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu et al.
  
  
• Language Models are Unsupervised Multitask Learners,
  by Radford, Alec, Wu, Jeffrey, Child, Rewon, Luan, David, Amodei, Dario and Sutskever, Ilya
  
  
• Unified Language Model Pre-training for Natural Language Understanding and Generation,
  by Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou et al.
  
  
• Large-Scale Transfer Learning for Natural Language Generation,
  by Sergey Golovanov, Rauf Kurbanov, Sergey I. Nikolenko, Kyryl Truskovskyi, Alexander Tselousov and Thomas Wolf
  
  
• Improving Neural Story Generation by Targeted Common Sense Grounding,
  by Huanru Henry Mao, Bodhisattwa Prasad Majumder, Julian J. McAuley and Garrison W. Cottrell
  
  
• MASS: Masked Sequence to Sequence Pre-training for Language Generation,
  by Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu and Tie-Yan Liu
  
  
• Generating Wikipedia by Summarizing Long Sequences,
  by Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser and Noam Shazeer
  
  
• Improving language understanding by generative pre-training,
  by Radford, Alec, Narasimhan, Karthik, Salimans, Tim, Sutskever, Ilya and others
  
  

• Quark: Controllable Text Generation with Reinforced Unlearning,
  by Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu and Yejin Choi
  
  
• Controllable Open-ended Question Generation with A New Question Type Ontology,
  by Shuyang Cao and Lu Wang
  
  
• A Distributional Approach to Controlled Text Generation,
  by Muhammad Khalifa, Hady Elsahar and Marc Dymetman
  
  
• A Plug-and-Play Method for Controlled Text Generation,
  by Damian Pascual, Beni Egressy, Clara Meister, Ryan Cotterell and Roger Wattenhofer
  
  
• Pre-training Text-to-Text Transformers for Concept-centric Common Sense,
  by Wangchunshu Zhou, Dong-Ho Lee, Ravi Kiran Selvam, Seyeon Lee and Xiang Ren
  
  
• Mention Flags (MF): Constraining Transformer-based Text Generators,
  by Yufei Wang, Ian D. Wood, Stephen Wan, Mark Dras and Mark Johnson
  
  
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation,
  by Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski and Rosanne Liu
  
  
• CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning,
  by Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi and Xiang Ren
  
  
• CTRL: A Conditional Transformer Language Model for Controllable Generation,
  by Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong and Richard Socher
  
  

• Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study,
  by Mingxu Tao, Yansong Feng and Dongyan Zhao
  
  
• Learning to Prompt for Continual Learning,
  by Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot et al.
  
  
• A Continual Learning Survey: Defying Forgetting in Classification Tasks,
  by Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory G. Slabaugh and Tinne Tuytelaars
  
  
• ELLE: Efficient Lifelong Pre-training for Emerging Data,
  by Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun and Jie Zhou
  
  
• Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora,
  by Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew O. Arnold and Xiang Ren
  
  
• Towards Continual Reinforcement Learning: A Review and Perspectives,
  by Khimya Khetarpal, Matthew Riemer, Irina Rish and Doina Precup
  
  
• Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network,
  by Zheng Gong, Kun Zhou, Xin Zhao, Jing Sha, Shijin Wang and Ji-Rong Wen
  
  
• LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5,
  by Chengwei Qin and Shafiq Joty
  We define a challenging yet practical problem as Lifelong Few-shot Language Learning and propose a unified framework for it based on prompt tuning of T5.
  
  
• Towards Continual Knowledge Learning of Language Models,
  by Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stanley Jungkyu Choi and Minjoon Seo
  We propose a novel continual learning formulation named Continual Knowledge Learning which allows large language models to constantly obtain new and updated knowledge while mitigating forgetting of previous learned time-invariant knowledge.
  
  
• Pretrained Language Model in Continual Learning: A Comparative Study,
  by Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi and Gholamreza Haffari
  To explore the layer-wise property of pretrained languge models in continual learning, we thoroughly compare the continual learning performance over the combination of 5 PLMs and 4 veins of CL methods on 3 benchmarks in 2 typical incremental settings.
  
  
• TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models,
  by Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim and Minjoon Seo
  
  
• StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models,
  by Adam Liska, Tom'as Kocisk'y, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d'Autume, Tim Scholtes et al.
  
  
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning,
  by Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu and Lei Shu
  NeurIPS 2021, The key component of CTR is the CL-plugin inserted in BERT. A CL-plugin is a capsule network with a new transfer routing mechanism to encourage knowledge transfer among tasks and also to isolate task-specific knowledge to avoid forgetting.
  
  
• Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning,
  by Jin, Xisen , Lin, Bill Yuchen , Rostami, Mohammad and Ren, Xiang
  We present a new learning setup, Continual Learning of Few-Shot Learners, to address challenges of both learning settings in a unified setup, with a hyper-network for task-specific adapter generation.
  
  
• Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models,
  by Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James R. Glass and Fuchun Peng
  Our major finding is that after standard finetuning, the model forgets some of the important language generation skills acquired during large-scale pretraining. We propose an intuitive finetuning strategy named “mix-review”: : For each finetuning epoch, we mix the target dialogue data with a random subset of the pretraining data, mix_ratio is 4, decay is 0.9.
  
  
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters,
  by Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang et al.
  We propose KADAPTER, a framework that retains the original parameters of the pre-trained model fixed
and supports the development of versatile
knowledge-infused model.
  
  
• Domain-Lifelong Learning for Dialogue State Tracking via Knowledge Preservation Networks,
  by Liu, Qingbin , Cao, Pengfei , Liu, Cao , Chen, Jiansong , Cai, Xunliang , Yang, Fan , He, Shizhu , Liu, Kang et al.
  This paper explores Domain-Lifelong Learning for Dialogue State Tracking, we propose Knowledge Preservation Network, which consists of multi-prototype enhanced retrospection and multi-strategy knowledge distillation, to solve the problems of expression diversity and combinatorial explosion in the DLL-DST task
  
  
• CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks,
  by Ke, Zixuan , Liu, Bing , Xu, Hu and Shu, Lei
  The key novelty is a contrastive continual learning method that enables both knowledge transfer across tasks and knowledge distillation from old tasks to the new task, which eliminates the need for task ids in testing.
  
  
• Lifelong Explainer for Lifelong Learners,
  by Situ, Xuelin , Maruf, Sameen , Zukerman, Ingrid , Paris, Cecile and Haffari, Gholamreza
  We propose a novel Lifelong Explanation approach that continuously trains a student explainer under the supervision of a teacher – an arbitrary explanation algorithm – on different tasks undertaken in LL. We also leverage the Experience Replay mechanism to prevent catastrophic forgetting in the student explainer.
  
  
• A Unified Speaker Adaptation Approach for ASR,
  by Yingzhu Zhao, Chongjia Ni, Cheung-Chi Leung, Shafiq R. Joty, Eng Siong Chng and Bin Ma
  Prefix-based user identifier, Continual ASR / Architecture Search / Network Pruning.
  
  
• Dynamic Language Models for Continuously Evolving Content,
  by Amba Hombaiah, Spurthi, Chen, Tao, Zhang, Mingyang, Bendersky, Michael and Najork, Marc
  
  
• Parameter-Efficient Transfer Learning with Diff Pruning,
  by Guo, Demi , Rush, Alexander and Kim, Yoon
  The approach learns a task-specific “diff” vector that extends the original pretrained parameters. As the number of tasks increases, diff pruning remains parameter-efficient, as it requires storing only a small diff vector for each task.
  
  
• Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction,
  by Cui, Li , Yang, Deqing , Yu, Jiaxin , Hu, Chengwei , Cheng, Jiayang , Yi, Jingjie and Xiao, Yanghua
  To fully utilize memorized samples, in this paper, we employ relation prototype to extract useful information of each relation.
  
  
• On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation,
  by He, Ruidan , Liu, Linlin , Ye, Hai , Tan, Qingyu , Ding, Bosheng , Cheng, Liying , Low, Jiawei , Bing, Lidong et al.
  we first show that adapter-based tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. Effectiveness: it tendsto outperform fine-tuning on both low-resource and cross-lingual tasks; 2 it demonstrates higher stability under different learning rates compared to fine-tuning.
  
  
• Rational LAMOL: A Rationale-based Lifelong Learning Framework,
  by Kanwatchara, Kasidis , Horsuwan, Thanapapas , Lertvittayakumjorn, Piyawat , Kijsirikul, Boonserm and Vateekul, Peerapon
  Rational LAMOL enhances LAMOL, a recent LL model, by applying critical freezing guided by human rationales. When the human rationales are not available, we propose exploiting unsupervised generated rationales as substitutions.
  
  
• Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution,
  by Garcia, Xavier , Constant, Noah , Parikh, Ankur and Firat, Orhan
  Introducing the catastrophic forgetting problem in incremental multi-language translation, and utilizing a vocabulary substitution manner to alleviate the above problem.
  
  
• Continual Learning for Text Classification with Information Disentanglement Based Regularization,
  by Huang, Yufan , Zhang, Yanzhe , Chen, Jiaao , Wang, Xuezhi and Yang, Diyi
  Proposing a regularization-based method for continual text classification, introducing the next sentence prediction and task id prediction as auxiliary tasks.
  
  
• Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System,
  by Xia, Congying , Yin, Wenpeng , Feng, Yihao and Yu, Philip
  Proposing a new setting and respective benchmark for few-shot incremental text classification, modeling continual text classification with text entailment.
  
  
• Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding,
  by Hua, Ting , Shen, Yilin , Zhao, Changsheng , Hsu, Yen-Chang and Jin, Hongxia
  Inspired by EWC and proposing a hyperparameter-free (Fisher information-based) sampling method for memory replay.
  
  
• Lifelong Knowledge-Enriched Social Event Representation Learning,
  by Vijayaraghavan, Prashanth and Roy, Deb
  Proposing a rehearsal-based method, i.e.,Domain-Representative Episodic Memory Replay (DR-EMR), for lifelong event representation with embedding alignment and external social commonsense knowledge.
  
  
• Lifelong Intent Detection via Multi-Strategy Rebalancing,
  by Qingbin Liu, Xiaoyan Yu, Shizhu He, Kang Liu and Jun Zhao
  We propose the lifelong intent detection task to handle continually emerging user intents. And, we propose multistrategy rebalancing to address multiple adverse effects caused by the data imbalance problem.
  
  
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting,
  by Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu and Xiangzhan Yu
  We propose a recall and learn mechanism, which adopts the idea of multi-task learning and jointly learns pretraining tasks and downstream tasks. Specifically, we introduce a Pretraining Simulation mechanism to recall the knowledge from pretraining tasks without data, and an Objective Shifting mechanism to focus the learning on downstream tasks gradually.
  
  
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning,
  by Zhaojiang Lin, Andrea Madotto and Pascale Fung
  Proposing an adapter-based method for continual learning in text generation. One of the insights is a frozen PLM can be well-applied in continual learning.
  
  
• An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training,
  by Arumae, Kristjan , Sun, Qing and Bhatia, Parminder
  We find that elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks.
  
  
• Visually Grounded Continual Learning of Compositional Phrases,
  by Jin, Xisen , Du, Junyi , Sadhu, Arka , Nevatia, Ram and Ren, Xiang
  A novel continual learning setting and a new benchmark for continual caption generation, evaluated with exiting rehearsal-based methods
  
  
• Incremental Event Detection via Knowledge Consolidation Networks,
  by Cao, Pengfei , Chen, Yubo , Zhao, Jun and Wang, Taifeng
  Proposing a hybrid continual learning method for event detection, combining experience replay and Knowledge Distillation, focusing on (1) semantic ambiguity in NLP and (2) data imbalance between memory and current task.
  
  
• A Multi-Task Incremental Learning Framework with Category Name Embedding for Aspect-Category Sentiment Analysis,
  by Dai, Zehui , Peng, Cheng , Chen, Huajie and Ding, Yadong
  Utilizing BERT for sentence and category encoding, preserving category encoding to prevent catastrophic forgetting.
  
  
• Efficient Meta Lifelong-Learning with Limited Memory,
  by Wang, Zirui , Mehta, Sanket Vaibhav , Poczos, Barnabas and Carbonell, Jaime
  A meta learning-enhanced version of MbPA (NeurIPS19), sharing the continual setting as well. Figure 1 is interesting.
  
  
• Lifelong Language Knowledge Distillation,
  by Chuang, Yung-Sung , Su, Shang-Yu and Chen, Yun-Nung
  Proposing a Knowledge Distillation-enhanced Method LLL based on LAMOL (ICLR 2020) model for continual learning, evaluated on text generation and text classification.
  
  
• Distill and Replay for Continual Language Learning,
  by Sun, Jingyuan , Wang, Shaonan , Zhang, Jiajun and Zong, Chengqing
  Proposing a distill and replay method (DnR) which follows the setting of LAMOL. As a distillation-based method, DnR also shows the ability in incrementally compressing the model size while still outperforming most of the baselines.
  
  
• ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding,
  by Sun, Yu, Wang, Shuohuan, Li, Yukun, Feng, Shikun, Tian, Hao, Wu, Hua and Wang, Haifeng
  In order to extract the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which incrementally builds pre-training tasks and then learn pre-trained models on these constructed tasks via continual multi-task learning.
  
  
• Episodic Memory in Lifelong Language Learning,
  by Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong and Dani Yogatama
  MbPA++. This paper proposes the use of memory (a fixed memory network) in life-long learning to prevent catastrophic forgetting by means of experience replay and local adaptation.
  
  

• On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex,
  by Terry Yue Zhuo, Zhuang Li, Yujin Huang, Yuan-Fang Li, Weiqing Wang, Gholamreza Haffari and Fatemeh Shiri
  
  
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,
  by Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith et al.
  
  
• ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design,
  by Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith and Douglas C. Schmidt
  
  
• Learning to Prompt for Continual Learning,
  by Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot et al.
  
  
• Do Prompt-Based Models Really Understand the Meaning of Their Prompts?,
  by Albert Webson and Ellie Pavlick
  
  
• Large Language Models Are Human-Level Prompt Engineers,
  by Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan and Jimmy Ba
  
  
• An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels,
  by Taylor Sorensen, Joshua Robinson, Christopher Michael Rytting, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda et al.
  
  
• Demystifying Prompts in Language Models via Perplexity Estimation,
  by Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith and Luke Zettlemoyer
  
  
• Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models,
  by Robert L. Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh and Sebastian Riedel
  
  
• Adversarial Soft Prompt Tuning for Cross-Domain Sentiment Analysis,
  by Hui Wu and Xiaodong Shi
  
  
• Fine-Grained Controllable Text Generation Using Non-Residual Prompting,
  by Fredrik Carlsson, Joey "Ohman, Fangyu Liu, Severine Verlinden, Joakim Nivre and Magnus Sahlgren
  
  
• MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators,
  by Zhixing Tan, Xiangwen Zhang, Shuo Wang and Yang Liu
  
  
• Noisy Channel Language Model Prompting for Few-Shot Text Classification,
  by Sewon Min, Mike Lewis, Hannaneh Hajishirzi and Luke Zettlemoyer
  
  
• SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer,
  by Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou' and Daniel Cer
  
  
• Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models,
  by Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen et al.
  
  
• Meta-Adapters: Parameter Efficient Few-shot Fine-tuning through Meta-Learning,
  by Trapit Bansal, Salaheddin Alzubi, Tong Wang, Jay-Yoon Lee and Andrew McCallum
  
  
• Sparse Structure Search for Delta Tuning,
  by Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu and Maosong Sun
  
  
• Ontology-enhanced Prompt-tuning for Few-shot Learning,
  by Hongbin Ye, Ningyu Zhang, Shumin Deng, Xiang Chen, Hui Chen, Feiyu Xiong, Xi Chen and Huajun Chen
  
  
• Pre-trained Language Models can be Fully Zero-Shot Learners,
  by Xuandong Zhao, Siqi Ouyang, Zhiguo Yu, Ming Wu and Lei Li
  
  
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models,
  by Denny Zhou, Nathanael Sch"arli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet et al.
  (1) 两阶段的prompt，第一阶段问题分解（通过in-context learning实现，context中包含了其他问题的分解示例），对于每个问题，分解出回答该问题需要先回答什么子问题；
(2) 在第二阶段中，从后往前依次解决子问题，同样通过in-context learing得到，每次LLM的回答会参与组成下一个问题的prompt。
  
  
• The unreliability of explanations in few-shot prompting for textual reasoning,
  by Ye, Xi and Durrett, Greg
  
  
• Ask Me Anything: A simple strategy for prompting language models,
  by Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel J. Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala et al.
  
  
• Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View,
  by Boxi Cao, Hongyu Lin, Xianpei Han, Fangchao Liu and Le Sun
  
  
• Reframing Instructional Prompts to GPTk's Language,
  by Daniel Khashabi, Chitta Baral, Yejin Choi and Hannaneh Hajishirzi
  
  
• Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?,
  by Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov and Luke Zettlemoyer
  
  
• Towards Unified Prompt Tuning for Few-shot Text Classification,
  by Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang et al.
  
  
• Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning,
  by Xiangyu Peng, Chen Xing, Prafulla Kumar Choubey, Chien-Sheng Wu and Caiming Xiong
  
  
• FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models,
  by Rakesh Chada and Pradeep Natarajan
  
  
• The Power of Scale for Parameter-Efficient Prompt Tuning,
  by Brian Lester, Rami Al-Rfou and Noah Constant
  
  
• Prefix-Tuning: Optimizing Continuous Prompts for Generation,
  by Xiang Lisa Li and Percy Liang
  
  
• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm,
  by Laria Reynolds and Kyle McDonell
  
  

• GPT-4 Technical Report,
  by OpenAI
  
  
• GPT-4 System Card,
  by OpenAI
  
  
• Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding,
  by Jianing Wang, Wenkang Huang, Minghui Qiu, Qiuhui Shi, Hongbin Wang, Xiang Li and Ming Gao
  
  
• VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding,
  by Dou Hu, Xiaolong Hou, Xiyang Du, Mengyuan Zhou, Lianxin Jiang, Yang Mo and Xiaofeng Shi
  
  
• Generating Training Data with Language Models: Towards Zero-Shot Language Understanding,
  by Yu Meng, Jiaxin Huang, Yu Zhang and Jiawei Han
  
  
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation,
  by Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang and Luo Si
  
  
• Unified Language Model Pre-training for Natural Language Understanding and Generation,
  by Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou et al.
  
  
• Improving language understanding by generative pre-training,
  by Radford, Alec, Narasimhan, Karthik, Salimans, Tim, Sutskever, Ilya and others
  
  

• Scaling Vision Transformers to 22 Billion Parameters,
  by Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron et al.
  
  
• PaLM-E: An Embodied Multimodal Language Model,
  by Driess, Danny, Xia, Fei, Sajjadi, Mehdi SM, Lynch, Corey, Chowdhery, Aakanksha, Ichter, Brian, Wahid, Ayzaan, Tompson, Jonathan et al.
  
  
• Learning Customized Visual Models with Retrieval-Augmented Knowledge,
  by Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee and Chunyuan Li
  
  
• Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models,
  by Wu, Chenfei, Yin, Shengming, Qi, Weizhen, Wang, Xiaodong, Tang, Zecheng and Duan, Nan
  
  
• Aligning Text-to-Image Models using Human Feedback,
  by Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh et al.
  
  
• CLIP-Event: Connecting Text and Images with Event Structures,
  by Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji et al.
  
  
• Are Visual-Linguistic Models Commonsense Knowledge Bases?,
  by Hsiu-Yu Yang and Carina Silberer
  
  
• Retrieval-Augmented Multimodal Language Modeling,
  by Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer et al.
  
  
• Contrastive Language-Image Pre-Training with Knowledge Graphs,
  by Xuran Pan, Tianzhu Ye, Dongchen Han, Shiji Song and Gao Huang
  
  
• CLSEBERT: Contrastive Learning for Syntax Enhanced Code Pre-Trained Model,
  by Xin Wang, Yasheng Wang, Pingyi Zhou, Fei Mi, Meng Xiao, Yadao Wang, Li Li, Xiao Liu et al.
  
  
• Less Is More: ClipBERT for Video-and-Language Learning via Sparse Sampling,
  by Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal and Jingjing Liu
  
  
• Transformer is All You Need: Multimodal Multitask Learning with a Unified Transformer,
  by Ronghang Hu and Amanpreet Singh
  
  
• Pre-training Graph Transformer with Multimodal Side Information for Recommendation,
  by Yong Liu, Susen Yang, Chenyi Lei, Guoxin Wang, Haihong Tang, Juyong Zhang, Aixin Sun and Chunyan Miao
  
  
• UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation,
  by Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen and Ming Zhou
  
  
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning,
  by Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng and Jingjing Liu
  
  
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision,
  by Hao Tan and Mohit Bansal
  
  
• Integrating Multimodal Information in Large Pretrained Transformers,
  by Wasifur Rahman, Md. Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency and Mohammed E. Hoque
  
  
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations,
  by Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei and Jifeng Dai
  
  
• VisualBERT: A Simple and Performant Baseline for Vision and Language,
  by Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh and Kai-Wei Chang
  
  
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks,
  by Jiasen Lu, Dhruv Batra, Devi Parikh and Stefan Lee
  
  
• VideoBERT: A Joint Model for Video and Language Representation Learning,
  by Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy and Cordelia Schmid
  
  

• GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models,
  by Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li and Kai-Wei Chang
  
  

• On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective,
  by Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang et al.
  
  
• Prompting GPT-3 To Be Reliable,
  by Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan L. Boyd-Graber and Lijuan Wang
  
  
• Plex: Towards Reliability using Pretrained Large Model Extensions,
  by Dustin Tran, Jeremiah Z. Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang et al.
  
  
• Revisiting the Calibration of Modern Neural Networks,
  by Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran and Mario Lucic
  
  
• Soft Calibration Objectives for Neural Networks,
  by Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C. Mozer and Becca Roelofs
  
  

• Pretrained Transformers Do not Always Improve Robustness,
  by Swaroop Mishra, Bhavdeep Singh Sachdeva and Chitta Baral
  
  
• Pretrained Transformers Improve Out-of-Distribution Robustness,
  by Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan and Dawn Song
  
  

• DiSTRICT: Dialogue State Tracking with Retriever Driven In-Context Tuning,
  by Praveen Venkateswaran, Evelyn Duesterwald and Vatche Isahagian
  
  
• Does GPT-3 Generate Empathetic Dialogues? A Novel In-Context Example Selection Method and Automatic Evaluation Metric for Empathetic Dialogue Generation,
  by Young-Jun Lee, Chae-Gyun Lim and Ho-Jin Choi
  
  
• Fusing Task-Oriented and Open-Domain Dialogues in Conversational Agents,
  by Tom Young, Frank Xing, Vlad Pandelea, Jinjie Ni and Erik Cambria
  
  
• GODEL: Large-Scale Pre-Training for Goal-Directed Dialog,
  by Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan et al.
  
  
• Mind the Knowledge Gap: A Survey of Knowledge-enhanced Dialogue Systems,
  by Sagi Shaier, Lawrence Hunter and Katharina Kann
  
  
• Dialogue State Tracking with a Language Model using Schema-Driven Prompting,
  by Chia-Hsuan Lee, Hao Cheng and Mari Ostendorf
  
  
• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems,
  by Andrea Madotto, Zhaojiang Lin, Genta Indra Winata and Pascale Fung
  
  
• Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems,
  by Derek Chen, Howard Chen, Yi Yang, Alexander Lin and Zhou Yu
  
  
• Fine-grained Post-training for Improving Retrieval-based Dialogue Systems,
  by Janghoon Han, Taesuk Hong, Byoungjae Kim, Youngjoong Ko and Jungyun Seo
  
  
• Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey,
  by Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, Vinay Adiga and Erik Cambria
  
  
• Slot Self-Attentive Dialogue State Tracking,
  by Fanghua Ye, Jarana Manotumruksa, Qiang Zhang, Shenghui Li and Emine Yilmaz
  
  
• Pretraining the Noisy Channel Model for Task-Oriented Dialogue,
  by Qi Liu, Lei Yu, Laura Rimell and Phil Blunsom
  
  
• UBAR: Towards Fully End-to-End Task-Oriented Dialog System with GPT-2,
  by Yunyi Yang, Yunhao Li and Xiaojun Quan
  
  
• End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2,
  by DongHoon Ham, Jeong-Gwan Lee, Youngsoo Jang and Kee-Eung Kim
  
  
• A Simple Language Model for Task-Oriented Dialogue,
  by Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz and Richard Socher
  
  

• Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System,
  by Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang and Jiawei Zhang
  
  
• A Survey on Knowledge Graph-Based Recommender Systems,
  by Qingyu Guo, Fuzhen Zhuang, Chuan Qin, Hengshu Zhu, Xing Xie, Hui Xiong and Qing He
  
  
• Are Graph Augmentations Necessary?: Simple Graph Contrastive Learning for Recommendation,
  by Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui and Quoc Viet Hung Nguyen
  
  
• Disentangled Representations Learning for Multi-Target Cross-Domain Recommendation,
  by Guo, Xiaobo, Li, Shaoshuai, Guo, Naicheng, Cao, Jiangxia, Liu, Xiaolei, Ma, Qiongxu, Gan, Runsheng and Zhao, Yunan
  
  
• Rethinking Reinforcement Learning for Recommendation: A Prompt Perspective,
  by Xin Xin, Tiago Pimentel, Alexandros Karatzoglou, Pengjie Ren, Konstantina Christakopoulou and Zhaochun Ren
  
  
• Advances and Challenges in Conversational Recommender Systems: A Survey,
  by Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke and Tat-Seng Chua
  
  
• Pre-training Graph Transformer with Multimodal Side Information for Recommendation,
  by Yong Liu, Susen Yang, Chenyi Lei, Guoxin Wang, Haihong Tang, Juyong Zhang, Aixin Sun and Chunyan Miao
  
  
• Towards Hands-Free Visual Dialog Interactive Recommendation,
  by Tong Yu, Yilin Shen and Hongxia Jin
  
  

• Word-Label Alignment for Event Detection: A New Perspective via Optimal Transport,
  by Amir Pouran Ben Veyseh and Thien Huu Nguyen
  
  
• Learning Cross-Task Dependencies for Joint Extraction of Entities, Events, Event Arguments, and Relations,
  by Minh Van Nguyen, Bonan Min, Franck Dernoncourt and Thien Nguyen
  
  
• CLIP-Event: Connecting Text and Images with Event Structures,
  by Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji et al.
  
  
• Augmenting Open-Domain Event Detection with Synthetic Data from GPT-2,
  by Amir Pouran Ben Veyseh, Minh Van Nguyen, Bonan Min and Thien Huu Nguyen
  
  
• SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup,
  by Rongzhi Zhang, Yue Yu and Chao Zhang
  
  
• Exploring Pre-trained Language Models for Event Extraction and Generation,
  by Sen Yang, Dawei Feng, Linbo Qiao, Zhigang Kan and Dongsheng Li
  
  

• Learning Cross-Task Dependencies for Joint Extraction of Entities, Events, Event Arguments, and Relations,
  by Minh Van Nguyen, Bonan Min, Franck Dernoncourt and Thien Nguyen
  
  
• Selecting Optimal Context Sentences for Event-Event Relation Extraction,
  by Hieu Man, Nghia Trung Ngo, Linh Ngo Van and Thien Huu Nguyen
  
  
• Multilingual SubEvent Relation Extraction: A Novel Dataset and Structure Induction Method,
  by Viet Dac Lai, Hieu Man, Linh Ngo Van, Franck Dernoncourt and Thien Nguyen
  
  
• Event Causality Identification via Derivative Prompt Joint Learning,
  by Shirong Shen, Heng Zhou, Tongtong Wu and Guilin Qi
  
  
• Salience-Aware Event Chain Modeling for Narrative Understanding,
  by Xiyang Zhang, Muhao Chen and Jonathan May
  
  
• Joint Constrained Learning for Event-Event Relation Extraction,
  by Haoyu Wang, Muhao Chen, Hongming Zhang and Dan Roth
  
  

• ChatAug: Leveraging ChatGPT for Text Data Augmentation,
  by Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu, Ninghao Liu et al.
  
  
• Combining Ensembles and Data Augmentation Can Harm Your Calibration,
  by Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W. Dusenberry, Jasper Snoek, Balaji Lakshminarayanan and Dustin Tran
  
  
• GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation,
  by Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee and Woo-Myoung Park
  
  
• SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup,
  by Rongzhi Zhang, Yue Yu and Chao Zhang
  
  

• Is GPT-3 a Good Data Annotator?,
  by Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq R. Joty and Boyang Li
  
  
• Want To Reduce Labeling Cost? GPT-3 Can Help,
  by Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu and Michael Zeng
  
  

• Zero-Shot Information Extraction via Chatting with ChatGPT,
  by Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu et al.
  
  
• Large language models are few-shot clinical information extractors,
  by Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim and David A. Sontag
  
  
• Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again,
  by Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun and Yu Su
  
  
• SPOT: Knowledge-Enhanced Language Representations for Information Extraction,
  by Jiacheng Li, Yannis Katsis, Tyler Baldwin, Ho-Cheol Kim, Andrew Bartko, Julian J. McAuley and Chun-Nan Hsu
  
  

• A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products,
  by Kesong Liu, Jianhui Jiang and Feifei Lyu
  
  
• Snapshot-Guided Domain Adaptation for ELECTRA,
  by Daixuan Cheng, Shaohan Huang, Jianfeng Liu, Yuefeng Zhan, Hao Sun, Furu Wei, Denvy Deng and Qi Zhang
  
  
• VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding,
  by Dou Hu, Xiaolong Hou, Xiyang Du, Mengyuan Zhou, Lianxin Jiang, Yang Mo and Xiaofeng Shi
  
  

• ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots,
  by Reham Omar, Omij Mangukiya, Panos Kalnis and Essam Mansour
  
  
• Evaluation of ChatGPT as a Question Answering System for Answering Complex Questions,
  by Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen and Guilin Qi
  
  
• Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries,
  by Xiao Liu, Shiyu Zhao, Kai Su, Yukuo Cen, Jiezhong Qiu, Mengdi Zhang, Wei Wu, Yuxiao Dong et al.
  
  
• Sequence-to-Sequence Knowledge Graph Completion and Question Answering,
  by Apoorv Saxena, Adrian Kochsiek and Rainer Gemulla
  
  
• RealTime QA: What's the Answer Right Now?,
  by Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir R. Radev, Noah A. Smith et al.
  
  
• Relation-aware Language-Graph Transformer for Question Answering,
  by Jinyoung Park, Hyeong Kyu Choi, Juyeon Ko, Hyeon-Jin Park, Ji-Hoon Kim, Jisu Jeong, Kyung-Min Kim and Hyunwoo J. Kim
  
  

• Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models,
  by Kung, Tiffany H, Cheatham, Morgan, Medenilla, Arielle, Sillos, Czarina, De Leon, Lorie, Elepa~no, Camille, Madriaga, Maria, Aggabao, Rimel et al.
  
  
• ChatGPT passing USMLE shines a spotlight on the flaws of medical education,
  by Mbakwe, Amarachi B, Lourentzou, Ismini, Celi, Leo Anthony, Mechanic, Oren J and Dagan, Alon
  
  
• BloombergGPT: A Large Language Model for Finance,
  by Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg et al.
  
  
• Towards the Generation of Musical Explanations with GPT-3,
  by Stephen James Krol, Maria Teresa Llano and Jon McCormack
  
  

• Meta-learning via Language Model In-context Tuning,
  by Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis and He He
  This paper proposes in-context tuning, which recasts task adaptation and prediction as a simple sequence prediction problem: to form the input sequence, concatenate the task instruction, labeled in-context examples, and the target input to predict; to meta train the model to learn from in-context examples, finetune a PLM to predict the target label given the input sequence on a collection of tasks (very similar to MetaICL). On LAMA and BinaryClfs, the proposed method outperforms MAML.
  
  
• MetaICL: Learning to Learn In Context,
  by Sewon Min, Mike Lewis, Luke Zettlemoyer and Hannaneh Hajishirzi
  MetaICL proposes a supervised meta-training framework to enable LMs to more effectively learn a new task in context. In MetaICL, each meta-training example includes several training examples from one task that will be presented together as a single sequence to the LM, and the prediction of the final example is used to calculate the loss.
  
  

• Conversation Regression Testing: A Design Technique for Prototyping Generalizable Prompt Strategies for Pre-trained Language Models,
  by J. D. Zamfirescu-Pereira, Bjoern Hartmann and Qian Yang
  
  
• Fine-tuning Pre-trained Language Models with Noise Stability Regularization,
  by Hang Hua, Xingjian Li, Dejing Dou, Cheng-Zhong Xu and Jiebo Luo
  
  
• Do Language Models Perform Generalizable Commonsense Inference?,
  by Peifeng Wang, Filip Ilievski, Muhao Chen and Xiang Ren
  
  

• Understanding Finetuning for Factual Knowledge Extraction from Language Models,
  by Mehran Kazemi, Sid Mittal and Deepak Ramachandran
  
  
• Can Language Models Serve as Temporal Knowledge Bases?,
  by Ruilin Zhao, Feng Zhao, Guandong Xu, Sixiao Zhang and Hai Jin
  
  
• Finding Structural Knowledge in Multimodal-BERT,
  by Victor Milewski, Miryam de Lhoneux and Marie-Francine Moens
  
  
• A Review on Language Models as Knowledge Bases,
  by Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona T. Diab and Marjan Ghazvininejad
  
  
• Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries,
  by Benjamin Heinzerling and Kentaro Inui
  
  
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts,
  by Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace and Sameer Singh
  
  
• Language Models as Knowledge Bases?,
  by Fabio Petroni, Tim Rockt"aschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu and Alexander H. Miller
  
  

• In-Context Retrieval-Augmented Language Models,
  by Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown and Yoav Shoham
  
  
• Learning Customized Visual Models with Retrieval-Augmented Knowledge,
  by Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee and Chunyuan Li
  
  
• REPLUG: Retrieval-Augmented Black-Box Language Models,
  by Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer and Wen-tau Yih
  
  
• Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning,
  by Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu et al.
  
  
• A Survey on Retrieval-Augmented Text Generation,
  by Huayang Li, Yixuan Su, Deng Cai, Yan Wang and Lemao Liu
  
  
• Retrieval-Augmented Multimodal Language Modeling,
  by Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer et al.
  
  
• Atlas: Few-shot learning with retrieval augmented language models,
  by Izacard, Gautier, Lewis, Patrick, Lomeli, Maria, Hosseini, Lucas, Petroni, Fabio, Schick, Timo, Dwivedi-Yu, Jane, Joulin, Armand et al.
  
  
• Training Language Models with Memory Augmentation,
  by Zexuan Zhong, Tao Lei and Danqi Chen
  
  
• Improving Language Models by Retrieving from Trillions of Tokens,
  by Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau et al.
  
  
• REALM: Retrieval-Augmented Language Model Pre-Training,
  by Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang
  
  
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,
  by Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K"uttler, Mike Lewis et al.
  
  

• Generating Sequences by Learning to Self-Correct,
  by Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi and Yejin Choi
  
  

• Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale,
  by Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff and Dan Roth
  
  
• Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations,
  by Swarnadeep Saha, Peter Hase, Nazneen Rajani and Mohit Bansal
  
  
• <img src=https://img.shields.io/badge/Findings_of_the_Association_for_Computational_Linguistics:_{ACL/IJCNLP} 2021,_Online_Event,_August_1--6,_2021-2021-blue alt="img" style="zoom:100%; vertical-align: middle" /> Prompting Contrastive Explanations for Commonsense Reasoning Tasks,
  by Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi and Luke Zettlemoyer
  
  

• Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning,
  by Gao, Jiahui, Pi, Renjie, Yong, LIN, Xu, Hang, Ye, Jiacheng, Wu, Zhiyong, ZHANG, WEIZHONG, Liang, Xiaodan et al.
  
  
• <img src=https://img.shields.io/badge/the_2022_Conference_on_Empirical_Methods_in_Natural Language_Processing,_{EMNLP}_2022,_Abu_Dhabi,_United_Arab_Emirates, December_7--11,_2022-2022-blue alt="img" style="zoom:100%; vertical-align: middle" /> ZeroGen: Efficient Zero-shot Learning via Dataset Generation,
  by Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu and Lingpeng Kong
  
  
• Generating Training Data with Language Models: Towards Zero-Shot Language Understanding,
  by Yu Meng, Jiaxin Huang, Yu Zhang and Jiawei Han
  
  

• Toward General Design Principles for Generative AI Applications,
  by Justin D. Weisz, Michael J. Muller, Jessica He and Stephanie Houde
  
  
• Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models,
  by Zijian Zhang, Zhou Zhao and Zhijie Lin
  
  
• Parsing as Pretraining,
  by David Vilares, Michalina Strzyz, Anders S\ogaard and Carlos G'omez-Rodr'\iguez
  
  
• Unsupervised Deep Learning via Affinity Diffusion,
  by Jiabo Huang, Qi Dong, Shaogang Gong and Xiatian Zhu
  
  
• HellaSwag: Can a Machine Really Finish Your Sentence?,
  by Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi and Yejin Choi
  
  
• Learning to detect unseen object classes by between-class attribute transfer,
  by Christoph H. Lampert, Hannes Nickisch and Stefan Harmeling
  
  

• Awesome-ChatGPT,
  ChatGPT资料汇总学习，持续更新......

• Awesome ChatGPT Prompts,
  In this repository, you will find a variety of prompts that can be used with ChatGPT.

• ChatRWKV,
  ChatRWKV is like ChatGPT but powered by my RWKV (100% RNN) language model, which is the only RNN (as of now) that can match transformers in quality and scaling, while being faster and saves VRAM. Training sponsored by Stability EleutherAI.

• ChatGPT-Hub,
  ChatGPT资源汇总

• PaLM-rlhf-pytorch,
  Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture.

• BAAI-WuDao/Data,
  “悟道”项目构建了高质量的数据集，用于支撑大模型的训练和测评工作，本仓库提供所有开源数据集的链接。

• Colossal-AI,
  Colossal-AI provides a collection of parallel components for you. We aim to support you to write your distributed deep learning models just like how you write your model on your laptop. We provide user-friendly tools to kickstart distributed training and inference in a few lines.

• Exploring Prompt Injection Attacks,
  by Jose Selvi
  Prompt Injection is a new vulnerability that is affecting some AI/ML models and, in particular, certain types of language models using prompt-based learning.

• ChatGPT发展历程、原理、技术架构详解和产业未来,
  by 陈巍
  本文将介绍ChatGPT的特点、功能、技术架构、局限、产业应用、投资机会和未来。作者本人曾担任华为系自然语言处理（ NLP ）企业的首席科学家。

• How does GPT Obtain its Ability?,
  by Yao Fu
  Tracing emergent abilities of language models to their sources.

• Open source solution replicates ChatGPT training process,
  Colossal-AI, as one of the hottest open-source solutions for large AI models, presents an open-source low-cost ChatGPT equivalent implementation process.

• CPM-Bee,
  CPM-Bee是一个开源的双语预训练语言模型，参数量为10B，拥有十余种原生能力和强大的通用语言能力，并支持结构化输入和输出。

• 久谦：ChatGPT纪要分享,

• 国泰君安ChatGPT研究框架,

• 哈尔滨工业大学：ChatGPT调研报告,

• Chain of Thought Prompting for Large Language Model Reasoning,

• Awesome-ChatGPT,
  ChatGPT资料汇总学习，持续更新......

• Awesome ChatGPT Prompts,
  In this repository, you will find a variety of prompts that can be used with ChatGPT.

• ChatRWKV,
  ChatRWKV is like ChatGPT but powered by my RWKV (100% RNN) language model, which is the only RNN (as of now) that can match transformers in quality and scaling, while being faster and saves VRAM. Training sponsored by Stability EleutherAI.

• ChatGPT-Hub,
  ChatGPT资源汇总

• PaLM-rlhf-pytorch,
  Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture.

• BAAI-WuDao/Data,
  “悟道”项目构建了高质量的数据集，用于支撑大模型的训练和测评工作，本仓库提供所有开源数据集的链接。

• Colossal-AI,
  Colossal-AI provides a collection of parallel components for you. We aim to support you to write your distributed deep learning models just like how you write your model on your laptop. We provide user-friendly tools to kickstart distributed training and inference in a few lines.

• Exploring Prompt Injection Attacks,
  by Jose Selvi
  Prompt Injection is a new vulnerability that is affecting some AI/ML models and, in particular, certain types of language models using prompt-based learning.

• ChatGPT发展历程、原理、技术架构详解和产业未来,
  by 陈巍
  本文将介绍ChatGPT的特点、功能、技术架构、局限、产业应用、投资机会和未来。作者本人曾担任华为系自然语言处理（ NLP ）企业的首席科学家。

• How does GPT Obtain its Ability?,
  by Yao Fu
  Tracing emergent abilities of language models to their sources.

• Open source solution replicates ChatGPT training process,
  Colossal-AI, as one of the hottest open-source solutions for large AI models, presents an open-source low-cost ChatGPT equivalent implementation process.

• CPM-Bee,
  CPM-Bee是一个开源的双语预训练语言模型，参数量为10B，拥有十余种原生能力和强大的通用语言能力，并支持结构化输入和输出。

• 久谦：ChatGPT纪要分享,

• 国泰君安ChatGPT研究框架,

• 哈尔滨工业大学：ChatGPT调研报告,

• Chain of Thought Prompting for Large Language Model Reasoning,

Knowledge Science and Engineering Lab is recruiting researchers! You are welcome to apply for the following positions:

• Research Assistant: Bachelor degree or above, proficient in Python/Java, familiar with machine learning espicially deep learning models.
• Postdoctoral Fellow: Doctoral research in Artificial Intelligence, published at least 3 high-quality papers.
• Lecturer, Associate Professor and Professor

If you are interested in our research and meet the above requirements, feel free to contact Prof. Guilin Qi.

知识科学与工程实验室正在招聘科研人员！欢迎申请以下岗位：

• 科研助理：本科学历以上，精通Python/Java，熟悉机器学习，特别是深度学习模型。
• 博士后：博士研究人工智能相关方向，发表至少3篇高水平论文。
• 讲师、副教授、教授等教职

如果您对我们的研究工作感兴趣并满足以上要求，欢迎您与漆桂林教授联系。