This is the official github repository for Towards Continual Knowledge Learning of Language Models, accepted at ICLR 2022.

Use the following to cite our paper:

@inproceedings{jang2022towards,
  title={Towards Continual Knowledge Learning of Language Models},
  author={Jang, Joel and Ye, Seonghyeon and Yang, Sohee and Shin, Joongbo and Han, Janghoon and Kim, Gyeonghun and Choi, Stanley Jungkyu and Seo, Minjoon},
  booktitle={ICLR},
  year={2022}
}

In order to reproduce our results, take the following steps:

conda create -n ckl python=3.8 && conda activate ckl
pip install -r requirements.txt

Also, make sure to install the correct version of pytorch corresponding to the CUDA version and environment: Refer to https://pytorch.org/

#For CUDA 10.x
pip3 install torch torchvision torchaudio
#For CUDA 11.x
pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

To download only the CKL benchmark dataset:

wget https://continual.blob.core.windows.net/ckl/ckl_data.zip

To download ALL of the data used for the experiments (required to reproduce results):

wget https://continual.blob.core.windows.net/ckl/data.zip

To download the (continually pretrained) model checkpoints of the main experiment (required to reproduce results):

wget https://continual.blob.core.windows.net/ckl/modelcheckpoints_main.zip

For the other experimental settings such as multiple CKL phases, GPT-2, we do not separately provide the continually pretrained model checkpoints.

We provide all the configs in order to reproduce the zero-shot results of our paper. We only provide the model checkpoints for the main experimental setting (full_setting) which can be downloaded with the command above.

configs
├── full_setting
│   ├── evaluation
│   |   ├── invariantLAMA
│   |   |   ├── t5_baseline.json
│   |   |   ├── t5_kadapters.json
│   |   |   ├── ...
│   |   ├── newLAMA
│   |   ├── newLAMA_easy
│   |   ├── updatedLAMA
│   ├── training
│   |   ├── t5_baseline.json
│   |   ├── t5_kadapters.json
│   |   ├── ...
├── GPT2
│   ├── ...
├── kilt
│   ├── ...
├── small_setting
│   ├── ...
├── split
│   ├── ...                    

• input_length (int) : the input sequence length
• output_length (int) : the output sequence length
• num_train_epochs (int) : number of training epochs
• output_dir (string) : the directory to save the model checkpoints
• dataset (string) : the dataset to perform zero-shot evaluation or continual pretraining
• dataset_version (string) : the version of the dataset ['full', 'small', 'debug']
• train_batch_size (int) : batch size used for training
• learning rate (float) : learning rate used for training
• model (string) : model name in huggingface models (https://huggingface.co/models)
• method (string) : method being used ['baseline', 'kadapter', 'lora', 'mixreview', 'modular_small', 'recadam']
• freeze_level (int) : how much of the model to freeze during traininig (0 for none, 1 for freezing only encoder, 2 for freezing all of the parameters)
• gradient_accumulation_steps (int) : gradient accumulation used to match the global training batch of each method
• ngpu (int) : number of gpus used for the run
• num_workers (int) : number of workers for the Dataloader
• resume_from_checkpoint (string) : null by default. directory to model checkpoint if resuming from checkpoint
• accelerator (string) : 'ddp' by default. the pytorch lightning accelerator to be used.
• use_deepspeed (bool) : false by default. Currently not extensively tested.
• CUDA_VISIBLE_DEVICES (string) : gpu devices that are made available for this run (e.g. "0,1,2,3", "0")
• wandb_log (bool) : whether to log experiment through wandb
• wandb_project (string) : project name of wandb
• wandb_run_name (string) : the name of this training run
• mode (string) : 'pretrain' for all configs
• use_lr_scheduling (bool) : true if using learning rate scheduling
• check_validation (bool) : true for evaluation (no training)
• checkpoint_path (string) : path to the model checkpoint that is used for evaluation
• output_log (string) : directory to log evaluation results to
• split_num (int) : default is 1. more than 1 if there are multile CKL phases
• split (int) : which CKL phase it is

This is an example of getting the invariantLAMA zero-shot evaluation of continually pretrained t5_kadapters

python run.py --config configs/full_setting/evaluation/invariantLAMA/t5_kadapters.json

This is an example of performing continual pretraining on CC-RecentNews (main experiment) with t5_kadapters

python run.py --config configs/full_setting/training/t5_kadapters.json

@article{jang2021towards,
  title={Towards Continual Knowledge Learning of Language Models},
  author={Jang, Joel and Ye, Seonghyeon and Yang, Sohee and Shin, Joongbo and Han, Janghoon and Kim, Gyeonghun and Choi, Stanley Jungkyu and Seo, Minjoon},
  journal={arXiv preprint arXiv:2110.03215},
  year={2021}
}