victorwz / longmem Goto Github PK

I am currently working with the repository, and I am facing difficulties in locating the scripts required for evaluating the model's performance on the SQuAD (Stanford Question Answering Dataset) task. It seems that the evaluation scripts for SQuAD are not present in the repository.

I kindly request that the repository maintainers consider adding or updating the repository with the necessary evaluation scripts for the SQuAD task.

Tx

Links in the article https://aka.ms/LongMem and on the arxive page lead to this empty repo.
Congratulations with your 'publication'!

Hi,
I followed the steps of the repository (set up the fairseq included and downloaded the checkpoints). Then, as instructed, I ran:

python eval_scripts/eval_longmem_icl.py --path checkpoints/gpt2_medium/checkpoint_last.pt

Which leads to the following error:

Traceback (most recent call last):
  File "/scratch/gpfs/ab4197/LongMem/eval_scripts/eval_longmem_icl.py", line 168, in <module>
    main(args)
  File "/scratch/gpfs/ab4197/LongMem/eval_scripts/eval_longmem_icl.py", line 99, in main
    model, _ = load_model_ensemble([args.path], arg_overrides=override_args, task=None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/gpfs/ab4197/anaconda3/.conda/envs/ab/lib/python3.11/site-packages/fairseq/checkpoint_utils.py", line 374, in load_model_ensemble
    ensemble, args, _task = load_model_ensemble_and_task(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/gpfs/ab4197/anaconda3/.conda/envs/ab/lib/python3.11/site-packages/fairseq/checkpoint_utils.py", line 443, in load_model_ensemble_and_task
    task = tasks.setup_task(cfg.task, from_checkpoint=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/gpfs/ab4197/anaconda3/.conda/envs/ab/lib/python3.11/site-packages/fairseq/tasks/__init__.py", line 44, in setup_task
    task is not None
AssertionError: Could not infer task type from {'_name': 'gpt_pretraining', 'data': '/mnt/msranlp/shaohanh/data/tnlg_config/', 'sample_break_mode': 'none', 'tokens_per_sample': 1024, 'output_dictionary_size': -1, 'self_target': False, 'future_target': False, 'past_target': False, 'add_bos_token': False, 'max_target_positions': None, 'shorten_method': 'none', 'shorten_data_split_list': '', 'pad_to_fixed_length': False, 'pad_to_fixed_bsz': False, 'gpt2_encoder_json': 'gpt2_bpe/encoder.json', 'gpt2_vocab_bpe': 'gpt2_bpe/vocab.bpe', 'required_batch_size_multiple': 1, 'batch_read_ahead': 5000, 'mlm_cut_length': 128, 'mlm_tokens_proportion': 0.23, 'gpt_dict_path': 'gpt2_bpe/dict.txt', 'seed': 1, 'batch_size': 4, 'batch_size_valid': 4, 'data_buffer_size': 10, 'tpu': False, 'use_plasma_view': False, 'plasma_path': '/tmp/plasma'}. Available argparse tasks: dict_keys(['hubert_pretraining', 'speech_unit_modeling', 'translation', 'multilingual_translation', 'translation_lev', 'translation_from_pretrained_bart', 'audio_pretraining', 'nlu_finetuning', 'denoising', 'speech_to_text', 'text_to_speech', 'frm_text_to_speech', 'translation_multi_simple_epoch', 'simul_speech_to_text', 'simul_text_to_text', 'span_masked_lm', 'sentence_ranking', 'audio_finetuning', 'language_modeling', 'speech_dlm_task', 'cross_lingual_lm', 'translation_from_pretrained_xlm', 'masked_lm', 'speech_to_speech', 'audio_classification', 'semisupervised_translation', 'sentence_prediction', 'sentence_prediction_adapters', 'online_backtranslation', 'multilingual_denoising', 'legacy_masked_lm', 'multilingual_language_modeling', 'multilingual_masked_lm', 'dummy_lm', 'dummy_masked_lm', 'dummy_mt']). Available hydra tasks: dict_keys(['hubert_pretraining', 'speech_unit_modeling', 'translation', 'translation_lev', 'audio_pretraining', 'nlu_finetuning', 'denoising', 'simul_text_to_text', 'span_masked_lm', 'audio_finetuning', 'language_modeling', 'speech_dlm_task', 'translation_from_pretrained_xlm', 'masked_lm', 'audio_classification', 'sentence_prediction', 'sentence_prediction_adapters', 'multilingual_denoising', 'multilingual_language_modeling', 'dummy_lm', 'dummy_masked_lm'])

It seems like the checkpoint should include a "task" key but doesn't - how may I fix this?

Can you give me a demo about how to train model

Hi, thank you for the awesome work!

However, when I tried to run some experiments with your code, I noticed that the Pile dataset is no longer available on their website. I'm wondering if you know any alternative methods to get this dataset.

Thank you for your help!

I got the following error trace when I run the evaluation with this
python eval_scripts/eval_longmem_icl.py --path ../LongMem_public_checkpoints/longmem_gpt2_medium/checkpoint_last.pt --pretrained-model-path ../LongMem_public_checkpoints/gpt2_medium/checkpoint_last.pt

Traceback (most recent call last): File "eval_scripts/eval_longmem_icl.py", line 171, in <module> main(args) File "eval_scripts/eval_longmem_icl.py", line 102, in main model, _ = load_model_ensemble([args.path], arg_overrides=override_args, task=None) File "/home/ubuntu/longmem/LongMem/fairseq/fairseq/checkpoint_utils.py", line 368, in load_model_ensemble ensemble, args, _task = load_model_ensemble_and_task( File "/home/ubuntu/longmem/LongMem/fairseq/fairseq/checkpoint_utils.py", line 443, in load_model_ensemble_and_task task = tasks.setup_task(cfg.task) File "/home/ubuntu/longmem/LongMem/fairseq/fairseq/tasks/__init__.py", line 49, in setup_task return task.setup_task(cfg, **kwargs) File "/home/ubuntu/longmem/LongMem/fairseq/fairseq/tasks/language_modeling.py", line 183, in setup_task dictionary, output_dictionary = cls.setup_dictionary(args, **kwargs) File "/home/ubuntu/longmem/LongMem/fairseq/fairseq/tasks/language_modeling.py", line 167, in setup_dictionary dictionary = Dictionary.load(os.path.join(paths[0], "dict.txt")) File "/home/ubuntu/longmem/LongMem/fairseq/fairseq/data/dictionary.py", line 226, in load d.add_from_file(f) File "/home/ubuntu/longmem/LongMem/fairseq/fairseq/data/dictionary.py", line 239, in add_from_file raise fnfe File "/home/ubuntu/longmem/LongMem/fairseq/fairseq/data/dictionary.py", line 236, in add_from_file with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd: FileNotFoundError: [Errno 2] No such file or directory: '/mnt/multimodal/data/text/data-bin/tnlg/dict.txt'

Thank you for adding more scripts.

In other words, am I wrong to say that this will bring long context lengths to the various existing OSS models?

(Exciting work either way, congrats!)

Hello, thank you for the work on the paper.

I notice in the revised GPT-2 model there is the use of scaled rotary (XPos), here and here, but the paper makes no mention of scaled rotary (or rotary) being applied or tested. I am wondering if an ablation was performed but not included in the paper?

Trying to load model from bigscience/bloom-1b7 result in:

Load Pre-trained GPT from bigscience/bloom-1b7
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/LongMem/fairseq/fairseq/tasks/language_modeling.py", line 202, in build_model
    model = super().build_model(args, from_checkpoint)
  File "/root/LongMem/fairseq/fairseq/tasks/fairseq_task.py", line 688, in build_model
    model = models.build_model(args, self, from_checkpoint)
  File "/root/LongMem/fairseq/fairseq/models/__init__.py", line 106, in build_model
    return model.build_model(cfg, task)
  File "/root/LongMem/fairseq/fairseq/models/transformer_lm_sidenet.py", line 409, in build_model
    decoder = TransformerDecoderSideNet(
  File "/root/LongMem/fairseq/fairseq/models/sidenet/transformer_decoder_sidenet.py", line 606, in __init__
    super().__init__(
  File "/root/LongMem/fairseq/fairseq/models/sidenet/transformer_decoder_sidenet.py", line 197, in __init__
    self.pretrained_model, _ = load_model_ensemble([self.pretrained_model_path], task=None, arg_overrides={"gpt2_vocab_bpe": os.path.join(cfg.gpt_encoder_path, "vocab.bpe"), "gpt2_encoder_json": os.path.join(cfg.gpt_encoder_path, "encoder.json"), "gpt_dict_path": os.path.join(cfg.gpt_encoder_path, "dict.txt"), "retrieval_layer_index": cfg.retrieval_layer_index})
  File "/root/LongMem/fairseq/fairseq/checkpoint_utils.py", line 367, in load_model_ensemble
    ensemble, args, _task = load_model_ensemble_and_task(
  File "/root/LongMem/fairseq/fairseq/checkpoint_utils.py", line 425, in load_model_ensemble_and_task
    state = load_checkpoint_to_cpu(filename, arg_overrides)
  File "/root/LongMem/fairseq/fairseq/checkpoint_utils.py", line 315, in load_checkpoint_to_cpu
    state = torch.load(f, map_location=torch.device("cpu"))
  File "/root/anaconda3/envs/longmem/lib/python3.8/site-packages/torch/serialization.py", line 815, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/root/anaconda3/envs/longmem/lib/python3.8/site-packages/torch/serialization.py", line 1033, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\xd2'.

Is the model different from model from huggingface?

I was trying to reproduce the results in your paper by running language modeling with newgpt class, where the codes failed to execute due to the error about new_batches in fairseq/fairseq/data/iterators.py. Is this function correct?

UnboundLocalError: local variable 'new_batches' referenced before assignment

And can you provide a demo/script on how to run the codes in order to reproduce your results? Thank you in advance! :)

Greetings! I have some questions regarding memory storage and retrieval in the provided code. Based on your code, it appears that you have stored the per-head embeddings of all chunks in the batch and subsequently retrieved the chunk embeddings for queries in the batch.

However, the paper states that each sample in the same batch originates from a different document. Thus, I find it puzzling that all chunks in the batch are mixed together and then retrieved. I have been unable to locate any specific code for reorganizing the retrieved chunks based on their original documents. As far as I understand, this reorganization may not be possible with faiss. Could you kindly provide some clarification on this matter? Thank you.

Hi!

Thanks for the great work. It seems the options.py is not right, and some important hyper-parameters (such as find_unused_parameters and memory_size) are not received by the script correctly. Could you please debug this?

Thanks!

I made a pile dataset, but how to divide the valid dataset. For my self-made validation set, I always get the error "Is a directory: '/home/mdz/pywork/LongMem/pile_preprocessed_binary/valid'

In your paper, the training dataset for memory adaptive learning includes 26B tokens, but I can't find the method for how to build the training dataset for adaptive learning.

Besides, the link of dataset can't download from its url: https://the-eye.eu/public/AI/pile/.
can you help me solve above problems. Thanks for you.

When running the code, an AttributeError occurs with the message "'NoneType' object has no attribute 'get_smoothed_values'". This issue prevents the code from executing successfully.

:) code is missing.

Hi, LongMem is an awesome work. I have the following questions

1. are you going to release the scripts you used to collect the training and evaluation dataset or the dataset itself?
2. are you going to release the a trained checkpoints?

Thanks!

sorry, i try to evaluate your trained model directly, but received this error:

Traceback (most recent call last):
File "eval_scripts/eval_longmem_icl.py", line 168, in
main(args)
File "eval_scripts/eval_longmem_icl.py", line 99, in main
model, _ = load_model_ensemble([args.path], arg_overrides=override_args, task=None)
File "/data/cwan38/trial/LongMem/fairseq/fairseq/checkpoint_utils.py", line 367, in load_model_ensemble
ensemble, args, _task = load_model_ensemble_and_task(
File "/data/cwan38/trial/LongMem/fairseq/fairseq/checkpoint_utils.py", line 425, in load_model_ensemble_and_task
state = load_checkpoint_to_cpu(filename, arg_overrides)
File "/data/cwan38/trial/LongMem/fairseq/fairseq/checkpoint_utils.py", line 315, in load_checkpoint_to_cpu
state = torch.load(f, map_location=torch.device("cpu"))
File "/data/cwan38/anaconda3/envs/longmem/lib/python3.8/site-packages/torch/serialization.py", line 797, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "/data/cwan38/anaconda3/envs/longmem/lib/python3.8/site-packages/torch/serialization.py", line 283, in init
super().init(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: Expected hasRecord("version") to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)