Working with larger datasets about diffuseq HOT 2 OPEN

It`s still not working properly. But I think that has something to do with padding and my sequence lengths. I have to investigate that further, but thank you for your help! :)
I found a small thing that accelerated the data loading time a lot:

here a line is loaded twice. By reading it once and then accessing the src and trg I saved a lot of time.

with open(path, 'r') as f_reader: 
     for row in f_reader: 
         line = json.loads(row)
         sentence_lst['src'].append(line['src'].strip()) 
         sentence_lst['trg'].append(line['trg'].strip())

from diffuseq.

Hi,
Maybe you can try to add keep_in_memory = True in function raw_datasets.map

If it doesn't work, you can try to split your datasets into separate folds and load them respectively in different training steps.

from diffuseq.