Tutorial for pretrain RoBERTa with custom data about cramming HOT 2 CLOSED

Hi, it's hard to write out all the steps without knowing where you're stuck. Did you have a look at https://github.com/JonasGeiping/cramming#data-handling?

Basically the steps are as follows for a custom dataset:

1. Create a new .yaml file in config/data, you can copy from one of the existing ones
2. Fill out the sources list (The first argument). You can either create a dataset from the sources already implemented (see config/data/sources), or add new sources yourself. The easiest sources to add are huggingface datasets (see https://github.com/JonasGeiping/cramming/blob/main/cramming/config/data/sources/bookcorpus.yaml for example)
3. Fill out all the other information, choosing normalization, tokenizer type etc.
4. Run preprocessing by starting a pretraining run. The run will check your base folder and prepare the dataset according to the config if it cannot be found. Depending on your settings and dataset size, this might require larger amounts of RAM, which you can control by setting impl.threads to a smaller number and impl.max_raw_chunk_size to a smaller number.

from cramming.

Hope this helps. Closing this for now.

from cramming.