I"m trying to pretrain Roberta Model with my own datasets and own tokenizer.
What I wonder is whether I can use my own WordPiece tokenized vocab when following fairseq pretraining guideline
In fact, I already encoded all my data using WordPiece tokenizer. So, according to the guideline, my next step is binarizing datasets for the language format and then training.
Unfortunately, when I pass my own dictionary.txt to dict.txt (which is the argument of the command line),
In my thought, this is because of the typing differences (such as [PAD]) between BPE and WordPiece.
18396, 18107, 4795, 19577, 5053, 18223, 6837, 18043, 6837, 4382, 22665, 18038, 1102, 1102, 24131
18312, 109, 21765, 4249, 22212, 5431, 31658, 20930, 4112, 6918, 4189, 6917, 9716, 6518, 9594, 18707, 17994, 7126, 23595, 9126, 18471, 6827, 7563, 9362, 29562, 112, 139, 21140, 9112, 117, 27432, 25164, 20476, 113, 25419, 4795, 30211, 18163, 18140, 8023, 4189, 25697, 4600, 4377, 18051, 6602, 27609, 9313, 18434, 19504, 8050, 23022, 17995, 7636, 22650, 25419, 5340, 18314, 19090, 4112, 4710, 4840, 118
18072, 8050, 5798, 7106, 6508, 6521, 18008, 18070, 6803, 4169, 4132, 17998
4594, 6803, 4913, 5340, 28949, 4795, 19185, 4795, 27101, 19179, 6602, 5467, 29407, 4979, 4795, 4894, 9282, 6837, 18017, 4377, 18051, 6837, 4840, 118
18045, 18202, 5053, 6602, 4795, 18717, 18401, 6506, 20548, 6061, 7103, 5471, 7103, 18357, 18037, 8059, 17997, 18202, 6070, 17994, 4795, 19255, 5723, 8026, 4135, 5467, 7258, 8125, 19802, 5000, 4135, 6686, 17999, 198, 19863, 6602, 6866, 22221, 18095, 6855, 6594, 18008, 5583, 9534, 18258, 6803, 18017, 6635, 17999, 198, 19863, 6870, 7271, 4230, 20124, 4446, 6709
Though my files is in proper directory, code can't detect it.
is this because I didn't binarize the dataset?