Broken encoding of vocab.json about ru-gpts HOT 5 CLOSED

That's something that weirded me off, too. If you try to train it, you may also look at features file which contains lots of broken encoding. I've mentioned this in my issue, too. Seems like it's somewhat readable for the generator, though? Are you sure you set your encoding for output to UTF-8?

from ru-gpts.

It could be readable for generator, but in this case there should be some preprocessing of my training data. If so, that would be strange.
As for encoding of the file, yeah, it's set to UTF-8 (and you can see it in the right bottom corner). By the way, there's a link to vocab.json above.
And yes, I might look at features file, but I'm not really familiar with this. So if you gave me some instructions, I'd try.

from ru-gpts.

I reviewed the source code and realized that I used custom tokenizer. Then I replaced it with tokenizer from transformers library and got my human-readable answers converted into the text similar to shown above. Now I'm testing whether it would work

from ru-gpts.

Yeah, I have no idea then. You might look at how data is preproccessed for GPT-3 in readme file, if you didn't preproccess your dataset. GPT-2 also needs dataset preproccessed, though it's not listed in the readme.
Seems like a really raw model implementation tbh.

from ru-gpts.

I've just tested GPT2Tokenizer from transformers, and now I see that the problem was in my custom tokenizer. So vocab.json seems to be ok, and it's encoding is all right

from ru-gpts.