How to create vocab for other language? about albert_zh HOT 5 CLOSED

hi, we used vocabulary from bert, it is a single word for Chinese.

you can check bert or xlnet for reference.

from albert_zh.

I have seen bert's vocab. They used Wordpiece for it. Which generates subwords like ##word but in sentencepiece thats not the case. Any suggestions?

from albert_zh.

in xlnet, it is using sentence piece. you can easily generate vocabulary based on your corpus. check here: xlnet, and then you will have to take a look of tokenizer

you can also check non-english support from bert: https://github.com/google-research/bert/blob/master/multilingual.md

from albert_zh.

@008karan SentencePiece does quite the opposite to WordPiece. From the documentation. SentencePiece first escapes the whitespace with a meta-symbol "▁" (U+2581) as follows:

Hello▁World.

Then, this text is segmented into small pieces, for example:

[Hello] [▁Wor] [ld] [.]

Subwords which occur after whitespace (which are also those that most words begin with) are prepended with '▁', while others are unchanged. This excludes subwords which only occur at the beginning of sentences and nowhere else. These cases should be quite rare, however.

So, in order to obtain a vocabulary analogous to WordPiece, we need to perform a simple conversion, removing "▁" from the tokens that contain it and adding "##" to the ones that don't.

def parse_sentencepiece_token(token):
    if token.startswith("▁"):
        return token[1:]
    else:
        return "##" + token

from albert_zh.

good job

from albert_zh.