pohanchi / albert-tf Goto Github PK

I am facing issue with tokenization while using this repo. Please help to come out of this issue.

cd ALBert-tf
/content/ALBert-tf

import tokenization

tokenizer=tokenization.FullTokenizer("/content/30k-cleanM.vocab")
tc=tokenizer.tokenize("Hello, my dog is cute")
ec=tokenizer.convert_tokens_to_ids(tc)

logs

KeyError Traceback (most recent call last)
in ()
1 tokenizer=tokenization.FullTokenizer("/content/30k-cleanM.vocab")
2 tc=tokenizer.tokenize("Hello, my dog is cute")
----> 3 ec=tokenizer.convert_tokens_to_ids(tc)

1 frames
/content/ALBERT/tokenization.py in convert_by_vocab(vocab, items)
209 output = []
210 for item in items:
--> 211 output.append(vocab[item])
212 return output
213

KeyError: '[UNK]'