Index Out of Range Error in tokenization using TF Hub for Pretrained Albert Models about albert HOT 9 CLOSED

'30k-clean.model' is 'spm_model_file', not 'vocab_file'.
The code changes are:

diff --git a/albert/run_classifier_with_tfhub.py b/albert/run_classifier_with_tfhub.py
index 92fef74..26f4339 100644                                                         
--- a/albert/run_classifier_with_tfhub.py                                             
+++ b/albert/run_classifier_with_tfhub.py                                             
@@ -156,6 +156,7 @@ def create_tokenizer_from_hub_module(albert_hub_module_handle):   
     with tf.Session() as sess:                                                       
       vocab_file, do_lower_case = sess.run([tokenization_info["vocab_file"],         
                                             tokenization_info["do_lower_case"]])     
+    FLAGS.spm_model_file = vocab_file                                                
   return tokenization.FullTokenizer(                                                 
       vocab_file=vocab_file, do_lower_case=do_lower_case,                            
       spm_model_file=FLAGS.spm_model_file)                                           

from albert.

I'm not sure what the python script is doing but the vocabulary is not simply a text file. It's a SentencePiece model file (which is basically a protobuf-serialized file) and must be deserialized using either protobuf directly or the SentencePiece python API.

from albert.

Upgrade your tensorflow to at least 1.15.

from albert.

I am getting the same errors on all the available TF hub models.

from albert.

Seems like this happens because of '\n' lines in vocab file, similar case described here. But it is not clear how to handle this, maybe the files are corrupted?

from albert.

Thanks @eaplatanios @KodairaTomonori .
I have resolved Index out of Range error but I got another error like below.

LookupError: No gradient defined for operation 'module_apply_tokens/bert/encoder/transformer/group_0_23/layer_23/inner_group_0/LayerNorm_1/batchnorm/add_1' (op type: AddV2)

Do you have any ideas? Thanks!

from albert.

Thanks @eaplatanios @KodairaTomonori .
I have resolved Index out of Range error but I got another error like below.

LookupError: No gradient defined for operation 'module_apply_tokens/bert/encoder/transformer/group_0_23/layer_23/inner_group_0/LayerNorm_1/batchnorm/add_1' (op type: AddV2)

Do you have any ideas? Thanks!

@YujiOshima Hi, I am still facing the same issue, can you help me with how did you resolve the issue?

from albert.

Thanks @eaplatanios @KodairaTomonori .
I have resolved Index out of Range error but I got another error like below.

LookupError: No gradient defined for operation 'module_apply_tokens/bert/encoder/transformer/group_0_23/layer_23/inner_group_0/LayerNorm_1/batchnorm/add_1' (op type: AddV2)

Do you have any ideas? Thanks!

@YujiOshima Hi, I am still facing the same issue, can you help me with how did you resolve the issue?

For version one, if you use the latest tensorflow 1 it'll fix both problems. However, I have not found a sensible solution for version 2 yet.

from albert.

We have fixed the issue with the v2 ALBERT modules on TF-Hub
We also updated the README.md with some details about --spm_model_file
Please let us know if you have other issues!

from albert.