Encoding/decoding NLP model in tensorflow lite (fine-tuned GPT2) about tflite-support HOT 7 CLOSED

Closing the issue. Feel free to reopen.

from tflite-support.

@lu-wang-g Hi Lu, it seems to be an issue related to GPT2Tokenizer, do you know which team maintains the GPT2Tokenizer?

from tflite-support.

You should be able to do step 2 by copying date from context_idx to input_data. Let me know what your problem is when doing so.

from tflite-support.

Dear @lu-wang-g ,

Many thanks for your answer.
When we try what you suggest, this is what we get:

ValueError                                Traceback (most recent call last)

<ipython-input-40-d8a95a34331a> in <module>()
     24 
     25 #3- feed input
---> 26 interpreter.set_tensor(input_details[0]['index'], input_data)
     27 
     28 #4- Run inference

/usr/local/lib/python3.7/dist-packages/tensorflow/lite/python/interpreter.py in set_tensor(self, tensor_index, value)
    570       ValueError: If the interpreter could not set the tensor.
    571     """
--> 572     self._interpreter.SetTensor(tensor_index, value)
    573 
    574   def resize_tensor_input(self, input_index, tensor_size, strict=False):

ValueError: Cannot set tensor: Dimension mismatch. Got 4 but expected 5 for dimension 1 of input 0.

So let me rephrase the questions we have now:

About the shapes of the input/output tensors:

I am not sure I understand what this 5 stands for in the input data shape printed. Should it not be the length of the sequence which may vary? Or is it a minimum length ?
Similarly, what are the output shape dimensions standing for ? Should not it be the size of the dictionary, and the length of the input fed?

About the input encoding:

Shall we encode the inputs in a specific way beforehand and feed encoder_outputs to the interpreter.set_tensors?
I saw that in hugging face library they first pass it to an encoder before feeding it to the model. Cf this link:
https://github.com/huggingface/transformers/blob/master/src/transformers/generation_tf_utils.py#L320
Is there a simple way to do this similarly in this context?
Is it a one-hot encoding we need simply?

About the output decoding:

Is the next_token_logits right as such ? I am not sure which part of output_data should be taken to be sampled (here, greedily).
In order to get the next token index. It seems it produce a string instead of one token currently.

You can find a colab with our process here:
https://colab.research.google.com/drive/1dPzO058qtS0VHO3BmrrJgGLxXjrH6CHd?usp=sharing

from tflite-support.

Dear @lu-wang-g ,

Many thanks for your answer.
When we try what you suggest, this is what we get:

ValueError                                Traceback (most recent call last)

<ipython-input-40-d8a95a34331a> in <module>()
     24 
     25 #3- feed input
---> 26 interpreter.set_tensor(input_details[0]['index'], input_data)
     27 
     28 #4- Run inference

/usr/local/lib/python3.7/dist-packages/tensorflow/lite/python/interpreter.py in set_tensor(self, tensor_index, value)
    570       ValueError: If the interpreter could not set the tensor.
    571     """
--> 572     self._interpreter.SetTensor(tensor_index, value)
    573 
    574   def resize_tensor_input(self, input_index, tensor_size, strict=False):

ValueError: Cannot set tensor: Dimension mismatch. Got 4 but expected 5 for dimension 1 of input 0.

So let me rephrase the questions we have now:

About the shapes of the input/output tensors:

I am not sure I understand what this 5 stands for in the input data shape printed. Should it not be the length of the sequence which may vary? Or is it a minimum length ?

According to the error message, the input shape is [batch, 5]. Your model doesn't seem to accept variable input length. You can verify it by printing out input_shape, where input_shape = input_details[0]['shape'], and see if each dimension has value other than -1. Normally for NLP models, it (5 in this case) means the maximum number of ids accepted. If your number of ids is less than 5, pad it with the PAD tokens in the vocabulary file. See our text classification app for example. Also see the input processing logic.

Similarly, what are the output shape dimensions standing for ? Should not it be the size of the dictionary, and the length of the input fed?

It depends on what your output tensor is. Seems like your model is a text-to-text model. So the output contains the ids of tokens that represent the output text. Similarly, if the output shape is [batch, N], N will be the maximum ids that can return by this model.

About the input encoding:

Shall we encode the inputs in a specific way beforehand and feed encoder_outputs to the interpreter.set_tensors?
I saw that in hugging face library they first pass it to an encoder before feeding it to the model. Cf this link:
https://github.com/huggingface/transformers/blob/master/src/transformers/generation_tf_utils.py#L320
Is there a simple way to do this similarly in this context?

Not that I'm aware of. What is the encoder in your link? Do you have pointers?

Is it a one-hot encoding we need simply?

Inference should use exactly the same encoder/decoder as the training script, and encoding varies from model to model.

About the output decoding:

Is the next_token_logits right as such ? I am not sure which part of output_data should be taken to be sampled (here, greedily).
In order to get the next token index. It seems it produce a string instead of one token currently.

You can find a colab with our process here:
https://colab.research.google.com/drive/1dPzO058qtS0VHO3BmrrJgGLxXjrH6CHd?usp=sharing

I'm out of context of what the output tensor is, i.e. what does each dimension in the shape (1, 5, 50257) referring to. It will be more helpful if you can provide information. In general, always refer to the training script for pre/post-processing. If you can point me to the training script, I can help to take a look as well.

from tflite-support.

Thanks @lu-wang-g, sorry I did not have time to look into it since last time.

from tflite-support.

@Guillaume-slize Hi, I know it is after a long time. Do you remember how you solved this issue with the output decoding?

from tflite-support.