akanyaani / gpt-2-tensorflow2.0 Goto Github PK

OpenAI GPT2 pre-training and sequence prediction implementation in Tensorflow 2.0

Home Page: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

License: MIT License

Python 76.73% Jupyter Notebook 23.27%

gpt-2 implementation gpt2 pretraining nlp transformer tensorflow2 tensorflow gpt openai

gpt-2-tensorflow2.0's Introduction

Hi 👋, I'm Abhay

A passionate Data Scientist, who loves training neural networks.

gpt-2-tensorflow2.0's People

Contributors

Stargazers

Watchers

Forkers

shivamvaibhav1466 hiyoung-asr qianrenjian aly-shmahell kangzhonghua lingsond sqkika vinnytwice wrapperband hundred06 awesome-archive trendingtechnology robotoil legendtianjin dlreseach dshwei kinred vmishra-uu 210010 tonysun9 andrewy12 nkotak aksanyaani trisongz vftens daabaa opendlf fedehr aartoul hemaoliang ravehun yunghans deepesp ksjae kmuellerjones dochoncho jzl0166 andjoer lmasiero manas-embold oxmah ck624 worldwidekatie hannahec1 josephpaulgiroux chytonpide tommasopetrolito sr3u shungjhon malinphy diaosidev wkoury googol-lab ipiyushsonar liaojz slash-under chease3640 dlperf tkahn c0ntradicti0n linzilongsaic sax0928 thisisanshgupta yuyouling ken2190 arkzv wesley-yang kimmorehvn peijiangkuang chiarawahsono valdex dersilberwulf xidchen dzzxjl viandmarket25 cicyer qcxyzff flyforfreedom david8862 uraltaf fenago kalsp manlinl chriss-0x01

gpt-2-tensorflow2.0's Issues

How can i train a model with language with not english symbols?

can not covert to pytorch model by using transformers

export OPENAI_GPT2_CHECKPOINT_PATH=./model/ckpt-301
transformers-cli convert --model_type gpt2 \

--tf_checkpoint $OPENAI_GPT2_CHECKPOINT_PATH
--pytorch_dump_output ./pytorch_dump_output/pytorch_model.bin
--config ./model/model_par.json \

INFO:transformers.modeling_gpt2:Converting TensorFlow checkpoint from /home/mycat/gpt-2-tensorflow2.0/model/ckpt-301
INFO:transformers.modeling_gpt2:Loading TF weight _CHECKPOINTABLE_OBJECT_GRAPH with shape []
Traceback (most recent call last):
File "/home/mycat/.local/bin/transformers-cli", line 11, in
load_entry_point('transformers', 'console_scripts', 'transformers-cli')()
File "/home/mycat/transformers/src/transformers/commands/transformers_cli.py", line 33, in main
service.run()
File "/home/mycat/transformers/src/transformers/commands/convert.py", line 134, in run
convert_gpt2_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
File "/home/mycat/transformers/src/transformers/convert_gpt2_original_tf_checkpoint_to_pytorch.py", line 38, in convert_gpt2_checkpoint_to_pytorch
load_tf_weights_in_gpt2(model, config, gpt2_checkpoint_path)
File "/home/mycat/transformers/src/transformers/modeling_gpt2.py", line 75, in load_tf_weights_in_gpt2
arrays.append(array.squeeze())
AttributeError: 'bytes' object has no attribute 'squeeze'

Probably some attributes are missing in GPT2 model.

why not epoch ? how to stop train model

hi, there is not epoch or max_train_steps in code ? how to stop training model?

sg.sample_sequence returns context after pre-trained model

First ofd all, thanks for providing this amazing repository providing a possibility for tf2!
Secondly, I were using the Readme to pre-train my model and eventually using sequence_generator.py to pass some context to the model.

However, the response is always 1:1 the same as the context but the capital letters are being replaced with ??s. The question now is, what am I doing wrong? Have I maybe forgotten a thing? Is there maybe a edge case leading to this point that could be prevented?

Please let me know any additional information you might need! Thanks a lot!

IndexError: Out of range: piece id is out of range.

When running sample_sequence, I get this error:

IndexError: Out of range: piece id is out of range.

Any idea on what exactly I am doing wrong?

pre processing and training went fine

Error in PredictCost() for the op: "Softmax"

I am running this code on my 3090 with tensorflow-gpu 2.5.0 and cuda 11.2

I can successfully run the training, but I get the error as follows:

Can someone give me a hint to solve this?

RuntimeError: Internal: /sentencepiece/src/trainer_interface.cc(336) [!sentences_.empty()]

Hi,
Fisrt thanks for your work.

When I am trying to do preprocessing. I get following error message:
RuntimeError: Internal: /sentencepiece/src/trainer_interface.cc(336) [!sentences_.empty()]

I am using a *.txt file uploaded on my colab.
I would like to know what does it mean and how to fix it.
Thanks

Vincent

Performance issues in data_pipeline.py(P2)

Hello,I found a performance issue in the definition of input_fn ,
data_pipeline.py,
dataset = dataset.map(parse_example) was called without num_parallel_calls.
I think it will increase the efficiency of your program if you add this.

Here is the documemtation of tensorflow to support this thing.

Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.

Error while training

Latest checkpoint restored...............
Running in graph mode.............
Traceback (most recent call last):
  File "train_gpt2.py", line 77, in <module>
    train()
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "train_gpt2.py", line 72, in train
    model.fit([train_dataset, test_dataset], graph_mode)
  File "/home/madrajib/workspace/projects/master_project/gpt-2-tensorflow2.0/gpt2_model.py", line 282, in fit
    step, loss, perplexity = train_func(inputs, targets)
  File "/home/madrajib/.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/home/madrajib/.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 840, in _call
    return self._stateless_fn(*args, **kwds)
  File "/home/madrajib/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/home/madrajib/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1843, in _filtered_call
    return self._call_flat(
  File "/home/madrajib/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1923, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/home/madrajib/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 545, in call
    outputs = execute.execute(
  File "/home/madrajib/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError:  indices[2,1] = 31868 is not in [0, 24512)
	 [[node gpt2/embeddings/embedding_layer/embedding/embedding_lookup (defined at /home/madrajib/workspace/projects/master_project/gpt-2-tensorflow2.0/layers/embedding_layer.py:41) ]] [Op:__inference_train_step_6798]

Errors may have originated from an input operation.
Input Source operations connected to node gpt2/embeddings/embedding_layer/embedding/embedding_lookup:
 Inputs (defined at /home/madrajib/workspace/projects/master_project/gpt-2-tensorflow2.0/gpt2_model.py:282)

Function call stack:
train_step

What is shared weight across layers?

Hi I found the TODO including: shared weight across layers

In which layers could this model share their weights?

Is Concatenation of Data Files Necessary?

I noticed that the first step of the pre-processing involves normalizing the text dataset and writing everything to one large txt file. Is it necessary to write everything to this txt file before training the tokenizer? My dataset is ~300GB; concatenating would take a very long time (unless there exists some way of concatenating files in parallel?).

Thoughts?

How can I load pretrained model from original OpenAI work?

Or, maybe, I need to take pretrained model somewhere else?

tensor mask shape may be different with tensor matmul_qk shape

Im reading this code, and wondering as follows
The tensor mask shape maybe ：[batch, 1, seq_len, seq_len]
by the code
`def get_padding_mask(seq):
with tf.name_scope("Padding_Mask"):
seq = tf.cast(tf.math.equal(seq, 0), tf.float32)

# add extra dimensions to add the padding
# to the attention logits.
# TODO by Ethan 2020-09-11, 周五, 23:1:here should be like this ? return seq[:, tf.newaxis, :]
return seq[:, tf.newaxis, tf.newaxis, :]  # (batch_size, 1, 1, seq_len)

def attention_mask(size):
"""
if size is 4 then it returns below matrix
[[0., 1., 1., 1.],
[0., 0., 1., 1.],
[0., 0., 0., 1.],
[0., 0., 0., 0.]]

"""
with tf.name_scope("attention_mask"):
mask = 1 - tf.linalg.band_part(tf.ones((size, size)), -1, 0)
return mask # (seq_len, seq_len)
def create_masks(inp):
with tf.name_scope("att_masking"):
att_mask = attention_mask(tf.shape(inp)[1])
padding_mask = get_padding_mask(inp)

TODO by Ethan 2020-09-11, 周五, 23:13: this shape will be [batch, 1, seq_len, seq_len]

mask = tf.maximum(padding_mask, att_mask)

return mask`

However, the tensor matmul_qk shape maybe ：[batch, seq_len_q, seq_len_k]
by the code
` def multihead_attention(self, q, k, v, training, mask=None):
matmul_qk = tf.matmul(q, k, transpose_b=True) # (..., seq_len_q, seq_len_k)
if self.scale:
dk = tf.cast(tf.shape(k)[-1], tf.float32)
matmul_qk = matmul_qk / tf.math.sqrt(dk)

if mask is not None:
    matmul_qk += (mask * -1e9)`

Im not sure. Somebody help me

Batch inference issue and left padding

Hi there,

Thanks for the project firstly. I saw a method called get_padding_mask in tf_utils.py file. It is combined with attention mask. It is designed to resolve the padding issue in batch inference or training only?

I see that with very a few padding in the left can still make good predictions/generations, but if use a large batch for inference, which cause a long left padding, the predictions get very incorrect. Do you have any suggestions for that?

Looking forward to your response. Thanks.

Mismatch in Arguments

In Line 180 of gpt2_model, we define the function as:
def distributed_train_step(self,inputs, targets,step,grad_clip=True, clip_value=1.0):

But In line 229, we instantiate it as:
train_loss = self.distributed_train_step(inputs, step)

The targets variable / argument is missing.

problem with training

Hi, I downloaded gpt-2-tensorflow2.0, so I could get gpt-2 working, but when I tried to install the requirements, I got this:

Requirement already satisfied: setuptools==41.0.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 1)) (41.0.1)
Collecting ftfy==5.6
  Using cached ftfy-5.6.tar.gz (58 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 14, in <module>
        File "/usr/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 20, in <module>
          from setuptools.dist import Distribution, Feature
        File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 34, in <module>
          from setuptools.depends import Require
        File "/usr/local/lib/python3.10/dist-packages/setuptools/depends.py", line 7, in <module>
          from .py33compat import Bytecode
        File "/usr/local/lib/python3.10/dist-packages/setuptools/py33compat.py", line 55, in <module>
          unescape = getattr(html, 'unescape', html_parser.HTMLParser().unescape)
      AttributeError: 'HTMLParser' object has no attribute 'unescape'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

, so I installed the requirements manually, and some still gave an error, but then I just installed the newest version, but when I tried to train the model, it gave me this:

2023-09-16 21:01:25.812726: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-09-16 21:01:25.812751: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
No. of tf records:-  2
2023-09-16 21:01:27.331253: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-16 21:01:27.331492: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-09-16 21:01:27.331561: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory
2023-09-16 21:01:27.331623: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory
2023-09-16 21:01:27.331684: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2023-09-16 21:01:27.331761: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2023-09-16 21:01:27.331820: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory
2023-09-16 21:01:27.331910: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory
2023-09-16 21:01:27.331978: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2023-09-16 21:01:27.331992: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Latest checkpoint restored...............
Running in eager mode.............
Traceback (most recent call last):
  File "/home/umikali/gpt-2/gpt-2-tensorflow2.0/train_gpt2.py", line 77, in <module>
    train()
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/umikali/gpt-2/gpt-2-tensorflow2.0/train_gpt2.py", line 72, in train
    model.fit([train_dataset, test_dataset], graph_mode)
  File "/home/umikali/gpt-2/gpt-2-tensorflow2.0/gpt2_model.py", line 282, in fit
    step, loss, perplexity = train_func(inputs, targets)
  File "/home/umikali/gpt-2/gpt-2-tensorflow2.0/gpt2_model.py", line 173, in _train_step
    predictions, _ = self(inputs, training=True)
  File "/usr/local/lib/python3.10/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/umikali/gpt-2/gpt-2-tensorflow2.0/gpt2_model.py", line 82, in call
    embedded_x = self.embedding(x)
  File "/home/umikali/gpt-2/gpt-2-tensorflow2.0/layers/embedding_layer.py", line 30, in call
    return self.embedding(inputs, scale=scale)
  File "/home/umikali/gpt-2/gpt-2-tensorflow2.0/layers/embedding_layer.py", line 41, in embedding
    embeddings = tf.nn.embedding_lookup(self.embedding_weights, inputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Exception encountered when calling layer "embedding_layer" (type EmbeddingLayer).

indices[14,4] = 31890 is not in [0, 24512) [Op:ResourceGather]

Call arguments received by layer "embedding_layer" (type EmbeddingLayer):
  • inputs=tf.Tensor(shape=(32, 241), dtype=int32)
  • mode=embedding
  • scale=False

, I tried doing it again i a virtual environment with sudo, but nothing worked. I haven't yet tried in a docker container tho.
EDIT: I tried in a container and it still didn't work

No module named sentencepiece

Why was WSWS.org taken out of the gpt-2 dataset, but Breitbart, fox etc was left in?

Just thought it was interesting lol. Its a source which is included in all the other GPT-2 repos at #500 and ranks highly, is authoritative, etc. It should be included before something like ahem Breitbart -- using GPT-2 to automatically create and disseminate fake news propaganda against "antifa" -- coronavirus / 5G conspiracy theories and such tends to be frowned upon. At least the wsws as a news report and editorial does not lie.

The Following Error is Generated

DESKTOP-NSQNAPV:/ai/gpt-2-tensorflow2.0$ pip3 install -r requirements.txt
Collecting setuptools==41.0.1
Using cached setuptools-41.0.1-py2.py3-none-any.whl (575 kB)
Collecting ftfy==5.6
Using cached ftfy-5.6.tar.gz (58 kB)
Collecting tqdm==4.32.1
Using cached tqdm-4.32.1-py2.py3-none-any.whl (49 kB)
Requirement already satisfied: Click==7.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 4)) (7.0)
Collecting sentencepiece==0.1.83
Using cached sentencepiece-0.1.83.tar.gz (497 kB)
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zv127a6c/sentencepiece/setup.py'"'"'; file='"'"'/tmp/pip-install-zv127a6c/sentencepiece/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-zv127a6c/sentencepiece/pip-egg-info
cwd: /tmp/pip-install-zv127a6c/sentencepiece/
Complete output (7 lines):
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-zv127a6c/sentencepiece/setup.py", line 29, in
with codecs.open(os.path.join('..', 'VERSION'), 'r', 'utf-8') as f:
File "/usr/lib/python3.8/codecs.py", line 905, in open
file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: '../VERSION'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
DESKTOP-NSQNAPV:/ai/gpt-2-tensorflow2.0$

Why Train_Accuracy is pretty low(about 0.2) ?

I follow the "README.md" trained the model on sample data

First: python pre_process.py
Second: python train_gpt2.py --num-layers=8 --embedding-size=768 --batch-size=32

Then, the training beigins, here us the Loss and Accurancy during training

eprecated and will be removed after 2020-07-01.
Instructions for updating:
`tf.python.eager.profiler` has deprecated, use `tf.profiler` instead.
Saving checkpoint for step 0 at xxx/GPT_2/GPT_tf/TF2/gpt-2-tensorflow2.0/model/ckpt-1
Step 10 Train_Loss 7.2324 Train_Accuracy 0.0832
Step 20 Train_Loss 6.5299 Train_Accuracy 0.1730
Step 30 Train_Loss 6.4850 Train_Accuracy 0.1768
Step 40 Train_Loss 6.1244 Train_Accuracy 0.1932
Step 50 Train_Loss 6.3007 Train_Accuracy 0.1790
Step 60 Train_Loss 6.3144 Train_Accuracy 0.1865
Step 70 Train_Loss 6.1924 Train_Accuracy 0.1648
Step 80 Train_Loss 6.2282 Train_Accuracy 0.1759
Step 90 Train_Loss 6.2466 Train_Accuracy 0.1744
Step 100 Train_Loss 6.1871 Train_Accuracy 0.1795
Step 110 Train_Loss 6.0732 Train_Accuracy 0.2064
Step 120 Train_Loss 5.7407 Train_Accuracy 0.2119
Step 130 Train_Loss 5.8436 Train_Accuracy 0.2077
Step 140 Train_Loss 5.7919 Train_Accuracy 0.1898
Step 150 Train_Loss 5.9080 Train_Accuracy 0.1661
Step 160 Train_Loss 5.8630 Train_Accuracy 0.1994
Step 170 Train_Loss 5.7625 Train_Accuracy 0.2076
---

Step 2740 Train_Loss 5.3913 Train_Accuracy 0.1958
Step 2750 Train_Loss 5.3359 Train_Accuracy 0.2195
Step 2760 Train_Loss 5.3394 Train_Accuracy 0.1973
Step 2770 Train_Loss 5.0865 Train_Accuracy 0.2501
Step 2780 Train_Loss 5.4709 Train_Accuracy 0.1929
Step 2790 Train_Loss 5.4672 Train_Accuracy 0.1946
Step 2800 Train_Loss 5.5116 Train_Accuracy 0.1962
Step 2810 Train_Loss 5.2981 Train_Accuracy 0.2346
Step 2820 Train_Loss 5.4803 Train_Accuracy 0.2078
Step 2830 Train_Loss 5.5752 Train_Accuracy 0.1869
Step 2840 Train_Loss 5.4528 Train_Accuracy 0.2158
Step 2850 Train_Loss 5.2158 Train_Accuracy 0.2377
Step 2860 Train_Loss 5.3771 Train_Accuracy 0.2202
Step 2870 Train_Loss 5.3635 Train_Accuracy 0.1965
Step 2880 Train_Loss 5.4944 Train_Accuracy 0.2296
Step 2890 Train_Loss 5.4714 Train_Accuracy 0.2068
Step 2900 Train_Loss 5.2218 Train_Accuracy 0.2330
Step 2910 Train_Loss 5.4696 Train_Accuracy 0.2070
Step 2920 Train_Loss 5.5928 Train_Accuracy 0.1947
Step 2930 Train_Loss 5.4761 Train_Accuracy 0.2173
Step 2940 Train_Loss 5.5963 Train_Accuracy 0.2022
Step 2950 Train_Loss 5.3133 Train_Accuracy 0.2197
Training Done................

python setup.py egg_info did not run successfully.

I'm a complete beginner in coding, so I'm lost in what to do.
I've been following the readme and during setup, when I write in anaconda prompt, pip install -r requirements.txt,
I get the error:

Collecting sentencepiece==0.1.83
  Using cached sentencepiece-0.1.83.tar.gz (497 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\ryyuj\AppData\Local\Temp\pip-install-1j97ilvg\sentencepiece_23e331750cc749e6b09f5118d946390c\setup.py", line 29, in <module>
          with codecs.open(os.path.join('..', 'VERSION'), 'r', 'utf-8') as f:
        File "C:\Users\ryyuj\anaconda3\lib\codecs.py", line 905, in open
          file = builtins.open(filename, mode, buffering)
      FileNotFoundError: [Errno 2] No such file or directory: '..\\VERSION'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Any ideas on what the problem might be, and what to do about it?
Thank you so much!

Extracting the graph

Since it's tf2, it's much harder for me to extract the forward graph (also the backward), can you help us with that?