I could not work this out directly from the code (I have only just started working wit

Loading converting PyTorch/ Lightning models about spago HOT 1 CLOSED

nlpodyssey commented on May 18, 2024

Loading converting PyTorch/ Lightning models

from spago.

Comments (1)

jimidle commented on May 18, 2024

I have worked out how to do this from code, once I realized I needed to save the Bert tokenizer. Now I have to explore the tokenizer from the bert model vs the Spago one. But I assume that as the vocab is loaded, it will work as advertised.

If I can use this package, then I will start to contribute, but in case anyone else looks for this, you need to do the following in Python code if you are using Torch lightning and a HuggingFace BERT model (and indeed any trained bert model).

You will have a class that inherits from pytorch_lightning.LightningModule and in your init method, you will be doing something like this:

import pytorch_lightning as pl
class MyClassifier(pl.LightningModule)

    def __init__(self, n_classes=5, steps_per_epoch=None, n_epochs=3, lr=2e-5):
        super().__init__()

        self.bert = BertModel.from_pretrained(BERT_MODEL_NAME, return_dict=True)
       # etc ...

When you are lightning training, you will have almost certainly have set up checkpoints, something like this:

    checkpoint_callback = ModelCheckpoint(
        monitor='val_loss',  # monitored quantity
        filename='tags-{epoch:02d}-{val_loss:.2f}',
        save_top_k=3,  # save the top 3 models
        mode='min',  #  monitored optimization quantity
    )

After training, you get the best model:

        model_path = checkpoint_callback.best_model_path

And you will have used the tokenizer from the pre-trained model:

     tokenizer = BertTokenizer.from_pretrained(BERT_MODEL_NAME)

Now you want to save the model and the vocab etc from your best trained model:

    classy = MyClassifier.load_from_checkpoint(model_path)
    classy.eval()
    tokenizer.save_pretrained('jimdir')
    # Note that we will overwrite the .bin that is saved next - should really just
    # get the config and save that to json directly as that is all we are keeping.
    # Also note, that you may need to modify the config yourself, such as add id2label and
    # label2id if you have a classifier.
    #
    QTmodel.bert.save_pretrained('/Users/jim/.gsl/model')
    # Make sure to save the fine tuned state and not the pre-trained state
    #
    torch.save(MyModel.state_dict(), 'jimdir/pytorch_model.bin')

Now, to do the conversion for Spago from this saved pre-trained model, in go code, you can do this (no error handling or params etc, but that's easy):

package bert

import (
	"github.com/nlpodyssey/spago/pkg/nlp/transformers/bert"
)

func ConvertJim() {

    bert.ConvertHuggingFacePreTrained("/path/to/jimdir")
}

You should see:

Serializing model to "/path/to/jimdir/spago_model.bin"... ok
BERT has been converted successfully!

With my model, I get some warnings, such as:

2021/06/29 12:17:17 WARNING!! `cls.predictions.transform.LayerNorm.weight` not initialized
2021/06/29 12:17:17 WARNING!! `classifier.bias` not initialized

But that has nothing to do with this conversion process.

from spago.

Loading converting PyTorch/ Lightning models about spago HOT 1 CLOSED

Comments (1)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent