Comments (1)
I have worked out how to do this from code, once I realized I needed to save the Bert tokenizer. Now I have to explore the tokenizer from the bert model vs the Spago one. But I assume that as the vocab is loaded, it will work as advertised.
If I can use this package, then I will start to contribute, but in case anyone else looks for this, you need to do the following in Python code if you are using Torch lightning and a HuggingFace BERT model (and indeed any trained bert model).
You will have a class that inherits from pytorch_lightning.LightningModule
and in your init method, you will be doing something like this:
import pytorch_lightning as pl
class MyClassifier(pl.LightningModule)
def __init__(self, n_classes=5, steps_per_epoch=None, n_epochs=3, lr=2e-5):
super().__init__()
self.bert = BertModel.from_pretrained(BERT_MODEL_NAME, return_dict=True)
# etc ...
When you are lightning training, you will have almost certainly have set up checkpoints, something like this:
checkpoint_callback = ModelCheckpoint(
monitor='val_loss', # monitored quantity
filename='tags-{epoch:02d}-{val_loss:.2f}',
save_top_k=3, # save the top 3 models
mode='min', # monitored optimization quantity
)
After training, you get the best model:
model_path = checkpoint_callback.best_model_path
And you will have used the tokenizer from the pre-trained model:
tokenizer = BertTokenizer.from_pretrained(BERT_MODEL_NAME)
Now you want to save the model and the vocab etc from your best trained model:
classy = MyClassifier.load_from_checkpoint(model_path)
classy.eval()
tokenizer.save_pretrained('jimdir')
# Note that we will overwrite the .bin that is saved next - should really just
# get the config and save that to json directly as that is all we are keeping.
# Also note, that you may need to modify the config yourself, such as add id2label and
# label2id if you have a classifier.
#
QTmodel.bert.save_pretrained('/Users/jim/.gsl/model')
# Make sure to save the fine tuned state and not the pre-trained state
#
torch.save(MyModel.state_dict(), 'jimdir/pytorch_model.bin')
Now, to do the conversion for Spago from this saved pre-trained model, in go code, you can do this (no error handling or params etc, but that's easy):
package bert
import (
"github.com/nlpodyssey/spago/pkg/nlp/transformers/bert"
)
func ConvertJim() {
bert.ConvertHuggingFacePreTrained("/path/to/jimdir")
}
You should see:
Serializing model to "/path/to/jimdir/spago_model.bin"... ok
BERT has been converted successfully!
With my model, I get some warnings, such as:
2021/06/29 12:17:17 WARNING!! `cls.predictions.transform.LayerNorm.weight` not initialized
2021/06/29 12:17:17 WARNING!! `classifier.bias` not initialized
But that has nothing to do with this conversion process.
from spago.
Related Issues (20)
- Multi-label BERT classifier from PyTorch HOT 3
- bart-large-mnli multi_class does not agree with Python version HOT 7
- Suitable for categorising intents and slots? HOT 2
- Differences in the output of zero shot classification between python & spago for the same model HOT 1
- Nearly 4 times the memory usage when compared to python for the same model HOT 3
- Gorgonia tensors
- QA Chinese model result does not match python version HOT 2
- v1.0.0-alpha.0: operator gradients are sometimes null in non-obvious ways
- The `deepset` dir must created by manually HOT 4
- flair models no longer work HOT 2
- How to train a lstm network with spago? HOT 1
- how to convert hf models to spago format? HOT 2
- Comparing python and go gives different embeddings HOT 4
- I fucked up HOT 1
- is:issue is:open Issue with bert compiling Q6A example HOT 2
- Moved methods HOT 2
- Detailed Instructions? HOT 2
- Examples and documentation
- Question server example should use http not https
- Help with German zero shot HOT 16
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spago.