Comments (5)
I recently tried it as follows.
- Prepare the SpanBERT model and its config.
Take model.pt
from https://dl.fbaipublicfiles.com/fairseq/models/spanbert_large_with_head.tar.gz (#44) and config.json
from https://dl.fbaipublicfiles.com/fairseq/models/spanbert_hf.tar.gz, and put them in a directory (e.g. spanbert_large_with_head
).
Then, convert model.pt
into pytorch_model.bin
using the following code so that it can be used from huggingface/transformers.
from collections import OrderedDict
import torch
def convert(input_path, output_path):
state_dict = OrderedDict()
for k, v in torch.load(input_path).items():
if k.startswith('decoder.'):
state_dict[k[8:]] = v
torch.save(state_dict, output_path)
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('input_path')
parser.add_argument('output_path')
args = parser.parse_args()
convert(args.input_path, args.output_path)
- Load SpanBERT using huggingface/transformers
from transformers import BertTokenizer, BertForMaskedLM
tokenizer = BertTokenizer.from_pretrained('bert-large-cased')
model = BertForMaskedLM.from_pretrained('./spanbert_large_with_head/')
seq = "Super Bowl 50 was {m} {m} {m} {m} to determine the champion".format(m=tokenizer.mask_token)
inputs = tokenizer(seq, return_tensors="pt")
outputs = model(**inputs).logits
print(tokenizer.decode([y if x == tokenizer.mask_token_id else x for x, y
in zip(inputs.input_ids[0], outputs.argmax(dim=2)[0])]))
In my env with transformers==4.1.1, this produced [CLS] Super Bowl 50 was the first football game to determine the champion [SEP]
.
The Hugging Face tutorial (https://huggingface.co/transformers/task_summary.html#masked-language-modeling) is also helpful.
I hope this will work for you.
from spanbert.
@chantera I'm not sure why, but this was my result: "[CLS] Super Bowl 50 was Trilogy trailers Singers 231 to determine the champion [SEP]" Do you have any idea as to why?
from spanbert.
Hi ! have you got the base mode with head?
from spanbert.
No, I haven't -- I don't think it has been published anywhere.
from spanbert.
This is indeed helpful. Another thing to figure out is using the masking strategy of SpanBERT. Have you figured that out too? SpanBERT is different than the original BERT in two main aspects: 1) MLM prediction, the SBO head, 2) masking spans. The SBO head is available here, but I'm trying to figure out how to add the masking strategy too
from spanbert.
Related Issues (20)
- What 's in train.txt and valid.txt these two files during preprocessing HOT 1
- Getting embeddings for spans
- How should i use train.py to load a SpanBERT model and continue pretraining it? HOT 2
- Could you provide uncased SpanBert? HOT 3
- spanBERT for Named Entity Recognition HOT 3
- About "Tokenization Step"
- About the pre-training train-test ratio
- How can i use the SpanBert to predict a span of tokens in some specific contex?
- continue pretraining HOT 1
- continue pretraining HOT 1
- TypeError: __new__() got multiple values for argument 'cls' HOT 3
- how to train spanBert NER ?
- How to continually pretrain SpanBERT starting from a model checkpoint? HOT 1
- pre-train model load failture
- Use existing pretrained model for retraining using SpanBERT
- MRQA datasets split
- Progress in Pretraining
- How to train model by changing the number of layers in the model?
- pretrained model
- Use spanbert model for Named Entity Recognition to predict on data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spanbert.