Comments (5)
The finetuned large models can be downloaded from here. Replace {task} with one of {squad1, squad2, tacred}
http://dl.fbaipublicfiles.com/fairseq/models/spanbert_{task}.tar.gz
I'll look into the other things you mentioned this week.
from spanbert.
@svjan5 I might not be 100% correct - From what I can tell, a 0.8 drop caused by removing --fp16
is not too surprising.
from spanbert.
Quick update:
I downloaded and evaluated the fine-tuned model you provided and it works perfectly!
I also noticed that, by copying the configuration provided in the README, I've used spanbert-base
as the base model instead of spanbert-large
. After retraining with the large model I get results very close to the paper scores.
Thanks again for providing the fine-tuned models.
from spanbert.
Hi,
I am unable to reproduce results on TACRED even with spanbert_tacred
model.
Firstly, I ran the following command for training the model:
setgpu 1; python code/run_tacred.py --do_eval --data_dir ~/tacred/data/json/ --model spanbert-large-tacred --train_batch_size 32 --eval_batch_size 32 --learning_rate 2e-5 --num_train_epochs 10 --max_seq_length 128 --output_dir tacred_dir --do_train
Then, to get performance on test split. I ran:
setgpu 1; python code/run_tacred.py --do_eval --data_dir ~/tacred/data/json/ --model spanbert-large-tacred --train_batch_size 32 --eval_batch_size 32 --learning_rate 2e-5 --num_train_epochs 10 --max_seq_length 128 --output_dir tacred_dir --eval_test
The performance on test is as follows:
***** Eval results ***** (Test)
accuracy = 0.8775549680830486
eval_loss = 0.50735904131968
f1 = 0.699196667658435
precision = 0.691786870768325
recall = 0.706766917293233
The obtained F1 score is around 0.8 less than what is reported. The only thing I changed is removing --fp16
argument because the code gives an error with that as reported above. @ChristophAlt solution does not work in my case. Can removing --fp16
lead to a reduction in performance or am I missing something else.
Thanks in advance!
from spanbert.
Thanks @danqi! with --fp16
I got around 70.33 which is reasonably close. If possible please give some idea how including --fp16
is changing performance.
from spanbert.
Related Issues (20)
- What 's in train.txt and valid.txt these two files during preprocessing HOT 1
- Getting embeddings for spans
- How should i use train.py to load a SpanBERT model and continue pretraining it? HOT 2
- Could you provide uncased SpanBert? HOT 3
- spanBERT for Named Entity Recognition HOT 3
- About "Tokenization Step"
- About the pre-training train-test ratio
- How can i use the SpanBert to predict a span of tokens in some specific contex?
- continue pretraining HOT 1
- continue pretraining HOT 1
- TypeError: __new__() got multiple values for argument 'cls' HOT 3
- how to train spanBert NER ?
- How to continually pretrain SpanBERT starting from a model checkpoint? HOT 1
- pre-train model load failture
- Use existing pretrained model for retraining using SpanBERT
- MRQA datasets split
- Progress in Pretraining
- How to train model by changing the number of layers in the model?
- pretrained model
- Use spanbert model for Named Entity Recognition to predict on data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spanbert.