Migrating to HuggingFace's Transformers about spanbert HOT 5 CLOSED

When I tried to load it using huggingface's AutoModel in your way, it gave me the same model as Bert Model. Are you able to get different weights from Bert?

from spanbert.

Hi Pedram. It seems like this is being caused by a change in torch/transformers API. Perhaps it might help to run the original code and figure out the dimensions?

from spanbert.

Hi Pedram. It seems like this is being caused by a change in torch/transformers API. Perhaps it might help to run the original code and figure out the dimensions?

Thanks for your response. Even running the original code results in the following error:

Traceback (most recent call last):
  File "code/run_tacred.py", line 571, in <module>
    main(args)
  File "code/run_tacred.py", line 512, in main
    model = BertForSequenceClassification.from_pretrained(args.output_dir, num_labels=num_labels)
  File "/Users/phosseini/PycharmProjects/SpanBERT/code/pytorch_pretrained_bert/modeling.py", line 654, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
	size mismatch for classifier.weight: copying a param with shape torch.Size([42, 1024]) from checkpoint, the shape in current model is torch.Size([3, 1024]).
	size mismatch for classifier.bias: copying a param with shape torch.Size([42]) from checkpoint, the shape in current model is torch.Size([3]).

This is the script I'm running:

import os

os.system("python code/run_tacred.py \
  --do_eval \
  --data_dir \"./data/tacred/\" \
  --model spanbert-base-cased \
  --train_batch_size 32 \
  --eval_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 5 \
  --max_seq_length 128 \
  --output_dir \"./models/tacred/\"")

I just simply created a dev and test set with 110 samples each in TACRED format and I'm running the original code in evaluation mode just to see if the code is running, but I still get the error. Any thoughts?

from spanbert.

I could resolve the issue I just mentioned by adding model = nn.DataParallel(model) before load(model, prefix=start_prefix) in the modeling.py class. Now I could run the original code, but still trying to figure out the first problem.

from spanbert.

I finally got a chance to write a transformers compatible version of run_tacred.py. You can find it here: https://github.com/phosseini/SpanBERT/blob/main/code/run_crest.py

I haven't tested it with fp16 yet. Since I have made other changes to my forked repository, I am not able to create a pull request for now, but meanwhile, I thought someone might find this code helpful.

from spanbert.