i am using this code only for multilabel classification of text into 26 labels but getting only 38% hamming score and 28% flat_score.
please suggest some changes in my cod so that can achieve above 90%

0it [00:00, ?it/s]
Epoch: 0, Loss: 0.607304036617279
102it [01:51, 1.04s/it]
Epoch: 0, Loss: 0.42913737893104553
202it [03:40, 1.04s/it]
Epoch: 0, Loss: 0.31689316034317017
278it [05:05, 1.10s/it]
0it [00:00, ?it/s]
Epoch: 1, Loss: 0.25217771530151367
102it [01:52, 1.04s/it]
Epoch: 1, Loss: 0.2359163910150528
202it [03:41, 1.04s/it]
Epoch: 1, Loss: 0.20691940188407898
278it [05:06, 1.10s/it]
0it [00:00, ?it/s]
Epoch: 2, Loss: 0.18726885318756104
102it [01:52, 1.05s/it]
Epoch: 2, Loss: 0.18049341440200806
202it [03:41, 1.05s/it]
Epoch: 2, Loss: 0.1675453633069992
278it [05:06, 1.10s/it]
0it [00:00, ?it/s]
Epoch: 3, Loss: 0.1804620772600174

loss is not descreasing no matter how much i increase the epochs and i also tried increasing my batch size to 64 also with 2 gpu support but not getting very good accuracy.

May I know the data location please!

I modify it and then predict the test.tsv and submit to kaggle sentiment competition. The score is quite low, much lower than BERT, so how much epoch will be better. For Bert, I just use 3 epoch, but increase epoch in this roberta seems not work.

The code I'm referring to is in cell 8 in https://github.com/DhavalTaunk08/NLP_scripts/blob/master/Transformers_multilabel_distilbert.ipynb
I tried to print before and after this line, and I see no difference.

print(text)
print(repr(text))
text = " ".join(text.split())
print(text)
print(repr(text))

Output:

proving there is no cure for stupidity.
'proving there is no cure for stupidity.'
proving there is no cure for stupidity.
'proving there is no cure for stupidity.'

Maybe it makes a difference in certain cases? Or can I just remove this line?

I noticed the fine-tuned roberta script saved the fine-tuned model locally by

model_to_save = model
torch.save(model_to_save, output_model_file)
tokenizer.save_vocabulary(output_vocab_file)

How to load this model from the local folder?
I try this:

from transformers import RobertaModel, RobertaTokenizer
# from the local folder
roberta_model = RobertaModel.from_pretrained("./")
roberta_tokenizer = RobertaTokenizer.from_pretrained('./', truncation=True, do_lower_case=True)

But failed with error: OSError: does not appear to have a file named config.json. Checkout 'https://huggingface.co/.//main' for available files.

To make predictions, how to load the model which were saved during the training process?
After load model, we can inference testing data.
thank you :)

The current code is unable to support multi-GPU environment.

I parallelized the model using torch.nn.DataParallel(model).cuda()

I got this error when I try :

Thank you. In the Transformers multilabel distillbert, in the function "train(epoch)", you clear gradients after the model feed-forward operation and before loss backward calculation, and you uses "optimizer.zero_grad()" twice. Will this clear the gradients that the backward calculation needs? Shall it be like this where we only need to clear gradients once before model feed-forward operation:

    optimizer.zero_grad()
    outputs = model(ids, mask, token_type_ids)

    loss = loss_fn(outputs, targets)
    if _%5000==0:
        print(f'Epoch: {epoch}, Loss:  {loss.item()}')

    loss.backward()
    optimizer.step()