How to handle Index Error Issue about bert_score HOT 5 CLOSED

@Tiiiger

from bert_score.

Hi @ShauryaUppal-1Mg,

Thank you for using our BERTScore.
The reason for this error is that both BERT and RoBERTa are trained with sentences with at most 512 tokens. Unlike the original Transformers, BERT and RoBERTa use learned positional embedding whose size is set during training.
BERTScore is commonly computed between a pair of sentences.
We would suggest that you split the documents into sentences before feeding them to BERT.
If there happen to be some sentences with more than 512 tokens, you can:

1. train a BERT model using longer sentences
2. cut sentences into multiple chunks and design a better way to aggregate them. This is one of the future directions to extend BERTScore, but we haven't studied it yet.
3. use XLNets which support longer inputs. However, they perform worse in our experiments.

Best,
Felix

from bert_score.

Will if work will it for sentence having length > 512, I remove stopwords and common words?

from bert_score.

But BERT as a service allows user to make max_sequence= ignore
Can't we do something with that?

https://github.com/hanxiao/bert-as-service

from bert_score.

As far as I know, they just trim down the sequence. Please see:
https://github.com/hanxiao/bert-as-service/blob/85690491d66fd1ca0d03924f8c9ead3d1cad90b1/server/bert_serving/server/__init__.py#L414-L422

We would like to follow huggingface's transformers and just raise an error instead.
We encourage users to deal with their own special cases.

Thank you for raising this issue. We will update the README to remind other users.

from bert_score.