Data leakage Train <=> Test? about pyabsa HOT 3 CLOSED

I will close this issue since I upload the lateset checkpoint

from pyabsa.

Thanks for your review, this problem exists indeed. The rest 15 should removed while it is included by rest16. The checkpoint could be updated later.

from pyabsa.

Hi,

when loading the ATEPC model (ATEPCTrainedModelManager.get_checkpoint(checkpoint_name="English")) it prints that the model was trained on the following datasetsplit:

dataset_file: {
'train':
['datasets\\atepc_datasets\\SemEval\\laptop14\\Laptops_Train.xml.seg.atepc',
'datasets\\atepc_datasets\\SemEval\\restaurant14\\Restaurants_Train.xml.seg.atepc',
'datasets\\atepc_datasets\\SemEval\\restaurant15\\restaurant_train.raw.atepc',
'datasets\\atepc_datasets\\SemEval\\restaurant16\\restaurant_train.raw.atepc'],
'test':
['datasets\\atepc_datasets\\SemEval\\laptop14\\Laptops_Test_Gold.xml.seg.atepc',
'datasets\\atepc_datasets\\SemEval\\restaurant14\\Restaurants_Test_Gold.xml.seg.atepc',
'datasets\\atepc_datasets\\SemEval\\restaurant15\\restaurant_test.raw.atepc',
'datasets\\atepc_datasets\\SemEval\\restaurant16\\restaurant_test.raw.atepc']
}

I noticed that some models/splits by other authors had a dataleak between train and test set when using the restaurant dataset from 15 and 16. More information here: https://github.com/lixin4ever/BERT-E2E-ABSA#updated-results-important

Can you trace whether this ATEPC model is affected by this issue as well?

Thanks

However, I try to upload a patch to fix the dataset selection before training. The checkpoint may update later.

from pyabsa.