Automatic Summary Generation and Evaluation for German Language

The models have been trained on these data-sets:

1. MLSUM
2. GeWiki
3. Sample of 1600 article/summary pairs from CNN/DM data-set along with german translations and human annotations originally taken from SummEval
4. Sample of 80 article/summary pairs from MLSUM data-set along with human annotations

1. baselines.ipynb contains code to load and pre-preprocess data-sets (MLSUM and GeWiki), generate baseline summaries (Random, Lead, TextRank) and evaluate them using different metrics (rouge, bleu, meteor, bert-score, mover-score, blanc, js-divergence, supert), and save the results to the file system.
2. bertsum.ipynb contains code to pre-process data for BertSum model, train BertSum model, and then predict both Oracle and BertSum summaries using the trained model for the test set and write them to the file system.
3. matchsum.ipynb contains code to pre-process data for MatchSum model, train MatchSum model, and then predict summaries using the trained model for the test set and write them to the file system.
4. quality_estimation.ipynb contains code to train our Quality Estimation models, report accuracy on the test set and save all the trained models to the file system.
5. data_analysis.ipynb contains code to predict the scores for the selected 60 summaries from MLSUM data-set using our trained Quality Estimation models, as well as to statistically analyse our previously saved evaluation results.

Model trained on MLSUM

Model trained on GeWiki

Model trained on MLSUM

Model trained on GeWiki

Trained Models

The project dependencies are defined in requirements.txt. We have borrowed code and datasets from the following repositories in this project:

1. BertSum
2. MatchSum
3. GeRouge