Metadata assisted finetuning with largepre-trained language models forabstractive text summarization : Multi-task finetuning with abstractive text summarization and categoryclassification

Abstract: Text summarization is time-consuming for humans to complete but is still required in many areas. Recent progress in machine learning research, especially in the natural language domain, has produced promising results. The introduction of the Transformer and the increased popularity of pre-trained language models have driven this improvement and led to a human-comparable performance in the automatic text summarization domain. However, the finetuning of pre-trained language models with multitask learning to increase the model’s performance is still new. This potential performance increase raises the question. How can Multitask finetuning affect the summarization performance of large pre-trained language models? To answer this, we finetune two pre-trained models in 3 variants each, one model as a benchmark and two models incorporating multitask finetuning with category classification as a complementary task to the abstractive text summarization. The results indicate decreased performance with multitask finetuning. However, extended finetuning of the models shows a more negligiblem difference between standard and multitask approaches, opening up for further hyperparameter tuning and a potential benefit from the multitask approach.