Evaluation of the Transformer Model for Abstractive Text Summarization

Abstract: Being able to generate summaries automatically could speed up the spread and retention of information and potentially increase productivity in several fields. Using RNN-based encoder-decoder models with attention have been successful on a variety of language-related tasks such as automatic summarization but also in the field of machine translation. Lately, the Transformer model has been shown to outperform RNN-based models with attention in the relatedfield of machine translation. This study compares the Transformer model to a LSTM-based encoderdecoder model with attention on the task of abstractive summarization. Evaluation is done both automatically, using ROUGE score, as well as using human evaluators to estimate the grammar and readability of the generated summaries. The results show that the Transformer model produces better summaries both in terms of ROUGE score and when evaluated with human evaluators.