Predicting average response sentiments to mass sent emails using RNN

Abstract: This study is concerned with using the popular Recurrent Neural Network (RNN) model, and its variants Gated Recurrent Unit (GRU) and Long-Short Term Memory (LSTM), on the novel problem of Sentiment Forecasting (SF). The goal of SF is to predict what the sentiment of a response will be in a conversation, using only the previous utterance. In more every day terms, we want to be able to predict the sentiment of person B’s response to something person A said, before B has said anything and using only A’s utterance. The RNN models were trained on a Swedish email database containing email conversations, where the task was to predict the average sentiment of the response emails to an initial mass-sent business email. The emails didn’t come with sentiment labels, so the Valence Aware Dictionary and sEntiment Reasoner (VADER) system was used to determine sentiments. Seventy-five training-and-testing experiments were run with varying RNN models and data conditions. The accuracy, precision, recall, and F1 scores were used to determine to what extent the models had been able to solve the problem. In particular, the F1 score of the models were compared to the F1 score of a dummy classifier that only answered with positive sentiment, with the success case being that a model was able to reach a higher F1 score than the dummy. The results led to the findings that the varying RNN models performed worse or comparably to the dummy classifier, with only 5 out of 75 experiments resulting in the RNN model reaching a higher F1 score than the positive classifier, and with the average performance of the rare succeeding models only going 2.6 percentage points over the positive only classifier, which isn’t considered worthwhile in relation to the time and resource investment involved in training RNNs. In the end, the results led to the conclusion that the RNN may not be able to solve the problem on its own, and a different approach might be needed. This conclusion is somewhat limited by the fact that more work could have been done on experimenting with the data and pre-processing techniques. The same experiments on a different dataset may show different results. Some of the observations showed that the RNN, particularly the Deep GRU, might be used as the basis for a more complex model. Complex models built on top of RNNs have been shown to be useful on similar research problems within Sentiment Analysis, so this may prove a valuable avenue of research.