Transfer learning in Swedish - Twitter sentiment classification

Abstract: Language models can be applied to a diverse set of tasks with great results, but training a language model can unfortunately be a costly task, both in time and money. By transferring knowledge from one domain to another, the costly training only has to be performed once, thus opening the door for more applications. Most current research is carried out with English as the language of choice, thus limiting the amount of available already trained language models in other languages. This thesis explores how the amount of data available for training a language model effects the performance on a Twitter sentiment classification task, and was carried out using Swedish as the language of choice. The Swedish Wikipedia was used as a source for pre-training the language models which were then transferred over to a domain consisting of Swedish tweets. Several models were trained using different amounts of data from these two domains in order to compare the performance of these models. The results of the model evaluation shows that transferring knowledge from the Swedish Wikipedia to tweets yield little to no improvements, while unsupervised fine-tuning on tweets give raise to large improvements in performance.