Evaluating machine learning models for text classification

Abstract: This thesis will explore the use of AWS machine learning services that enable natural language processing (NLP). More specifically, this work will focus on sentiment analysis of product and service reviews written in Swedish. To find the most efficient solution to this task, the ready-made sentiment analysis tool available on AWS (Amazon Comprehend) was compared to custom-made solutions built on the AWS platform Amazon SageMaker. Several model options were considered for the custom-made alternative, ranging from simple regression to state-of-the-art language models, along with different ways of deploying the best performing model to the cloud so that its insights could be accessed in the most efficient way. For the evaluation, a use-case was put forward that is deemed realistic for how Sigma Technology Cloud would use such a service. Based on this, the thesis findings suggest that a scaled-down version of the BERT model called DistilBERT is the best alternative of the models evaluated. Furthermore, this model should be set up as an asynchronous endpoint with a policy allowing it to auto-scale down to zero when it is not invoked. Doing this on the Amazon SageMaker platform results in a solution that is both cheaper and that shows better performance than other alternatives. Finally, this comparison between the services based on their performance, ease of use, and cost efficiency was put forward as a recommendation as to what model and configuration should be used by Sigma Technology Cloud.