Understand me, do you? : An experiment exploring the natural language understanding of two open source chatbots

Abstract: What do you think of when you hear the word chatbot? A helpful assistant when booking flight tickets? Maybe a frustrating encounter with a company’s customer support, or smart technologies that will eventually take over your job? The field of chatbots is under constant development and bots are more and more taking a place in our everyday life, but how well do they really understand us humans?  The objective of this thesis is to investigate how capable two open source chatbots are in understanding human language when given input containing spelling errors, synonyms or faulty syntax. The study will further investigate if the bots get better at identifying what the user’s intention is when supplied with more training data to base their analysis on.  Two different chatbot frameworks, Botpress and Rasa, were consulted to execute this experiment. The two bots were created with basic configurations and trained using the same data. The chatbots underwent three rounds of training and testing, where they were given additional training and asked control questions to see if they managed to interpret the correct intent. All tests were documented and scores were calculated to create comparable data. The results from these tests showed that both chatbots performed well when it came to simpler spelling errors and syntax variations. Their understanding of more complex spelling errors were lower in the first testing phase but increased with more training data. Synonyms followed a similar pattern, but showed a minor tendency towards becoming overconfident and producing incorrect results with a high confidence in the last phase. The scores pointed to both chatbots getting better at understanding the input when receiving additional training. In conclusion, both chatbots showed signs of understanding language variations when given minimal training, but got significantly better results when provided with more data. The potential to create a bot with a substantial understanding of human language is evident with these results, even for developers who are previously not experienced with creating chatbots, also taking into consideration the vast possibilities to customise your chatbot.