A Case Study of a Chatbot Aimed at Healthcare Employees

Abstract: This case study investigates a chatbot and its responsiveness to user expression, utility and the users' experience. A user study was performed with both a questionnaire and user interviews. A program to automatically categorize user input messages was made in order to analyze the correlations and reasons for failed predictions and misunderstandings. The current utility is not reaching the return on investment, although expansions are planned and the launch is underway. The strongest and most meaningful correlations with unsuccessful messages were with messages that were out of the chatbot's scope, inside of the scope but unsolved, messages that lacked training and test data or that contained at least one unknown word. The user rated the chatbot's ability to understand them slightly above the middle of the scale, even though the rate of unsuccessful messages was high since the chatbot is still under development. Misspellings, vernacular and organization specific words did not have a particularly strong correlation to unsuccessful messages. The opinions on humanlike interaction were split both in regard to its current likeness and what would be desirable in the future. However, most users seemed to want the conversation to feel natural as if from a human and to receive nuanced answers.