Utilizing GPT for Interactive Dialogue-based Learning Scenarios : A Comparative Analysis with Rasa

Abstract: This thesis explores the use of advanced language models, specifically OpenAI’s Generative Pretrained Transformer (GPT), in the context of interactive tutoring systems built within a Unity-based game environment. The central problem addressed is whether the recent advancements in large language models make them feasible and useful to function as tutors specifically in providing meaningful, engaging, and educationally rich user interactions on a dialogue based learning platform developed by Fictive Reality. There is also a comparison on the effectiveness of GPT versus the model that previously powered the learning platform built in Rasa. The importance of this problem lies in offering people learning opportunities that might not otherwise be available to them, and in seeing if recent advancements in generative AI are sufficient for developing useful interactive AI tutors of soft skills. The Fictive Reality learning platform is powered by a Rasa model that generates appropriate responses to users in the context of roleplay-based learning scenarios while keeping an internal state of the progress of the dialogue. The project entails replacing this model with GPT and a comparison of their performance and respective merits. We also explored the potential for a hybrid model, leveraging the strengths of both systems. Using Rasa for internal state tracking and answering simpler queries, and utilizing GPT to handle those queries whose intent Rasa cannot determine. The first part of this project was integrating GPT with the existing functionality of the platform, this includes changes to the platform that allow people to create and play GPT powered learning scenarios and adopting the existing features and user interface. Additionally, prompt engineering GPT to act as a tutor and to stay within the context of a learning environment. Changes had to be made to the platform so that the already existing features of Rasa scenarios could be replicated in GPT scenarios. Finally there is a systematic comparison of the user experience and performance metrics when interacting with either a GPT or a Rasa chatbot in a learning scenario. Specifically these metrics are determined from the conversational flow between bot and user, the context and continuity, finish rate, chit-chat handling and length of average session. The results suggest a distinct user preference for the GPT model due to its superior conversational capabilities, despite Rasa’s faster response times and state-tracking feature. The study suggest that GPT is sufficient for creating useful learning scenarios in restricted contexts. Therefore we suggest that large language models can be leveraged in interactive learning systems, with potential impacts on edtech, AI in education, and conversational AI.