Question answering on introductory Java programming concepts using the Transformer

Abstract: AI applications for education could help students learn in their introductory programming courses. Many applications for education try to simulate a humantutoring session that engages the student in a dialogue. During the session, they can ask questions and have them answered while working throughan exercise. Refining the question-answering capability of such applicationsmay prove to be a base for supplementary education tools. These could be usedby students in introductory programming courses to ask questions to reviewconcepts in programming, facilitating the teaching done by professors. This thesis investigates question-answering on introductory Java programming using The Transformer model. The focus is on the extent to which the model can answer questions on Java concepts when trained on questions and answers from the online programming forum Stack Overflow. A total of five Transformer models with default parameters were trained on posts segmented with different granularities using byte-pair encoding. Each model was evaluated using perplexity as an automatic metric and a qualitative evaluation done by the author. The automatic metric evaluation scores a low perplexity indicating a hig hquality model. However, the qualitative evaluation shows that the generated responses are short, generic, repetitive, and even contradicting, with the most common response being “You can do it like this:“. That is, the model exhibits a fundamental inability to answers questions on Java programming concepts.