Facial activity recognition as predictor for learner engagement of robot-lead language cafes

Abstract: Sweden is a country being extra exposed to immigration and, therefore, suffer from a shortage of teachers within second language learning (L2). Other L2 solutions have increased in popularity but require many volunteers. This fact motivated the project CORALL at KTH, Stockholm, which aims to use robot-assisted language learning (RALL) to simulate a language cafe setting. In order to learn, one has to be engaged. This study thereby aims to examine whether the engagement level of students being exposed to RALL can be classified. This is a contribution to the project’s long term goal of being able to automatically adapt the robot to the learners’ engagement levels. The study also aims to analyze the project’s current status, in order to evaluate its future, by examining teachers’ and the CORALL project leader’s responses to a survey, and then utilizing the SWOT framework to draw conclusions. The technical approach is to use the open source toolkit OpenFace to extract facial features from each frame of video recordings of RALL participants. By annotating each frame with one out of four engagement levels, supervised machine learning algorithms are then used to try and estimate students’ engagement. The results from the technical study are deemed inadequate. The models produced, however, were decent at classifying high engagement levels, which could be of use for further investigation. The results from the investigative survey and the SWOT analysis suggested that the internal and external views of RALL align well with the current focal points of the project. The matter of engagement tracking was deemed as an important factor by all parties. While other circumstances could have improved the technical results, important takeaways for future work were discovered.