Automated Feedback on Learning Progress : An exploratory study of opportunities and risks associated with automatically computed metrics and recommendations

Abstract: For the past 100 years, attempts have been made to develop teaching machines that support students’learning. Judging by the contemporary increase in funding for educational technology companies,there is now more than ever an interest in technology that facilitates learning. As a result of studentsbeing able to interact with learning material via internet-connected devices, it has become feasible toanalyze large quantities of interactional data in order to provide learners with individually tailoredlearning paths. This is one of the services provided by the company Sana Labs. Their algorithms aresaid to equip learners with actionable insights in the form of metrics and recommendations based oninteractions with digital learning platforms. However, the company has limited insight into how thetechnology can be used formatively in a classroom environment.As formative assessment includes clarifying what a student knows in relation to a curriculum goaland modifying learning activities accordingly to improve the student’s prospects to meet that goal, onespecific area of research within formative actions is the notion of feedback. Feedback has in metastudiesshown to be associated with high effect sizes in relation to learning, and how feedback can beconceptualized and implemented in digital learning environments has been investigated specifically.This exploratory study uses design science research to examine potential opportunities and risksassociated with automatically computed feedback in the form of visual metrics and recommendations.When analyzing external factors that affect learning, a behaviorist perspective is applied, and as forinternal processes such as meta-cognition and self-regulation, a constructivist perspective of learningis used. An artifact is developed in the form of a lo-fi prototype including three functions forautomated feedback, which are then evaluated by upper secondary mathematics teachers in Stockholmand a research manager at Sana Labs.The results indicate that it is possible to implement digital feedback strategies according to theliterature of feedback effects, and the technology available as of 2020. The interviewed teachersexpress how the functionality presented in the prototype could be helpful when working withformative assessment. Furthermore, the research manager deemed all of the proposed feedbackfunctions to be technically feasible. Regarding risks accompanying these strategies, a concern of theteachers could be identified, that the automated metrics and recommendations would be based on alimited view of the student’s knowledge. As a direction for further research, additional development ofthe prototyped functions is proposed, along with studies on the relationship between goal-setting andautomated feedback.