Adaptive Comparative Judgement in Practice : A case study of the formative use of ACJ in a collaborative open-ended design project between Swedish upper primary schools under authentic classroom conditions

Abstract: Digital assessment has the potential to increase equity in the assessment by facilitating a smother collaboration between teachers. It also allows peer assessment between students from different classes or even schools, which allows the students to view a potentially larger and more varied set of examples of the same assignment. In this study, we have examined the potential of using Adaptive Comparative Judgement (ACJ) as a collaborative method of formative assessment of student portfolios created in the open-ended design project: Public Spaces. In the project, the students were given the task of creating an improvement suggestion for a public space from an Agenda 2030 perspective. ACJ has been used with positive results in several similar studies, but many of these have been conducted in more controlled environments with fewer variables. This study, therefore, focuses on examining ACJ from a more practice-based perspective with younger students. The students in the study were taught by their regular teachers who were instructed to plan their lessons according to the teaching material created for Public Spaces within this master thesis, which can be found in appendixes A, B and C. This study was conducted through a collaboration between four upper primary school classes from different municipalities in Sweden. The students and teachers participating in this study completed two ACJ assessment sessions, one in the middle of the project and one at the end. During the assessments, they were also instructed to provide written feedback. When analysing the content of the feedback it became clear that although the majority of the comments could be classified as specific, meaning they clearly specified which area of the assignment the feedback referred to, the feedback was often rather shallow and potentially not very helpful. The students and teachers rarely motivate their feedback, and the students rarely provided clear improvement suggestions. When analysing the reliability of the portfolio rankings from the ACJ assessments we found that they were notably lower than in many of the previous studies. Furthermore, we noticed that several of the teachers seemed to have struggled to find the time to read the teaching material. This problem could potentially have been alleviated by communicating the material more clearly at the start of the project.