Variation in Assessment : A Coh-Metrix Analysis of Evaluation of Written English in the Swedish Upper Secondary School

Abstract: Reliable evaluation is an important part of language education. However, reliable evaluation of student writing is notoriously difficult to achieve nationally. To better understand what evaluators value in upper secondary English education in Sweden this study has examined correlations between grading and linguistic variables in student writing using Coh-Metrix, a natural language processor. Previous studies in Hong Kong and the U.S. have shown that linguistic and lexical sophistication have a high correlation with good grades, while explicit cohesive devices have a negative correlation with grades. Results have varied depending on level of proficiency and there are indications that evaluators have cultural differences. Seventy-nine essays from the national test for the course English 6, from two different schools and teachers were analysed in this study. The study asked: What language variables correlate with grades, and how do two teachers differ in their value of quality? In order to answer this, correlations between grades and all 106 Coh-Metrix indices were calculated and t-tested for significance. The ten most highly correlating variables were then compiled, with very similar indices excluded. Correlation scores of the same variables but for both teachers separately were then compiled for comparison. The ten highest correlations for both teachers separately were also compiled along with scores for the same indices for the other teacher. The results showed similarities with previous studies. Lexical sophistication correlated significantly with grades and indices measuring explicit cohesive devices showed significant negative correlations with grades. Surprisingly however, syntactic complexity did not correlate clearly with grades. The highest correlating indices differed between teachers but were mostly connected to cohesion or word choice. In all, the results showed that while teachers seem to differ in their evaluative focus, they do in fact value similar aspects of student writing.