Initial Development and Validation of Language-Based Assessments for Meaningful Change

Abstract: Meaningful change has been discussed in multiple studies, with the recurring question of how it could be conceptualized and assessed to identify what determines meaningful change and where it occurs. Previous studies have conducted statistical analyses based on traditional rating scales (i.e., the PHQ-9) to assess meaningful change. There is no evidence to be found of previous studies attempting to assess meaningful change through language-based assessments. This study intended to examine whether language-based assessments could be utilized in assessing meaningful change, and if so, to what extent. This study has utilized scores from human-rated meaningful change assessments of natural language responses (NLR) and self-reported scores from the open-source Patient Health Questionnaire-9 (PHQ-9). The study conducted analyses in R-studio based on the text-package and included the large-language model RoBERTa for word embedding, correlation testing for examining reliability and validity, and ridge regression to train the model. The analyses showed results of inter-rater reliability in human-rated assessments (r = .64, p <.001, N = 100), correlation between human-rated assessments and PHQ-9 difference scores (r = .36, p <.001, N = 298), the strongest trained model (r = .39, p <.001, N = 298), and correlation between language-based assessment and PHQ-9 difference scores (r = .29, p <.001, N = 298). These findings suggest that language-based assessments can be further developed to assess meaningful change, and preferably by including human-rated assessment.