Engagement Evaluation of Content of an Internet-based Cognitive Behavioral Therapy (iCBT) Mobile Application: An Observational, Quantitative Study of Usage Data

Abstract: Background: Digital mental health interventions (DMHI) hold promise for addressing mental health needs on a large scale. Among these interventions, Internet-based Cognitive Behavioral Therapy (iCBT) has proven effective in tackling various mental health issues. However, the effectiveness of these interventions is hindered by a lack of sustained user engagement. A lack of knowledge on what specific iCBT content is the most engaging and the absence of a standardized approach to measure engagement hampers progress in this area. Aim: To conduct an evaluation of engagement based on usage data to learn which and what kind of iCBT content from a self-tailored blended iCBT application is the most engaging in order to inform strategies and enhance efforts to improve the overall effectiveness of the intervention. Methods: An observational, quantitative study of real-world usage data was conducted to evaluate engagement levels across various iCBT modules available in the app. Four engagement metrics were obtained from the users´ usage data: Adherence Rate, Rate of Max. Progress, Return Rate, and Average Time Spent to create rank and compare engagement levels across the iCBT modules. Results: The results included data from the 1st of January to the 30th of April of 2023 for 138 iCBT modules. The mean engagement score for all iCBT modules was 56.41±9.85, with a median of 57.30 and a range of 67.97. The most engaging iCBT modules were “Thought traps and questioning thoughts” (81.4), “A model for social anxiety” (80.5), and “Safety Behaviors” (78.1). The most engaging iCBT module classifications were “Social phobia” (92.10), “Depressive disorder” (80.70), and “Post-traumatic stress disorder” (80.40). Conclusions: Based on real-world usage data of patients, it was possible to determine the extent to which users engaged with different iCBT modules comprehensively. The findings shed light on the strengths and weaknesses of the iCBT modules regarding engagement levels. The quantitative evaluation of “in the wild” patient usage data proved useful for assessing engagement levels of specific content within the app.