Standardizing Usability Evaluation : Case study of objective measures as complements to user satisfaction surveys

Abstract: Standardized methodologies for usability evaluation have been of interest to the Human-Computer Interaction(HCI) community for years due to their perceived advantages over non-standardized ones. Such methodologies include standardized questionnaires designed to measure usability, although it has been argued that such questionnaires can only measure user satisfaction and not usability in general. This study was conducted together with the editorial experience (EDX)team at Sveriges Television AB (SVT) in order to identify appropriate standardized methods for usability evaluation of the content management system they are currently developing, while at the same time exploring how questionnaire formats such as the System Usability Scale(SUS) could benefit from being complemented by objective measurement methods such as the keystroke-level model (KLM) and productive time. The project ran test sessions with participants using a benchmark version of the current content management system and a prototype of a redesigned version. SUS, KLM and productive time were then applied to both versions and compared. The results showed no significant difference in the usability of the two versions, although the study lacks reliable data to draw any strong conclusions. However, the data does raise interesting questions that may provide the groundwork for future research. This includes findings that may question the validity of SUS when used for the evaluation of prototypes and the benefits of objective measures to self-administered questionnaires.