Data Visualization of Software Test Results : A Financial Technology Case Study

Abstract: With the increasing pace of development, the process of interpreting software test results data has become more challenging and time-consuming. While the test results provide valuable insights into the software product, the increasing complexity of software systems and the growing volume of test data pose challenges in effectively analyzing this data to ensure quality. To address these challenges, organizations are adopting various tools. Visualization dashboards are a common approach used to streamline the analysis process. By aggregating and visualizing test results data, these dashboards enable easier identification of patterns and trends, facilitating informed decision-making. This study proposes a management dashboard with visualizations of test results data as a decision support system. A case study was conducted involving eleven quality assurance experts with a number of various roles, including managers, directors, testers, and project managers. User interviews were conducted to evaluate the need for a dashboard and identify relevant test results data to visualize. The participants expressed the need for a dashboard, which would benefit both newcomers and experienced employees. A low-fidelity prototype of the dashboard was created and A/B testing was performed through a survey to prioritize features and choose the preferred version of the prototype. The results of the user interviews highlighted pass-rate, executed test cases, and failed test cases as the most important features. However, different professions showed interest in different test result metrics, leading to the creation of multiple views in the prototype to accommodate varying needs. A high-fidelity prototype was implemented based on feedback and underwent user testing, leading to iterative improvements. Despite the numerous advantages of a dashboard, integrating it into an organization can pose challenges due to variations in testing processes and guidelines across companies and teams. Hence, the dashboards require customization. The main contribution of this study is twofold. Firstly, it provides recommendations for relevant test result metrics and suitable visualizations to effectively communicate test results. Secondly, it offers insights into the visualization preferences of different professions within a quality assurance team that were missing in previous studies.