Meaningful Metrics in Software Engineering : The Value and Risks of Using Repository Metrics in a Company

Abstract: Many large companies use various business intelligence solutions to filter, process, and visualize their software source code repository data. These tools focus on improving continuous integration and are used to get insights about people, products, and projects in the organization. However, research has shown that the quality of measurement programs in software engineering often is low since the science behind them is unexplored. In addition, code repositories contain a considerable amount of information about the developers, and several ethical and legal aspects need to be considered before using these tools, such as compliance with GDPR. This thesis aims to investigate how companies can use repository metrics and these business intelligence tools in a safe and valuable way. In order to answer the research questions, a case study was conducted in a Swedish company, and repository metrics from a real business intelligence tool were analyzed based on several questions. These questions were related to software measurement theory, ethical and legal aspects of software engineering and metrics, and institutionalized theory. The results show how these metrics could be of value to a company in different ways, for instance by visualization collaboration in a project or by differentiating between read and active repositories. These metrics could also be valuable by linking them to other data in the company such as bug reports and repository downloads. The findings show that the visualizations could potentially be perceived as a type of performance monitoring by developers, causing stress and unhealthy incitements in the organization. In addition, repository metrics are based on identifiable data from Git, which according to the GDPR is classified as personal data. Further, there is a risk that these tools are used simply because they are available, as a way to legitimize the company. In order to mitigate these risks, the thesis states that the metrics should be anonymized, and the focus of the metrics should be on teams and processes rather than individual developers. The teams themself should be a part of creating the Goal-Question-Metrics that link the metrics to what the teams wish to establish.