Big Data in Performance Measurement: : Towards a Framework for Performance Measurement in a Digital and Dynamic Business Climate

Abstract: In today’s business climate permeated by Big Data, an opportunity to drive performance lies in analysing consumer behaviour from user data. In particular for online content providers, user data is available in abundance and logged continuously. This leads to new possibilities for design and usage of metrics, as businesses can benefit from smart and timely decision-making. However, in order to profit from user data in performance measurement (PM), it is critical to identify metrics that truly guide decisions. Thus, an effective and efficient PM process is imperative. Despite its promise, Big Data’s role in PM has been scarcely researched. Research has studied user behaviour from data, for instance in the context of video or audio streaming and web search, but primarily with a focus on technical performance. In addition, the research on online content providers’ PM is fragmented, and has mainly been conducted by practitioners. Thus, the PM field needs to be updated to reflect today’s dynamic and digital business climate. Therefore, the purpose of this research was to explore how online content providers, generating a large amount of user data, work with PM, and also practically illustrate how metrics can be designed from user data. The research was carried out as a case study at an audio streaming company, but empirics was also gathered from other online content providers with the aim to increase the generalisability. The illustration of metric design was based on quantitative analysis of commuters’ in-car audio streaming. For commuters’ audio streaming it was found that suitable metrics should capture the habitual nature. Therefore engagement metrics were found to be applicable, for instance the fraction having sessions both in the morning and afternoon, and the fraction having more than one day commuting with the streaming service per week. In regard to online content providers’ PM process, this research contributes with a proposed framework, which was developed from three existing frameworks; HEART reflected as important measurement dimensions and translation of goals to metrics, OKR which sets the focus in terms of high-level goals, and design-implement-use reflected as the process’ phases. It was found that insights from user data and explicit user feedback are complementary and can arise throughout the whole process, and that mutual communication between data scientists and product managers is crucial. Further, four types of iterations were identified in the process; modifying a metric, designing new metrics, completely changing a metric, and starting new initiatives. Moreover, metrics were found to be highly context dependent. Additionally, four important aspects were identified in metric design; data availability and proxy assessment, characteristics and form of metric, metric trade-offs, and metric movement interpretation.