ROI Calculation on Online Controlled Experiment

Abstract: As online services such as e-commerce and mobile applications keeps growing, the need of optimizing the user experience does as well. By conducting Online Controlled Experiments, companies can get an insight to which features, design choices and implementations that users enjoy the most. This projects explores two areas in relation to Return on Investment (ROI) calculations for Online Controlled Experiments. First, challenges and pitfalls with calculating a reliable ROI metric are described as well as references to research working to mitigate these challenges. The challenges presented are all related to how to accurately measure the effect between two candidate variants which sums up the main challenges with ROI calculation. Secondly, a model based on expected return on investment is constructed and explored in order to investigate whether the model can help to optimize test parameters for a two sample t-test in the setting of Online Controlled Experiments. The results of the model analysis shows that the model has limited practical use since it maximizes the ROI - quotient without taking into account to magnitude of potential revenue increase as well as potential cost.