Prioritizing Tests with Spotify’s Test & Build Data using History-based, Modification-based & Machine Learning Approaches

Abstract: This thesis intends to determine the extent to which machine learning can be used to solve the regression test prioritization (RTP) problem. RTP is used to order tests with respect to probability of failure. This will optimize for a fast failure, which is desirable if a test suite takes a long time to run or uses a significant amount of computational resources. A common machine learning task is to predict probabilities; this makes RTP an interesting application of machine learning. A supervised learning method is investigated to train a model to predict probabilities of failure, given a test case and a code change. The features investigated are chosen based on previous research of history- based and modification-based RTP. The main motivation for looking at these research areas is that they resemble the data provided by Spotify. The result of the report shows that it is possible to improve how tests run with RTP using machine learning. Nevertheless, a much simpler history- based approach is the best performing approach. It is looking at the history of test results, the more failures recorded for the test case over time, the higher priority it gets. Less is sometimes more.