Test Selection Based on Historical Test Data

Abstract: This thesis is the culmination of an action research project conducted at Axis Communications, which tries to enable test selection based on historical test data. In order to realize this, a prototype program -- the Difference Engine -- was created, which can parse and analyze historical test data drawn from a large database. The analysis produces a weighted correlation between code packages and test packages, and this correlation can then be used to select only a recommended subset of the full regression test suite in order to minimize the time spent running tests which are not related to the code that was changed. The evaluation results of the Difference Engine shows that its algorithm for finding correlations is robust, and running only the recommended tests is more effective than running the full regression test suite by a significant factor. On average it successfully identifies nearly all relevant tests (median success rate is 100%), and it discards 14 times more irrelevant tests given the code that has been changed compared to the default selection strategy of running all tests. Furthermore, the average size of the recommended subset is only 5% of the size of the full regression test suite, meaning that the time spent running tests can be decreased by a factor of 20.