Methods For Test Case Prioritization Based On Test Case Execution History

Abstract: Motivation: Test case prioritization can prioritize test cases, optimize the test execution, save time and cost. There are many different methods for test case prioritization, test case prioritization method based on test case execution history is one kind of them. Based on the test case execution history, it’s easier to increase the rate of fault detection, hence we want to do a study about test case prioritization methods based on the test case execution history. Meanwhile, executing the feasible methods to compare the effectiveness of them. For the motivation of the thesis may be regarded as an example for experiencing approach for comparing test case prioritizations based on test case execution history, or as a study case for identifying the suitable methods to use and help improve the effectiveness of the testing process. Objectives: The aim of this thesis is to look for a suitable test case prioritization method that can support risk based testing, in which test case execution history is employed as the key criterion of evaluation. For this research, there are three main objectives. First, explore and summarize methods of test case prioritization based on test case history. Next, identify what are differences among the test case prioritization methods. Finally, execute the methods which we selected, and compare the effectiveness of methods. Methods: To achieve the first and the second study objectives, a systematic literature review has been conducted using Kitchenham guidelines. To achieve the third study objective, an experiment was conducted following Wohlin guidelines. Results: In our thesis: 1) We conducted a systematic literature review and selected 15 relevant literatures. We extracted data of the literatures and then we synthesized the data. We found that the methods have different kinds of inputs, test levels, maturity levels, validation and "automated testing or manual testing". 2) We selected two feasible methods from those 15 literatures, Method 1 is Adaptive test-case prioritization and Method 2 is Similarity-based test quality metric. We executed the methods within 17 test suites. Comparing the result of two methods and non-prioritization, the mean Average Percentage of Defects Found (APFD) of Adaptive test-case prioritization execution result (86.9%) is significantly higher than non-prioritization (51.5%) and Similarity-based test quality metric (47.5%), it means that the Adaptive test-case prioritization has higher effectiveness. Conclusion: In our thesis, existing test case prioritization methods based on test case execution history are extracted and listed out through systematic literature review. The summary of them and the description of differences can be available in the thesis. The 15 relevant literatures and the synthesized data may be as a guideline for relevant software researchers or testers. We did the statistical test for the experimental result, we can see two different test case prioritization methods have different effectiveness.