Solutions to the equivalent mutants problem: A systematic review and comparative experiment

Abstract: Context: Mutation testing is a fault-based technique for measuring the effectiveness of a test set in terms of its ability to detect faults. Mutation testing seeds artificial faults into an application and checks whether a test suite can detect them. If these faults are not found, the test suite is still not considered to be 'good enough'. However, there are also mutations which keep the program semantics unchanged and thus cannot be detected by any test suite. Finding a way to assess these mutations is also known as the equivalent mutant problem (EMP). Objectives: The main objective of this thesis is to conduct a systematic literature review in the field of mutation testing, to identify and classify existing methods for equivalent mutants detection. In addiction, other objectives are: analyze possibilities to improve existing methods for equivalent mutant detection, implement new or improved method and compare it with existing ones. Methods: Based on the systematic literature review method we have went over publications from six electronic databases and one conference proceedings. Standard method was extended by scanning lists of references and some alternative sources: searching in Google Scholar, checking personal websites of relevant authors and contacting all of them. We have performed all the systematic literature review steps such as the protocol development, initial selection, final selection, quality assessment, data extraction and data synthesis. In the second part of this thesis - an experiment, we have implemented four second order mutation testing strategies and compared them each other from four different perspectives: mutants reduction, equivalent mutants reduction, fault detection loss and mutation testing process time reduction. Results: The search identified 17 relevant techniques in 22 articles. Three categories of techniques can be distinguished: detecting (DEM), suggesting (SEM) and avoiding equivalent mutants generation (AEMG). Furthermore, for each technique current state of development and some ideas on how to improve it are provided. The experiment proved that DifferentOperators strategy gives the best results in all four investigated areas. In addition, time for manual mutants classification against equivalence was measured. Assessing one first order mutant takes 11 minutes 49 seconds, while for the second order mutants classification time is 9 minutes 36 seconds in average. Conclusions: After three decades of studies, results obtained for techniques from the DEM group are still far from perfection (best one is detecting 47,63% of equivalent mutants). Thus, new paths for the solution have been developed - SEM and AEMG group. Methods from both categories help in dealing with EMP, however from SEM provide only mutants likely to equivalent, while from AEMG cause some loss of test effectiveness. The conclusion from the experiment is that DifferentOperators strategy gives the best results among all proposed.