Test Process Evaluation by Combining ODC and Test Technique Effectiveness

Abstract: This report discusses the importance of test process evaluation in order to improve a test model and to provide developer- and management feedback. The report results in a test evaluation framework, developed in cooperation with a department at Ericsson Software Technology in Karlsrona. The framework is a result of discussions with the developers regarding performed testing, studying defect types from past projects and by analyzing the result from a small survey answered by some of the developers at Ericsson. The overall project aim was to evaluate performed testing in order to improve the test model. This requires a good insight of the test process, which is provided by the developed test evaluation framework. The test process is visualized by extracting test process data, making it possible to achieve the project aim. The project aim can be divided into the three following areas: Firstly to evaluate if the current test model is followed as expected, for example are all test techniques used according to the test model? Secondly to evaluate how well the test model fulfills predefined expectations, i.e. is a defect detected with the expected test technique and in the expected test phase? Finally to evaluate if there are any problematic defects that should receive extra attention during a project such as if one or several defect types occurs more frequently than others? The framework is based on another framework, Orthogonal Defect Classification [Chillarege92], combined with the research area Test Technique Effectiveness. The aim of this combination was to support the developed framework. Further a specific part of the framework is focusing on developer- and management feedback.