Evidence and perceptions on GUI test automation : An explorative Multi-Case study

Abstract: Context. GUI-based automation testing is a costly and tedious activity in practice. As GUIs are well-known for being modified and redesigned throughout the development process, the corresponding test scripts are not valid anymore thereby being a hindrance to automation. Hence, substantial effort is invested in maintaining GUI test scripts which often leads to rework or waste due to improper decisions. As a result, practitioners have identified the need for decision support regarding when should GUI automation testing begin and how to make it easier and also identify what are the factors leading to waste in GUI-based automation testing. The current literature provides solutions relating to automation in general and few answers for GUI based-automation testing. Such generic answers might not be applicable to GUI test automation and also industries new to GUI development and testing. Thus, it is necessary to validate if the general solutions are applicable to GUI test automation and find additional answers that are not identified previously from practitioners opinions in an industrial context. Objectives. Capture relevant information regarding the current approach for GUI test automation within the subsystems from a case company. Next, identify the criteria for when to begin automation, testability requirements and factors associated with waste from literature and practice. Methods. We conducted a multiple-case study to explore opinions of practitioners in two subsystems at a Swedish telecommunication industry implementing GUI-automation testing. We conducted a literature review to identify answers from scientific literature prior to performing a case study.A two-phased interview was performed with different employees to collect their subjective opinions and also gather their opinions on the evidence collected from the literature. Later, Bayesian synthesis method was used to combine subjective opinions of practitioners with research-based evidence to produce context-specific results. Results. We identified 12 criteria for when to begin automation, 16 testability requirements and 15 factors associated with waste in GUI test automation.Each of them is classified into categories namely SUT-related,test-process related, test-tool related, human and organizational, environment and cross-cutting. New answers which were not present in the existing literature in the domain of the research are found. Conclusions. On validating the answers found in literature, it was revealed that the answers applicable for software test automation, in general, are valid for GUI automation testing as well. Since we incorporated subjective opinions to produce context specific results, we gained an understanding that every practitioner has their own way of working. Hence, this study aids in developing a common understanding to support informed subjective decisions based on evidence.