Acceptance Testing in Agile Software Development - Perspectives from Research and Practice

Abstract: Context: Acceptance testing is an important activity that verifies the conformance of a system to its acceptance criteria. It aims to provide a detailed communication of domain knowledge and is used to evaluate whether the customer requirements are met. Existing literature lacks empirical evidence for acceptance testing. Especially in the context of industry practice, it is not in the authors' consideration, except for a few studies, where the authors have investigated the state of practice in a specific domain. Objective: This study aims to recognize the state of research and practice of acceptance testing in Agile Software Development and investigate the similarities and differences in both perspectives. The study contributes to identify the industry-academia gap in the context of acceptance testing. Research Method: To identify the acceptance testing practices and challenges from research, I have conducted a literature review. For the industry perspective on acceptance testing practices and challenges, I have conducted an interview-based survey of the practitioners working in the Agile Software Development environment. I followed the snowball search strategy to search the primary studies, whereas to select the respondents, I used the convenience and snowball sampling method. For data analysis, I followed the approach of thematic synthesis. Results: The results of this thesis are the outcome of a literature review of 20 selected studies and an interview-based survey with 12 practitioners representing10 companies. I identified acceptance testing practices and challenges from research and industry. In the research, the most recommended form of acceptance testing is acceptance test-driven development (ATDD), and the majority of the studies are referring to the use of FIT for acceptance testing. Customer involvement in different phases of acceptance testing is recommended in research. From the interviews, I come across that acceptance testing is manual at large in the industry, and the most challenging aspect is the customer’s involvement. Conclusions: From the findings of this thesis, it is concluded that there is a gap between the research and industry perspective of acceptance testing practices. Currently, acceptance testing in the industry is mostly manual, the research is not focusing on this aspect of acceptance testing. Despite the differences, there are some commonalities as well. Especially, most challenges of acceptance testing are similar in both perspectives. Researchers have to consider the commonalities, and they have to look at how they can minimize the acceptance testing challenges from the perspective of the industry.