A Case Study on the Limitations of Automated Duplicate Bug Report Detection

Abstract: Identifying duplicate bug reports is crucial in software development as it helps streamline the debugging process, reduce redundancy, and enhance overall efficiency. By addressing the challenges associated with existing automated techniques and leveraging testers’ expertise, the tool proposed in this study aims to improve the accuracy of duplicate detection, saving valuable time and resources while ensuring that potential duplicates are not overlooked. While various automated techniques for identifying duplicate bug reports have been previously described in literature, they often result in false positives or identified duplicates that testers would not consider as such [8]. To address this we propose Bugle, a software tool that incorporates a state-of-the-art large language model and involves the opinions of testers when evaluating potential duplicate bug reports in real-time. Our approach leverages testers’ tacit knowledge and intuition to improve the accuracy of duplicate detection and reduces the amount of false positives by letting the tester evaluate a ranked list of recommended duplicates with the highest semantic textual similarity. We evaluate the tool in a case study at TestScouts, a testing consulting company, and analyze the recommendations of the tool against the judgement of a group of testers at the company. Besides Bugle, we contribute to the state-of-the-art by incorporating testers’ opinions and provide insights into the limitations of automated techniques for duplicate bug report detection.