Exploring the Efficacy of ChatGPT in Generating Requirements: An Experimental Study

Abstract: This thesis explores the efficacy of ChatGPT in generating software requirements and compares its performance to human participants through an experimental study. The study addresses three main research questions (RQs), examining how ChatGPT-generated requirements align with human-written requirements, the variation in quality between different versions of ChatGPT using two additional sub-questions that look at improvement in quality from feedback and consistency of quality when the same prompt is queried multiple times, and the capacity of the Content at Scale AI detector in identifying AI-generated requirements. Our findings reveal that ChatGPT, in both its free and premium versions, holds potential for generating software requirements, albeit it lags behind human experts in some quality attributes. However, the use of feedback mechanisms helps enhance the AI’s performance. We also found that the premium version of ChatGPT outperforms its free counterpart in consistency and overall quality, except in an initial trial where the free version showed superior performance on certain attributes. ChatGPT demonstrated remarkable time efficiency compared to human participants without compromising significantly on the quality of generated requirements. Despite this, requirements produced by experienced human practitioners were found to be more detailed and comprehensive. This suggests that AI tools like ChatGPT should supplement rather than replace human expertise in requirement engineering. Our study also assessed the Content at Scale AI detector’s proficiency in recognizing AI-generated requirements, uncovering high precision but low recall. This highlights a need for improved detectors for reliable identification of AI-generated requirements. In summary, this thesis underscores the importance of AI tools like ChatGPT in efficient requirement generation while emphasizing the irreplaceable value of human expertise and feedback mechanisms in optimizing AI’s performance. Furthermore, it identifies a crucial need for the development of enhanced AI detectors for the accurate identification of AI-generated requirements.