ChatGPT: A gateway to AI generated unit testing

Abstract: This paper studies how the newly released AI ChatGPT can be used to reduce the time and effort software developers spend on writing unit tests, more specifically if ChatGPT can generate quality unit tests. Another aspect of the study is how the prompting of ChatGPT can be optimized for generating unit tests, by creating a prompt framework. Lastly how the generated unit tests of ChatGPT compare to human written tests was tested. This was done by conducting an experiment where ChatGPT was prompted to generate unit tests for predefined code written in C# or Typescript which was then evaluated and rated. After the generated unit test had been rated, the next steps were determined, and the process was repeated. The results were logged following a diary study. The rating system was constructed with the help of previous research and interviews with software developers working in the industry which defined what a high-quality unit test should include. The interviews also helped in understanding ChatGPT’s perceived capabilities. The experiment showed that ChatGPT can generate unit tests that are of quality, though with certain issues. For example, reusing the same prompt multiple times revealed that the consistency in the responses was lacking. Inconsistencies included different testing approaches (how setup methods were used for example), testing areas and sometimes quality. The inconsistencies were reduced by using the deduced prompt framework, but the issue could be a current limitation of ChatGPT which could be handled with a future release.