Automatic Description of AI Patches With AST Analysis

Abstract: Using artificial intelligence (AI) to write source code is soon to be reality. One research areais to use AI to automatically generate a patch that fixes a bug. However, most such tools onlygenerate the patch without any natural language description of it, making it difficult for thereviewer to understand the patch. Therefore in our work, we aim to develop a program thatgenerates descriptions of code patches generated by artificial intelligence. The program takestwo Java files as input and analyses the differences in the abstract syntax tree representationsof the code. First, the program finds what changes have been made and where those changesare located. Then, it generates a description of the change. The program has been tested on 50patches from open-source projects from GitHub. The generated comments were comparedwith comments generated by ChatGPT as well as those written by humans in a survey. Thesurvey also asked the participants to rate how useful the generated comments were in helpingthem understand the patches generated by artificial intelligence. The result shows that thecomments generated with our approach received 34.7 % of the participant votes whereas theexisting human-written descriptions received 20.8 % of the votes, although the generateddescriptions were sometimes thought to be too long when describing simple changes. Theprogram-generated descriptions were also rated a 3.09 out of 5 on average in how helpfulthey were in understanding patches generated by artificial intelligence.