Amplifying heap overflow vulnerability detection with reinforcement learning

Abstract: The extensive development of cyberspace and the increasing potential for cybersecu-rity vulnerabilities demand the constant production of improved methods for detect-ing and mitigating vulnerabilities in software. In a perfect world, there would be atool that detects and mitigates all types of vulnerabilities in all types of software, butunfortunately, that is not the reality. Most methods need to be specific to have goodperformance. The tool we use in our paper specializes in detecting vulnerabilities inexecutable programs, specifically heap buffer overflow vulnerabilities.In this master thesis, we focus on the problem of detecting heap buffer overflowvulnerabilities in executable programs. We conducted two experiments to answertwo research questions related to this problem. The first research question aims toevaluate the performance of a unit-based symbolic execution method for detectingsuch vulnerabilities in terms of accuracy and execution time. The second researchquestion investigates whether the performance of the method from the first questioncan be improved through the use of the machine learning method Q-learning.In the first experiment, we used the 90 included test programs to evaluate theoriginal version of the tool. For our second experiment, we used 100 other testprograms that we selected from the NIST database, together with the original versionof the tool and our modified version with integrated Q-learning functionality. Thefindings from our experiments show that unit-based symbolic execution tools arecomplex, and the accuracy of these tools can be improved through the use of machinelearning algorithms. However, the use of these algorithms comes at the cost ofexecution time.Overall, this thesis contributes to the field of software security by providing in-sights into the performance and potential improvements of symbolic execution meth-ods for detecting heap buffer overflow vulnerabilities. Our findings suggest that theuse of machine learning algorithms can enhance the accuracy of unit-based symbolicexecution tools, which can be useful for detecting security vulnerabilities in software.