Compiler Testing by Random Source Code Generation

Abstract: Most software projects today are written using programming languages. Compilers in turn translate programs written in these higher level languages into machine code, executable on actual hardware. Ensuring that these compilers function correctly is therefore paramount. Manually written test suites make sure that compilers functions correctly for some inputs, but can never hope to cover every possible use case. Thus, it is of interest to find out how other testing techniques can be applied. This project aimed to implement a random test program generator for Configura Magic (CM), a proprietary programming language used at Configura. Our tool is inspired by the widely successful C program generator Csmith. It is implemented by randomly generating an abstract syntax tree (AST) and unparsing it to produce correct code. Our tool found about 3 bugs in the CM compiler, Configura Virtual Machine (CVM), during its development.CVM was instrumented to get code coverage data after compiling programs. Compiling the CVM test suite (CTS) and Configura's main product CET (Configura Extension Technology)cover about 23% and 19% of the compiler respectively, while compiling programs generated by our tool only cover about 6%. But on the other hand, our generated programs uniquely cover about 0.2% that is not covered by CTS and CET. A backend for our tool that generates C-code was also implemented, to compare it against Csmith. The results show that on average (100 program generations 30 times, for a total of 3000 programs), our tool gets about 45% coverage while Csmith gets about 50% on the small C compiler TinyCC. Although our tool was mildly successful in finding bugs, the comparison between it and Csmith shows its potential to be even more effective.