Type-directed Generation and Shrinking for Imperative Programming Languages

Abstract: Optimizing compilers are large and complex systems, potentially consisting of millions of lines of code. As a result, the implementation of an industry standard compiler can contain several serious bugs. This is alarming given the critical part that software plays in modern society. Therefore, we investigate type-directed program generation and shrinking for the testing of compilers for imperative programming languages. We implement a type-directed generator and shrinker for a subset of C in Haskell. The generator is capable of generating type-correct programs that contain variable mutations and challenging program constructs, such as while-statements and if statements. We evaluate the quality of the generated programs by measuring statement coverage of a reference interpreter and by comparing the results to reference test suite. The results show that the generated programs can surpass the test suite in terms of quality by one percentage point of coverage. To test the bug finding capabilities of the generator and the shrinker, we use them to test seven interpreters and manage to identify a total of seven bugs. Furthermore, we test the same interpreters with the reference test suite and find only three bugs. The implications of the results lead us to question the reliability of code coverage as a measure of quality for compiler test suites. Finally, we conclude that type-directed generation and shrinking is a powerful method that can be used for the testing of compilers for imperative programming languages.