Non-deterministic tests and where to find them : Empirically investigating the relationship between flaky tests and test smells by examining test order dependency

Abstract: Flaky tests are non-deterministic tests that both pass and fail when no new code changes have been introduced to the code under test. This is a widespread problem for the Continuous Integration community where the underlying idea is that code is only integrated if all test cases pass. Debugging and eradicating flaky tests is hard and time-consuming and if ignored may introduce bugs. Due to flaky tests, Continuous Integration systems are at risk of decreasing either productivity or quality, properties it claims to provide. Prior research suggests a link between 4 root causes for flaky tests and 5 different test smells, i.e. anti-code patterns. Unfortunately, difficulties in reproducing flaky tests found in the study make this connection ambiguous. This thesis intends to validate this relationship by re-implementing the test smells in a static analyser named FlakyHoover and detecting them on a new dataset consisting of order-dependent and non-order-dependent flaky tests. Analysis of the test smell distribution over order and non-order-dependent flaky tests makes it possible to determine the relationship between test smells and root causes for flaky tests. The findings suggest that there may exist a correlation between 3/5 test smells and certain types of root causes for flaky tests. However further research is required to determine the studied test smells relation to flaky tests.