Automatic analysis for continuous integration test failures

Abstract: CI (Continuous Integration) is a software development practice which became more and more popular in last decade. Ericsson followed the trends and used CI several years. Because of the complexity of RBS (Radio Base Station) software few levels of CI have been implemented there. In RCS (RBS Control System) module CI there are many automatic JCAT (Java Common Auto Tester) test loops running every day and some of them failed. This thesis tries to find a way to classify these test failures automatically, so efficiency and lead time can be improved. Two methods are presented and investigated in this report, rule matching and machine learning. After analysis and comparisons rule matching approach is selected because it does not require huge effort in the initial phase and rule matched data can be used as labeled data for machine learning. This approach requires manual work to add new rules continuously but with correctly defined rules the accuracy is 100%, if the rule is general it can classify one type of issue including the ones which never happen before. One analysis system is designed and implemented, and only small update is required to the result report block of the CI flow. One matching example is showed and according to estimation this method could save many man hours every year.