An Evaluation of Model-Based Testing for an Industrial Train Control Software

Abstract: Currently, the increasing complexity of software and the short release cycles are becoming a challenge for testing software in an efficient and effective way. Traditionally, creating tests is done manually by engineers, which are then automatically or manually executed on the actual software. Manually creating test cases is a time-consuming effort. For the last couple of decades, researchers have proposed ways to improve this process by automating parts of the testing steps. One of these approaches that have gained a lot of attention is called Model-Based Testing (MBT). MBT has been suggested as a way of automatically creating tests at a lower cost. Nonetheless, it is not very well studied how MBT is actually applied in industrial contexts and how these tests compare to manually written ones. This is particularly true for industrial control software such as the one found in the train domain, where strict requirements on testing are in place. In this thesis, we investigate the literature and review the related work on case studies on the MBT use in industry and its evaluation. We perform a case study to evaluate MBT on a train control management system provided by Bombardier Transportation. We use Comformiq Creator MBT Tool to create models for functional requirements of a master controller function and generate test cases. We provide the result of the modeling approach as well as a comparison between automatic test cases created by Conformiq Creator and manual test cases written by industrial engineers at Bombardier Transportation using the following metrics: test coverage and time spent on testing. The results of this comparison suggest that test coverage of MBT is higher and test cases are more detailed than manual testing. Our results are not conclusive in regard to the cost of using MBT, mainly because this depends on different testing scenarios and how testing is performed. We show that MBT is a suitable approach for modeling the functional requirements of a realistic industrial control software function. In this thesis work, we focus on system-level testing. As future work, applying MBT on lower levels of testing can be a promising way forward for evaluation. In addition, the transformation of these test cases into executable test scripts and the possible problems needs to be investigated further.