Test Maintenance for Machine Learning Systems: A Case Study in the Automotive Industry

Abstract: Machine Learning (ML) is widely used nowadays, including safety-critical systems in the automotive industry. Consequently, testing is essential to ensure the quality of these systems. Compared to traditional software systems, ML systems introduce new testing challenges with the potential to affect test maintenance—the action of updating test cases. Test maintenance is important to keep the tests consistent with the system under test and reduce the number of false-positive tests. This exploratory case study was conducted with Zenseact, a company in the automotive industry with a focus on software for autonomous driving. The study included a thematic interview analysis and an artifact evaluation to identify factors that affect test maintenance for ML systems, examine the influence of flaky tests on test maintenance, and provide recommendations on how to improve the test maintenance process. We identified 14 factors, including five especially relevant for ML systems—the most frequently mentioned factor was non-determinism. There were no clear indications that flaky tests have a different effect on test maintenance for ML systems compared to traditional systems. We also proposed ten recommendations on improving test maintenance, including four specifically suitable for ML systems. The most important recommendation is the use of oracle tolerances to deal with variations caused by non-determinism, rather than using a strictly-defined oracle. The study’s findings can help fill an existing gap in the literature about test maintenance factors. Practitioners can profit from industry insights that increase awareness of potential test maintenance issues and give inspiration on how to approach challenges in the context of test maintenance for ML systems.