Challenges in Specifying Safety-Critical Systems with AI-Components

Abstract: Safety is an important feature in automotive industry. Safety critical system such as Advanced Driver Assistance System (ADAS) and Autonomous Driving (AD) follows certain processes and procedures in order to perform the desired function safely. Many ADAS applications relies significantly on Machine Learning and data needed to perform the desired function. Data quality, more specifically the information content of the data, can highly impact the effectiveness of the model and its function. It is important to select the right data to train the model. Furthermore, monitoring the safety critical system during runtime helps to understand the data which the model receives. Such information helps further to create and update machine model. There are uncertainties and challenges in defining the requirements for finding the right information content of the data such that the desired and a safe behaviour of the system is ensured. This case study investigates and explores the challenges experienced in creating the requirements for proper selection of training data. It also analyzes challenges when specifying runtime monitoring and the relation between requirements on runtime monitoring and the training data. This case study follows the approach of qualitative and exploratory research. The analysis for this study is based on ten interviews with experts from different field. Moreover, a workshop has been conducted with academic and industry experts to validate the results from our interview analysis. Based on the qualitative analysis of data, the case study shows that there is lack of clarity in defining requirements, lack of communication, no clear scope of design domain, missing guidelines for data selection and safety requirements, and a lack of metrics for defining the right variety of data and runtime monitors. The results outline challenges experienced by practitioners when specifying data and defining requirements for runtime monitors for safety critical systems.