Sync your tests : A qualitative study on strategies for testing asynchronous code

Abstract: The rise of Electrical control units inside of vehicles has led to strict safety regulationsin terms of testing specific software. When the software indicates errors due to inconsistenttests, instead of actual faults, problems arise and a product’s credibility and reliability arequestioned. Previous studies have highlighted that developers spend up to 70% of theirtime maintaining already existing code and that testing accounts for half of the total timedeveloping. As for testing itself, unpredictable tests are a widely researched field with results indicating that there are multiple underlying factors as to why flakiness occurs, oneof which is asynchronous function calls. By applying a design-based research approach wefurther investigate how developers approach testing of asynchronous functionality. Withgathered knowledge, we explore alternative approaches to mitigate the impact of such unpredictable tests while abiding to safety and coding guidelines in the automotive industry.A conducted code base analysis followed by input from experienced developers lead tomultiple implemented improvement points. Subsequently, data extracted from tests facilitated an evaluation where the main focus was put on reliability, performance, and maintainability. Results show that having a general company-wide approach for testing asynchronous code could not only lead to higher maintainability but also increase performance.Major findings indicate that careful selection of various parameters can lead to a test execution time reduction of up to 23% while maintaining high reliability. Our solutions exceedour expectations in terms of maintainability by lowering the amount of physical source linesof code needed by roughly 90%.