Comparing Static and Dynamic Synchronization of GUI-based tests: An Industrial study

Abstract: Background. Speed is getting more and more critical in modern Software Engineering to be able to respond to users’ expectations of product development. One practice that takes a significant amount of time in the process of releasing software to the customers is testing. It is a clear trend that organizations are increasing the amount of test automation compared to manual testing. However, manual testing is still prominent in GUI-based testing due to challenges interacting with a GUI from test scripts. One of the most prominent challenges is synchronizing test script execution with the system under test. Objectives. This research aims to compare static and dynamic synchronization of GUI-based tests. This comparison will be conducted by replicating an existing Selenium test suite using static synchronization to Playwright, which will use dynamic synchronization and run the test suites against a web-based application. These test suites will then be used to compare the two types of synchronization from the perspective of test execution efficiency, test output correctness, and maintenance cost.Methods. The research methodology we chose is experiment. We have chosen Selenium to represent static synchronization and Playwright to represent dynamic synchronization. We used an existing test suite in Selenium that was translated into Playwright. There are a total of 81 tests in the test suite. The test suites were used to compare test scripts that use static and dynamic synchronization from the perspective of test execution efficiency, test output correctness, and maintenance cost.Results. The data collected from the experiments shows that execution efficiency for test scripts using dynamic synchronization is significantly faster than static synchronization. A mean difference between the test suites showed a decrease of 87%. For defect identification, ten defects where used and both test suites managed to identify all of them, resulting in no difference could be found. Test maintenance cost was found that test scripts using dynamic synchronization had a positive effect with an average of 60% less time spent on maintenance.Conclusions. Based on the result, we found that test scripts using dynamic synchronization improved execution efficiency and maintenance costs without sacrificing the test output correctness.