Automated GUI regression tests via remote desktop with Robot Framework and Sikuli

Abstract: To test software over a remote desktop is an increasingly common practice since outsourcing and offshoring of testing activities are on the rise. Most companies useproprietary testing tools which require an agent to be installed on the remote machine.This installation opens several attack surfaces and has thus the potential to compromise security. In addition license, support and other proprietary test tool costs might reduce the return on investment of automated tests to an unprofitable level, especially for relatively short projects.Robot Framework with Sikuli integration (RFSI) is a concept based on two open source tools which offer solutions concerning the security and cost issues. RFSI based tests do not need any further installations, access rights or network configurations on the remote machine, because of the fact that the tests are based on visual input. Therefore tests with RFSI cannot access the AUT (Application Under Test) or the remote machine in any other way than a manual tester could. Open-source tools, being freely available, also escape the problem of licence costs. The purpose of the proof of concept pilot study wasto give an answer on the question if RFSI can furthermore be considered an alternative to proprietary tools in terms of requirements for tool quality characteristics and for the external context of test automation projects.The methodology of this research followed the broader Goal-Question-Metrics method(GQM) described by Rini van Solingen and Egon Berghout, based on the GQM approachby Victor Basili and David Weiss. The requirements and metrics for the evaluation of RFSI in the study were selected through a literature study of ISO/IEC-TR9126-3 and other articles in the field. They comprehend six categories: functionality, reliability,maintainability and stability, efficiency, understandability and portability. For each category,metrics were established and adapted to the context of the proof of concept. The scope of the project was given by six use cases and test specifications, established in cooperation with system experts at the company CGI Sundsvall. These were then translated into RFSI test scripts and implemented with the requirements for successful remote GUI testing in mind.The results of the measurements for the proof of concept study showed that RFSI fulfilles the internal and external requirements for successful automated testing completely for functionality, understandability and portability. The requirements are partially satisfiedfor reliability, maintainability and stability, and efficiency.In conclusion, RFSI satisfies enough requirements to be considered a viable alternative to commercial testing tools for tests of GUI (Graphical User Interface) desktop applications over a remote desktop.