Automated testing of firmware installation and update scenarios for peripheral devices

Abstract: This research presents an approach to transition from manual to automated testing of hardware specific firmware. The manual approach for firmware testing can be repetitive and time consuming. A significant proportion of the time is spent on cleaning and re-installing operating systems so that old firmware does not interfere with the newer firmware that is being tested. The approach in this research utilizes virtual machines and presents an automation framework. One component of the automation framework is an application to imitate connected peripheral devices to bypass hardware dependencies of firmware installers. The framework also consists of automation and pipeline scripts with the objective to execute firmware installers and detect errors and abnormalities in the installation and updating processes. The framework can run on locally hosted virtual machines, but is most applicable using cloud hosted virtual machines, where it is part of a continuous integration that builds, downloads, installs, updates and tests new firmware versions, in a completely automated manner. The framework is evaluated by measuring and comparing execution times with manually conducted installation and updating tests, and the result shows that the framework complete tests much faster than the manual approach. Another evaluation in terms of scalability is made where speedup is measured. It is however concluded that scalability does not necessarily depend on the performance and resource allocation of systems, but rather how well it can be expanded and used for future devices and functionalities.