Evaluation of different runner set-ups for CI/CD pipelines

Abstract: DevOps and continuous practices are increasingly popular development practices aiming at bridging the gap between software development and IT operations with the indented outcome of shorter development life cycles while maintaining a high software quality. A fundamental part of many DevOps systems is a CI/CD (continuous integration/deployment) pipeline allowing for automatic building, testing and deployment of software. The use of continuous practices have been shown to achieve the desired outcomes, whereas the adopting of such practices has been attributed with the challenges of lacking expertise and skill as well as lacking available tools and technology. Execution of commands in a CI/CD pipeline are handled by a runner application, which can be configured in different ways allowing for different levels of the quality attributes performance, response time, throughput, robustness, stability, resource constraints, cost and maintainability. Five different types of runner infrastructure were implemented and evaluated on the quality attributes. These were: one single-machine implementation, one serverless implementation and three autoscaling implementations. For robustness and stability autoscaling implementations achieved the best results. Performance and throughput were affected by resource constraints which in turn affected the cost. Similar results were found for response time for all but one of the three autoscaling implementations, and for the serverless implementation. Finally, all implementations had similar results for reliability.