Comparative Study of Open-Source Performance Testing tools versus OMEXUS

Abstract: With the development of service digitalization and the increased adoption of web services, modern large-scale software systems often need to support a large volume of concurrent transactions. Therefore, performance testing focused on evaluating the performance of systems under workload has gained greater attention in current software development. Although there are many performance testing tools available for providing assistance in load generation, there is a lack of a systematic evaluation process to provide guidance and parameters for tool selection for a specific domain. Focusing on business operations as the specific domain and the Nasdaq Central Securities Depository (NCSD) system as an example of large-scale software systems, this thesis explores opportunities and challenges of existing open- source performance testing tools as measured by usability and feasibility metrics. The thesis presents an approach to evaluate performance testing tools concerning requirements from the business domain and the system under test. This approach consists of a user study conducted with four quality assurance experts discussing general performance metrics and specific analytical needs. The outcome of the user study provided the assessment metrics for a comparative experimental evaluation of three open-source performance testing tools (JMeter, Locust, and Gatling) with a realistic test scenario. These three tools were evaluated in terms of their affordance and limitations in presenting analytical details of performance metrics, efficiency of load generation, and ability to implement realistic load models. The research shows that the user study with potential tool users provided a clear direction when evaluating the usability of the three tools. Additionally, the realistic test case was sufficient to reveal each tool’s capability to achieve the same scale of performance as the Nasdaq’s in-house testing tool OMEXUS and provide additional value with realistic simulation of user population and user behavior during performance testing with regard to the specified requirements.