Performance of message brokers in event-driven architecture: Amazon SNS/SQS vs Apache Kafka

Abstract: Microservice architecture, which involves breaking down applications into smaller and loosely coupled components, is becoming increasingly common in the development of modern systems. Connections between these components can be established in various ways. One of these approaches is event-driven architecture, where components in the system communicate asynchronously with each other through message queues.  AWS, Amazon Web Services, the largest provider of cloud-based services, offers such a messaging queue: SQS, Amazon Simple Queue Service, which can be integrated with SNS, Amazon Simple Notification Service, to enable "one-to-many" asynchronous communication.  An alternative tool is Apache Kafka, created by LinkedIn and later open-sourced under the Apache Software Foundation. Apache Kafka is an event logging and streaming platform that can also function as a message queue in an event-driven architecture.  The authors of this thesis have been commissioned by Scania to compare and evaluate the performance of these two tools and investigate whether there are use cases where one might be more suitable than the other. To achieve this, two prototypes were developed, each prototype consisting of a producer microservice and a consumer microservice. These prototypes were then used to conduct latency and load tests by producing messages and measuring the time interval until they were consumed.  The results of the tests show that Apache Kafka has a lower average latency than SNS/SQS and scales more efficiently with increasing data volumes, making it more suitable for use cases involving real-time data streaming. Its potential as a message bus for loosely coupled components in the system is also evident. In this context, SNS/SQS is equally valuable, as it operates as a dedicated message bus with good latency and offers a user-friendly and straightforward setup process.