A Real-Time Reactive Platform for Data Integration and Event Stream Processing

Abstract: This thesis presents a Real-time Reactive platform for Data Integration and Event Stream Processing. The Data Integration part is composed of data pullers that incrementally pull data changes from REST data sources and propagates them as streams of immutable events across the system according to the Event-Sourcing principle. The Stream Processing part is a Tree-like structure of event-sourced stream processors where a processor can react in various ways to events sent by its parent and send derived sub-streams of events to child processors. A processor use case is maintaining a pre-computed view on aggregated data, which allows to define low read latency business dashboards that are updated in real-time. The platform follows the Reactive architecture principles to maximize performance and minimize resource consumption using an asynchronous nonblocking architecture with an adaptive push-pull stream processing model with automatic back-pressure. Moreover, the platform uses functional programming abstractions for simple and composable asynchronous programming. Performance tests have been performed on a prototype application, which validates the architecture model by showing expected performance patterns concerning event latency between the top of the processing tree and the leaves, and expected fault-tolerance behaviours with acceptable recovery times.