Implementation & architecture of a cloud-based data analytics pipeline

Abstract: Organizations can gain deeper insight into the usage of their products and services by analyzing data from events as they happen in real-time. Examples of such data include user login attempts, feature usage, sign-ups, client logs as well as other domain and application specific events. The high frequency at which these types of events are generated combined with the large volume of data they will occupy once stored provides many interesting challenges. Most importantly, it is difficult to predict what type of analytics data will provide useful insights in the future. This master thesis documents the architecture of a data analytics pipeline based on the principles of decoupled components to meet this goal. It is shown that by extending the concepts of the publish & subscribe pattern and by leveraging cloud-based services the development of such a system is achievable even for smaller organizations.