Secure log-management for an Apache Kafka-based data-streaming service

Abstract: This thesis aims to investigate the prospect of using Apache Kafka to manage data streams based on secrecy/classification level and separate these data streams in order to meet the requirement set by the secrecy/classification levels. Basalt AB has the responsibility of managing classified data for private and state actors, including the Swedish Armed Forces and other organizations. There is interest in a data-streaming solution that can securely stream large amounts of data while coordinating different data classifications and managing user access. This thesis work examines the viability of logically and physically separating producer data streams into categories based on the classification level of the data in an Apache Kafka cluster. Additionally, the thesis examines the viability of managing access control through the use of Access Control Lists. To protect against embedded attackers this thesis examines the viability of using Shamir Secret Sharing (SSS) algoritm to segment messages to on top of that use multi-factor authentication to ensure that messages cannot be read by a lone individual. The work seeks to contribute to the existing body of knowledge by improving the security, and ensuring the integrity of data through the application of detailed or granular user management of event-logs in an Apache Kafka cluster. This is of interest to organizations that require protection from both external and internal potential attackers. Our results indicate that Apache Kafka is an appropriate tool for data streaming secret data, we used a secret sharing algorithm to segment data and used Simple Authentication and Security Layer to build a multi-factor authentication system.