Optimizing Consensus Protocols with Machine Learning Models : A cache-based approach

Abstract: Distributed systems offer a reliable and scalable solution for tackling massive and complex tasks that cannot be handled by a single computer. However, standard consensus protocols used in such systems often replicate data without considering the workload, leading to unnecessary retransmissions. This thesis proposes using machine learning (ML) to optimize consensus protocols and make them adaptable to recurring workloads. It introduces a cache that encodes frequently-transmitted data between nodes to reduce network traffic. To implement this, the thesis builds a caching layer at all nodes using the decided logs, which represent a consistent view of the application history. The cache can encode and decode incoming log entries to reduce the average message size and improve throughput under limited network bandwidth. The thesis selects an ML-based model that combines various caching policies and adapts to changing access patterns in the workload. Experimental results show that this approach can improve throughput up to 250%, assuming negligible preprocessing overhead.