Design of a Network Library for Continuous Deep Analytics

Abstract: Data-intensive stream processing applications have escalated in popularity in recent years, producing numerous designs and implementations for handling unbounded streams of high-volume data. The sheer size and dimensionality of these types of data requires multiple machines to push processing throughput of hundreds of millions events per second at low latencies. Advances in the fields of distributed deep learning and stream processing have highlighted networking-specific challenges and requirements such as flow control and scalable communication abstractions. Existing stream processing frameworks, however, only address subsets of these requirements. This thesis proposes a design and implementation in the Rust programming language for a modular networking library able to address these requirements together. The design entails protocol framing, buffer management, stream multiplexing, flow control, and stream prioritization. The implemented prototype handles multiplexing of logical streams and credit-based flow control through a flexible application programming interface. The prototype is tested for overall throughput and round-trip latency in a distributed environment, displaying promising results in both categories.