Information-Theoretic Framework for Network Anomaly Detection: Enabling online application of statistical learning models to high-speed traffic

Abstract: With the current proliferation of cyber attacks, safeguarding internet facing assets from network intrusions, is becoming a vital task in our increasingly digitalised economies. Although recent successes of machine learning (ML) models bode the dawn of a new generation of intrusion detection systems (IDS); current solutions struggle to implement these in an efficient manner, leaving many IDSs to rely on rule-based techniques. In this paper we begin by reviewing the different approaches to feature construction and attack source identification employed in such applications. We refer to these steps as the framework within which models are implemented, and use it as a prism through which we can identify the challenges different solutions face, when applied in modern network traffic conditions. Specifically, we discuss how the most popular framework -- the so called flow-based approach -- suffers from significant overhead being introduced by its resource heavy pre-processing step. To address these issues, we propose the Information Theoretic Framework for Network Anomaly Detection (ITF-NAD); whose purpose is to facilitate online application of statistical learning models onto high-speed network links, as well as provide a method of identifying the sources of traffic anomalies. Its development was inspired by previous work on information theoretic-based anomaly and outlier detection, and employs modern techniques of entropy estimation over data streams. Furthermore, a case study of the framework's detection performance over 5 different types of Denial of Service (DoS) attacks is undertaken, in order to illustrate its potential use for intrusion detection and mitigation. The case study resulted in state-of-the-art performance for time-anomaly detection of single source as well as distributed attacks, and show promising results regarding its ability to identify underlying sources.