Unsupervised real-time anomaly detection on streaming data for large-scale application deployments

Abstract: Anomaly detection is the classification of data points that do not adhere to the familiar pattern; in previous studies there existed a need for extensive human interactions with either labelling or sorting normal and abnormal data from one another. In this thesis, we want to go one step further and apply machine learning techniques on time-series data in order to have a deeper understanding of the properties of a given data point without any sorting and labelling. In this thesis, a method is presented that can successfully find anomalies in both real and synthetic datasets. The method uses a combination of three algorithms from various disciplines, Hierarchical temporal memory and Restricted Boltzmann machines from machine learning and Autoregressive integrated moving average from regression. Each algorithm is specialised in finding a particular type of anomalies. The combination finds all anomalies with little to no gap from the occurrence of an anomaly to its detection.