A Framework for Anomaly Detection withApplications to Sequences

Abstract: Anomaly detection is an important issue in data mining and analysis, with applications in almost every area of science, technology and business that involves data collection.The development of general, automated anomaly detection methods could therefore have a large impact on data analysis across many domains. Due to the highly subjective nature of anomaly detection, there are no generally applicable methods, and for each new application a large number of possible methods must be evaluated. In spite of this, little work has been done to automate the process of anomaly detection research for new applications. In this report, a novel approach to anomaly detection research is presented, in which the task of finding appropriate anomaly detection methods for some specific application is formulated as an optimisation problem over a set of possible problem formulations. In order to facilitate the application of this optimisation problem to applications, a high-level framework for classifying and reasoning about anomaly detection problems is introduced. An application of this optimisation problem to anomaly detection in sequences is presented; algorithms for solving general anomaly detection problems in sequences are given, along with tractable formulations of the optimisation problem for the two major anomaly detection tasks in sequences. Finally, a software implementation of the optimisation problem and framework is presented, along with a preliminary investigation into how it can be used to facilitate anomaly detection research.