Ensemble based unsupervised anomaly detection

Abstract: A methodology as well as a suggested solution to the problem of unsupervised anomaly detection for contextual anomalies is presented. Using a combination of statistical and clustering approaches, an ensemble of algorithms provide automatic anomaly detection in an Application-to-person networking environment which can be scaled to different domains using hierarchical time series data. The aim of this thesis is to further advance the field of anomaly detection and to provide conclusions with regards to the usability, maintainability and trustworthiness of unsupervised anomaly detection frameworks. Applications in the domain of unsupervised anomaly detection are hard to evaluate, thus methods as well as future work, which can be used to further create unmitigated assertions about any data set, is investigated. An introduction to the concepts underlying anomaly detection as well as an implementation of the concepts are presented. Principles of machine learning are applied using static thresholds and assumptions about the data set being monitored. No active learning or dynamic adjustments of the anomaly detection framework is applied with the drawback of limiting the resulting classification but still providing clear and robust insights into the analyzed data. It is shown that purely statistical or naive probabilistic assumptions about any data monitored is inconclusive in producing a fair estimation of anomalies. For a setting where the utility of an anomaly detection framework are not adamant to the survival of a monitoring system, the proposed solution works adequately. Since the results have not been validated, no conclusions can be drawn with regards to recall and precision metrics.