Detection of Vulnerability Scanning Attacks using Machine Learning : Application Layer Intrusion Detection and Prevention by Combining Machine Learning and AppSensor Concepts

Abstract: It is well-established that machine learning techniques have been used with great success in other domains and has been leveraged to deal with sources of evolving abuse, such as spam. This study aims to determine whether machine learning techniques can be used to create a model that detects vulnerability scanning attacks using proprietary real-world data collected from tCell, a web application firewall. In this context, a vulnerability scanning attack is defined as an automated process that detects and classifies security weaknesses and flaws in the web application. To test the hypothesis that machine learning techniques can be used to create a detection model, twenty four models were trained. The models showed a high level of precision and recall, ranging from 91% to 0.96% and 85% to 0.93%, respectively. Although the classification performance was strong, the models were not calibrated sufficiently which resulted in an underconfidence in the predictions. The results can therefore been viewed as a performance baseline. Nevertheless, the results demonstrate an advancement over the simplistic threshold-based techniques developed in the early days of the internet, but require further research and development to tune and calibrate the models.