Automated cyber security compliance assessment

Abstract: Companies and organizations seek to comply to various cybersecurity standards in order to improve their security levels, to followthe requirements of their customers or sometimes obliged by regulatorylaws.The compliance process rely on human assessors and could becomemore effective by automating a subset of the underlying tasksthat constitute it.This study is aimed at creating a software advisor that assessesan IT architecture for compliance to the NERC-CIP standard. Toachieve this a two step process was followed.At the first step specific NERC-CIP requirements were modeledand mapped using the SecuriCAD cyber security modeling tool.Then a software expert able to interpret and compare the Securi-CAD’s abstractions was created. Comparing the mapped rules withthe attributes of a modeled architecture the expert passes judgmenton the compliance status of the system. The second stage validated the advisor and measured its accuracy.That was achieved by conducting a Turing test. During the testanswers produced by the software advisor were compared to thoseof five human domain experts within cyber security for the same assessmenttask. The comparison was assigned to another domainexpert who evaluated the responses without being aware of theirorigin. The responses were graded and ranked from best to worst.The results show that the software expert was able to surpasshuman expertise for the given task and was ranked first along withanother human expert.Also the study contains a section that describes a method of extractingmetrics characterizing the NERC-CIP standard. This derivesfrom the combination of the modeled standard requirementsand the ability of the SecuriCAD tool to simulate cyber attacks andproduce probabilistic security metrics for a given architecture. Thatwas achieved by creating 50 random NERC-CIP compliant architecturesand extracting the average time that a successful adversaryneeds to compromise the system. The results show that half of the the successful attacks againstour compliant sample succeed on 23.19 days on average, while a5% of them succeed in 5.02 days on average.As indicated by the results software assessors can be as mucheffective as human assessors. They can help an organization toprepare for a scheduled assessment and assist human complianceexperts with their judgments. Finally the suggested method of extractingsecurity metrics could be a base for extending to other standardsand making comparisons among them. This would be anadded variable during the process of selecting which cyber securitystandard certification an organization should pursue.