Empirical test of a tool for cyber security vulnerability assessment

Abstract: This report describes a study aimed at verifying a cyber security modeling language named the Predictive, Probabilistic Cyber Security Modelling Language. This modeling language together with the Enterprise Architecture Analysis Tool acts as a tool for cyber security evaluations of system architectures. To verify the accuracy and readiness of the tool, a generic model of a real life Supervisory Control And Data Acquisition System’s system architecture was modeled using the tool and later evaluated. The evaluation process consisted of a Turing test, which was the same method used for evaluation of the Predictive, Probabilistic Cyber Security Modelling Language predecessor the Cyber Security Modelling Language. For the Turing test, interviews were held with five domain experts within cyber security. Four of which were tasked with creating attack paths given a scenario in the modeled system architecture. The Predictive, Probabilistic Cyber Security Modelling Language was given the same task as the four experts. The attack paths created were consolidated in a standardized form for the last internal company expert within cyber security to evaluate. An expert evaluator was tasked with grading the attack paths produced by the four experts and the Predictive, Probabilistic Cyber Security Modelling Language. The grading was based on how probable the attack paths were perceived by the internal expert.  The conclusion was made that given the limitations of the study, the Predictive, Probabilistic Cyber Security Modelling Language produced a cyber security evaluation that was as probable as those created by the human cyber security experts. The results produced were also consistent with the results produced by the Predictive, Probabilistic Cyber Security Modelling Language predecessor the Cyber Security Modelling Language in a previous study. Suggestions for further studies were also introduced which could complement this study and further strengthen the results. This thesis was a collaboration between ABB Enterprise Software and the members of the team behind the Predictive, Probabilistic Cyber Security Modelling Language at ICS at KTH.