Comparison of adversary emulation tools for reproducing behavior in cyber attacks

Abstract: As cyber criminals can find many different ways of gaining unauthorized access to systems without being detected, it is of high importance for organizations to monitor what is happening inside their systems. Adversary emulation is a way to mimic behavior of advanced adversaries within cyber security, which can be used to test detection capabilities of malicious behavior within a system of an organization. The emulated behavior can be based on what have been observed in real cyber attacks - open source knowledge bases such as MITRE ATT&CK collect this kind of intelligence. Many organizations have in recent years developed tools to simplify emulating the behavior of known adversaries. These tools are referred to as adversary emulation tools in this thesis. The purpose of this thesis was to evaluate how noisy different adversary emulation tools are. This was done through measurements on the amount of event logs generated by Sysmon when performing emulations against a Windows system. The goal was to find out which tool was the least noisy. The different adversary emulation tools included in this thesis were Invoke-AtomicRedTeam, CALDERA, ATTPwn and Red Team Automation. To make sure the correlation between the adversary emulation tools and the generated event logs could be identified, a controlled experiment was selected as the method for the study. Five experiments were designed including one emulation scenario each, executed by the different adversary emulation tools included in each experiment. After each emulation, event logs were collected, filtered, and measured for use in the comparison. Three experiments were conducted which compared Invoke-AtomicRedTeam, CALDERA, and a manual emulation. The results of the first three experiments indicated that Invoke-AtomicRedTeam team was the noisiest, followed by CALDERA, and the manual emulation was the least noisy. On average, the manual emulation generated 83,9% fewer logs than Invoke-AtomicRedTeam and 78,4% fewer logs than CALDERA in experiments 1-3. A fourth experiment compared Red Team Automation and Invoke-AtomicRedTeam, where Red Team Automation was the least noisy tool. The final fifth experiment compared ATTPwn and CALDERA, and the results indicated that these were similarly noisy but in different ways. It was also concluded that a main difference between the adversary emulation tools was that the number of techniques available differed between the tools which could limit the ability to emulate the behavior of real adversaries. However, as the emulation tools were implemented in different ways, this thesis could be one starting point for future development of silent adversary emulation tools or to assist in selecting an existing adversary emulation tool.