Protocol-aware fuzzing of DTLS

Abstract: Communication protocols like TCP and UDP enable transfer of databetween sender and receiver during online communication. To provide security from eavesdropping, message forgery and data tampering during online communication, communication protocols likeTLS and DTLS are run on top of TCP/UDP. It is therefore important tokeep the implementations of these security-providing communication protocols exploit free to ensure secure communication. One way to keep implementations exploit free is to fuzz them to find anypotential memory leaks, crashes and failed assertions. In this project we do stateful fuzzing of tinyDTLS [3], that is a library for the Datagram Transport Layer Security (DTLS) [16] protocol that is widely used in applications such as VOIP, MMO gaming, etc. We look athow a grammar-aware fuzzer, in our case AFLSmart [15], compares to anon grammar-aware fuzzer, in our case AFL [20], and if it can provide better fuzzing results. The fuzzers are compared on paths found and lines, functions and branches covered after a 24 hour fuzzing run.Stateful fuzzing allows us to put the protocol implementation in aparticular state before starting fuzzing. To perform stateful fuzzing a framework for tinyDTLS [1] will be utilized, which allows fuzzing from any state of the tinyDTLSimplementation. The tinyDTLS harness extends TLS-Attacker [19]. The main work of the project will be to write a grammar file for all the communication messages that will be used by AFLSmart for fuzzing, run experiments with the grammar and analyze the performance of the two fuzzers.