Analysis of DTLS Implementations Using State Fuzzing

Abstract: Despite being unreliable, UDP is the prefered choice for a growing number ofimplementations due to its low latency. Hence, a means for protecting sensitive datasent over UDP is essential. The protocol most commonly used for this purpose is Datagram Transport Layer Security (DTLS). DTLS is an extension of thecryptographic TLS protocol used to secure datagram transport protocols such asUDP. The challenges of supporting unreliable transport protocols increases thecomplexity of DTLS implementations. While implementations of TLS have received alot of scrutiny, the same cannot be said for DTLS, leaving DTLS implementationspotentially vulnerable. An analysis framework, dtls-fuzzer, has been developed toanalyze server implementations of DTLS. The framework uses state fuzzing, atechnique for automatically learning a state-machine model of a system and checkingthe model against the system’s specification to find bugs. Using this framework,several bugs and security vulnerabilities were found in popular DTLS serverimplementations.DTLS client implements were not covered, however, leaving a gap to fill. In orderto plug this gap, I extend the dtls-fuzzer framework to also allow for comprehensiveanalysis of client implementations through the use of state fuzzing. I then use theextended framework to learn state-machine models of four client implementations:Contiki-NG TinyDTLS, MbedTLS, OpenSSL and Scandium. The learned modelsuncover several bugs highlighting deviations from the DTLS standard, such asincorrect client authentication in MbedTLS and OpenSSL, and a message sequencebug in OpenSSL. These bugs hint at structural flaws within the implementations,indicating that further testing of client implementations is called for.