Nuking Duke Nukem : Reaching the Stack via a Glboal Buffer Overflow in DOS Protected Mode

Abstract: Control-flow hijack attacks on software exploit vulnerabilities in the software’s memory handling. Over the years, various security mitigations have been developed to counter these attacks. However, compatibility issues have hindered the adoption of such measures in some legacy systems. This thesis focuses on the case of the legacy DOS system and examines whether a DOS system running the DOS/4GW protected mode extender can provide control-flow protection against an attack exploiting a buffer overflow vulnerability in the well-known retro game Duke Nukem3D. To investigate this, three model programs were created, and designed with memory models that share memory layout characteristics with the target retro game’s executable. Experimental attacks were then conducted on these models, aiming to identify an effective attack vector for the target vulnerability. The underlying theory suggests that memory models that segregate application data into distinct memory segments could potentially safeguard against the demonstrated attack. However, attempts to implement such a memory model within an application proved unsuccessful. The challenge that remains is to prove the existence of memory models under DOSprotected mode that can effectively shield Duke Nukem 3D, or other legacy games, from the control-flow hijack attack demonstrated in this thesis.