Foveated Facial Interaction Using Neural Networks and Eye Gaze

Abstract: Non-player characters (NPC) in games are highly scripted and seldom feel natural, one aspect that is missing to produce more human-like NPCs is more believable eye movements. But scripting eye movements for every single char- acter is time-consuming, and the end result may still feel unnatural due to the complexity of human gaze behaviour. Machine learning is an alternative to scripts, by using a large amount of data to find complex patterns and return predictions on new data. By gathering data in a real-life environment and using machine learning, a reactive gaze generation prototype coupled with a demo in VR has been developed. The thesis shows the processes and modules needed to produce such a system and provides insight into the process of developing a machine learning model for a complex problem. The resulting prototype together with the demo presents organic gaze features such as Mutual gaze, Gaze aversion and Joint attention in a stochastic manner. However the NPC in the demo could not be said to have human behaviour, even though you could attach personalities to the NPC, more work has to be conducted to produce acceptable natural eye gaze behaviour by adding more features, evaluate the use of other datasets and try other algorithms. The difficulties of classifying realism was encountered as a problem to be able to further develop the prototype, and larger experiments have to be conducted to evaluate the results for further improvements.