The State of Live Facial Puppetry in Online Entertainment

Abstract: Avatars are used more and more in online communication, in both games and socialmedia. At the same time technology for facial puppetry, where expressions of the user aretransferred to the avatar, has developed rapidly. Why is it that facial puppetry, despite this,is conspicuous by its absence? This thesis analyzes the available and upcoming solutions for facial puppetry, if a com-mon framework or library can exist and what can be done to simplify the process for de-velopers who wants to implement facial puppetry. A survey was conducted to get a better understanding of the technology. It showedthat there is no standard yet for how to describe facial expressions, but part of the marketis converging towards a common format. It also showed that there is no existing inter-face that can handle communication with tracking devices or translation between differentexpression formats. Several prototypes for recording and streaming facial expression data from differentsources were implemented as a practical test. This was done to evaluate the complexity ofimplementing real-time facial puppetry. It showed that it is not always possible to integratethe available tracking solutions into an existing project. When integration was possible itrequired a lot of work. The best way to get tracking right now seems to be to implement astandalone program for tracking that streams the tracked data to the main application. In summary it is the poor integrability of the solutions that makes it problematic forthe developers, together with a wide variety of facial expression formats. A software thatacts like a bridge between the tracking solutions and the game could allow for translationbetween different formats and simplify implementation of support. In the future, instead of working towards making all tracking solutions output stan-dardized tracking data, research further how to build a framework that can handle differ-ent configurations.