Emotion recognition with Deep Learning and its correlation with veracity

Abstract: Emotion recognition is an interesting and important field of study in many professional fieldssuch as medicine and law enforcement. This topic of study has many different challenges toexplore and we chose a few of these to expand upon. We created and trained a ConvolutionalNeural Network (CNN) that could classify the 7 basic emotions (anger, disgust, fear, happiness,sadness, surprise, and neutral). We explored what parts of the face contribute to discerningexpressions, and we also looked into whether there was a meaningful correlation between the emotions present and the veracity of a person’s statements.  Using the Facial Expression Recognition 2013 (FER2013) data set and the Expressions in the Wild (ExpW) data set we achieved an accuracy of 65.5% on the FER2013 data, which was in the same area as the human accuracy on the FER2013 data set which was said to be between 65-68% accurate [1]. For the ExpW data we looked at both the colour and greyscale images and achieved 79.21% top1k and 88.64% top2k accuracy on the colour images, and 83.07% top1k and 90.87% top2k accuracy on the greyscale. With the use of the attention map GradCam and occluded images we could see that the eyes and mouth were very important when classifying expressions and that with the exclusion of the eyes, accuracy could drop to zero when discerning the more negative emotions such as anger, disgust, and fear. The Deception in Real-life (2016) data set with videos from court cases was used to find that while there were some clear trends with what emotions were more prominent whether the data was truthful or deceitful, the results were not decisive enough that given an emotion or emotion distribution we could definitively say that the person was being truthful or not. To do this it would have to be in conjunction with other techniques for lie detection.