The Perception of Mixed Emotions From Dynamic Video-Only and Audio-Only Expressions

Abstract: Although the concept of mixed emotions has shown to be a robust and measurable phenomenon, the perception of mixed emotions has rarely been addressed in previous studies and remains poorly understood. Based on this clear research gap, the present study aimed to explore the perception of mixed emotions expressed through video-only and audio-only modalities. The study tested a novel database consisting of various dynamic clips of portrayed visually and vocally expressed mixed emotions. Fifty-one Swedish-speaking subjects rated mixed expressions of anger, disgust, happiness, sadness, and fear in an online experiment. Emotion ratings were used to analyze how clearly and accurately participants perceived different mixed emotions in both modalities. The results revealed that the perception was determined by an interaction between modality and emotion combination with moderate effects sizes, indicating that the perception varied depending on how emotions were expressed and in which combination emotions occurred. Participants were generally able to perceive a variety of mixed emotions clearly and accurately in both modalities, whereby visually expressed mixed emotions were perceived more clearly and with greater accuracy than vocally expressed mixed emotions. The findings provide additional evidence for the assumption that the affect system is more complex and flexible than previously thought. The results have important clinical relevance since the accurate recognition of mixed emotions in psychiatric disorders can facilitate correct diagnosis and thereby improve treatment outcomes. However, further research is needed to understand the complexity of the underlying mechanisms and to explore the benefits associated with mixed emotion recognition accuracy.