Virtual character facial expressions influence human brain and facial EMG activity in a decision-making game

Abstract

We examined the effects of the emotional facial expressions of a virtual character (VC) on human frontal electroencephalographic (EEG) asymmetry (putatively indexing approach/withdrawal motivation), facial electromyographic (EMG) activity (emotional expressions), and social decision making (cooperation/defection). In a within-subjects design, the participants played the Iterated Prisoner's Dilemma game with VCs with different dynamic facial expressions (predefined or dependent on the participant's electrodermal and facial EMG activity). In general, VC facial expressions elicited congruent facial muscle activity. However, both frontal EEG asymmetry and facial EMG activity elicited by an angry VC facial expression varied as a function of preceding interactional events (human collaboration/defection). Pre-decision inner emotional-motivational processes and emotional facial expressions were dissociated, suggesting that human goals influence pre-decision frontal asymmetry, whereas display rules may affect (pre-decision) emotional expressions in human-VC interaction. An angry VC facial expression, high pre-decision corrugator EMG activity, and relatively greater left frontal activation predicted the participant's decision to defect. Both post-decision frontal asymmetry and facial EMG activity were related to reciprocal cooperation. The results suggest that the justifiability of VC emotional expressions and the perceived fairness of VC actions influence human emotional responses.