Hierarchical Convolutional Neural Network for Emotion Recognition Using EEG and Facial Expressions

Abstract

Emotion recognition is crucial for advancing human–computer interaction (HCI) by enabling systems to interpret complex affective states. While Electroencephalogram (EEG) signals provide direct insights into neural activity, facial expressions offer external emotional cues. However, unimodal systems often struggle with robustness and generalization across diverse subjects. This study presents a Hierarchical Convolutional Neural Network (HCNN) framework that integrates EEG and facial expressions through multi-level convolutional feature extraction and featurelevel fusion. The proposed model combines deep hierarchical representations with handcrafted temporal–frequency and texture-based descriptors to form a unified feature vector. Experiments on the MAHNOB-HCI and DEAP datasets show that the HCNN achieves accuracies of 91.40% and 88.09%, outperforming CNN-, LSTM-, and SVM-based methods. The results demonstrate the model’s ability to effectively capture complementary cross-modal correlations while reducing feature redundancy and computational complexity. The HCNN framework shows great promise for real-time emotion recognition applications, offering a scalable, interpretable, and data-efficient solution for multimodal emotion recognition in next-generation HCI systems.

Document