2022 • In Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference, 2022, p. 3170 - 3174
Reproducibility of Results; Electroencephalography/methods; Emotions; Brain activity; Dual model; Emotion estimation; Feature map; Features fusions; Features vector; Fusion model; Image-based representation; Innovative method; Power; Electroencephalography; Signal Processing; Biomedical Engineering; Computer Vision and Pattern Recognition; Health Informatics
Abstract :
[en] Among the different modalities to assess emotion, electroencephalogram (EEG), representing the electrical brain activity, achieved motivating results over the last decade. Emotion estimation from EEG could help in the diagnosis or rehabilitation of certain diseases. In this paper, we propose a dual model considering two different representations of EEG feature maps: 1) a sequential based representation of EEG band power, 2) an image-based representation of the feature vectors. We also propose an innovative method to combine the information based on a saliency analysis of the image-based model to promote joint learning of both model parts. The model has been evaluated on four publicly available datasets: SEED-IV, SEED, DEAP and MPED. The achieved results outperform results from state-of-the-art approaches for three of the proposed datasets with a lower standard deviation that reflects higher stability. For sake of reproducibility, the codes and models proposed in this paper are available at https://github.com/VDelv/Emotion-EEG.
Disciplines :
Electrical & electronics engineering
Author, co-author :
Delvigne, Victor; ISIA Lab, Faculty of Engineering, University of Mons, Belgium ; IMT Nord Europe, CRIStAL UMR CNRS 9189, France
Facchini, Antoine; IMT Nord Europe, CRIStAL UMR CNRS 9189, France
Wannous, Hazem; IMT Nord Europe, CRIStAL UMR CNRS 9189, France
Ris, Laurence ; Université de Mons - UMONS > Faculté de Médecine et de Pharmacie > Service de Neurosciences
Vandeborre, Jean-Philippe; IMT Nord Europe, CRIStAL UMR CNRS 9189, France
Language :
English
Title :
A Saliency based Feature Fusion Model for EEG Emotion Estimation.
Publication date :
July 2022
Journal title :
Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference
eISSN :
2694-0604
Publisher :
Institute of Electrical and Electronics Engineers Inc., United States
R550 - Institut des Sciences et Technologies de la Santé R300 - Institut de Recherche en Technologies de l'Information et Sciences de l'Informatique
Funders :
Verasonics
Funding text :
Research jointly supported by the University of Mons and Institut Mines-Télécom Nord Europe. The content is solely the responsibility of the authors. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s). This work has been made in collaboration with the Centre de Recherche et de Formation Interdisciplinaire en Psychophysiologie et Electrophysiologie de la Cognition (CiPsE). The authors would like to thank Nathan Hubens and Luca La Fisca for their collaboration.
G. L. Ahern and G. E. Schwartz. Differential lateralization for positive and negative emotion in the human brain: EEG spectral analysis. Neuropsychologia, 23(6):745-755, January 1985.
P. Bashivan, I. Rish, M. Yeasin, and N. Codella. Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks. arXiv:1511.06448 [cs], 2015. arXiv: 1511.06448.
A. Craik, Y. He, and J. L. Contreras-Vidal. Deep learning for electroencephalogram (EEG) classification tasks: a review. Journal of Neural Engineering, 16(3):031001, April 2019. Publisher: IOP Publishing.
Yong D., W. Wang, and L. Wang. Hierarchical recurrent neural network for skeleton based action recognition. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. ISSN: 1063-6919.
R. Duan, J. Zhu, and B. Lu. Differential entropy feature for EEGbased emotion classification. In 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), 2013.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37, ICML'15, Lille, France, 2015. JMLR.org.
M. Hatt, B. Laurent, A. Ouahabi, H. Fayad, Sh. Tan, L. Li, W. Lu, V. Jaouen, C. Tauber, J. Czakon, F. Drapejkowski, W. Dyrka, S. Camarasu-Pop, F. Cervenansky, P. Girard, T. Glatard, M. Kain, Y. Yao, C. Barillot, A. Kirov, and D. Visvikis. The first MICCAI challenge on PET tumor segmentation. Medical Image Analysis, 2018.
Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing, pages 18-31, 2011.
Zirui Lan, Olga Sourina, Lipo Wang, Reinhold Scherer, and Gernot R Müller-Putz. Domain adaptation techniques for eeg-based emotion recognition: a comparative study on two public datasets. IEEE Transactions on Cognitive and Developmental Systems, 11(1):85-94, 2018.
W.-Ch. L. Lew, D. Wang, K. Shylouskaya, Z. Zhang, J.-H. Lim, K. K. Ang, and A-H. Tan. EEG-based Emotion Recognition Using Spatial-Temporal Representation via Bi-GRU. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC), pages 116-119, July 2020. ISSN: 2694-0604.
R. Li, J. S. Johansen, H. Ahmed, T. V. Ilyevsky, R. B. Wilbur, H. M. Bharadwaj, and J. M. Siskind. The Perils and Pitfalls of Block Design for EEG Classification Experiments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1):316-333, January 2021. Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
Y. Li, W. Shi, and W. W. Wasserman. Genome-wide prediction of cis-regulatory regions using supervised deep learning methods. BMC Bioinformatics, 2018.
Y. Li, L. Wang, W. Zheng, Y. Zong, L. Qi, Z. Cui, T. Zhang, and T. Song. A Novel Bi-hemispheric Discrepancy Model for EEG Emotion Recognition. IEEE Transactions on Cognitive and Developmental Systems, 2020.
S. Muthukumaraswamy. High-frequency brain activity and muscle artifacts in MEG/EEG: A review and recommendations. Frontiers in Human Neuroscience, 7, 2013. Publisher: Frontiers.
Lucas Antón Pastur-Romay, Francisco Cedrón, Alejandro Pazos, and Ana Belén Porto-Pazos. Deep artificial neural networks and neuromorphic chips for big data analysis: pharmaceutical and bioinformatics applications. International journal of molecular sciences, 17(8):1313, 2016.
Vishal M Patel, Hien Van Nguyen, and René Vidal. Latent space sparse subspace clustering. In Proceedings of the IEEE international conference on computer vision, pages 225-232, 2013.
N. Phan, D. Dou, B. Piniewski, and D. Kil. A deep learning approach for human behavior prediction with explanations in health social networks: social restricted Boltzmann machine (SRBM+). Social network analysis and mining, 2016.
S. Sabour, N. Frosst, and G. E. Hinton. Dynamic Routing Between Capsules. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30. Curran Associates, Inc., 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556 [cs], 2015. arXiv: 1409.1556.
T. Song, W. Zheng, C. Lu, Y. Zong, X. Zhang, and Z. Cui. MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition. IEEE Access, 2019.
T. Song, W. Zheng, P. Song, and Z. Cui. EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks. IEEE Transactions on Affective Computing, 2018.
Kris van Noord, Wenjin Wang, and Hailong Jiao. Insights of 3d input cnn in eeg-based emotion recognition. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 212-215. IEEE, 2021.
W.-L. Zheng, B.-N. Dong, and B.-L. Lu. Multimodal emotion recognition using EEG and eye tracking data. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2014.
W.-L. Zheng, W. Liu, Y. Lu, B.-L. Lu, and A. Cichocki. Emotion-Meter: A Multimodal Framework for Recognizing Human Emotions. IEEE Transactions on Cybernetics, 2019.
W.-L. Zheng and B.-L. Lu. Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks. IEEE Transactions on Autonomous Mental Development, 2015.
P. Zhong, D. Wang, and C. Miao. EEG-Based Emotion Recognition Using Regularized Graph Neural Networks. IEEE Transactions on Affective Computing, 2020.
Y. Zong, Y. Li, W. Zheng, Z. Cui, and T. Zhang. A Novel Neural Network Model based on Cerebral Hemispheric Asymmetry for EEG Emotion Recognition. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), 2018.
