PAD-based multimodal affective fusion

[en] The study of multimodality is comparatively less developed for Affective interfaces than for their traditional counterparts. However, one condition for the successful development of Affective interface technologies is the development of frameworks for the real-time multimodal fusion. In this paper, we describe an approach to multimodal affective fusion, which relies on a dimensional model, Pleasure-Arousal-Dominance (PAD) to support the fusion of affective modalities, each input modality being represented as a PAD vector. We describe how this model supports both affective content fusion and temporal fusion within a unified approach. We report results from early user studies which confirm the existence of a correlation between measured affective input and user temperament scores.

Disciplines :

Computer science

Author, co-author :

Gilroy, Stephen W.

Cavazza, Marc

Niiranen, Marcus

André, Elisabeth

Vogt, Thurid

Urbain, Jérôme ; Université de Mons > Faculté Polytechnique > Information, Signal et Intelligence artificielle

Benayoun, Maurice

Seichter, Hartmut

Billinghurst, Mark

Language :

English

Title :

PAD-based multimodal affective fusion

Publication date :

10 September 2009

Event name :

Third International Conference on Affective Computing and Intelligent Interaction

Event place :

Amsterdam, Netherlands

Event date :

2009

Research unit :

F105 - Information, Signal et Intelligence artificielle

Research institute :

R450 - Institut NUMEDIART pour les Technologies des Arts Numériques

Available on ORBi UMONS :

since 20 January 2011

Statistics

Number of views

49 (0 by UMONS)

Number of downloads

0 (0 by UMONS)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

Bibliography

M. D. Ainley. The factor structure of curiosity measures: Breadth and depth of interest curiosity styles. Australian Journal of Psychology, 39:53-59, 1987.
I. Albrecht, M. Schröder, J. Haber, and H.-P. Seidel. Mixed feelings: expression of non-basic emotions in a muscle-based talking head. Virtual Reality, 8(4):201-212, 2005.
J. Andreassi. Psychophysiology: Human Behavior and Physiological Response. Routledge, 2006.
N. Bianchi-Berthouze and A. Kleinsmith. A categorical approach to affective gesture recognition. Connection Science, 15(4):259-269, 2003.
R. A. Bolt. "put-that-there": Voice and gesture at the graphics interface. In SIGGRAPH '80: Proceedings of the 7th annual conference on Computer graphics and interactive techniques, pages 262-270, New York, NY, USA, 1980. ACM.
C. Breazeal. Affective interaction between humans and robots. In ECAL '01: Proceedings of the 6th European Conference on Advances in Artificial Life, pages 582-591, London, UK, 2001. Springer-Verlag.
C. S. Carver. Pleasure as a sign you can attend to something else: Placing positive feelings within a general model of affect. Cognition and Emotion, 17(2):241-261, 2003.
G. Castellano, L. Kessous, and G. Caridakis. Multimodal emotion recognition from expressive faces, body gestures and speech. In R. C. Fiorella de Rosis, editor, Proceedings of the Doctoral Consortium of 2nd International Conference on Affective Computing and Intelligent Interaction, September 2007.
L. Chen, H. Tao, T. Huang, T. Miyasato, and R. Nakatsu. Emotion recognition from audiovisual information. In Proceedings of the IEEE Workshop on Multimedia Signal Processing, pages 83-88, 1998.
M. Chen and J. A. Bargh. Consequences of automatic evaluation: Immediate behavioral predispositions to approach or avoid the stimulus. Personality and Social Psychology Bulletin, 25(2):215-224, 1999.
S. K. Christian Becker and I. Wachsmuth. Simulating the emotion dynamics of a multimodal conversational agent. In Affective Dialogue Systems, volume 3068 of LNCS, pages 154-165. Springer Berlin / Heidelberg, 2004.
J. Coutaz, L. Nigay, D. Salber, A. Blandford, J. May, and R. Young. Four easy pieces for assessing the usability of multimodal interaction: The CARE properties. In Proceedings of INTERACT'95, pages 115-120. Chapman & Hall, 1995.
C. Duarte and L. Carrico̧. A conceptual framework for developing adaptive multimodal applications. In IUI '06: Proceedings of the 11th international conference on Intelligent user interfaces, pages 132-139, New York, NY, USA, 2006. ACM.
P. Ekman and H. Oster. Facial expressions of emotion. Annual Review of Psychology, 30(1):527-554, 1979.
J. Förster and F. Strack. Influence of overt head movements on memory for valenced words: A case of conceptual?motor compatibility. Journal of Personality and Social Psychology, 71:421-430, 1996.
P. Gebhard. Alma: a layered model of affect. In AAMAS '05: Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, pages 29-36, New York, NY, USA, 2005. ACM.
S. W. Gilroy, M. Cavazza, R. Chaignon, S.-M. Mäkelä, M. Niranen, E. André, T. Vogt, J. Urbain, M. Billinghurst, H. Seichter, and M. Benayoun. E-tree: emotionally driven augmented reality art. In MM '08: Proceedings of the 16th ACM international conference on Multimedia, pages 945-948, New York, NY, USA, 2008. ACM.
S. Hidi and K. A. Renninger. Interest, a motivational variable that combines affective and cognitive functioning. In D. Y. Dai and R. J. Sternberg, editors, Motivation, Emotion and Cognition: Integrative Perspectives on Intellectual Functioning and Development, pages 89-115. Laurence Erlbaum Associates, 2004.
C. H. Hillman, K. S. Rosengren, and D. P. Smith. Emotion and motivated behavior: postural adjustments to affective picture viewing. Biological Psychology, 66:51-62, 2004.
S. A. T. Jr. and P. J. Silvia. Must interesting things be pleasant? a test of competing appraisal structures. Emotion, 6(4), 2006.
A. Kapoor, W. Burleson, and R. Picard. Automatic prediction of frustration. International Journal of Human-Computer Studies, 65(8):724-736, 2007.
A. Kleinsmith and N. Bianchi-Berthouze. Recognizing affective dimensions from body posture. In Affective Computing and Intelligent Interaction - ACII 2007, volume 4738 of LNCS. Springer-Verlag, 2007.
A. Kleinsmith, P. R. D. Silva, and N. Bianchi-Berthouze. Grounding affective dimensions into posture features. In Affective Computing and Intelligent Interaction - ACII 2005, volume 3784 of LNCS. Springer-Verlag, 2005.
C.-H. J. Lee, C.-Y. I. Jang, T.-H. D. Chen, J.Wetzel, T.-H. D. Shen, and T. Selker. Attention meter: a vision-based input toolkit for interaction designers. In CHI '06: CHI '06 extended abstracts on Human factors in computing systems, pages 1007-1012, New York, NY, USA, 2006. ACM.
C. L. Lisetti and F. Nasoz. Maui: a multimodal affective user interface. In MULTIMEDIA '02: Proceedings of the tenth ACM international conference on Multimedia, pages 161-170, New York, NY, USA, 2002. ACM.
L. Maat and M. Pantic. Gaze-x: adaptive affective multimodal interface for single-user office scenarios. In ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces, pages 171-178, New York, NY, USA, 2006. ACM.
M. Mansoorizadeh and N. M. Charkari. Bimodal person-dependent emotion recognition comparison of feature level and decision level information fusion. In International Conference on Pervasive Technologies Related to Assistive Environments - PETRA '08, 2008.
A. Mehrabian. Non-verbal communication. Aldine-Atherton, 1972.
A. Mehrabian. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology, 14:261-292, 1996.
W. A. Melder, K. P. Truong, M. D. Uyl, D. A. Van Leeuwen, M. A. Neerincx, L. R. Loos, and B. S. Plum. Affective multimodal mirror: sensing and eliciting laughter. In HCM '07: Proceedings of the international workshop on Human-centered multimedia, pages 31-40, New York, NY, USA, 2007. ACM.
R. Nakatsu, M. Rauterberg, and P. Vorderer. A new framework for entertainment computing: From passive to active experience. In Entertainment Computing - ICEC 2005, volume 3711 of LNCS. Springer-Verlag, 2005.
L. Nigay and J. Coutaz. A generic platform for addressing the multimodal challenge. In CHI '95: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 98-105, New York, NY, USA, 1995. ACM Press/Addison-Wesley Publishing Co.
S. Oviatt and P. Cohen. Perceptual user interfaces: multimodal interfaces that process what comes naturally. Communations of the ACM, 43(3):45-53, 2000.
S. Oviatt, A. DeAngeli, and K. Kuhn. Integration and synchronization of input modes during multimodal human-computer interaction. In CHI '97: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 415-422, New York, NY, USA, 1997. ACM.
M. Pantic and L. J. Rothkrantz. Towards an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91(9):1370-1390, September 2003.
C. Pereira. Dimensions of emotion meaning in speech. In Proceedings of the ISCA Workshop on Speech and Emotion, pages 25-28, 2000.
N. Pfleger. Context based multimodal fusion. In ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces, pages 265-272, New York, NY, USA, 2004. ACM.
R. W. Picard. Affective Computing. The MIT Press, 1997.
J. A. Russell and A. Mehrabian. Evidence for a three-factor theory of emotions. Journal of Research in Personality, 11:273-294, 1977.
N. Sebe, I. Cohen, T. Gevers, and T. S. Huang. Emotion recognition based on joint visual and audio cues. In Proceedings of the ICPR International Conference on Pattern Recognition, pages 1136-1139, 2006.
R. Sharma, V. I. Pavlovic, and T. S. Huang. Toward multimodal human-computer interface. Proceedings of the IEEE, pages 853-869, May 1998.
L. C. D. Silva and P. C. Ng. Bimodal emotion recognition. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, pages 332-335, 2000.
T. Vogt and E. André. Improving automatic emotion recognition from speech via gender differentiation. In Proceedings of Language Resources and Evaluation Conference (LREC 2006), 2006.
H. G. Wallbott. Bodily expression of emotion. European Journal of Social Psychology, 28:879-896, 1998.
K. C. Wassermann, K. Eng, and P. F. M. J. Verschure. Live soundscape composition based on synthetic emotions. IEEE MultiMedia, 10(4):82-90, 2003.