Audiovisual Facial Action Unit Recognition using Feature Level Fusion

Zibo Meng, Shizhong Han, Min Chen and Yan Tong
Additional contact information
Zibo Meng: University of South Carolina, Columbia, SC, USA
Shizhong Han: University of South Carolina, Columbia, SC, USA
Min Chen: Computing and Software Systems, School of STEM, University of Washington Bothell, Bothell, WA, USA
Yan Tong: University of South Carolina, Columbia, SC, USA

International Journal of Multimedia Data Engineering and Management (IJMDEM), 2016, vol. 7, issue 1, 60-76

Abstract: Recognizing facial actions is challenging, especially when they are accompanied with speech. Instead of employing information solely from the visual channel, this work aims to exploit information from both visual and audio channels in recognizing speech-related facial action units (AUs). In this work, two feature-level fusion methods are proposed. The first method is based on a kind of human-crafted visual feature. The other method utilizes visual features learned by a deep convolutional neural network (CNN). For both methods, features are independently extracted from visual and audio channels and aligned to handle the difference in time scales and the time shift between the two signals. These temporally aligned features are integrated via feature-level fusion for AU recognition. Experimental results on a new audiovisual AU-coded dataset have demonstrated that both fusion methods outperform their visual counterparts in recognizing speech-related AUs. The improvement is more impressive with occlusions on the facial images, which would not affect the audio channel.

Date: 2016
References: Add references at CitEc
Citations:

Downloads: (external link)
http://services.igi-global.com/resolvedoi/resolve. ... 18/IJMDEM.2016010104 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:igg:jmdem0:v:7:y:2016:i:1:p:60-76

Access Statistics for this article

International Journal of Multimedia Data Engineering and Management (IJMDEM) is currently edited by Chengcui Zhang

More articles in International Journal of Multimedia Data Engineering and Management (IJMDEM) from IGI Global
Bibliographic data for series maintained by Journal Editor ().