Attention-Based Multimodal Neural Network for Automatic Evaluation of Press Conferences

Shengzhou Yi, Koshiro Mochitomi, Isao Suzuki, Xueting Wang and Toshihiko Yamasaki
Additional contact information
Shengzhou Yi: The University of Tokyo, Japan
Koshiro Mochitomi: PRAP Japan, Inc., Japan
Isao Suzuki: PRAP Japan, Inc., Japan
Xueting Wang: The University of Tokyo, Japan
Toshihiko Yamasaki: The University of Tokyo, Japan

International Journal of Multimedia Data Engineering and Management (IJMDEM), 2020, vol. 11, issue 3, 1-19

Abstract: In the study, a multimodal neural network is proposed to automatically predict the evaluation of a professional consultant team for press conferences using text and audio data. Seven publicly available press conference videos were collected, and all the Q&A pairs between speakers and journalists were annotated by the consultant team. The proposed multimodal neural network consists of a language model, an audio model, and a feature fusion network. The word representation is made up by a token embedding using ELMo and a type embedding. The language model is an LSTM with an attention layer. The audio model is based on a six-layer CNN to extract segmental feature as well as an attention network to measure the importance of each segment. Two approaches of feature fusion are proposed: a shared attention network and the production of text features and audio features. The former can explain the importance between speech content and speaking style. The latter achieved the best performance with the average accuracy of 60.1% for all evaluation criteria.

Date: 2020
References: Add references at CitEc
Citations:

Downloads: (external link)
http://services.igi-global.com/resolvedoi/resolve. ... 18/IJMDEM.2020070101 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:igg:jmdem0:v:11:y:2020:i:3:p:1-19

Access Statistics for this article

International Journal of Multimedia Data Engineering and Management (IJMDEM) is currently edited by Chengcui Zhang

More articles in International Journal of Multimedia Data Engineering and Management (IJMDEM) from IGI Global
Bibliographic data for series maintained by Journal Editor ().