3D-CNN-Based Fused Feature Maps with LSTM Applied to Action Recognition

Sheeraz Arif, Jing Wang, Tehseen Ul Hassan and Zesong Fei
Additional contact information
Sheeraz Arif: Information and Communication Engineering, Beijing Institute of Technology, Beijing 100081, China
Jing Wang: Information and Communication Engineering, Beijing Institute of Technology, Beijing 100081, China
Tehseen Ul Hassan: Information and Communication Engineering, Beijing Institute of Technology, Beijing 100081, China
Zesong Fei: Information and Communication Engineering, Beijing Institute of Technology, Beijing 100081, China

Future Internet, 2019, vol. 11, issue 2, 1-17

Abstract: Human activity recognition is an active field of research in computer vision with numerous applications. Recently, deep convolutional networks and recurrent neural networks (RNN) have received increasing attention in multimedia studies, and have yielded state-of-the-art results. In this research work, we propose a new framework which intelligently combines 3D-CNN and LSTM networks. First, we integrate discriminative information from a video into a map called a ‘motion map’ by using a deep 3-dimensional convolutional network (C3D). A motion map and the next video frame can be integrated into a new motion map, and this technique can be trained by increasing the training video length iteratively; then, the final acquired network can be used for generating the motion map of the whole video. Next, a linear weighted fusion scheme is used to fuse the network feature maps into spatio-temporal features. Finally, we use a Long-Short-Term-Memory (LSTM) encoder-decoder for final predictions. This method is simple to implement and retains discriminative and dynamic information. The improved results on benchmark public datasets prove the effectiveness and practicability of the proposed method.

Keywords: action recognition; fused features; 3D convolution neural network; motion map; long short-term-memory (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2019
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/11/2/42/pdf (application/pdf)
https://www.mdpi.com/1999-5903/11/2/42/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:11:y:2019:i:2:p:42-:d:205468

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().