Real-Time Human Fault Detection in Assembly Tasks, Based on Human Action Prediction Using a Spatio-Temporal Learning Model

Zhujun Zhang, Gaoliang Peng, Weitian Wang and Yi Chen
Additional contact information
Zhujun Zhang: State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China
Gaoliang Peng: State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China
Weitian Wang: Department of Computer Science, Montclair State University, Montclair, NJ 07043, USA
Yi Chen: Department of Automotive Engineering, Clemson University, Greenville, SC 29607, USA

Sustainability, 2022, vol. 14, issue 15, 1-26

Abstract: Human fault detection plays an important role in the industrial assembly process. In the current unstructured industrial workspace, the definition of human faults may vary over a long sequence, and this vagueness introduces multiple issues when using traditional detection methods. A method which could learn the correct action sequence from humans, as well as detect the fault actions based on prior knowledge, would be more appropriate and effective. To this end, we propose an end-to-end learning model to predict future human actions and extend it to detect human faults. We combined the auto-encoder framework and recurrent neural network (RNN) method to predict and generate intuitive future human motions. The convolutional long short-term memory (ConvLSTM) layer was applied to extract spatio-temporal features from video sequences. A score function was implemented to indicate the difference between the correct human action sequence and the fault actions. The proposed model was evaluated on a model vehicle seat assembly task. The experimental results showed that the model could effectively capture the necessary historical details to predict future human actions. The results of several fault scenarios demonstrated that the model could detect the faults in human actions based on corresponding future behaviors through prediction features.

Keywords: assembly; fault detection; human action prediction; spatio-temporal; machine learning; autonomous (search for similar items in EconPapers)
JEL-codes: O13 Q Q0 Q2 Q3 Q5 Q56 (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2071-1050/14/15/9027/pdf (application/pdf)
https://www.mdpi.com/2071-1050/14/15/9027/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jsusta:v:14:y:2022:i:15:p:9027-:d:869745

Access Statistics for this article

Sustainability is currently edited by Ms. Alexandra Wu

More articles in Sustainability from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().