RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets

Wenjun Ruan, Wenbo Zhu (), Zhijia Zhao, Kai Wang, Qinghua Lu, Lufeng Luo and Wei-Chang Yeh
Additional contact information
Wenjun Ruan: School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, China
Wenbo Zhu: School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, China
Zhijia Zhao: School of Mechanical and Electrical Engineering, Guangzhou University, Guangzhou 510006, China
Kai Wang: School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, China
Qinghua Lu: School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, China
Lufeng Luo: School of Mechatronic Engineering and Automation, Foshan University, Foshan 528225, China
Wei-Chang Yeh: Department of Industrial Engineering and Engineering Management, National Tsing Hua University, Hsinchu 30013, Taiwan

Mathematics, 2023, vol. 11, issue 18, 1-16

Abstract: Accurate grasping state detection is critical to the dexterous operation of robots. Robots must use multiple modalities to perceive external information, similar to humans. The direct fusion method of visual and tactile sensing may not provide effective visual–tactile features for the grasping state detection network of the target. To address this issue, we present a novel visual–tactile fusion model (i.e., RFCT) and provide an incremental dimensional tensor product method for detecting grasping states of weak-stiffness targets. We investigate whether convolutional block attention mechanisms (CBAM) can enhance feature representations by selectively attending to salient visual and tactile cues while suppressing less important information and eliminating redundant information for the initial fusion. We conducted 2250 grasping experiments using 15 weak-stiffness targets. We used 12 targets for training and three for testing. When evaluated on untrained targets, our RFCT model achieved a precision of 82.89%, a recall rate of 82.07%, and an F1 score of 81.65%. We compared RFCT model performance with various combinations of Resnet50 + LSTM and C3D models commonly used in grasping state detection. The experimental results show that our RFCT model significantly outperforms these models. Our proposed method provides accurate grasping state detection and has the potential to provide robust support for robot grasping operations in real-world applications.

Keywords: visual–tactile fusion perception; target grasping state detection; grasping; multimodal perception (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2023
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/11/18/3969/pdf (application/pdf)
https://www.mdpi.com/2227-7390/11/18/3969/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:11:y:2023:i:18:p:3969-:d:1242735

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().