A Two-Stage Facial Kinematic Control Strategy for Humanoid Robots Based on Keyframe Detection and Keypoint Cubic Spline Interpolation

Ye Yuan, Jiahao Li, Qi Yu, Jian Liu, Zongdao Li, Qingdu Li and Na Liu ()
Additional contact information
Ye Yuan: Institute of Machine Intelligence, University of Shanghai for Science and Technology, Shanghai 200093, China
Jiahao Li: Institute of Machine Intelligence, University of Shanghai for Science and Technology, Shanghai 200093, China
Qi Yu: Institute of Machine Intelligence, University of Shanghai for Science and Technology, Shanghai 200093, China
Jian Liu: Institute of Machine Intelligence, University of Shanghai for Science and Technology, Shanghai 200093, China
Zongdao Li: Institute of Machine Intelligence, University of Shanghai for Science and Technology, Shanghai 200093, China
Qingdu Li: Institute of Machine Intelligence, University of Shanghai for Science and Technology, Shanghai 200093, China
Na Liu: Institute of Machine Intelligence, University of Shanghai for Science and Technology, Shanghai 200093, China

Mathematics, 2024, vol. 12, issue 20, 1-16

Abstract: A plentiful number of facial expressions is the basis of natural human–robot interaction for high-fidelity humanoid robots. The facial expression imitation of humanoid robots involves the transmission of human facial expression data to servos situated within the robot’s head. These data drive the servos to manipulate the skin, thereby enabling the robot to exhibit various facial expressions. However, since the mechanical transmission rate cannot keep up with the data processing rate, humanoid robots often suffer from jitters in the imitation. We conducted a thorough analysis of the transmitted facial expression sequence data and discovered that they are extremely redundant. Therefore, we designed a two-stage strategy for humanoid robots based on facial keyframe detection and facial keypoint detection to achieve more natural and smooth expression imitation. We first built a facial keyframe detection model based on ResNet-50, combined with optical flow estimation, which can identify key expression frames in the sequence. Then, a facial keypoint detection model is used on the keyframes to obtain the facial keypoint coordinates. Based on the coordinates, the cubic spline interpolation method is used to obtain the motion trajectory parameters of the servos, thus realizing the robust control of the humanoid robot’s facial expression. Experiments show that, unlike before where the robot’s imitation would stutter at frame rates above 25 fps, our strategy allows the robot to maintain good facial expression imitation similarity (cosine similarity of 0.7226), even at higher frame rates.

Keywords: humanoid robots; facial expression imitation; facial keyframe detection; facial keypoint detection; cubic spline interpolation (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/12/20/3278/pdf (application/pdf)
https://www.mdpi.com/2227-7390/12/20/3278/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:12:y:2024:i:20:p:3278-:d:1501919

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().