Face Swapping Consistency Transfer with Neural Identity Carrier

Kunlin Liu, Ping Wang, Wenbo Zhou, Zhenyu Zhang, Yanhao Ge, Honggu Liu, Weiming Zhang and Nenghai Yu
Additional contact information
Kunlin Liu: School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China
Ping Wang: School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China
Wenbo Zhou: School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China
Zhenyu Zhang: Tencent Youtu, Shanghai 200233, China
Yanhao Ge: Tencent Youtu, Shanghai 200233, China
Honggu Liu: School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China
Weiming Zhang: School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China
Nenghai Yu: School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China

Future Internet, 2021, vol. 13, issue 11, 1-14

Abstract: Deepfake aims to swap a face of an image with someone else’s likeness in a reasonable manner. Existing methods usually perform deepfake frame by frame, thus ignoring video consistency and producing incoherent results. To address such a problem, we propose a novel framework Neural Identity Carrier (NICe), which learns identity transformation from an arbitrary face-swapping proxy via a U - N e t . By modeling the incoherence between frames as noise, NICe naturally suppresses its disturbance and preserves primary identity information. Concretely, NICe inputs the original frame and learns transformation supervised by swapped pseudo labels. As the temporal incoherence has an uncertain or stochastic pattern, NICe can filter out such outliers and well maintain the target content by uncertainty prediction. With the predicted temporally stable appearance, NICe enhances its details by constraining 3D geometry consistency, making NICe learn fine-grained facial structure across the poses. In this way, NICe guarantees the temporal stableness of deepfake approaches and predicts detailed results against over-smoothness. Extensive experiments on benchmarks demonstrate that NICe significantly improves the quality of existing deepfake methods on video-level. Besides, data generated by our methods can benefit video-level deepfake detection methods.

Keywords: deepfake generation; face swapping; consistency transfer (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/13/11/298/pdf (application/pdf)
https://www.mdpi.com/1999-5903/13/11/298/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:13:y:2021:i:11:p:298-:d:685122

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().