Knowledge-Based Scene Graph Generation with Visual Contextual Dependency

Lizong Zhang, Haojun Yin, Bei Hui, Sijuan Liu and Wei Zhang
Additional contact information
Lizong Zhang: School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Haojun Yin: School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Bei Hui: School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Sijuan Liu: Research Institute of Social Development, Southwestern University of Finance and Economics, Chengdu 611130, China
Wei Zhang: School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China

Mathematics, 2022, vol. 10, issue 14, 1-20

Abstract: Scene graph generation is the basis of various computer vision applications, including image retrieval, visual question answering, and image captioning. Previous studies have relied on visual features or incorporated auxiliary information to predict object relationships. However, the rich semantics of external knowledge have not yet been fully utilized, and the combination of visual and auxiliary information can lead to visual dependencies, which impacts relationship prediction among objects. Therefore, we propose a novel knowledge-based model with adjustable visual contextual dependency. Our model has three key components. The first module extracts the visual features and bounding boxes in the input image. The second module uses two encoders to fully integrate visual information and external knowledge. Finally, visual context loss and visual relationship loss are introduced to adjust the visual dependency of the model. The difference between the initial prediction results and the visual dependency results is calculated to generate the dependency-corrected results. The proposed model can obtain better global and contextual information for predicting object relationships, and the visual dependencies can be adjusted through the two loss functions. The results of extensive experiments show that our model outperforms most existing methods.

Keywords: scene graph generation; external knowledge; context fusion; computer vision; visual dependency constraint (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2022
References: View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.mdpi.com/2227-7390/10/14/2525/pdf (application/pdf)
https://www.mdpi.com/2227-7390/10/14/2525/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:10:y:2022:i:14:p:2525-:d:867157

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().