Im2Graph: A Weakly Supervised Approach for Generating Holistic Scene Graphs from Regional Dependencies

Swarnendu Ghosh (), Teresa Gonçalves () and Nibaran Das
Additional contact information
Swarnendu Ghosh: Department of CSE (AIML), Institute of Engineering and Management, Kolkata 700091, WB, India
Teresa Gonçalves: Department of Informatics, University of Évora, 7000-671 Évora, Portugal
Nibaran Das: Department of CSE, Jadavpur University, Kolkata 700032, WB, India

Future Internet, 2023, vol. 15, issue 2, 1-21

Abstract: Conceptual representations of images involving descriptions of entities and their relations are often represented using scene graphs. Such scene graphs can express relational concepts by using sets of triplets ⟨ s u b j e c t — p r e d i c a t e — o b j e c t ⟩ . Instead of building dedicated models for scene graph generation, our model tends to extract the latent relational information implicitly encoded in image captioning models. We explored dependency parsing to build grammatically sound parse trees from captions. We used detection algorithms for the region propositions to generate dense region-based concept graphs. These were optimally combined using the approximate sub-graph isomorphism to create holistic concept graphs for images. The major advantages of this approach are threefold. Firstly, the proposed graph generation module is completely rule-based and, hence, adheres to the principles of explainable artificial intelligence. Secondly, graph generation can be used as plug-and-play along with any region proposition and caption generation framework. Finally, our results showed that we could generate rich concept graphs without explicit graph-based supervision.

Keywords: scene graph generation; weakly supervised deep learning; explainable AI; dependency parsing; knowledge graphs (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/15/2/70/pdf (application/pdf)
https://www.mdpi.com/1999-5903/15/2/70/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:15:y:2023:i:2:p:70-:d:1064160

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().