Measuring Semantic Coherence of RAG-Generated Abstracts Through Complex Network Metrics

Bady Gana (), Wenceslao Palma, Freddy A. Lucay (), Cristóbal Missana, Carlos Abarza and Hector Allende-Cid
Additional contact information
Bady Gana: Escuela de Ingeniería Informática, Pontificia Universidad Católica de Valparaíso, Avenida Brasil 2241, Valparaiso 2362807, Chile
Wenceslao Palma: Escuela de Ingeniería Informática, Pontificia Universidad Católica de Valparaíso, Avenida Brasil 2241, Valparaiso 2362807, Chile
Freddy A. Lucay: Escuela de Ingeniería Informática, Pontificia Universidad Católica de Valparaíso, Avenida Brasil 2241, Valparaiso 2362807, Chile
Cristóbal Missana: Escuela de Ingeniería Informática, Pontificia Universidad Católica de Valparaíso, Avenida Brasil 2241, Valparaiso 2362807, Chile
Carlos Abarza: Escuela de Ingeniería Informática, Pontificia Universidad Católica de Valparaíso, Avenida Brasil 2241, Valparaiso 2362807, Chile
Hector Allende-Cid: Escuela de Ingeniería Informática, Pontificia Universidad Católica de Valparaíso, Avenida Brasil 2241, Valparaiso 2362807, Chile

Mathematics, 2025, vol. 13, issue 21, 1-35

Abstract: The exponential growth of scientific literature demands scalable methods to evaluate large-language-model outputs beyond surface-level fluency. We present a two-phase framework that separates generation from evaluation: a retrieval-augmented generation system first produces candidate abstracts, which are then embedded into semantic co-occurrence graphs and assessed using seven robustness metrics from complex network theory. Two experiments were conducted. The first varied model, embedding and prompt configurations, achieved results showing clear differences in performance; the best family combined gemma-2b-it, a prompt inspired by chain-of-Thought reasoning, and all-mpnet-base-v2, achieving the highest graph-based robustness. The second experiment refined the temperature setting for this family, identifying τ = 0.2 as optimal, which stabilized results (sd = 0.12 ) and improved robustness relative to retrieval baselines ( Δ E G = + 0.08 , Δ ρ = + 0.55 ). While human evaluation was limited to a small set of abstracts, the results revealed a partial convergence between graph-based robustness and expert judgments of coherence and importance. Our approach contrasts with methods like GraphRAG and establishes a reproducible, model-agnostic pathway for the scalable quality control of LLM-generated scientific content.

Keywords: RAG; complex networks; semantic graphs; weighted kappa; graph robustness (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/21/3472/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/21/3472/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:21:p:3472-:d:1784075

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().