AI-Driven Data Observability: A Hybrid Approach Using Graph Neural Networks and Bayesian Anomaly Detection

Ranjeet Kumar (), Mohan Vamsi Musunuru () and Swaminathan Sethuraman ()

Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023, 2024, vol. 6, issue 1, 634-648

Abstract: In modern data-intensive enterprises, ensuring reliability, accuracy, and trust in data pipelines is critical for downstream analytics and decision-making. Traditional monitoring systems often struggle to detect complex dependencies and subtle anomalies across distributed data ecosystems, leading to delayed remediation and increased operational risk. This research introduces a hybrid AI-driven data observability framework that combines Graph Neural Networks (GNNs) for structural dependency modeling with Bayesian anomaly detection for probabilistic uncertainty estimation. The proposed approach leverages GNNs to capture high-dimensional relational patterns across datasets, workflows, and metadata, while Bayesian inference provides robust anomaly detection under uncertainty and sparse conditions. Experiments conducted on enterprise-scale synthetic and real-world datasets demonstrate significant improvements in anomaly detection accuracy, early incident identification, and false-positive reduction compared to conventional rule-based and statistical monitoring techniques. By unifying structural learning with probabilistic reasoning, this hybrid framework enhances trust in data pipelines, reduces mean-time-to-detection (MTTD), and ensures resilient data operations. The findings highlight the potential of AI-driven observability as a foundational capability for next-generation data reliability engineering.

Keywords: Data Observability; Graph Neural Networks (GNNs); Bayesian Anomaly Detection; Hybrid AI Framework; Data Reliability Engineering; Anomaly Detection in Data Pipelines; AI-Driven Monitoring (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://newjaigs.com/index.php/JAIGS/article/view/412 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:das:njaigs:v:6:y:2024:i:1:p:634-648:id:412

Access Statistics for this article

Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023 is currently edited by Justyna Żywiołek

More articles in Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023 from Open Knowledge
Bibliographic data for series maintained by Open Knowledge ().