AI-Augmented Data Pipeline Optimization for Scalable Cloud Systems

Rajendar Reddy Sama

International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 2026, vol. 12, issue 4, 359-366

Abstract: Deterministic ETL architectures - scheduled, fixed, and failure-reactive - cannot sustain the operational requirements of modern enterprise data environments, where volume growth, schema instability, and SLA pressure compound continuously. This paper presents a five-layer AI-augmented pipeline operating model that replaces reactive recovery with proactive, adaptive operation. The model integrates intelligent scheduling, continuous anomaly detection, and an operational copilot capability within a coherent Azure-native reference architecture anchored by a persistent feedback store. A structured implementation pathway and a three-dimensional evaluation framework - covering operational reliability, data quality, and delivery performance - are provided alongside the architectural specification. The model is grounded in operational observability as a prerequisite for automation, with governance controls embedded as non-optional cross-cutting elements.

Keywords: anomaly detection; Azure Data Factory; cloud data architecture; data pipeline reliability; feedback loop; intelligent scheduling; machine learning operations; pipeline optimisation (search for similar items in EconPapers)
Date: 2026
Note: Article URL: https://ijsrcseit.com/home/article/view/CSEIT2612419
References: Add references at CitEc
Citations:

Downloads: (external link)
https://ijsrcseit.com/home/article/view/CSEIT2612419 Article URL (text/html)
https://ijsrcseit.com/home/article/download/CSEIT2612419/CSEIT2612419 Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:jbh:ijsrcs:v12:y2026:i4:id:2143

DOI: 10.32628/CSEIT2612419

Access Statistics for this article

More articles in International Journal of Scientific Research in Computer Science, Engineering and Information Technology from International Journal of Scientific Research in Computer Science, Engineering and Information Technology
Bibliographic data for series maintained by Pankaj Sharma (USA) ().