Proactive Data Pipeline Maintenance via Machine Learning-Driven Anomaly Detection

Akash Vijayrao Chaudhari and Pallavi Ashokrao Charate

International Journal of Scientific Research in Science and Technology, 2025, vol. 12, issue 2, 1041-1053

Abstract: Modern data pipelines are the backbone of data-driven enterprises, feeding analytics and machine learning systems with timely and accurate data. Ensuring these pipelines operate reliably is critical, as failures or data quality issues can propagate downstream and lead to significant business losses. Traditional pipeline maintenance is largely reactive—engineers respond to broken jobs or corrupted data after the fact. In this paper, we propose a proactive maintenance framework that leverages machine learning-driven anomaly detection to continuously monitor data pipelines and address issues before they escalate. The approach integrates real-time anomaly detection on both pipeline operational metrics and data quality indicators to flag deviations from normal behavior. We outline how advanced algorithms (including time-series models, unsupervised outlier detection, and reinforcement learning agents) can identify subtle pipeline issues such as data schema changes, upstream delays, and data drift. The framework further incorporates automated diagnosis and remediation strategies, aiming for self-healing pipelines that reduce downtime. We demonstrate the effectiveness of this approach using synthetic data pipeline experiments, where an anomaly detection model achieves 100% recall in identifying injected pipeline faults with minimal false alarms. We also survey relevant literature and industry solutions, including recent works by Chaudhari and colleagues on AI-driven ETL and multi-agent anomaly resolution, to situate our contributions. Results from both our experiments and prior studies show that ML-driven monitoring can intercept issues in real-time – enabling maintenance that is not only reactive but truly proactive. The proposed approach can significantly improve pipeline reliability, reduce manual intervention, and ultimately ensure the consistent delivery of high-quality data for critical applications.

Keywords: Machine Learning-Driven Anomaly; Learning-Driven Anomaly Detection; Data Pipeline; Pipeline Maintenance; Maintenance Via Machine (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://ijsrst.com/home/article/view/IJSRST251222663 Abstract page (text/html)
https://ijsrst.com/home/article/download/IJSRST251222663/IJSRST251222663 Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:etm:ijsrst:v12:y2025:i2:id:759

DOI: 10.32628/IJSRST251222663

Access Statistics for this article

More articles in International Journal of Scientific Research in Science and Technology from Technoscience Academy
Bibliographic data for series maintained by Pankaj Sharma ().