eBPF-Enhanced Streaming Observability for Flink Pipelines in Kubernetes

Srikanth Gorle (), Pradeep Manivannan () and Karthik Mani ()

Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023, 2024, vol. 4, issue 1, 463-502

Abstract: Real-time observability in distributed stream processing systems is critical for ensuring performance, reliability, and rapid incident response. This paper introduces a novel observability framework that leverages extended Berkeley Packet Filter (eBPF) technology to enhance runtime visibility into Apache Flink pipelines deployed in Kubernetes environments. Unlike traditional monitoring approaches that rely on static logs or intrusive agents, our eBPF-based solution enables low-overhead, in-kernel telemetry collection at both the system and application levels. We integrate this telemetry with Flink’s metrics and Kubernetes orchestration data to deliver a unified, fine-grained view of pipeline behavior, network dynamics, and resource usage. Experimental evaluations show that our approach improves detection of performance anomalies, reduces monitoring latency, and incurs minimal system overhead, making it highly suitable for production-grade stream processing in cloud-native architectures.

Keywords: eBPF; Apache Flink; Kubernetes; Stream Processing; Observability; Telemetry; Real-Time Monitoring; Cloud-Native Architecture; Performance Debugging (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://newjaigs.com/index.php/JAIGS/article/view/384 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:das:njaigs:v:4:y:2024:i:1:p:463-502:id:384

Access Statistics for this article

Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023 is currently edited by Justyna Żywiołek

More articles in Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023 from Open Knowledge
Bibliographic data for series maintained by Open Knowledge ().