Choosing Real- Time Over Batch: Architectural Considerations for Streaming Pipeline in Cloud

Vikrant Sikarwar ()

Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023, 2024, vol. 2, issue 1, 368-395

Abstract: Real-time data processing has become a critical requirement for modern cloud-native applications, where timely insights directly influence business decisions, system responsiveness, and user experience. This paper explores the architectural considerations involved in choosing real-time streaming pipelines over traditional batch processing in cloud environments. It highlights key factors such as data ingestion patterns, latency requirements, scalability demands, and cost–performance trade-offs. Various cloud-native tools and services for streaming—such as managed messaging queues, event hubs, and distributed stream-processing engines—are examined to illustrate how they support high-throughput and low-latency workloads. The study provides a comparative discussion on fault tolerance, state management, and system complexity between batch and streaming architectures. Finally, a set of best practices is proposed to guide architects and developers in designing efficient, resilient, and scalable real-time data pipelines in the cloud.

Keywords: Real-time processing; Streaming pipeline; Cloud architecture; Batch processing; Low-latency computing; Data ingestion; Scalability (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://newjaigs.com/index.php/JAIGS/article/view/434 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:das:njaigs:v:2:y:2024:i:1:p:368-395:id:434

Access Statistics for this article

Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023 is currently edited by Justyna Żywiołek

More articles in Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023 from Open Knowledge
Bibliographic data for series maintained by Open Knowledge ().