Event Driven System Design with High Availability

Nikhil Sagar Miriyala

International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 2024, vol. 10, issue 6, 2470-2477

Abstract: Modern event-driven architectures rely on frameworks like Apache Kafka to enable real-time asynchronous processing. However, ensuring high availability across distributed systems, particularly in a multi-region environment, poses challenges such as avoiding duplication of data and achieving seamless failover with minimal manual intervention. This paper proposes a robust solution that leverages AWS Managed Streaming for Apache Kafka (MSK) to design an active-active highly available system. By implementing a smart consumer-based idempotent mechanism with a globally shared database and message-level UUIDs, the system ensures zero message duplication while maintaining fault tolerance. This design effectively minimizes latency, supports system resilience, and addresses the challenges of distributed event streaming systems. Practical considerations, including scalability, failure recovery, and latency optimization, are also discussed to provide a comprehensive guide to the deployment of resilient event-driven architectures in cloud-based infrastructures.

Keywords: Kafka; AWS MSK; High Availability; Idempotent Consumers; Fault Tolerance (search for similar items in EconPapers)
Date: 2024
Note: Article URL: https://ijsrcseit.com/home/article/view/CSEIT251112158
References: Add references at CitEc
Citations:

Downloads: (external link)
https://ijsrcseit.com/home/article/view/CSEIT251112158 Article URL (text/html)
https://ijsrcseit.com/home/article/download/CSEIT251112158/CSEIT251112158 Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:jbh:ijsrcs:v10:y2024:i6:id:808

DOI: 10.32628/CSEIT251112158

Access Statistics for this article

More articles in International Journal of Scientific Research in Computer Science, Engineering and Information Technology from International Journal of Scientific Research in Computer Science, Engineering and Information Technology
Bibliographic data for series maintained by Pankaj Sharma ().