Scalable Distributed Systems for Real-Time Big Data Processing in Financial Technology

Ruilin Zhang

Simen Owen Academic Proceedings Series, 2026, vol. 3, 198-206

Abstract: Real-time stream processing in regulated financial environments requires simultaneous guarantees of low latency, data confidentiality, and auditability, requirements that existing systems struggle to satisfy jointly. Prior approaches either sacrifice performance for security or omit compliance mechanisms entirely, leaving a gap in practical, production-ready solutions. To address this, we propose a co-designed architecture integrating lightweight secure aggregation (LSA), adaptive micro-batching, and LSTM-based predictive autoscaling within Apache Flink. Evaluated on a real-world dataset of anonymized payment transactions, our system achieves a 99th-percentile latency of 178 ± 6 ms at a sustained throughput of 89k ± 1.2k events/sec, thereby meeting a strict 200-ms service-level objective while maintaining 100% compliance completeness. In contrast, a baseline employing homomorphic encryption (CryptoStream) incurs a significantly higher latency of 312 ± 18 ms and consumes roughly four times the CPU resources. Another secure baseline (Flink-SGX), while meeting the latency target (192 ± 9 ms), exhibits operational fragility under load. Ablation studies confirm the necessity of each component for balancing performance, stability, and regulatory adherence. Collectively, the results demonstrate a feasible path toward confidential, auditable, and high-performance stream processing for real-world financial infrastructure.

Keywords: secure stream processing; adaptive batching; lightweight cryptography; regulatory compliance; real-time analytics (search for similar items in EconPapers)
Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://soapubs.com/index.php/SOAPS/article/view/1597/1461 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:axf:soapsa:v:3:y:2026:i::p:198-206

Access Statistics for this article

More articles in Simen Owen Academic Proceedings Series from Scientific Open Access Publishing
Bibliographic data for series maintained by Yuchi Liu ().