Multi-Agent Big-Data Lambda Architecture Model for E-Commerce Analytics

Gautam Pal, Gangmin Li and Katie Atkinson
Additional contact information
Gautam Pal: Department of Computer Science, The University of Liverpool, Liverpool L69 7ZX, UK
Gangmin Li: Department of Computer Science, Xi’an Jiaotong-Liverpool University, Wuzhong 215123, China
Katie Atkinson: Department of Computer Science, The University of Liverpool, Liverpool L69 7ZX, UK

Data, 2018, vol. 3, issue 4, 1-15

Abstract: We study big-data hybrid-data-processing lambda architecture, which consolidates low-latency real-time frameworks with high-throughput Hadoop-batch frameworks over a massively distributed setup. In particular, real-time and batch-processing engines act as autonomous multi-agent systems in collaboration. We propose a Multi-Agent Lambda Architecture (MALA) for e-commerce data analytics. We address the high-latency problem of Hadoop MapReduce jobs by simultaneous processing at the speed layer to the requests which require a quick turnaround time. At the same time, the batch layer in parallel provides comprehensive coverage of data by intelligent blending of stream and historical data through the weighted voting method. The cold-start problem of streaming services is addressed through the initial offset from historical batch data. Challenges of high-velocity data ingestion is resolved with distributed message queues. A proposed multi-agent decision-maker component is placed at the MALA stack as the gateway of the data pipeline. We prove efficiency of our batch model by implementing an array of features for an e-commerce site. The novelty of the model and its key significance is a scheme for multi-agent interaction between batch and real-time agents to produce deeper insights at low latency and at significantly lower costs. Hence, the proposed system is highly appealing for applications involving big data and caters to high-velocity streaming ingestion and a massive data pool.

Keywords: Lambda Architecture; e-commerce analytics; real-time data analytics; real-time data ingestion; real-time machine leaning; recommender engine; online k-means clustering (search for similar items in EconPapers)
JEL-codes: C8 C80 C81 C82 C83 (search for similar items in EconPapers)
Date: 2018
References: View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.mdpi.com/2306-5729/3/4/58/pdf (application/pdf)
https://www.mdpi.com/2306-5729/3/4/58/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jdataj:v:3:y:2018:i:4:p:58-:d:187063

Access Statistics for this article

Data is currently edited by Ms. Cecilia Yang

More articles in Data from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().