Topology-aware task allocation for online distributed stream processing applications with latency constraints

Xiaohui Wei, Xun Wei and Hongliang Li

Physica A: Statistical Mechanics and its Applications, 2019, vol. 534, issue C

Abstract: There have been increasing demands for real time processing of the ever-growing data. In order to meet this requirement and ensure the reliable processing of streaming data, a variety of distributed stream processing architectures and platforms have been developed, which handles the fundamental task of allocating processing tasks to the currently available physical resources and routing streaming data between these resources. However, many stream processing systems lack an intelligent scheduling mechanism, in which their default schedulers allocate tasks without taking resource demands and availability, or the transfer latency between resources into consideration. Besides, stream processing has a strict request for latency. Thus it is important to give latency guarantee for distributed stream processing. In this paper, we propose two new algorithms for stream processing with latency guarantee, both the algorithms consider transfer latency and resource demand in task allocation. Both algorithms can guarantee latency constraints. Algorithm AHA reduces more than 21.3% and 58.9% resources compared with the greedy and the round-robin algorithms, and algorithm PHA further improves the resource utilization to 32.1% and 73.2%.

Keywords: Distributed stream processing; Task allocation; Transfer latency; Latency constraint; Heuristic approach; Critical path (search for similar items in EconPapers)
Date: 2019
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0378437119311604
Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:phsmap:v:534:y:2019:i:c:s0378437119311604

DOI: 10.1016/j.physa.2019.122024

Access Statistics for this article

Physica A: Statistical Mechanics and its Applications is currently edited by K. A. Dawson, J. O. Indekeu, H.E. Stanley and C. Tsallis

More articles in Physica A: Statistical Mechanics and its Applications from Elsevier
Bibliographic data for series maintained by Catherine Liu ().