Improving the Performance of MapReduce for Small-Scale Cloud Processes Using a Dynamic Task Adjustment Mechanism

Tzu-Chi Huang, Guo-Hao Huang and Ming-Fong Tsai
Additional contact information
Tzu-Chi Huang: Department of Electronic Engineering, Lunghwa University of Science and Technology, Taoyuan 333, Taiwan
Guo-Hao Huang: Department of Electronic Engineering, Lunghwa University of Science and Technology, Taoyuan 333, Taiwan
Ming-Fong Tsai: Department of Electronic Engineering, National United University, Miaoli 360, Taiwan

Mathematics, 2022, vol. 10, issue 10, 1-17

Abstract: The MapReduce architecture can reliably distribute massive datasets to cloud worker nodes for processing. When each worker node processes the input data, the Map program generates intermediate data that are used by the Reduce program for integration. However, as the worker nodes process the MapReduce tasks, there are differences in the number of intermediate data created, due to variation in the operating-system environments and the input data, which results in the phenomenon of laggard nodes and affects the completion time for each small-scale cloud application task. In this paper, we propose a dynamic task adjustment mechanism for an intermediate-data processing cycle prediction algorithm, with the aim of improving the execution performance of small-scale cloud applications. Our mechanism dynamically adjusts the number of Map and Reduce program tasks based on the intermediate-data processing capabilities of each cloud worker node, in order to mitigate the problem of performance degradation caused by the limitations on the Google Cloud platform (Hadoop cluster) due to the phenomenon of laggards. The proposed dynamic task adjustment mechanism was compared with a simulated Hadoop system in a performance analysis, and an improvement of at least 5% in the processing efficiency was found for a small-scale cloud application.

Keywords: dynamic task adjustment mechanism; intermediate-data processing cycle prediction algorithm; small-scale cloud application (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2022
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/10/10/1736/pdf (application/pdf)
https://www.mdpi.com/2227-7390/10/10/1736/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:10:y:2022:i:10:p:1736-:d:819052

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().