Performance Analysis of Hadoop YARN Job Schedulers in a Multi-Tenant Environment on HiBench Benchmark Suite

Kamalakant Laxman Bawankule, Rupesh Kumar Dewang and Anil Kumar Singh
Additional contact information
Kamalakant Laxman Bawankule: Motilal Nehru National Institute of Technology Allahabad, Prayagraj, India
Rupesh Kumar Dewang: Motilal Nehru National Institute of Technology Allahabad, Prayagraj, India
Anil Kumar Singh: Motilal Nehru National Institute of Technology Allahabad, Prayagraj, India

International Journal of Distributed Systems and Technologies (IJDST), 2021, vol. 12, issue 3, 64-82

Abstract: Big data processing technology marks a prominent place in today's market. Hadoop is an efficient open-source distributed framework used to process big data with fewer expenses utilizing a cluster of commodity machines (nodes). In Hadoop, YARN got introduced for effective resource utilization among the jobs. Still, YARN over-allocates the resources for some tasks of a job and keeps the cluster resources underutilized. This paper has investigated the CAPACITY and FAIR schedulers' practical utilization of resources in a multi-tenancy shared environment using the HiBench benchmark suite. It compares the above MapReduce job schedulers' performance in two scenarios and proposes some open research questions (ORQ) with potential solutions to help the upcoming researchers. On average, the authors found that CAPACITY and FAIR schedulers utilize 77% of RAM and 82% of CPU cores. Finally, the experimental evaluation proves that these schedulers over-allocate the resources for some of the tasks and keep the cluster resources underutilized in different scenarios.

Date: 2021
References: Add references at CitEc
Citations:

Downloads: (external link)
http://services.igi-global.com/resolvedoi/resolve. ... 018/IJDST.2021070104 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:igg:jdst00:v:12:y:2021:i:3:p:64-82

Access Statistics for this article

International Journal of Distributed Systems and Technologies (IJDST) is currently edited by Nik Bessis

More articles in International Journal of Distributed Systems and Technologies (IJDST) from IGI Global
Bibliographic data for series maintained by Journal Editor ().