Dynamic Job Ordering and Slot Configurations for Mapreduce Workloads Using Heuristic Algorithm

M. Praveen Kumar, S. P. Santhoshkumar and S. Syed Shajahaan

International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 2017, vol. 2, issue 5, 67-72

Abstract: MapReduce is a popular parallel computing paradigm for large-scale data processing in clusters and data centers. A MapReduce workload generally contains a set of jobs, each of which consists of multiple map tasks followed by multiple reduce tasks. Due to 1) that map tasks can only run in map slots and reduce tasks can only run in reduce slots, and 2) the general execution constraints that map tasks are executed before reduce tasks, different job execution orders and map/reduce slot configurations for a MapReduce workload have significantly different performance and system utilization. This paper proposes two classes of algorithms to minimize the makespan and the total completion time for an offline MapReduce workload. Our first class of algorithms focuses on the job ordering optimization for a MapReduce workload under a given map/reduce slot configuration. In contrast, our second class of algorithms considers the scenario that we can perform optimization for map/reduce slot configuration for a MapReduce workload. We perform simulations as well as experiments on Amazon EC2 and show that our proposed algorithms produce results that are up to 15 _ 80 percent better than currently unoptimized Hadoop, leading to significant reductions in running time in practice

Keywords: MapReduce; Hadoop; Flow-Shops; Scheduling Algorithm; Job Ordering. (search for similar items in EconPapers)
Date: 2017
Note: Article URL: https://ijsrcseit.com/CSEIT17252
References: Add references at CitEc
Citations:

Downloads: (external link)
https://ijsrcseit.com/CSEIT17252 Article URL (text/html)
https://ijsrcseit.com/paper/CSEIT17252.pdf Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:jbh:ijsrcs:v2:y2017:i5:id:hcseit17252

Access Statistics for this article

More articles in International Journal of Scientific Research in Computer Science, Engineering and Information Technology from International Journal of Scientific Research in Computer Science, Engineering and Information Technology
Bibliographic data for series maintained by Pankaj Sharma ().