DMCA
The Impact of Capacity Scheduler Configuration Settings on MapReduce Jobs
BibTeX
@MISC{Chauhan_theimpact,
author = {Jagmohan Chauhan and Dwight Makaroff and Winfried Grassmann},
title = {The Impact of Capacity Scheduler Configuration Settings on MapReduce Jobs},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Abstract—MapReduce is a parallel programming paradigm used for processing huge datasets on certain classes of distributable problems using a cluster. Budgetary constraints and the need for better usage of resources in a MapReduce cluster often influence an organization to rent or share hardware resources for their main data processing and analysis tasks. Thus, there may be many competing jobs from different clients performing simultaneous requests to the MapReduce framework on a particular cluster. Schedulers like Fair Share and Capacity have been specially designed for such purposes. Administrators and users run into performance problems, however, because they do not know the exact meaning of different task scheduler settings and what impact they can have with respect to the application execution time and resource allocation policy decisions. Existing work shows that the performance of MapReduce jobs depends on the cluster configuration, input data type and job configuration settings. However, that work fails to take into account the task scheduler settings. We show, through experimental evaluation, that task scheduler configuration parameters make a significant difference to the performance of the cluster and it is important to understand the influence of such parameters. Based on our findings, we also identified some of the open issues in the existing area of research. Keywords-MapReduce, Task Scheduler, Performance I.
Keyphrases
mapreduce job capacity scheduler configuration setting significant difference huge datasets share hardware resource mapreduce framework certain class parallel programming paradigm fair share distributable problem main data processing experimental evaluation task scheduler different task scheduler setting work show abstract mapreduce particular cluster task scheduler setting task scheduler configuration parameter analysis task budgetary constraint application execution time allocation policy decision data type simultaneous request cluster configuration open issue job configuration setting performance problem mapreduce cluster different client exact meaning
