DMCA
Optimizing Completion Time and Resource Provisioning of Pig Programs (2012)
| Venue: | In Workshop on Cloud Computing Optimization (CCOPT |
| Citations: | 4 - 2 self |
BibTeX
@INPROCEEDINGS{Zhang12optimizingcompletion,
author = {Zhuoyao Zhang and Ludmila Cherkasova and Abhishek Verma and Boon Thau Loo},
title = {Optimizing Completion Time and Resource Provisioning of Pig Programs},
booktitle = {In Workshop on Cloud Computing Optimization (CCOPT},
year = {2012}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Abstract—As cloud computing continues to mature, IT managers have started concentrating on the support of additional performance requirements: quality of service and tailored resource allocation for achieving service performance goals. In this paper 1, we consider the popular Pig framework that provides a high-level SQL-like abstraction on top of MapReduce engine for processing large data sets. Programs written in such frameworks are compiled into directed acyclic graphs (DAGs) of MapReduce jobs. Often, data processing applications have to produce results by a certain time deadline. We design a performance modeling framework for Pig programs that solves two inter-related problems: (i) estimating the completion time of a Pig program as a function of allocated resources; (ii) estimating the amount of resources (a number of map and reduce slots) required for completing a Pig program with a given (soft) deadline. To achieve these goals, we first, optimize a Pig program execution by enforcing the optimal schedule of its concurrent jobs. This optimization reduces a program completion time (10%-27 % in our experiments), and moreover, it eliminates possible non-determinism in the DAGs execution. Based on our optimization, we propose an accurate performance model for Pig programs. This approach leads to significant resource savings (20%-60 % in our experiments) compared with the original, unoptimized solution. We validate our approach in a 66-node Hadoop cluster using two workload sets: TPC-H queries and a set of customized queries mining a collection of HP Labs ’ web proxy logs. Keywords-MapReduce, Pig, performance model, resource allocation, execution optimization, job scheduling. I.
Keyphrases
pig program completion time resource provisioning resource allocation workload set 66-node hadoop cluster tpc-h query proxy log mapreduce job large data set customized query data processing application high-level sql-like abstraction allocated resource significant resource saving performance model popular pig framework dag execution concurrent job certain time deadline possible non-determinism performance modeling framework hp lab optimal schedule program completion time service performance goal acyclic graph pig program execution unoptimized solution execution optimization mapreduce engine additional performance requirement inter-related problem accurate performance model job scheduling
