Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience

DMCA

Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience (2009)

Cached

Download Links

by Alan F. Gates , Olga Natkovich , Shubham Chopra , Pradeep Kamath , Shravan M. Narayanamurthy , Christopher Olston , Benjamin Reed , Santhosh Srinivasan , Utkarsh Srivastava

Venue:	IN VLDB '09: PROCEEDINGS OF THE VLDB ENDOWMENT
Citations:	102 - 5 self

BibTeX

@INPROCEEDINGS{Gates09buildinga,
    author = {Alan F. Gates and Olga Natkovich and Shubham Chopra and Pradeep Kamath and Shravan M. Narayanamurthy and Christopher Olston and Benjamin Reed and Santhosh Srinivasan and Utkarsh Srivastava},
    title = {Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience},
    booktitle = {IN VLDB '09: PROCEEDINGS OF THE VLDB ENDOWMENT},
    year = {2009},
    pages = {1414--1425},
    publisher = {}
}

OpenURL

Abstract

Increasingly, organizations capture, transform and analyze enormous data sets. Prominent examples include internet companies and e-science. The Map-Reduce scalable dataflow paradigm has become popular for these applications. Its simple, explicit dataflow programming model is favored by some over the traditional high-level declarative approach: SQL. On the other hand, the extreme simplicity of Map-Reduce leads to much low-level hacking to deal with the many-step, branching dataflows that arise in practice. Moreover, users must repeatedly code standard operations such as join by hand. These practices waste time, introduce bugs, harm readability, and impede optimizations. Pig is a high-level dataflow system that aims at a sweet spot between SQL and Map-Reduce. Pig offers SQL-style high-level data manipulation constructs, which can be assembled in an explicit dataflow and interleaved with custom Map- and Reduce-style functions or executables. Pig programs are compiled into sequences of Map-Reduce jobs, and executed in the Hadoop Map-Reduce environment. Both Pig and Hadoop are open-source projects administered by the Apache Software Foundation. This paper describes the challenges we faced in developing Pig, and reports performance comparisons between Pig execution and raw Map-Reduce execution. 1.

Keyphrases

high-level dataflow system pig experience explicit dataflow enormous data set custom map pig program hadoop map-reduce environment extreme simplicity traditional high-level declarative approach map-reduce scalable dataflow paradigm raw map-reduce execution sweet spot internet company organization capture sql-style high-level data manipulation construct map-reduce job open-source project report performance comparison much low-level hacking reduce-style function standard operation practice waste time apache software foundation impede optimization pig execution harm readability prominent example

DMCA