Pig Latin: A Not-So-Foreign Language for Data Processing

DMCA

Pig Latin: A Not-So-Foreign Language for Data Processing

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Christopher Olston , Benjamin Reed , Utkarsh Srivastava , Ravi Kumar , Andrew Tomkins

Citations:

606 - 13 self

BibTeX

@MISC{Olston_piglatin:,
    author = {Christopher Olston and Benjamin Reed and Utkarsh Srivastava and Ravi Kumar and Andrew Tomkins},
    title = {Pig Latin: A Not-So-Foreign Language for Data Processing},
    year = {}
}

OpenURL

Abstract

There is a growing need for ad-hoc analysis of extremely large data sets, especially at internet companies where innovation critically depends on being able to analyze terabytes of data collected every day. Parallel database products, e.g., Teradata, offer a solution, but are usually prohibitively expensive at this scale. Besides, many of the people who analyze this data are entrenched procedural programmers, who find the declarative, SQL style to be unnatural. The success of the more procedural map-reduce programming model, and its associated scalable implementations on commodity hardware, is evidence of the above. However, the map-reduce paradigm is too low-level and rigid, and leads to a great deal of custom user code that is hard to maintain, and reuse. We describe a new language called Pig Latin that we have designed to fit in a sweet spot between the declarative style of SQL, and the low-level, procedural style of map-reduce. The accompanying system, Pig, is fully implemented, and compiles Pig Latin into physical plans that are executed over Hadoop, an open-source, map-reduce implementation. We give a few examples of how engineers at Yahoo! are using Pig to dramatically reduce the time required for the development and execution of their data analysis tasks, compared to using Hadoop directly. We also report on a novel debugging environment that comes integrated with Pig, that can lead to even higher productivity gains. Pig is an open-source, Apache-incubator project, and available for general use. 1.

Keyphrases

pig latin data processing not-so-foreign language map-reduce implementation general use new language declarative style scalable implementation map-reduce paradigm large data set sweet spot internet company data analysis task productivity gain procedural style apache-incubator project procedural programmer great deal procedural map-reduce programming model novel debugging environment parallel database product custom user code physical plan compiles pig latin ad-hoc analysis sql style commodity hardware

DMCA