Hive- A Warehousing Solution Over a Map-Reduce Framework

DMCA

Hive- A Warehousing Solution Over a Map-Reduce Framework (2009)

Cached

Download Links

by Ashish Thusoo , Joydeep Sen Sarma , Namit Jain , Zheng Shao , Prasad Chakka , Suresh Anthony , Hao Liu , Pete Wyckoff , Raghotham Murthy

Venue:	IN VLDB '09: PROCEEDINGS OF THE VLDB ENDOWMENT
Citations:	265 - 1 self

BibTeX

@INPROCEEDINGS{Thusoo09hive-a,
    author = {Ashish Thusoo and Joydeep Sen Sarma and Namit Jain and Zheng Shao and Prasad Chakka and Suresh Anthony and Hao Liu and Pete Wyckoff and Raghotham Murthy},
    title = {Hive- A Warehousing Solution Over a Map-Reduce Framework},
    booktitle = {IN VLDB '09: PROCEEDINGS OF THE VLDB ENDOWMENT},
    year = {2009},
    pages = {1626--1629},
    publisher = {}
}

OpenURL

Abstract

The size of data sets being collected and analyzed in the industry for business intelligence is growing rapidly, mak- ing traditional warehousing solutions prohibitively expen- sive. Hadoop [3] is a popular open-source map-reduce im- plementation which is being used as an alternative to store and process extremely large data sets on commodity hard- ware. However, the map-reduce programming model is very low level and requires developers to write custom programs which are hard to maintain and reuse. In this paper, we present Hive, an open-source data ware- housing solution built on top of Hadoop. Hive supports queries expressed in a SQL-like declarative language - HiveQL, which are compiled into map-reduce jobs executed on Hadoop. In addition, HiveQL supports custom map-reduce scripts to be plugged into queries. The language includes a type sys- tem with support for tables containing primitive types, col- lections like arrays and maps, and nested compositions of the same. The underlying IO libraries can be extended to query data in custom formats. Hive also includes a system catalog, Hive-Metastore, containing schemas and statistics, which is useful in data exploration and query optimization. In Facebook, the Hive warehouse contains several thousand tables with over 700 terabytes of data and is being used ex- tensively for both reporting and ad-hoc analyses by more than 100 users.

Keyphrases

warehousing solution map-reduce framework custom program hive warehouse type sys tem business intelligence low level col lections large data set popular open-source map-reduce im plementation custom map-reduce script system catalog data exploration commodity hard ware open-source data ware housing solution custom format expen sive map-reduce job io library sql-like declarative language hiveql primitive type traditional warehousing solution nested composition map-reduce programming model query optimization data set ad-hoc analysis

DMCA