DMCA
Scale-up vs Scale-out for Hadoop: Time to rethink?
| Citations: | 17 - 1 self |
BibTeX
@MISC{Gkantsidis_scale-upvs,
author = {Christos Gkantsidis and Dushyanth Narayanan and Orion Hodson and Antony Rowstron},
title = {Scale-up vs Scale-out for Hadoop: Time to rethink?},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
In the last decade we have seen a huge deployment of cheap clusters to run data analytics workloads. The conventional wisdom in industry and academia is that scaling out using a cluster of commodity machines is better for these workloads than scaling up by adding more resources to a single server. Popular analytics infrastructures such as Hadoop are aimed at such a cluster scaleout environment. Is this the right approach? Our measurements as well as other recent work shows that the majority of realworld analytic jobs process less than 100 GB of input, but popular infrastructures such as Hadoop/MapReduce were originally designed for petascale processing. We claim that a single “scale-up ” server can process each of these jobs and do as well or better than a cluster in terms of performance, cost, power, and server density. We present an evaluation across 11 representative Hadoop jobs that shows scale-up to be competitive in all cases and significantly better in some cases, than scale-out. To achieve that performance, we describe several modifications to the Hadoop runtime that target scale-up configuration. These changes are transparent, do not require any changes to application code, and do not compromise scale-out performance; at the same time our evaluation shows that they do significantly improve Hadoop’s scale-up performance. ∗ Work done while on internship from Vrije Universiteit Amsterdam Copyright c ○ 2013 by the Association for Computing Machinery, Inc. (ACM). Permission to make digital or hard copies of portions of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page in print or the first screen in digital media. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission
Keyphrases
scale-up v scale-out data analytics workload first page last decade commercial advantage digital medium recent work single server realworld analytic job process petascale processing hadoop scale-up performance cheap cluster hadoop mapreduce hadoop runtime commodity machine right approach representative hadoop job popular analytics infrastructure popular infrastructure single scale-up server prior specific permission conventional wisdom full citation application code first screen scale-out performance hard copy scale-up configuration huge deployment cluster scaleout environment vrije universiteit amsterdam copyright several modification classroom use
