Citation Context

...ense. Hadoop [2], of course, is a popular open source implementation of the MapReduce platform; Hyracks targets the same sorts of applications that Hadoop is used for. The recent Nephele/PACTS system =-=[28]-=- extends the MapReduce programming model by adding additional secondorder operators (e.g., match, cross, and co-group) with “your function goes here” plug-in points. Such an extension could be impleme...

Citation Context

...reasonable job annotations automatically by a higher level-programming model layered on top of Nephele. Rather than describing jobs as arbitrary DAGs, this higher level programming model called PACTs =-=[4]-=- is centered around the concatenation of second-order functions, e.g., like the map and reduce function from the well-known MapReduce programming model. Developers can write custom first-order functio...

Citation Context

... an improved version of Hadoop that executes their jobs faster. As a result, we reject all design alternatives that require operator writers to either implement their jobs following special templates =-=[3]-=- or provide special inputs such as cost functions for their operators [21]. Instead, SkewTune should operate on unchanged MapReduce jobs. Mitigation Transparency. Today, MapReduce makes certain guaran...

Citation Context

...tasks as are needed or requested by the user. Dataflow systems include Dryad [20] and its extension DryadLINQ [31] (Microsoft), Clustera [15] (U. Wisconsin), Hyracks [8] (UC Irvine), and Nephele/PACT =-=[7]-=- (T. U. Berlin). Other extensions of map-reduce are high-level languages that compile into a sequence of map-reduce jobs. PIG [24] is an implementation of relational algebra from Yahoo!. Hive [6] (ope...

Citation Context

...rallel databases [17,19], MapReduce [16] is the best known representative, popular for its applicability beyond relational data. Several other systems, like Dryad [23], Hyracks [11], and Stratosphere =-=[7]-=-, follow that trend and push the paradigm further, eliminating many shortcomings of MapReduce. While dataflow systems were originally built for tasks like indexing, filtering, transforming, or aggrega...

Citation Context

... of MapReduce itself to achieve better query evaluation performance. For example, MapReduce Online [9] allows pipelined job interconnections to avoid intermediate result materialization. A PACT model =-=[4]-=- extends the MapReduce concept for complex relational operations. Our work, as well as work [27] on set similarity join, work [25] on Theta-join, all fall in the first category. We briefly survey some...

Citation Context

...t sufficiently pushed the limits of MapReduce in general and the Hadoop implementation in particular. In my opinion, it may be premature to declare it obsolete and call for a fresh ground-up redesign =-=[4, 8]-=-. MapReduce is less than ten years old, and Hadoop is even younger. There has already been plenty of interesting work within the confines of Hadoop, just from the database perspective: integration wit...

Citation Context

...rior specific permission and/or a fee. EDBT 2012, March 26–30, 2012, Berlin, Germany. Copyright 2012 ACM 978-1-4503-0790-1/12/03 ...$10.00 In order to provide a simple programming environment for the users, MapReduce offers a limited choice of execution strategies, which can adversely affect performance. For example, mappers checkpoint after every split, all map outputs are sorted and written to file, and reducers read statically determined partitions. To gain more flexibility, new MapReduce-inspired massive data processing platforms have emerged: Dryad [11], Hyracks [12], Spark [13], Nephele [14], Ciel [15] - all include elements of MapReduce, but have more choices in runtime query execution. In contrast to these projects, we chose to enhance MapReduce, to leverage existing investment in the Hadoop framework and in the query processing systems built on top of it, such as Jaql [16], Pig [4], and Hive [5]. While adding new runtime options to Hadoop, we made them adaptive to the runtime environment, to avoid making performance tuning any harder than it already is [17]. Adaptive algorithms demonstrate superior performance stability, as they are robust to tuning errors and changing runtime...

Citation Context

...motivated by these breakthroughs, new big data analysis systems have emerged to serve the aforementioned needs. Many of these systems such as Hyracks [11], Dryad [25], and our own Stratosphere system =-=[7]-=- adopt a data flow abstraction, where a data analysis program is specified as a directed acyclic graph (DAG) of smaller components that contain arbitrary user code. Even though some of these systems o...