Citation Context

... manageable administrative overhead: Yahoo has announced one that uses 10,000 cores [29]. Although the original motivation of MapReduce’s designers was scalability for Web-scale bulk-processing tasks =-=[11]-=-, Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the VLDB copyright notice and the title o...

Citation Context

...uce [12]. The critical contribution of Manimal is to detect these selections automatically in unmodified developer code. Manimal currently uses a B+Tree, but in the future could also employ an R-Tree =-=[15]-=- or some other indexing technique when appropriate. Projection optimizations modify the on-disk data file to only store bytes that are actually necessary for executing the user’s code. For example, th...

Citation Context

...ate for MapReduce programs that are “program-specific,” with code that is directly related to the user’s end-goals for the program. For tools layered on top of MapReduce, such as the Pig query system =-=[21]-=-, we believe a better approach is for the tools to give Manimal explicit hints about program semantics. We discuss this issue in more depth in Appendix A. Background There has been a recent surge of i...

Citation Context

...in MapReduce systems. Some projects have applied MapReduceinspired techniques to building a traditional relational database [2, 26], but most have focused on improving MapReduce execution performance =-=[3, 8, 12, 27, 28]-=-. However, most of these projects are either low-level system techniques that are semantics-free, or ask the user to modify their code to expose more program semantics. To the best of our knowledge, M...

Citation Context

...file data – overlap with traditional relational workloads. However, MapReduce systems lag far behind RDBMSes in their query-processing sophistication and runtime efficiency. For example, Pavlo et al. =-=[22]-=- showed that a MapReduce program can run 2-50x slower than a similar relational query run on an RDBMS, using identical hardware. A recent paper by Anderson and Tucek [5] suggested that Hadoop performe...

Citation Context

...em can use an alternate serialized version of the data that stores only the needed fields for a program, thereby reducing the overall number of bytes that must be processed (similar to a column-store =-=[25]-=- or an ondisk binary association table [7]). The goal of Manimal is to(SELECT, V.rank(), V.rank() > 1) /logs/log.1 /logs/log.2 ... /logs/.log.1.idx /logs/.log.2.idx ... Manimal catalog select src... ...

Citation Context

...in MapReduce systems. Some projects have applied MapReduceinspired techniques to building a traditional relational database [2, 26], but most have focused on improving MapReduce execution performance =-=[3, 8, 12, 27, 28]-=-. However, most of these projects are either low-level system techniques that are semantics-free, or ask the user to modify their code to expose more program semantics. To the best of our knowledge, M...

Citation Context

...e depth in Appendix A. Background There has been a recent surge of interest in MapReduce systems. Some projects have applied MapReduceinspired techniques to building a traditional relational database =-=[2, 26]-=-, but most have focused on improving MapReduce execution performance [3, 8, 12, 27, 28]. However, most of these projects are either low-level system techniques that are semantics-free, or ask the user...

Citation Context

...of the data that stores only the needed fields for a program, thereby reducing the overall number of bytes that must be processed (similar to a column-store [25] or an ondisk binary association table =-=[7]-=-). The goal of Manimal is to(SELECT, V.rank(), V.rank() > 1) /logs/log.1 /logs/log.2 ... /logs/.log.1.idx /logs/.log.2.idx ... Manimal catalog select src... select src... ... (SELECT, log1.idx, V.ran...

Citation Context

...everal compression techniques, but in all cases applies the technique to the entire data file. Instead, Manimal enables two semantics-aware forms of compression, both previously used by Abadi, et al. =-=[1]-=-: First, delta-compression efficiently stores runs of numeric values, by only keeping differences between values, instead of the absolute values. Storing just small deltas, when combined with a size-s...

Citation Context

...in MapReduce systems. Some projects have applied MapReduceinspired techniques to building a traditional relational database [2, 26], but most have focused on improving MapReduce execution performance =-=[3, 8, 12, 27, 28]-=-. However, most of these projects are either low-level system techniques that are semantics-free, or ask the user to modify their code to expose more program semantics. To the best of our knowledge, M...

Citation Context

... N/A 0 Table 2: Overall performance improvement provided by Manimal, across the Pavlo benchmark tasks. Manimal’s analyzer employs compiler techniques for databasestyle optimizations. Chilimbi, et al. =-=[10]-=- tried to reorder inmemory data representations to improve cache behavior, a systems-level improvement suggested by program semantics. They did not examine disk-based approaches. Manimal has some qual...

Citation Context

...mount of recent work on MapReduce [3, 8, 12, 27, 28], though none that takes system’s wholly-automated approach to optimization. Several efforts have explored the problem of scheduling task execution =-=[16, 28]-=-. Afrati and Ullman [3] investigated how to efficiently perform joins using MapReduce. Yang, et al. [27] extended the programming model to Map-Reduce-Merge, allowing the user to express different join...

Citation Context

...in MapReduce systems. Some projects have applied MapReduceinspired techniques to building a traditional relational database [2, 26], but most have focused on improving MapReduce execution performance =-=[3, 8, 12, 27, 28]-=-. However, most of these projects are either low-level system techniques that are semantics-free, or ask the user to modify their code to expose more program semantics. To the best of our knowledge, M...

Citation Context

...in MapReduce systems. Some projects have applied MapReduceinspired techniques to building a traditional relational database [2, 26], but most have focused on improving MapReduce execution performance =-=[3, 8, 12, 27, 28]-=-. However, most of these projects are either low-level system techniques that are semantics-free, or ask the user to modify their code to expose more program semantics. To the best of our knowledge, M...

Citation Context

...c analysis to automatically apply well-known optimizations to a novel language. There has also been work in integrating data-manipulation primitives with traditional programming languages, as in LINQ =-=[19]-=-. Such tools side-step the need for Manimal’s analyzer but also require the programmer to use a potentiallynovel programming language. 6. CONCLUSIONS We have described the Manimal system for optimizin...

Citation Context

...he behavior, a systems-level improvement suggested by program semantics. They did not examine disk-based approaches. Manimal has some qualities in common with work optimizing XQuery (e.g., Ré, et al. =-=[24]-=-). In particular, Manimal follows the same general approach of using static analysis to automatically apply well-known optimizations to a novel language. There has also been work in integrating data-m...

Citation Context

...e system to use data semantics-driven optimizations without requiring any code changes from developers. We previously presented an outline of the Manimal architecture and a single experimental result =-=[9]-=-. This paper substantially expands on that earlier work, with new optimization techniques, a much more detailed technical discussion, more complete discussions of MapReduce workloads, and full experim...

Citation Context

...e depth in Appendix A. Background There has been a recent surge of interest in MapReduce systems. Some projects have applied MapReduceinspired techniques to building a traditional relational database =-=[2, 26]-=-, but most have focused on improving MapReduce execution performance [3, 8, 12, 27, 28]. However, most of these projects are either low-level system techniques that are semantics-free, or ask the user...

Citation Context

...cy. For example, Pavlo et al. [22] showed that a MapReduce program can run 2-50x slower than a similar relational query run on an RDBMS, using identical hardware. A recent paper by Anderson and Tucek =-=[5]-=- suggested that Hadoop performed bulk data processing at a rate of less than 5 megabytes per second per node (and barely more than half a megabyte per second per core) [5]. Thus, MapReduce systems may...

Citation Context

...o explicitly-declared metadata. Further, MapReduce systems have shown they can operate at extremely large scale with manageable administrative overhead: Yahoo has announced one that uses 10,000 cores =-=[29]-=-. Although the original motivation of MapReduce’s designers was scalability for Web-scale bulk-processing tasks [11], Permission to copy without fee all or part of this material is granted provided th...

Citation Context

...educe programs. There have been several recent index-style attempts to improve MapReduce performance, such as Dittrich, et al. [12]’s Hadoop++ system, and the column-oriented work of Floratou, et al. =-=[14]-=-. The former requires explicit support from the programmer, while the latter only requires physical storage reorganization; both could be used as targets for Manimal.Test Description Space Overhead H...