Results 1  10
of
68
GraphChi: Largescale Graph Computation On just a PC
 In Proceedings of the 10th USENIX conference on Operating Systems Design and Implementation, OSDI’12
, 2012
"... Current systems for graph computation require a distributed computing cluster to handle very large realworld problems, such as analysis on social networks or the web graph. While distributed computational resources have become more accessible, developing distributed graph algorithms still remains c ..."
Abstract

Cited by 115 (6 self)
 Add to MetaCart
(Show Context)
Current systems for graph computation require a distributed computing cluster to handle very large realworld problems, such as analysis on social networks or the web graph. While distributed computational resources have become more accessible, developing distributed graph algorithms still remains challenging, especially to nonexperts. In this work, we present GraphChi, a diskbased system for computing efficiently on graphs with billions of edges. By using a wellknown method to break large graphs into small parts, and a novel parallel sliding windows method, GraphChi is able to execute several advanced data mining, graph mining, and machine learning algorithms on very large graphs, using just a single consumerlevel computer. We further extend GraphChi to support graphs that evolve over time, and demonstrate that, on a single computer, GraphChi can process over one hundred thousand graph updates per second, while simultaneously performing computation. We show, through experiments and theoretical analysis, that GraphChi performs well on both SSDs and rotational hard drives. By repeating experiments reported for existing distributed systems, we show that, with only fraction of the resources, GraphChi can solve the same problems in very reasonable time. Our work makes largescale graph computation available to anyone with a modern PC. 1
Mizan: A system for dynamic load balancing in largescale graph processing
 In EuroSys ’13
, 2013
"... Pregel [23] was recently introduced as a scalable graph mining system that can provide significant performance improvements over traditional MapReduce implementations. Existing implementations focus primarily on graph partitioning as a preprocessing step to balance computation across compute node ..."
Abstract

Cited by 21 (0 self)
 Add to MetaCart
(Show Context)
Pregel [23] was recently introduced as a scalable graph mining system that can provide significant performance improvements over traditional MapReduce implementations. Existing implementations focus primarily on graph partitioning as a preprocessing step to balance computation across compute nodes. In this paper, we examine the runtime characteristics of a Pregel system. We show that graph partitioning alone is insufficient for minimizing endtoend computation. Especially where data is very large or the runtime behavior of the algorithm is unknown, an adaptive approach is needed. To this end, we introduce Mizan, a Pregel system that achieves efficient load balancing to better adapt to changes in computing needs. Unlike known implementations of Pregel, Mizan does not assume any a priori knowledge of the structure of the graph or behavior of the algorithm. Instead, it monitors the runtime characteristics of the system. Mizan then performs efficient finegrained vertex migration to balance computation and communication. We have fully implemented Mizan; using extensive evaluation we show that—especially for highlydynamic workloads— Mizan provides up to 84 % improvement over techniques leveraging static graph prepartitioning. 1.
epiC: an Extensible and Scalable System for Processing Big Data
"... The Big Data problem is characterized by the so called 3V features: Volume a huge amount of data, Velocity a high data ingestion rate, and Variety a mix of structured data, semistructured data, and unstructured data. The stateoftheart solutions to the Big Data problem are largely based on the ..."
Abstract

Cited by 13 (5 self)
 Add to MetaCart
(Show Context)
The Big Data problem is characterized by the so called 3V features: Volume a huge amount of data, Velocity a high data ingestion rate, and Variety a mix of structured data, semistructured data, and unstructured data. The stateoftheart solutions to the Big Data problem are largely based on the MapReduce framework (aka its open source implementation Hadoop). Although Hadoop handles the data volume challenge successfully, it does not deal with the data variety well since the programming interfaces and its associated data processing model is inconvenient and inefficient for handling structured data and graph data. This paper presents epiC, an extensible system to tackle the Big Data’s data variety challenge. epiC introduces a general Actorlike concurrent programming model, independent of the data processing models, for specifying parallel computations. Users process multistructured datasets with appropriate epiC extensions, the implementation of a data processing model best suited for the data type and auxiliary code for mapping that data processing model into epiC’s concurrent programming model. Like Hadoop, programs written in this way can be automatically parallelized and the runtime system takes care of fault tolerance and intermachine communications. We present the design and implementation of epiC’s concurrent programming model. We also present two customized data processing model, an optimized MapReduce extension and a relational model, on top of epiC. Experiments demonstrate the effectiveness and efficiency of our proposed epiC. 1.
Optimizing Graph Algorithms on Pregellike Systems
, 2014
"... We study the problem of implementing graph algorithms efficiently on Pregellike systems, which can be surprisingly challenging. Standard graph algorithms in this setting can incur unnecessary inefficiencies such as slow convergence or high communication or computation cost, typically due to structu ..."
Abstract

Cited by 10 (2 self)
 Add to MetaCart
We study the problem of implementing graph algorithms efficiently on Pregellike systems, which can be surprisingly challenging. Standard graph algorithms in this setting can incur unnecessary inefficiencies such as slow convergence or high communication or computation cost, typically due to structural properties of the input graphs such as large diameters or skew in component sizes. We describe several optimization techniques to address these inefficiencies. Our most general technique is based on the idea of performing some serial computation on a tiny fraction of the input graph, complementing Pregel’s vertexcentric parallelism. We base our study on thorough implementations of several fundamental graph algorithms, some of which have, to the best of our knowledge, not been implemented on Pregellike systems before. The algorithms and optimizations we describe are fully implemented in our opensource Pregel implementation. We present detailed experiments showing that our optimization techniques improve runtime significantly on a variety of very large graph datasets.
An Experimental Comparison of Pregellike Graph Processing Systems∗
"... The introduction of Google’s Pregel generated much interest in the field of largescale graph data processing, inspiring the development of Pregellike systems such as Apache Giraph, GPS, Mizan, and GraphLab, all of which have appeared in the past two years. To gain an understanding of how Pregel ..."
Abstract

Cited by 8 (1 self)
 Add to MetaCart
(Show Context)
The introduction of Google’s Pregel generated much interest in the field of largescale graph data processing, inspiring the development of Pregellike systems such as Apache Giraph, GPS, Mizan, and GraphLab, all of which have appeared in the past two years. To gain an understanding of how Pregellike systems perform, we conduct a study to experimentally compare Giraph, GPS, Mizan, and GraphLab on equal ground by considering graph and algorithm agnostic optimizations and by using several metrics. The systems are compared with four different algorithms (PageRank, single source shortest path, weakly connected components, and distributed minimum spanning tree) on up to 128 Amazon EC2 machines. We find that the system optimizations present in Giraph and GraphLab allow them to perform well. Our evaluation also shows Giraph 1.0.0’s considerable improvement since Giraph 0.1 and identifies areas of improvement for all systems. 1.
Optimizations and Analysis of BSP Graph Processing Models on Public Clouds
"... Abstract — Largescale graph analytics is a central tool in many fields, and exemplifies the size and complexity of Big Data applications. Recent distributed graph processing frameworks utilize the venerable Bulk Synchronous Parallel (BSP) model and promise scalability for large graph analytics. Thi ..."
Abstract

Cited by 7 (3 self)
 Add to MetaCart
(Show Context)
Abstract — Largescale graph analytics is a central tool in many fields, and exemplifies the size and complexity of Big Data applications. Recent distributed graph processing frameworks utilize the venerable Bulk Synchronous Parallel (BSP) model and promise scalability for large graph analytics. This has been made popular by Google’s Pregel, which offers an architecture design for BSP graph processing. Public clouds offer democratized access to mediumsized compute infrastructure with the promise of rapid provisioning with no capital investment. Evaluating BSP graph frameworks on cloud platforms with their unique constraints is less explored. Here, we present optimizations and analysis for computationally complex graph analysis algorithms such as betweennesscentrality and allpairs shortest paths on a native BSP framework we have developed for the Microsoft Azure Cloud, modeled on the Pregel graph processing model. We propose novel heuristics for scheduling graph vertex processing in swaths to maximize resource utilization on cloud VMs that lead to a 3.5x performance improvement. We explore the effects of graph partitioning in the context of BSP, and show that even a well partitioned graph may not lead to performance improvement due to BSP's barrier synchronization. We end with a discussion on leveraging cloud elasticity for dynamically scaling the number of BSP workers to achieve a better performance than a static deployment, and at a significantly lower cost. Keywords Graph analytics; Cloud computing; Pregel;
Pregelix: Big(ger) Graph Analytics on A Dataflow Engine
"... There is a growing need for distributed graph processing systems that are capable of gracefully scaling to very large graph datasets. Unfortunately, this challenge has not been easily met due to the intense memory pressure imposed by processcentric, message passing designs that many graph process ..."
Abstract

Cited by 6 (0 self)
 Add to MetaCart
(Show Context)
There is a growing need for distributed graph processing systems that are capable of gracefully scaling to very large graph datasets. Unfortunately, this challenge has not been easily met due to the intense memory pressure imposed by processcentric, message passing designs that many graph processing systems follow. Pregelix is a new open source distributed graph processing system that is based on an iterative dataflow design that is better tuned to handle both inmemory and outofcore workloads. As such, Pregelix offers improved performance characteristics and scaling properties over current open source systems (e.g., we have seen up to 15× speedup compared to Apache Giraph and up to 35 × speedup compared to distributed GraphLab), and more effective use of available machine resources to support Big(ger) Graph Analytics. 1.
Recent advances in graph partitioning
, 2013
"... We survey recent trends in practical algorithms for balanced graph partitioning together with applications and future research directions. ..."
Abstract

Cited by 6 (2 self)
 Add to MetaCart
(Show Context)
We survey recent trends in practical algorithms for balanced graph partitioning together with applications and future research directions.
Scalable matrix computations on large scalefree graphs using 2d graph partitioning
 in Supercomputing
"... Scalable parallel computing is essential for processing large scalefree (powerlaw) graphs. The distribution of data across processes becomes important on distributedmemory computers with thousands of cores. It has been shown that twodimensional layouts (edge partitioning) can have significant a ..."
Abstract

Cited by 6 (1 self)
 Add to MetaCart
(Show Context)
Scalable parallel computing is essential for processing large scalefree (powerlaw) graphs. The distribution of data across processes becomes important on distributedmemory computers with thousands of cores. It has been shown that twodimensional layouts (edge partitioning) can have significant advantages over traditional onedimensional layouts. However, simple 2D block distribution does not use the structure of the graph, and more advanced 2D partitioning methods are too expensive for large graphs. We propose a new twodimensional partitioning algorithm that combines graph partitioning with 2D block distribution. The computational cost of the algorithm is essentially the same as 1D graph partitioning. We study the performance of sparse matrixvector multiplication (SpMV) for scalefree graphs from the web and social networks using several different partitioners and both 1D and 2D data layouts. We show that SpMV run time is reduced by exploiting the graph’s structure. Contrary to popular belief, we observe that current graph and hypergraph partitioners often yield relatively good partitions on scalefree graphs. We demonstrate that our new 2D partitioning method consistently outperforms the other methods considered, for both SpMV and an eigensolver, on matrices with up to 1.6 billion nonzeros using up to 16,384 cores. Keywords parallel computing, graph partitioning, scalefree graphs, sparse matrixvector multiplication, twodimensional distribution ∗Sandia is a multiprogram laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of
FACADE: A compiler and runtime for (almost) objectbounded big data applications
 In ASPLOS (2015
"... The past decade has witnessed the increasing demands on datadriven business intelligence that led to the proliferation of dataintensive applications. A managed objectoriented programming language such as Java is often the developer’s choice for implementing such applications, due to its quick dev ..."
Abstract

Cited by 5 (2 self)
 Add to MetaCart
(Show Context)
The past decade has witnessed the increasing demands on datadriven business intelligence that led to the proliferation of dataintensive applications. A managed objectoriented programming language such as Java is often the developer’s choice for implementing such applications, due to its quick development cycle and rich community resource. While the use of such languages makes programming easier, their automated memory management comes at a cost. When the managed runtime meets Big Data, this cost is significantly magnified and becomes a scalabilityprohibiting bottleneck. This paper presents a novel compiler framework, called FACADE, that can generate highlyefficient data manipulation code by automatically transforming the data path of an existing Big Data application. The key treatment is that in the generated code, the number of runtime heap objects created for data types in each thread is (almost) statically bounded, leading to significantly reduced memory management cost and improved scalability. We have implemented FACADE and used it to transform 7 common applications on 3 realworld, already welloptimized Big Data frameworks: GraphChi, Hyracks, and GPS. Our experimental results are very positive: the generated programs have (1) achieved a 3%–48 % execution time reduction and an up to 88 × GC reduction; (2) consumed up to 50 % less memory, and (3) scaled to much larger datasets.