by Kavitha Ranganathan, Ian Foster
in Proceedings of the 11th IEEE Symposium on High Performance Distributed Computing (HPDC
http://www.cs.uchicago.edu/~krangana/papers/decoupling_comp_data.ps
Add To MetaCart
Abstract:
In high energy physics, bioinformatics, and other disciplines, we encounter applications involving numerous, loosely coupled jobs that both access and generate large data sets. Socalled Data Grids seek to harness geographically distributed resources for such large-scale data-intensive problems. Yet effective scheduling in such environments is challenging, due to a need to address a variety of metrics and constraints (e.g., resource utilization, response time, global and local allocation policies) while dealing with multiple, potentially independent sources of jobs and a large number of storage, compute, and network resources. We describe a scheduling framework that addresses these problems. Within this framework, data movement operations may be either tightly bound to job scheduling decisions or, alternatively, performed by a decoupled, asynchronous process on the basis of observed data access patterns and load. We develop a family of job scheduling and data movement (replication) algorithms and use simulation studies to evaluate various combinations. Our results suggest that while it is necessary to consider the impact of replication on the scheduling strategy, it is not necessary to couple data movement and computation scheduling. Instead, these two activities can be addressed separately, thus significantly simplifying the design and implementation of the overall Data Grid system.
Citations
|
1187
|
Globus: A Metacomputing Infrastructure Toolkit, Int
– Foster
- 1997
|
|
1131
|
The Anatomy of the Grid: Enabling Scalable Virtual Organizations
– Foster, Kesselman, et al.
|
|
982
|
The Grid: Blueprint for a New Computing Infrastructure
– Foster, Kesselman
- 1999
|
|
433
|
Summary cache: A scalable wide-area web cache sharing protocol
– Fan, Cao, et al.
- 1998
|
|
373
|
Grid Information Services for Distributed Resource Sharing
– Czajkowski, Fitzgerald, et al.
- 2001
|
|
221
|
On the Scale and Performance of Cooperative Web Proxy Caching
– Wolman, Voelker, et al.
- 1999
|
|
193
|
Dynamically forecasting network performance to support dynamic scheduling using the network weather service
– Wolski
- 1997
|
|
138
|
TheAppLeS Parameter Sweep Template: User-Level Middleware for the Grid
– Casanova, Obertelli, et al.
|
|
121
|
GASS: A Data Movement and Access Service for Wide Area Computing Systems
– Bester, Foster, et al.
- 1999
|
|
64
|
Identifying Dynamic Replication Strategies for a High Performance
– Ranganathan, Foster
- 2001
|
|
59
|
The Kangaroo Approach to Data Movement on the Grid
– Thain, Basney, et al.
- 2001
|
|
55
|
Demand-based document dissemination to reduce tra c and balance load in distributed information systems
– Bestavros
- 1995
|
|
42
|
Evaluation of JobScheduling Strategies for Grid Computing
– Hamscher, Schwiegelshohn, et al.
- 2000
|
|
41
|
The GriPhyN Project: Towards Petascale Virtual Data Grids
– Avery, Foster
- 2001
|
|
32
|
Performance study of a collaborative method for hierarchical caching in proxy servers
– Yu, MacNair
- 1998
|
|
23
|
Gathering at the well: Creating communities for grid I/O
– Thain, Bent, et al.
- 2001
|
|
18
|
Utilizing Widely Distributed Computational Resources Efficiently with Execution Domains
– Basney, Livny, et al.
- 2001
|
|
17
|
Harnessing the Capacity of Computational Grids for High Energy Physics
– Basney, Livny, et al.
- 2000
|
|
15
|
A unified resource scheduling framework for heterogeneous computing environments
– Alhusaini, Prasanna, et al.
- 1999
|
|
8
|
Dynamic-level scheduling for heterogeneous processor networks
– Sih, Lee
- 1990
|
|
6
|
CMS Requirements for the Grid
– Holtman
- 2001
|
|
1
|
A Taxonomy of scheduling in general-purpose distributed computing systems
– Braun
- 1998
|
|
1
|
H.J.Siegel and D.Hensgen, Dynamic Matching and Scheduling of a Class of Independent Tasks onto Heterogeneous Computing Systems
– Maheswaran
- 1999
|
