DMCA
Computation and Data Scheduling for Large-Scale Distributed Computing
BibTeX
@MISC{Ranganathan_computationand,
author = {Kavitha Ranganathan and Ian Foster},
title = {Computation and Data Scheduling for Large-Scale Distributed Computing},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
In high-energy physics, bioinformatics, and other disciplines, we encounter applications involving numerous, loosely coupled jobs that both access and generate large data sets. So-called Data Grids seek to harness geographically distributed resources for such large-scale data-intensive problems. Yet effective scheduling in such environments is challenging, due to a need to address a variety of metrics and constraints (e.g., resource utilization, response time, global and local allocation policies) while dealing with multiple, potentially independent sources of jobs and a large number of storage, compute, and network resources. We describe a scheduling framework that addresses these problems. Within this framework, data movement operations may be either tightly bound to job scheduling decisions or, alternatively, performed by a decoupled, asynchronous process on the basis of observed data access patterns and load. We develop a family of job scheduling and data movement (replication) algorithms and use simulation studies to evaluate various combinations. Our results suggest that while it is necessary to consider the impact of replication on the scheduling strategy, it is not always necessary to couple data movement and computation scheduling. Instead, these two activities can be addressed separately, thus significantly simplifying the design and implementation of the overall Data Grid system.
Keyphrases
data scheduling large-scale distributed computing data movement asynchronous process scheduling framework use simulation study observed data access pattern computation scheduling large number network resource large data set response time scheduling strategy various combination distributed resource so-called data grid seek large-scale data-intensive problem local allocation policy overall data grid system data movement operation independent source high-energy physic job scheduling resource utilization effective scheduling
