DMCA
Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks (2007)
| Venue: | In EuroSys |
| Citations: | 762 - 27 self |
BibTeX
@INPROCEEDINGS{Isard07dryad:distributed,
author = {Michael Isard and Mihai Budiu and Yuan Yu and Andrew Birrell and Dennis Fetterly},
title = {Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks},
booktitle = {In EuroSys},
year = {2007}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Dryad is a general-purpose distributed execution engine for coarse-grain data-parallel applications. A Dryad applica-tion combines computational “vertices ” with communica-tion “channels ” to form a dataflow graph. Dryad runs the application by executing the vertices of this graph on a set of available computers, communicating as appropriate through files, TCP pipes, and shared-memory FIFOs. The vertices provided by the application developer are quite simple and are usually written as sequential programs with no thread creation or locking. Concurrency arises from Dryad scheduling vertices to run simultaneously on multi-ple computers, or on multiple CPU cores within a computer. The application can discover the size and placement of data at run time, and modify the graph as the computation pro-gresses to make efficient use of the available resources. Dryad is designed to scale from powerful multi-core sin-gle computers, through small clusters of computers, to data centers with thousands of computers. The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between ver-tices.
Keyphrases
sequential building block data-parallel program communica-tion channel multiple cpu core dryad applica-tion combine computational vertex thread creation run time available resource application developer dataflow graph coarse-grain data-parallel application general-purpose distributed execution engine powerful multi-core sin-gle computer computation pro-gresses difficult problem dryad execution engine shared-memory fifo available computer small cluster multi-ple computer sequential program data center computer failure concurrent application efficient use
