D. Kotz. Disk-directed I/O for MIMD Multiprocessors. In Proc. of the First USENIX Symp. on Operating Systems Design and Implementation, 1994.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....Introduction Data intensive applications have presented challenging problems to computational scientists. A major problem is the I O performance when many datasets are stored and frequently accessed. Over the years, scientists have developed techniques for I O optimizations such as collective I O [14, 38, 23, 10], prefetching [15, 13] data sieving [38, 10] caching [8] and so on. But the performance problem still persists because data intensive applications are Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL 60208, choudhar ece.nwu.edu generating more data and ....

D. Kotz. Disk-directed i/o for mimd multiprocessors. In Proc. the 1994.

....and complex scientific data. The user program can send query requests to MDMS to obtain information about data structures that will be accessed. Then, the user can use this information in accessing the HSS in an optimal manner, taking advantage of powerful I O optimizations like collective I O [35, 8, 23], prefetching [19] prestaging [14] and so on. The user program can also send access pattern hints to the MDMS and let the MDMS to decide the best I O strategy considering the storage layout of the data in question. These access pattern hints span a wide spectrum that contains inter processors ....

D. Kotz. Disk-directed i/o for mimd multiprocessors. In Proc. the 1994.

....certain optimizations. Another advantage is that if any processor needs to access some data which was previously modified by some other processor, it can be done using just a read call without any additional synchronization. The idea of collective I O has also been used in other schemes such as in [1, 11, 10]. In the next section, we describe the Extended Two Phase Method for reading sections of out of core arrays. The method for writing sections is analogous and is discussed in Section 6. 5 Reading Sections of Out of Core Array Let us assume that each processor needs to read some regular section ....

....proposed in this paper provides similar functionality using only a single routine for reading sections and a single routine for writing sections, which can be implemented very easily. It does not incur the overhead of managing many processes and the associated context switching. Disk directed I O [11] is another technique which uses collective I O. In disk directed I O, compute processors collectively send a single request to all I O processors, which then perform I O efficiently, and send the data to the compute processors. In other words, it is the I O processors that decide the best way of ....

D. Kotz. Disk-directed I/O for MIMD Multiprocessors. In Proceedings of the First Symposium on Operating Systems Design and Implementation (OSDI), November 1994.

....code reuse, and ease of programming. Notice also that al though the parallel I O libraries (e.g. 6] built on top of parallel file systems have the potential to provide both ease of use and high performance through the use of advanced I O optimization techniques such as col lective I O [14,20], and array chunking [26] their ex tensibility is severely limited by the design principles and the programming language used in their implemen tation [19] At the other end of the spectrum are database man agement systems (DBMS) They provide a layer on top of file systems, which is ....

D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proc. the 1994.

....are run on a 16 node IBM SP2 at CPDC of Northwestern University. Each node of the SP2 is RS 6000 Model 390 processor. The HPSS is located at SDSC. We also use the performance model in the previous section to analyze the results that we obtained. 4. 1 Remote Collective I O Collective I O [9, 2, 5] is a technique to glean individual I O requests of each processor into a global picture and then decide how the combined I O requests should be best performed. The two phase I O [9, 2] is one of significant implementation techniques of collective I O. In the two phase scheme, the data is accessed ....

D. Kotz. Disk-directed i/o for mimd multiprocessors. In Proc. the 1994.

....some I O systems perform best when the number of simultaneous accesses is limited [KRY 93, NIT 92] These systems in particular benefit from this approach. 2.3. Disk directed I O Disk directed I O (DDIO) is a combination of a number of other techniques for data transfer in parallel I O systems [KOT 97] DDIO was developed after both the data sieving and two phase techniques, and it relies on both noncontiguous and collective I O primitives in the file system. Additionally the I O servers must be able to map file locations into disk block positions and must be capable of reasonably predicting ....

KOTZ D., "Disk-directed I/O for MIMD Multiprocessors", ACM Transactions on Computer Systems, vol. 15, num. 1, 1997, p. 41--74, ACM Press.

....component in the system, and few of them provide a programming environment, leading to simple static partitioning approaches in the programs. More advanced parallel file systems have specified higher level interfaces to data via collective I O (a similar concept is expressed with two phase I O) Kot94, CBH 94] In the original paper, Kotz found that many scientific codes show tremendous improvement by aggregating I O requests and then shipping them to the underlying I O system; the I O nodes can then schedule the requests, and often noticeably increase delivered bandwidth. Because requests ....

David Kotz. Disk-directed I/O for MIMD multiprocessors. In Proceedings of the 1994 Symposium on Operating Systems Design and Implementation, pages 61--74. USENIX Association, November 1994. 104

....number of timesteps has been completed. Efficient transfer of this output from memory to local or remote disks is very important to achieve high performance for such applications. Often the output data are disjoint subsets of a logically shared data set, encouraging the use of collective I O [1, 3, 7, 9, 11]. In this approach, all the processors cooperate to transfer data between disk and memory. Information about the on disk and in memory layouts of the data set is used to plan efficient file operations, and reorganize the data across the memory of the processors if necessary. Most collective I O ....

D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proceedings of the Symposium on Operating Systems Design and Implementation, November 1994. Through the Center for Simulation of Advanced Rockets (CSAR) at UIUC.

....the user program and the HSS. The user program can send query requests to MDMS to obtain information about data structures that will be accessed. Then, the user can use this information in accessing the HSS in an optimal manner, taking advantage of powerful I O optimizations like collective I O [36, 8, 23], prefetching [20] prestaging [13] and so on. The user program can also send access pattern hints to the MDMS and let the MDMS to decide the best I O strategy considering the storage layout of the data in question. These access pattern hints span a wide spectrum that contains inter processors ....

D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proc. the 1994.

....to create a single, big I O request and sent to the storage system. As a result, the effective I O bandwidth is 2 P2 P3 Collective Request Access P1 P2 P3 P4 Multi Collective P2 P3 P2 P3 Figure 1: Overview of the MCIO. significantly increased. This optimization has many variants [19, 13, 22]; although any CIO technique can be used for MCIO, the one used in this study is two phase I O as implemented in ROMIO, a portable implementation of MPI IO from Argonne National Laboratories [26] ROMIO has been incorporated into several MPI libraries, including the MPI implementations of several ....

....is, each data item is delivered to its final destination) Several variations of the collective I O technique have been proposed in previous research. In node grouping [20] nodes making I O requests are partitioned into groups. Then, they take turns in performing the I O. In disk directed I O [13], one compute node sends the col P2 P3 P4 Figure 4: An access pattern involving four processors and eight files. lective request to all I O nodes. I O nodes optimize the access, perform the request, and send the data directly back to all compute nodes. 4 Multi Collective I O In this ....

Kotz, D. Disk-directed I/O for MIMD multiprocessors. In Proc. the 1994.

....increasing number of I O nodes. And the buffer which each I O node required will have similar variation. This implies that there is an optimal number of I O nodes to a given I O workload. These are consistent with previous I O study by S. J. Baylor, C. E. Wu, and D. Kotz, Nils in [BBY94] BW95] [DK94]. The I O rate(throughput) vary with I O workloads and number of I O nodes. There is a maximum I O rate for a given I O subsystem configuration 5 Conclusion Our discussion shows that the hybrid modelings technique (involves both analytic modeling and simulation modeling) is an effective way for ....

David Kotz, "Disk-directed I/O for MIMD multiprocessors", Dartmouth College Technical Report PCS-TR94-226.

....and parallel disks. One of the most important optimizations in MPI IO [7] an emerging parallel I O standard, is collective I O, an optimization that allows each processor to do I O on behalf of other processors if doing so improves the overall performance [5] This optimization has many variants [16, 13, 18]; the one used in this study is two phase I O as implemented in MPI 2 standard [14] In this implementation, I O is performed in two phases: an I O phase and a communication phase. In the I O phase, processors perform I O in a way that is most beneficial from the storage layout point of view. In ....

D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proc. the 1994.

....to create a single, big I O request and sent to the storage system. As a result, the effective I O bandwidth is P2 P3 Collective Request Access P1 P2 P3 P4 Multi Collective P2 P3 P2 P3 Figure 1: Overview of the MCIO. significantly increased. This optimization has many variants [19, 13, 22]; although any CIO technique can be used for MCIO, the one used in this study is two phase I O as implemented in ROMIO, a portable implementation of MPI IO from Argonne National Laboratories [26] ROMIO has been incorporated into several MPI libraries, including the MPI implementations of several ....

....is, each data item is delivered to its final destination) Several variations of the collective I O technique have been proposed in previous research. In node grouping [20] nodes making I O requests are partitioned into groups. Then, they take turns in performing the I O. In disk directed I O [13], one compute node sends the col P2 P3 P4 Figure 4: An access pattern involving four processors and eight files. lective request to all I O nodes. I O nodes optimize the access, perform the request, and send the data directly back to all compute nodes. 4 Multi Collective I O In this ....

Kotz, D. Disk-directed I/O for MIMD multiprocessors. In Proc. the 1994.

....code to execute on or near the data servers, when that helps to reduce network traffic. There are many good reasons to allow applicationspecific functionality to run on processors other than the clients. Application specific data distribution patterns, perhaps to support disk directed I O [21], need to execute on the data servers. If the distribution of data across clients or across disks is dependent on the value of the data, moving that function to the data server can halve network traffic [22] Processors near the data servers can filter data in an application specific way, passing ....

D. Kotz, Disk-directed I/O for MIMD multiprocessors, ACM Trans. Comput. Syst. 15 (1) (1997) 41--74.

No context found.

D. Kotz. Disk-directed I/O for MIMD Multiprocessors. In Proc. of the First USENIX Symp. on Operating Systems Design and Implementation, 1994.

No context found.

D. Kotz. Disk-Directed I/O for MIMD Multiprocessors. In Hai Jin, Tomi Cortes, and R. Buyya, editors, High Performance Mass Storage and Parallel Input/Output: Technologies and Applications, pages 513--535. IEEE Computer Society and John Wiley & Sons, 2001.

No context found.

David Kotz. Disk-directed I/O for MIMD multiprocessors. In Hai Jin, Toni Cortes, and Rajkumar Buyya, editors, High Performance Mass Storage and Parallel I/O: Technologies and Applications, chapter 35, pages 513--535. IEEE Computer Society Press and John Wiley & Sons, 2001.

No context found.

D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proceedings of the 1994.

No context found.

D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proceedings of the 1994.

No context found.

D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proceedings of the 1994.

No context found.

David Kotz. Disk-directed I/O for MIMD multiprocessors. In Proceedings of the 1994 Symposium on Operating Systems Design and Implementation, pages 61-- 74. USENIX Association, November 1994. Updated as Dartmouth TR PCS-TR94-226 on November 8, 1994.

No context found.

D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proceedings of the 1994.

No context found.

D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proc. the 1994.

No context found.

D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proc. the 1994.

No context found.

David Kotz. Disk-directed I/O for MIMD multiprocessors, February 1997. 69

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC