E. Miller and R. Katz. Input/Output Behavior of Supercomputer Applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....Accelerator Center. Ramakrishnan et al. 28] performed a study of the I O access patterns in a commercial computing environment on a VAX VMS system. Jensen and Reed [17] studied the patterns of access to the file archive of the National Center for Supercomputing Applications. Miller and Katz [22] traced certain I O intensive applications on the Cray Y MP at NASA Ames Research Center. Pasquale and Polyzos [25, 26] studied the file access characteristics of scientific applications on the Cray Y MP and Cray C90 at San Diego Supercomputer Center. Only recently have there been any efforts to ....

E. Miller and R. Katz. Input/Output Behavior of Supercomputer Applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

....requests and sequential prefetching can hide input output latency for this workload. Because the workload is relatively uniform, a single set of policies provides good performance for the majority of accesses. Supercomputer input output is dominated by accesses to very large, sequential files [55, 64]. Input output is due to compulsory reads of initialization data and writes to large checkpoint files. Therefore, sequential supercomputer input output systems are typically optimized for high throughput. Uniprocessor input output, whether from scientific applications or engineering workstations, ....

....Studies of typical engineering workloads on workstations under UNIX 4.2 BSD [61] and the Sprite distributed file system [3] reveal that most file accesses are small, sequential, and read only. Scientific workloads on sequential supercomputers have also been extensively studied. Miller and Katz [55] characterized the behavior of seven computational fluid dynamics and climate applications. File accesses were sequential and input output was bursty due to the iterative nature of the underlying algorithms. They identified three main categories of input output 9 requests: compulsory, cyclic ....

[Article contains additional citation context not shown here]

Miller, E. L., and Katz, R. H. Input/Output Behavior of Supercomputer Applications. In Proceedings of Supercomputing '91 (November 1991), pp. 567--576.

....Notable examples include Lawrie and Randell s study [8] of automatic file migration algorithms, Stritter s analysis [25] of file lifetime distributions, Smith s study [24] of mainframe file access behavior, and Reed and Jensen s study [6] of file archive accesses. More recently, Miller and Katz [10] captured detailed traces of application file accesses from a suite of Cray applications, identifying compulsory, checkpoint, and staging input output. Pasquale and Polyzos [11, 12] followed with two additional studies of vector workloads, concluding that most input output had regular behavior. ....

Miller, E. L., and Katz, R. H. Input /Output Behavior of Supercomputer Applications. In Proceedings of Supercomputing '91 (November 1991), pp. 567--576.

....computing environment, on a VAX VMS platform. There have been a few studies of I O from scientific workloads. Del Rosario and Choudhary [10] provided an informal characterization of some grand challenge applications. Powell [26] concentrated mainly on files sizes on a Cray 1. Miller and Katz [21] and Pasquale and Polyzos [24] studied I O intensive Cray applications. Jensen and Reed traced file archive activity on a Cray at NCSA [16] Experimental studies of I O from parallel scientific programs running on multiprocessors have been rather limited. Crockett [7] and Kotz and Ellis [18] ....

E. L. Miller and R. H. Katz. Input/Output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

.... Experience has shown that this simple model of a file is well suited to uniprocessor applications that tend to access files in a simple, sequential fashion [OCH 85] It has similarly proven to be appropriate for scientific, vector applications that also tend to access files sequentially [MK91] Results in [KN94] however, show that sequential access to consecutive portions of a file is much less common in a multiprocessor environment. So, while the simple Unix like interface has worked well in the past, it is clear that it is not well suited to parallel applications, which have more ....

....from the same compute node, and a consecutive request to be a sequential request that begins where the previous request ended. A common characteristic of many file system workloads, particularly scientific file system workloads, is that files are accessed consecutively [OCH 85, BHK 91, MK91] In the parallel file system workload, we found that while almost 93 of all files were accessed sequentially, consecutive access was primarily limited to those files that were only opened by one compute node. When a file was opened by just a single node, 93 of those nodes accessed the file ....

[Article contains additional citation context not shown here]

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

....computing environment, on a VAX VMS platform. There have been a few studies of I O from scientific workloads. Del Rosario and Choudhary [10] provided an informal characterization of some grand challenge applications. Powell [26] concentrated mainly on file sizes on a Cray 1. Miller and Katz [21] and Pasquale and Polyzos [24] studied I O intensive Cray applications. Jensen and Reed traced file archive activity on a Cray at NCSA [16] Experimental studies of I O from parallel scientific programs running on multiprocessors have been rather limited. Crockett [7] and Kotz and Ellis [18] ....

E. L. Miller and R. H. Katz. Input/Output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

....SIO API and the design issues that are important for parallel I O performance. Finally, x7 summarizes our observations. 2 Related Work The first I O characterization efforts of scientific applications on vector supercomputers concluded that I O behavior is regular, recurrent, and predictable [8, 10], characteristics that were attributed to the iterative nature of such applications. In contrast to the results on vector systems, recent analyses on the Intel Paragon XP S [4] the Intel iPSC 860 [6] and the CM 5 [12] showed greater irregularity in I O access patterns, with the majority of file ....

Miller, E. L., and Katz, R. H. Input/Output behavior of supercomputer applications. In Supercomputing '91 (November 1991), pp. 567--576.

....for the noncommercialreproductionof the complete work for educational or research purposes. was designed for a general purpose workload [OCH 85] rather than for a parallel, scientific workload. Scientific applications, on the other hand, use larger files and have more sequential access [MK91, GGL93, PP93] Parallel scientific programs access the file with patterns not seen in uniprocessor or distributed system workloads, in particular, complex strided access to discontiguous pieces of the file [KN94, NK94] Finally, scientific applications use files for more than loading raw data ....

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

....(CPs) and I O processors (IOPs) Disks attach only to IOPs, which run the file system code. Applications run only on the CPs. for a general purpose workload [OCH 85] rather than for a parallel, scientific workload. Scientific applications use larger files and have more sequential access [MK91, GGL93, PP93, PP94] Parallel scientific programs access the file with patterns not seen in uniprocessor or distributed system workloads. Although there seems to be a wide variety of access patterns [NKP 96, CACR95, SACR96] we have noticed many patterns accessing small, discontiguous pieces ....

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

.... 95, NKP 95] Specifically, it was commonly believed that parallel, scientific applications would have behavior similar to that of existing sequential and vector scientific applications [Pie89, PFDJ89, LIN 93] These applications tend to access large files in large, consecutive chunks [MK91, PP93] Studies of Copyright c fl1996 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage ....

.... has shown that the simple, Unix like model of a file is well suited to uniprocessor applications that tend to access files in a simple, sequential fashion [OCH 85] It has similarly proven to be appropriate for scientific, vector applications that also tend to access files sequentially [MK91] Until recently, however, there had been no investigation into whether this file model and interface were well suited to massively parallel scientific applications. To determine whether this model was appropriate, we examined the file system workloads on two different massively parallel ....

Ethan L. Miller and Randy H. Katz. Input /output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

.... Experience has shown that this simple model of a file is well suited to uniprocessor applications that tend to access files in a simple, sequential fashion [OCH 85] It has similarly proven to be appropriate for scientific, vector applications that also tend to access files sequentially [MK91] Results in [KN94, PEK 95] however, show that sequential access to consecutive portions of a file is much less common in a multiprocessor environment. So, while the simple Unix like interface has worked well in the past, it is clear that it is not well suited to parallel applications, which ....

....from the same compute node, and a consecutive request to be a sequential request that begins where the previous request ended. A common characteristic of many file system workloads, particularly scientific file system workloads, is that files are accessed consecutively [OCH 85, BHK 91, MK91] In the parallel file system workload, we found that while almost 93 of all files were accessed sequentially, consecutive access was primarily limited to those files that were only opened by one compute node. When files were opened by just a single node, 93 of those files were accessed ....

[Article contains additional citation context not shown here]

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

....earlier version of this paper appears in the First Symposium on Operating Systems Design and Implementation (OSDI) November 1994. a general purpose workload [OCH 85] rather than for a parallel, scientific workload. Scientific applications use larger files and have more sequential access [MK91, GGL93, PP93] Parallel scientific programs access the file with patterns not seen in uniprocessor or distributed system workloads, in particular, complex strided access to discontiguous pieces of the file [KN94, NK94] Finally, scientific applications use files for more than loading raw data ....

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

....but makes no qualitative changes. is often poor. Poor performance is not surprising because the Unix file system was designed for a general purpose workload [OCH 85] rather than for a parallel, scientific workload. Scientific applications use larger files and have more sequential access [MK91, GGL93, PP93] Parallel scientific programs access the file with patterns not seen in uniprocessor or distributed system workloads, in particular, complex strided access to discontiguous pieces of the file [KN94, NK94] Finally, scientific applications use files for more than loading raw data ....

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

.... File System (CFS) and found that its caching improved performance, but they were not able to compare several policies [PFDJ89, FPD91] More recent studies found that, in general, despite its cache performance, CFS performance lags well behind hardware capabilities [Nit92, BCR93] Miller and Katz [MK91] used application traces from a Cray supercomputer to drive cache simulations, and found that there was not enough temporal locality for a file system cache to provide significant performance gains. When caching alone does not work well, prefetching can be useful for reads. French et al. studied ....

E. L. Miller and R. H. Katz. Input/Output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

....systems [17] Furthermore, workload characterization is critical to effective benchmarking and capacity planning. One of the first I O characterization efforts of scientific applications was conducted on vector supercomputers and concluded that I O behavior is cyclic, bursty, and predictable [8, 12], characteristics that were attributed to the iterative nature of the applications. I O activity for these applications is mainly observed at the beginning and or the end of program execution and is classified as compulsory, while periodic checkpointing and data staging is also reported [8] ....

....[8, 12] characteristics that were attributed to the iterative nature of the applications. I O activity for these applications is mainly observed at the beginning and or the end of program execution and is classified as compulsory, while periodic checkpointing and data staging is also reported [8]. Subsequent characterization efforts that focused exclusively on scientific applications included either measurements from synthetic benchmarks that emulated the behavior of common scientific codes [11] or studies that focused on the behavior of individual applications and on ways to restructure ....

[Article contains additional citation context not shown here]

E. L. Miller and R. H. Katz. Input/Output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

....applications. Studies of I O from scientific workloads have been fairly limited in scope and generality. Del Rosario and Choudhary [dC94] provided an informal characterization of some grand challenge applications. Powell [Pow77] concentrated mainly on files sizes on a Cray 1. Miller and Katz [MK91] and Pasquale and Polyzos [PP93] studied I O intensive Cray applications. Experimental studies of I O from parallel scientific programs running on multiprocessors have been rather limited. Crockett [Cro89] and Kotz and Ellis [Kot93] described hypothetical characterizations of a parallel ....

....large I O s to accommodate this bipolar workload. Figures 11 and 12 show read request sizes from CMF and CMMD applications and they exhibit behavior similar to the write requests. 4. 4 Sequentiality Uniprocessor scientific applications access files in a predominantly sequential manner [MK91] For parallel applications we need to define sequential access carefully. As in the iPSC study, we define a sequential request to be one at a higher file offset than the previous request, and a consecutive request to be one that begins exactly at the same offset where the previous request ....

E. L. Miller and R. H. Katz. Input/Output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

.... the development of most parallel file systems were incorrect [KN94, NK95, PEK 95] It was commonly believed that parallel scientific applications would behave like sequential and vector scientific applications: accessing large files in large, consecutive chunks [Pie89, PFDJ89, LIN 93, MK91] Studies of two parallel file system workloads, running a variety of applications in a variety of scientific domains, at two sites on two architectures, under both data parallel and control parallel programming models, show that many applications make many small, regular, but non consecutive ....

Ethan L. Miller and Randy H. Katz. Input /output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

....SIO API and the design issues that are important for parallel I O performance. Finally, x7 summarizes our observations. 2 Related Work The first I O characterization efforts of scientific applications on vector supercomputers concluded that I O behavior is regular, recurrent, and predictable [16, 21], characteristics that were attributed to the iterative nature of such applications. In contrast to the results on vector systems, recent analyses on the Intel Paragon XP S [7] the Intel iPSC 860 [13] and the CM 5 [23] showed greater irregularity in I O access patterns, with the majority of ....

Miller, E. L., and Katz, R. H. Input/Output behavior of supercomputer applications. In Proceedings of Supercomputing '91 (November 1991), pp. 567--576.

....examined scientific workloads. Del Rosario and Choudhary provide an informal characterization of grand challenge applications [10] Powell measured file sizes on a Cray 1 file system [31] Miller and Katz traced specific I O intensive Cray applications to determine the per file access patterns [25], focusing primarily on access rates. Pasquale and Polyzos studied I O intensive Cray applications, focusing on patterns in the I O rate [29] All of these studies are limited to uniprocess applications on vector supercomputers. Scientific parallel applications. Crockett [7] and Kotz [20] ....

.... that the throughput of CFS can be disappointing relative to the capabilities of the hardware [27, 3] Miller and Katz drove a cache simulation using traces from a Cray supercomputer and found that access locality was not high enough for significant benefits to be realized from a file system cache [25]. 2.4 Intel iPSC 860 and CFS The iPSC 860 is a distributed memory, messagepassing, MIMD machine. The compute nodes are based on the Intel i860 processor and are connected by a hypercube network. I O is handled by dedicated I O nodes, which are each connected to a single compute node rather than ....

[Article contains additional citation context not shown here]

E. L. Miller and R. H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, Nov. 1991.

.... and Choudhary provide an informal characterization of grand challenge applications [dC94] Powell measured a set of static characteristics (file sizes) of a Cray 1 file system [Pow77] Miller and Katz traced specific I O intensive Cray applications to determine the per file access patterns [MK91] focusing primarily on access rates. Miller and Katz also measured secondary tertiary file migration patterns on a Cray [MK93] giving a good picture of long term, whole file access patterns. Pasquale and Polyzos studied I O intensive Cray applications, focusing on patterns in the I O rate ....

.... throughput of CFS can be disappointing relative to the capabilities of the hardware [Nit92, BCR93] Miller and Katz drove a cache simulation using traces from a Cray supercomputer and found that access locality was not high enough for significant benefits to be realized from a file system cache [MK91] 2.4 Intel iPSC 860 and CFS The iPSC 860 is a distributed memory, message passing, MIMD machine. The compute nodes are based on the Intel i860 processor and are connected by a hypercube network. I O is handled by dedicated I O nodes, which are each connected to a single compute node rather ....

[Article contains additional citation context not shown here]

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

.... For example, it was generally assumed that scientific applications designed to run on a multiprocessor would behave in the same fashion as scientific applications designed to run on sequential and vector supercomputers: accessing large files in large, consecutive chunks [Pie89, PFDJ89, LIN 93, MK91] Instead, our observations show that many scientific applications make many small, regular, but non consecutive requests to the file system. Using the results from our workload characterizations and from performance evaluations of existing multiprocessor file systems, we have developed Galley. ....

....del Rosario and Choudhary provide an informal characterization of grand challenge applications. Powell measured a set of static characteristics (file sizes) of a Cray 1 file system [Pow77] Miller and Katz traced specific I O intensive Cray applications to determine the per file access patterns [MK91] focusing primarily on access rates. Miller and Katz also measured secondary tertiary file migration patterns on a Cray [MK93] giving a good picture of long term, whole file access patterns. Pasquale and Polyzos studied I O intensive Cray applications, focusing on patterns in the I O rate ....

[Article contains additional citation context not shown here]

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

.... Kotz Technical Report PCS TR93 190 Department of Math and Computer Science Dartmouth College Hanover, NH 03755 3551 David.Kotz Dartmouth.edu May 14, 1993 Fast file systems are critical for high performance scientific computing, since many scientific applications have tremendous I O requirements [MK91] Many parallel supercomputers have only recently obtained fully parallel I O architectures and file systems, which are necessary for scalable I O performance. Scalability aside, I show here that many systems lack sufficient absolute performance. I examined several the papers in the literature ....

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

....Uniprocessor and distributed system file access patterns have been measured many times [7, 13, 1] Sequential access is most common. Supercomputer file access patterns (a scientific workload) involve huge files (tens to thousands of megabytes) accessed primarily sequentially, sometimes repeatedly [11]. Five parallel scientific applications, chosen from the PERFECT benchmarks [16] and parallelized for an eight processor Alliant, have only sequential access patterns. 18] This is only a small sample, however, and the programs are parallelized sequential programs, not parallel programs per se. ....

E. L. Miller and R. H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, Nov. 1991.

.... PEK 95] It was generally assumed that scientific applications designed to run on a multiprocessor would behave in the same fashion as scientific applications designed to run on sequential and vector supercomputers: accessing large files in large, consecutive chunks [Pie89, PFDJ89, LIN 93, MK91] Studies of two different multiprocessor file system workloads, This research was funded by NSF under grant number CCR 9404919 and by NASA Ames under agreement numbers NCC 2 849 and NAG 2 936. running a variety of applications in a variety of scientific domains, on two architectures, under ....

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

.... file systems were incorrect [12,18,25] It was generally assumed that scientific applications designed to run on a multiprocessor would behave in the same fashion as scientific applications designed to run on sequential and vector supercomputers: accessing large files in large, consecutive chunks [23,24,15,16]. Studies of two different multiprocessor file system workloads, running a variety of applications in a variety of scientific domains, on two architectures, under both data parallel and control parallel programming models, show that many applications make many small, regular, but non consecutive ....

Ethan L. Miller and Randy H. Katz. Input/output behavior of supercomputer applications. In Proceedings of Supercomputing '91, pages 567--576, November 1991.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC