David Kotz and Nils Nieuwejaar, "Dynamic file-access characteristics of a production parallel scientific workload," in Proceedings of Supercomputing '94, Washington, DC, 1994, pp. 640--649, IEEE Computer Society Press.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....are properly equipped to specify detailed goals. There is some hope that availability and reliability goals can be derived from various business and insurance costs (related to risk management) but the proper way to set performance targets is a different problem. Workload characterization [4, 8] can provide a first estimate of performance requirements and can even be used to guide storage system design [2, 3] but further tuning by the administrator will be necessary. Providing an easy to use interface for this refinement is a necessity. We believe that using a system of complaints from ....

David Kotz and Nils Nieuwejaar. Dynamic file-access characteristics of a production parallel scientific workload. ACM Symposium on Operating System Principles (Washington, DC, USA, 14--18 November 1994.

....represents a significant obstacle to achieving good performance. The problem is generally not with the hardware; many parallel I O subsystems offer excellent performance. Rather, the problem arises from other factors, primarily the I O patterns exhibited by many parallel scientific applications [5, 14]. In particular, each processor tends to make a large number of small I O requests, incurring the high cost of I O on each such request. One reason for this access pattern is that parallel scientific codes frequently involve large arrays distributed across the processor s local memory. After a ....

Kotz, D. and N. Nieuwejaar. Dynamic file-access characteristics of a production parallel scientific workload. In Supercomputing '94 pages 640-649, November 1994.

....represents a significant obstacle to achieving good performance. The problem is generally not with the hardware; many parallel I O subsystems offer excellent performance. Rather, the problem arises from other factors, primarily the I O patterns exhibited by many parallel scientific applications [5, 14]. In particular, each processor tends to make a large number of small I O requests, incurring the high cost of I O on each such request. One reason for this access pattern is that parallel scientific codes frequently involve large arrays distributed across the processor s local memory. After a ....

Kotz, D. and N. Nieuwejaar. Dynamic file-access characteristics of a production parallel scientific workload. In Supercomputing '94 pages 640-649, November 1994.

....in order to run operations on distributed data structures in parallel. CHARISMA (CHARacterize I O in Scientific Multiprocessor Applications) CHARISMA is a project to characterize I O in scientific multiprocessor applications from a variety of production parallel computing platforms and sites [23]. It recorded individual read and write requests in live, multiprogramming workloads. It turned out that most files were accessed in complex, highly regular patterns. checkpointing Checkpointing allows processes to save their state from time to time so that they can be restarted in case of ....

David Kotz and Nils Nieuwejaar. Dynamic file-access characteristics of a production parallel scientific workload. In Proceedings of Supercomputing '94, pages 640--649, Washington, DC, November 1994. IEEE Computer Society Press.

....and represents a significant obstacle to achieving good performance. The problem is often not with the hardware; many parallel I O subsystems offer excellent performance. Rather, the problem arises from other factors, primarily the I O patterns exhibited by many parallel scientific applications [9, 18, 2, 3, 22, 24, 25, 28]. In particular, each processor tends to make a large number of small I O requests, incurring the high cost of I O on each such request. One reason for this access pattern is that parallel scientific codes frequently involve large arrays distributed across the processor s local memory. After a ....

Kotz, D. and N. Nieuwejaar. Dynamic File-Access Characteristics of a Production Parallel Scientific Workload. In Supercomputing '94 pages 640--649, November 1994.

....and represents a significant obstacle to achieving good performance. The problem is often not with the hardware; many parallel I O subsystems offer excellent performance. Rather, the problem arises from other factors, primarily the I O patterns exhibited by many parallel scientific applications [9, 18, 2, 3, 22, 24, 25, 28]. In particular, each processor tends to make a large number of small I O requests, incurring the high cost of I O on each such request. One reason for this access pattern is that parallel scientific codes frequently involve large arrays distributed across the processor s local memory. After a ....

Kotz, D. and N. Nieuwejaar. Dynamic File-Access Characteristics of a Production Parallel Scientific Workload. In Supercomputing '94 pages 640--649, November 1994.

....general parallel file operations. We say files relying on this generality use Independent Buffering Mode (IB) because each VP s stream is managed separately. Many debugged, production quality, data parallel applications do not require such generality; they perform very 5 regular file operations [1, 2, 5, 7, 8, 10]. When regular operations are performed, all VPs read or write the same amount of data at the same time. In such instances, a single value can act as the file pointer for all the VP streams. During a write, each VP copies data from its parallel variable into its file buffer. Under the assumption ....

D. Kotz and N. Nieuwejaar. Dynamic file-access characteristics of a production parallel scientific workload. In Proceedings of Supercomputing '94, pages 640--649, November 1994.

....6, and draw conclusions and outline the future work in section 7. 2 Panda basics Panda is designed for large scale scientific applications performing collective array I O operations on distributedmemory multiprocessors, a commonly observed I O pattern in large scale scientific applications [26, 21]. In this paper, we base our discussion on Panda 2.1 with each node either dedicated to application computation or I O. The dedicated I O nodes are called Panda servers and the compute nodes are called Panda clients . During each I O request, Panda servers read or write array data from or to ....

D. Kotz and N. Nieuwejaar. Dynamic file-access characteristics of a production parallel scientific workload. In Proceedings of Supercomputing '94, pages 640--649, Washington, DC, November 1994. IEEE Computer Society Press.

....Trace Availability The trace used in this study is available upon request from Bill Nitzberg, at nitzberg nas.nasa.gov. Acknowledgements This study was initiated using the CHARISMA traces, collected by David Kotz and Nils Nieuwejaar of Dartmouth College to study I O behavior of applications [14]. These traces were later replaced by an accounting trace collected at NASA Ames, which had the advantage that it included user and job information, and that it covered a much longer period. The generosity and help provided by Dave and Nils at the initial stages of this work are greatly ....

D. Kotz and N. Nieuwejaar, "Dynamic file-access characteristics of a production parallel scientific workload". In Supercomputing '94, pp. 640--649, Nov 1994.

....that Java offers as a programming language. To be useful as a language for high performance computing, however, Java must not only have good support for computation but must also be able to provide high performance file I O, as many scientific applications have significant I O requirements [6, 22, 34]. In this paper, we investigate the I O capabilities of Java for high performance computing and provide suggestions for relatively simple changes to the Java I O model that can improve performance significantly. We first examine several approaches that attempt to provide high performance I O in ....

....and Computer Science Division, Argonne National Laboratory, Argonne, IL 60439. thakur mcs.anl.gov 1 1. 1 I O in High Performance Computing Many computationally intensive scientific applications also need to access large amounts of data, and I O is often the bottleneck in such applications [6, 22, 34]. A common I O requirement is as follows. The application has some large data structures, say multidimensional arrays, distributed among processes in some fashion. The arrays must be read from or written to a file containing the global array. The program may begin by reading in an input array and ....

Kotz, D. and N. Nieuwejaar. Dynamic File-Access Characteristics of a Production Parallel Scientific Workload. In Proceedings of Supercomputing '94, pages 640-649, November 1994.

....represents a significant obstacle to achieving good performance. The problem is generally not with the hardware; many parallel I O subsystems offer excellent performance. Rather, the problem arises from other factors, primarily the I O patterns exhibited by many parallel scientific applications [1, 5] In particular, each processor tends to make a large number of small I O requests, incurring the high cost of I O on each such request. The technique of collective I O has been developed to better utilize the parallel I O subsystem [2, 7, 8] In this approach, the processors exchange information ....

Kotz, D. and N. Nieuwejaar. Dynamic file-access characteristics of a production parallel scientific workload. In Supercomputing '94 pages 640-649, November 1994.

....the Java I O model in Section 6. Related work is discussed in Section 7, followed by conclusions in Section 8. 2. I OINHIGH PERFORMANCECOMPUTING Many computationally intensive scientific applications also need to access large amounts of data, and I O is often the bottleneck in such applications [3, 13, 22]. A common I O requirement is as follows. The application has some large data structures, say multidimensional arrays, distributed among processes in some fashion. The arrays must be read from or written to a file containing the global array. The program may begin by reading in an input array and ....

Kotz, D. and N. Nieuwejaar. Dynamic File-Access Characteristics of a Production Parallel Scientific Workload. In Proceedings of Supercomputing '94, pages 640-649, November 1994.

....suggestions for improving the Java I O model in Section 6. In Section 7 we discuss related work, and we conclude in Section 8. 2 I O in High Performance Computing Many large scientific applications also need to access large amounts of data, and I O is often the bottleneck in such applications [3, 13, 21]. A common I O requirement is as follows. The application has some large data structures, say multidimensional arrays, distributed among processes in some fashion. The arrays must be read from or written to a file containing the global array. The program may begin by reading in an input array and ....

Kotz, D. and N. Nieuwejaar. Dynamic File-Access Characteristics of a Production Parallel Scientific Workload. In Proceedings of Supercomputing '94, pages 640-649, November 1994.

....(BLOCK, and ( BLOCK) from left to right. The server meshes are indicated in the figure. 4 Related Work To design a good parallel i o subsystem we must understand its intended usage. Thus there are many recent studies characterizing dynamic i o patterns in scientific applications ([Kotz94a, Thakur95, Pura94, Pasquale94, Miller91]) They show that many scientific applications have regular patterns of i o behavior, such as physical periodicity in strided access to multidimensional arrays, or temporal periodicity in checkpoint and restart operations. In both cases all compute nodes participate in collective i o operations to ....

D. Kotz, and N. Nieuwejaar, Dynamic file-access characteristics of a production parallel scientific workload, Supercomputing, 1994.

....due to inadequate file system performance; those that do exhibit architecture centric access patterns. Several studies attempt to address this dearth of characterization data by gathering application input output access pattern information at the system and application level. The CHARISMA project [44, 60] has examined input output accesses on the iPSC 860 Concurrent File System (CFS) and the CM5 Scalable Disk Array to obtain some generalizations of access patterns in parallel input output workloads. They have observed predominantly write accesses, small request sizes, and generally sequential ....

Kotz, D., and Nieuwejaar, N. Dynamic File-Access Characteristics of a Production Parallel Scientific Workload. In Proceedings of Supercomputing '94 (November 1994), pp. 640--649.

....and the user interfaces provide special I O modes that allow multiple processors to simultaneously access a file. Unfortunately, most of the parallel file system interfaces are derived from UNIX. As a result, these interfaces often fail to express access patterns generated in parallel workloads [KN94] This often leads to situations where processors are forced to make a large number of small non contiguous data requests. It has been shown that in such cases, a technique called Collective I O can be effectively used to improve the I O performance [BdC93, Kot94] The primary focus of this work ....

David Kotz and Nils Nieuwejaar. Dynamic File-Access Characteristics of a Production Parallel Scientific Workload. In Proceedings of Supercomputing '94, pages 640--649, November 1994.

....permit a proper discussion of all the work related to Panda; we will limit our discussion to papers from the HPCC community that appeared in 1993 or later and that focus on parallel implementations of collective i o. Recent studies characterizing dynamic i o patterns in scientific applications [Kotz94a, Purakayastha94, Pasquale94] show that many scientific applications have regular patterns of i o behavior, such as physical periodicity in strided access to multidimensional arrays, or temporal periodicity in checkpoint and restart operations. In both cases all application compute nodes ....

D. Kotz, and N. Nieuwejaar, Dynamic file-access characteristics of a production parallel scientific workload. In Proceedings of Supercomputing '94, pages 640-649, November 1994.

....with no hardware protection. 3 Semantics of Parallel I O Operations In conventional (Unix) systems, files are nearly never shared at the same time by more than one process (at least not for writing) 3] In parallel systems such sharing is the norm, including extensive sharing at the block level [38]. Thus there is an urgent need to define what happens when multiple processes open and access the same file. For example, if p processes write hello world to a file, what should happen Options are that p copies be written, that only a single copy be written, or that the multiple copies be ....

....the next set of p p processes takes over. The net effect is that the access is serialized and involves a lot of redundant synchronization. The problems in implementing the desired access patterns based on a scatter gather file mode has led most programmers to use the independent mode instead [38]. In this mode, each process can seek to a different offset in the file, and access the desired data irrespective of what the other processes are doing. For example, each set of p p processes can seek ahead to their part of the matrix in the block block decomposition, rather than participating in ....

[Article contains additional citation context not shown here]

D. Kotz and N. Nieuwejaar, "Dynamic file-access characteristics of a production parallel scientific workload". In Supercomputing '94, pp. 640--649, Nov 1994.

....simultaneous access by multiple processes is the rule rather than the exception. Moreover, parallel processes often access the file in an interleaved manner, where each process accesses a fragmented subset of the file, while other processes access the parts that the first process does not access [19]. UNIX file operations provide no support for such access, and in particular, do not allow access to multiple non contiguous parts of the file in a single operation. Parallel file systems and programming environments have typically solved the problems of data partitioning and collective access by ....

....being accessed by a single process. In a parallel environment, we must decide whether a file pointer is shared by multiple processes or if an individual file pointer will be maintained by each process. In addition, parallel programs do not generally exhibit locality of reference within a file [19]. Instead, they tend to move between distinct noncontiguous regions of a file. This means that the process must seek on almost every read or write operation. In addition, in multithreaded environments or when performing I O asynchronously, it is difficult to ensure that the file pointer will be in ....

David Kotz and Nils Nieuwejaar. Dynamic file-access characteristics of a production parallel scientific workload. In Proceedings of Supercomputing '94, pages 640--649, November 1994.

....5 which has fine grained I O spread throughout the application. The second application uses large grained I O operations. Finally, the conclusions are presented. 2 Related Work There have been several studies of parallel I O for applications using real data. Some examples of these are found in [10, 25, 29, 30, 34, 35]. These studies look at traces of actual parallel applications doing I O using specific parallel I O libraries and architectures. Characterizing well understood parallel programs under controlled conditions facilitates development of optimization techniques. Parallel I O systems such as PIOUS[27] ....

D. Kotz and N. Nieuwejaar, "Dynamic File-Access Characteristics of a Production Parallel Scientific Workload," In proceedings of Supercomputing `94, pp. 640649, Washington, DC, 1994.

No context found.

David Kotz and Nils Nieuwejaar, "Dynamic file-access characteristics of a production parallel scientific workload," in Proceedings of Supercomputing '94, Washington, DC, 1994, pp. 640--649, IEEE Computer Society Press.

No context found.

D. Kotz and N. Nieuwejaar. Dynamic file-access characteristics of a production parallel scientific workload. In Proc. Supercomputing '94, pages 640--649, 1994.

No context found.

D. Kotz and N. Nieuwejaar, Dynamic file-access characteristics of a production parallel scientific workload, in Proceedings of Supercomputing 94, November 1994.

No context found.

David Kotz and Nils Nieuwejaar, "Dynamic file-access characteristics of a production parallel scientific workload," in Proceedings of Supercomputing '94, Washington, DC, 1994, pp. 640--649, IEEE Computer Society Press.

No context found.

D. Kotz and N. Nieuwejaar, Dynamic File-Access Characteristics of a Production Parallel Scientific Workload, Proc. Supercomputing '94, pp. 640649, Nov. 1994.

First 50 documents  Next 50

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC