T. M. Madhyastha and D. A. Reed. Intelligent, adaptive file system policy selection. Frontiers of Massively Parallel Computation, October 1996.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....in the market. For example, parallel file systems [11, 30, 9] might be effective for applications whose I O access patterns fit a few specific forms. They achieve impressive performance for these applications by utilizing smart I O optimization techniques such as prefetching [19] caching [24, 6], and parallel I O [17, 12] However, there are serious obstacles preventing the parallel file systems from becoming a global solution to the data management problem. First of all, user interfaces of the file systems are in general low level [22] allowing the users to express access patterns of ....

....5: A typical MDMS execution flow. sometimes delaying hints and issuing them to MDMS collectively might be a better choice. Of course, only the correlated hints must be issued together. While passing (access pattern) hints to file systems and runtime systems was proposed by other researchers [27, 24, 29], we believe that this is the first study that considers a large spectrum (variety) of performance oriented hints in a unified framework. The functions used by the MDMS to manipulate the database tables are given in Table 2. Figure 5, on the other hand, shows a typical flow of calls using the ....

T. Madhyastha and D. Reed. Intelligent, adaptive file system policy selection. In Proc. Frontiers of Massively Parallel Computing, pages 172--179, 1996.

....there are high performance parallel file systems (e.g. Intel s PFS [25] and IBM s Vesta [8] that have been built to exploit the parallel I O capabilities provided by modern architec tures. They achieve this goal by adopting smart I O op timization techniques such as prefetching [17] caching [22,5], and parallel I O [15,10] However, there are se rious obstacles preventing the file systems from becom ing a real solution to the high level data management problem. First of all, user interfaces of the file systems are low level [21] They force the users to express ac cess patterns of their ....

T. Madhyastha and D. Reed. Intelligent, adaptive file system policy selection. In Proc. Frontiers of Massively Parallel Computing, Annapolis, MD, pp. 172 179, Oct 1996.

....in the market. For example, parallel file systems [9, 31, 11] might be effective for applications whose I O access patterns fit a few specific forms. They achieve impressive performance for these applications by utilizing smart I O optimization techniques such as prefetching [20] caching [25, 7], and parallel I O [18, 12] However, there are serious obstacles preventing the parallel file systems from becoming a global solution to the data management problem. First of all, user interfaces of the file systems are in general low level [24] allowing the users to express access patterns of ....

....we considered the next hint. Therefore, sometimes delaying hints and issuing them to MDMS collectively might be a better choice. Of course, only the correlated hints must be issued together. While passing (access pattern) hints to file systems and runtime systems was proposed by other researchers [28, 25, 30], we believe that this is the first study that considers a large spectrum (variety) of performance oriented hints in a unified framework. The functions used by the MDMS to manipulate the database tables are given in Table 2. Figure 5, on the other hand, shows a typical flow of calls using the ....

T. Madhyastha and D. Reed. Intelligent, adaptive file system policy selection. In Proc. Frontiers of Massively Parallel Computing, Annapolis, MD, pp. 172--179, Oct 1996.

....been proposed to improve the performance of a wide array of system components, encompassing compilers, memory, I O, and network communication subsys tems. They range from simple sequential read aheads for virtual storage systems [4] to prefetches of more complex data patterns for parallel systems [32, 37, 61] and wide area networks [22, 28] The techniques exploit either application specific information, supplied in the form of hints, or past reference patterns to infer the future. They can be broadly divided into three categories: offiine, online, and mixed offiine online analyses. We will examine ....

....consumption, at the cost of extra hardware support. 2.2. 2 Non compiler and Compiler Techniques To detect patterns that are more complex than just sequential, Madhyastha deployed a feed forward artificial neural network to automatically identify and qualitatively classify I O access patterns [37]. Classification results provide qualitative information to describe different types of file requests (read or write) their access orders (sequential, strided or random) and their sizes (uniform, small, medium, large) By using classification knowledge to select file policies, significant ....

[Article contains additional citation context not shown here]

MADHYASTHA, T. M., AND REED, D. A. Intelligent, Adaptive File System Policy Selection. In Proceedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation (Oct. 1996), IEEE Computer Society Press, pp. 172-179.

....group has developed a number of techniques for analyzing workloads and choosing policies based on this analysis. This technique has been integrated into their parallel file system as well. Both their selection system and the file system will be covered here. 2.4. 1 Adaptive Policy Selection In [30] they focus on an artificial neural network (ANN) approach to classification of I O access patterns in parallel systems. In [31] they improve on this approach by using hidden Markov models (HMMs) to perform classification, which provides more thorough control over policies than the neural network. ....

Tara M. Madhyastha and Daniel A. Reed. Intelligent, adaptive file system policy selection. In Proceedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation, pages 172--179. IEEE Computer Society Press, October 1996.

....that shows the best I O performance. 32 64 32 64 32 64 0.0 20.0 40.0 60.0 80.0 100.0 I O Bandwidth (MB Sec. Original) Level 1) Level 2 3) SDM Figure 7. I O bandwidth for RT 5. Related Work Several efforts have sought to optimize I O in parallel file systems and runtime libraries [3, 5, 6, 14, 16, 18, 22, 27, 31]. SRB (Storage Resource Broker) 2] provides an uniform interface to access various storage systems, such as file systems, Unitree, HPSS and database objects. However, it does not fully support the optimizations implemented in MPIIO. Shoshani et al. 28, 29] describe an architecture for op6 ....

T. M. Madhyastha and D. A. Reed. Intelligent, Adaptive File System Policy Selection. In Proceedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation, pages 172--179. IEEE Computer Society Press, October 1996.

....le systems provide users with some way to customize the le system policies. For example, Intel Paragon s PFS [31] and IBM SP2 s PIOFS [32] allow users to dictate certain le distribution parameters such as striping widths and striping units. First generation PPFS (Portable Parallel File System) [7, 33, 34, 35, 36] is an input output library, which is portable across parallel systems and workstation clusters. PPFS has a rich interface for application control of data placement and le system policies. Yet, to achieve performance gains with PPFS, as is the case in PFS and PIOFS, the application writer must ....

....the PPFS input output cost model. However, several characterization studies have shown that developers often do not know their le access patterns in sucient detail to correctly choose le policies. As a result, some studies have proposed techniques that can automatically classify access patterns [34, 35] and dynamically choose appropriate policies [33, 36] Computational steering. Interactive application steering [37, 36] is studied extensively, particularly in the context of scienti c applications and immersive visualization. Several techniques for automated decision making have been proposed, ....

T. M. Madhyastha and D. A. Reed, \Intelligent, Adaptive File System Policy Selection," in Proceedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation, pp. 172-179, IEEE Computer Society Press, Oct 1996.

....Caching Prefetching Policies. Parallel applications exhibit such a wide variation in access patterns that any one caching prefetching policy is unlikely to perform well for all applications [27] The file system must therefore either detect and automatically adapt to changing access patterns [16, 17] or provide an interface for the user to specify the access pattern or caching prefetching policy [2, 22] 10. File Preallocation. It is easy and inexpensive for a file system to provide a function to preallocate disk space for a file. If such a function is not provided, the MPI IO function MPI ....

T. Madhyasthaand D. Reed. Intelligent, Adaptive File System Policy Selection. In Proceedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation, pages 172-- 179. IEEE Computer Society Press, October 1996.

....tuning in parallel I O systems. A recent effort has focused on automatically selecting efficient file system caching and prefetching policies in PPFS using two I O access pattern classification approaches, i.e. a trained neural network approach and a hidden Markov model approach ( 22] and [23]) Automatically classifying the I O access patterns serves as the first step towards automatic performance optimization. In Panda, we used an application profiler and system microbenchmarking for this purpose. We also took one step further to devised a rule based and randomized search based ....

T.M. Madhyasta and D.A. Reed. Intelligent, adaptive file system policy selection. In Proceedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation, pages 172--179. IEEE Computer Society Press, October 1996.

.... subfiles, and forks [35] PPFS is a parallel file system developed at the University of Illinois for clusters of workstations [23] The developers use it as a testbed for research on various aspects of file system design, such as caching prefetching policies and automatic adaptive policy selection [29, 30]. PVFS is a parallel file system for Linux clusters developed at Clemson University [64] PVFS stripes files across the local disks of machines in a Linux cluster and provides the look and feel of a single Unix file system. The regular Unix commands, such as rm, ls, and mv, can be used on PVFS ....

....access pattern, the desired striping parameters, or the desired caching prefetching policies, or the file system can be designed to automatically detect and adapt its policies to the access pattern of the application. Various research efforts have demonstrated the benefits of such optimization [6, 29, 30, 41]. As mentioned above, hints can also be used to vary the sizes of temporary buffers used internally by the implementation for various optimizations. Choosing the right buffer size can improve performance considerably, as demonstrated in Section 13.5.2 and in [66] The hints mechanism in MPI IO ....

Tara M. Madhyastha and Daniel A. Reed. Intelligent, adaptive file system policy selection. In Proceedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation, pages 172--179. IEEE Computer Society Press, October 1996.

....there are highperformance parallel file systems (e.g. Intel s PFS [25] and IBM s Vesta [8] that have been built to exploit the parallel I O capabilities provided by modern architectures. They achieve this goal by adopting smart I O optimization techniques such as prefetching [17] caching [22,5], and parallel I O [15,10] However, there are serious obstacles preventing the file systems from becoming a real solution to the high level data management problem. First of all, user interfaces of the file systems are low level [21] They force the users to express access patterns of their codes ....

T. Madhyastha and D. Reed. Intelligent, adaptive file system policy selection. In Proc. Frontiers of Massively Parallel Computing, Annapolis, MD, pp. 172--179, Oct 1996.

....in the market. For example, parallel file systems [10, 29, 8] might be effective for applications whose I O access patterns fit a few specific forms. They achieve impressive performance for these applications by utilizing smart I O optimization techniques such as prefetching [18] caching [23, 6], and parallel I O [16, 11] However, there are serious obstacles preventing the parallel file systems from becoming a global solution to the data Visualization Analysis Simulation Archive Data Decomposition Mesh Adjust Generation Parameters Domain Cycle Figure 1: A typical ....

....we considered the next hint. Therefore, sometimes delaying hints and issuing them to MDMS collectively might be a better choice. Of course, only the correlated hints must be issued together. While passing (access pattern) hints to file systems and runtime systems was proposed by other researchers [26, 23, 28], we believe that this is the first study that considers a large spectrum (variety) of performance oriented hints in a unified framework. The functions used by the MDMS to manipulate the database tables are given in Table 2. Figure 5, on the other hand, shows a typical flow of calls using the ....

T. Madhyastha and D. R. Intelligent. adaptive file system policy selection. In Proc. Frontiers of Massively Parallel Computing, pages 172--179, 1996.

....storage, retrieval and processing of very large multi dimensional datasets. An initial discussion of a framework for scientific data management similar to the one described in this paper is given in [6] Several efforts have involved optimizing I O in parallel file systems and runtime libraries [3, 4, 7, 13, 16, 18, 22, 27, 31]. However, file systems and libraries have a lower level interface than SDM, requiring more work from the user. 6 Conclusions and Future Work We have presented the design and implementation of an environment for high performance scientific data management, called Scientific Data Manager (SDM) ....

Tara M. Madhyastha and Daniel A. Reed. Intelligent, Adaptive File System Policy Selection. In Proceedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation, pages 172--179. IEEE Computer Society Press, October 1996.

....not much work has been done in providing automatic support for performance tuning in parallel I O systems. A recent effort has focused on automatically selecting efficient file system caching and prefetching policies in PPFS using two different I O access pattern classification approaches. In (Madhyasta et al. 1996) and (Madhyasta and Reed, 1996) a trained neural network is used to recognize the application I O access patterns based on a pre defined classification of patterns. A Hidden SPEEDING UP AUTOMATIC PARALLEL I O PERFORMANCE OPTIMIZATION 13 The cost of optimization using different annealing scales ....

....in providing automatic support for performance tuning in parallel I O systems. A recent effort has focused on automatically selecting efficient file system caching and prefetching policies in PPFS using two different I O access pattern classification approaches. In (Madhyasta et al. 1996) and (Madhyasta and Reed, 1996), a trained neural network is used to recognize the application I O access patterns based on a pre defined classification of patterns. A Hidden SPEEDING UP AUTOMATIC PARALLEL I O PERFORMANCE OPTIMIZATION 13 The cost of optimization using different annealing scales with 8 compute nodes 171.3 ....

Madhyasta, T. and Reed, D. (1996). Intelligent, adaptive file system policy selection.

.... programmer, detailed workload characterization may be a source of suggestions for code restructuring by identifying system bottlenecks and consequently hinting ways to remove them [18] For the system designer, the characterization is instrumental for the design and tuning of file system policies [7] that respond well to diverse workload requirement and on a more broader sense for the design and evaluation of general resource management policies (e.g. processor scheduling) in parallel systems [17] Furthermore, workload characterization is critical to effective benchmarking and capacity ....

T. Madhyastha and D. A. Reed. Intelligent, adaptive file system policy selection. In Proceedings of Frontiers'96, 1996.

....and fail to exploit bursty behavior to aggressively prefetch data during idle periods. For example, we observed that the combination of PFS policies and data distributions across disks eliminated almost all access locality present in the application pattern. Automatic access pattern classification [9], coupled with performance directed adaptive control for policy selection [17] could dynamically tailor policies to access patterns. Fourth, despite the temptation to sacrifice input output systems for additional processors or primary memory, high performance parallel systems can realize their ....

....Simply put, no single file policy or data distribution is optimal for all application access patterns. Based on this analysis, we are exploring three approaches to input output optimization: qualitative access pattern classification based on trained neural networks and hidden Markov models [9], flexible policy selection using fuzzy logic techniques [17] and adaptive storage formats based on redundant representations. 5 Clearly, for some application domains this is not the case. Acknowledgments We thank Evgenia Smirni, Christopher Elford, Tara Madhyastha and Ruth Aydt for their ....

Madhyastha, T., and Reed, D. A. Intelligent, Adaptive File System Policy Selection. In Proceedings of Frontiers'96 (1996).

....DAVT63 91 C 0029, DABT63 93C 0040 and DABT63 94 C 0049 (SIO Initiative) system architectures by isolating input optimization decisions within a retargetable file system infrastructure. We have shown this approach to be successful, using an artificial neural network (ANN) based classifier [18]. This paper describes a complementary classification technique that uses hidden Markov models (HMMs) 24, 1] for modeling input output access patterns, using training data from previous application executions. As we shall see, this method offers significant advantages over ANN access pattern ....

....patterns, we can improve file system performance. To be useful for controlling file system policies, a file access pattern description need not be a perfect predictor of future accesses. It simply needs to provide the file system with enough information to select suitable policies. To this end, in [18] we proposed an artificial neural network (ANN) classification framework that processes statistics calculated from a short sequence of input output requests and generates qualitative, categorical classifications of access patterns (e.g. strided or random, read only or read write) A neural ....

[Article contains additional citation context not shown here]

Madhyastha, T. M., and Reed, D. A. Intelligent, Adaptive File System Policy Selection. In Proceedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation (1996), pp. 172--179.

....and software platforms and would increase achieved performance by choosing and configuring those resource management algorithms best matched to temporally varying application behavior. This view of adaptive runtime systems is buttressed by recent experiences with flexible input output policies [12,10,19] and by adaptive runtime systems for wide area computing [6] In both cases, use of real time performance data to adapt to changing resource demands and availability has yielded order of magnitude performance improvements. Based on this thesis, this paper describes the design and prototype ....

.... for real time measurement and adaptive control are based on our experience with the Pablo performance analysis environment [17,16] and extensions to support real time performance monitoring, qualitative classification of file access patterns [13] and table driven selection of file policies [19,12]. Building on these lessons and components of our existing Pablo software, we have designed and are implementing the Autopilot toolkit for real time adaptive control of distributed and parallel computations. Below, we describe the components of the Autopilot toolkit and our implementation ....

[Article contains additional citation context not shown here]

Tara M. Madhyastha and Daniel A. Reed. Intelligent, Adaptive File System Policy Selection. In Proceedings of Frontiers '96, pages 172--179, October 1996.

....I O request sizes and patterns suggests that achieving high performance is unlikely with a single file system policy. Instead, one needs a file system API via which users can inform the file system of expected access patterns. Using such hints or an automatic access pattern classification scheme [7], an adaptive file system could then choose those file policies and policy parameters best matched to the access pattern. For example, via user controls and hints one might advise the file system that the file access pattern is read only, write only, mixed read write sequential, or strided. For ....

Madhyastha, T., and Reed, D. A. Intelligent, adaptive file system policy selection. In Proceedings of Frontiers'96 (1996).

....described in x4. We present our experimental results in x5. Finally, x6 x7 place this work in context and summarize our results. 2 Access Pattern Classification Rationale In previous work we showed the performance benefits of using local access pattern classification to tune file system policies [16] for sequential applications. In a parallel file system, the complexities of efficiently servicing concurrent, related, input output request streams make exploitation of global behavior even more important to overall performance. Global access pattern information can be used to select globally ....

....at file open based on previous execution information, while neural network classification must observe some window of accesses before detecting local and global patterns. 3. 3 Global Classification Local classification is powerful tool for tuning file system policies in a sequential file system [16]. However, local classification is a small part of a global classification problem. As we noted earlier, local access patterns within the parallel program merge during program execution to create a global access pattern; it is essential to recognize qualitative patterns within the interleavings of ....

Madhyastha, T. M., and Reed, D. A. Intelligent, Adaptive File System Policy Selection. In Proceedings of the Sixth Symposium on the Frontiers of Massively Parallel Computation (1996), pp. 172--179.

.... analysis environment [15, 14, 16] with input output analysis software [2] flexible parallel file system policies [5] immersive virtual environments for interactive file system policy selection and optimization [18, 20] and an infrastructure for automatic classification of input output behavior [8]. We also briefly describe Pablo s role in the Scalable I O Initiative [13] a wide ranging project to characterize application input output performance, develop new language interfaces and file systems, and prototype input output software. The remainder of this paper is organized as follows. In ....

....selection, performance sensors, placed in the file system read routines, would provide the data needed to choose cache sizes and prefetch distances. As one step toward automatic policy selection, we have begun exploration of general techniques for qualitative classification of file access patterns [8]. We extended the PPFS software infrastructure to include a trained feed forward artificial neural network that accepts file request attributes (e.g. read or write, file offset, and request size) The result of the classification is a qualitative classification along three axes: sequentiality, ....

[Article contains additional citation context not shown here]

Madhyastha, T., and Reed, D. A. Intelligent, Adaptive File System Policy Selection. In submitted for publication (1995).

No context found.

T. M. Madhyastha and D. A. Reed. Intelligent, adaptive file system policy selection. Frontiers of Massively Parallel Computation, October 1996.

No context found.

Tara M. Madhyastha and Daniel A. Reed. Intelligent, adaptive file system policy selection. Frontiers of Massively Parallel Computation, October 1996.

No context found.

T. M. Madhyastha and D. A. Reed. Intelligent, adaptive file system policy selection. Frontiers of Massively Parallel Computation, October 1996.

No context found.

T. Madhyastha and D.A. Reed, Intelligent, Adaptive File System Policy Selection, Proc. Frontiers '96, 1996.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC