Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....39 40 41 42 43 44 45 46 47 48 Chapter 9 I O 9.1 Introduction POSIX provides a model of a widely portable file system, but the portability and optimization needed for parallel I O cannot be achieved with the POSIX interface. The significant optimizations required for efficiency (e.g. grouping [15], collective buffering [1, 2, 16, 19, 22] and disk directed I O [13] can only be implemented if the parallel I O system provides a high level interface supporting partitioning of file data among processes and a collective interface supporting complete transfers of global data structures between ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....and high performance I O libraries for parallel processing. 5.4 By Others Most commercial multiprocessor file systems are based on the Unix linear file model. Some simply provide a Unix like interface, and some provide the full semantics required by Unix standards. Intel Concurrent File System [17] is frequently cited as the canonical first generation parallel file system. It was written for the IPSC family of parallel machines. Its successor, PFS, is similar and was written for the Paragon [18] Thinking Machines developed a file system called sfs, which supports data parallel I O on CM 5 ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....these environments. Parallel File Systems We now turn our attention to the large body of work in parallel file systems. Most systems have focussed on extracting high performance from a set of uniform disks, including PPFS [65] Bridge [46] Panda [110] Galley [89] Vesta [36] Swift [28] CFS [90], SFS [84] the SIO specification [15] and SPIFFI [50] Some common features include scatter gather transfers, asynchronous interfaces, layout control, prefetching, and caching support at the client or server or both. Most of these parallel file systems stripe data naively across the set disks in ....

....or both. Most of these parallel file systems stripe data naively across the set disks in the I O subsystem, which can have undesirable performance properties. Shared File Pointers: One interesting feature provided by some of these systems is the notion of a shared file pointer, as found in CFS [90] and SPIFFI [50] With a shared file pointer, multiple processes on different machines can access a file concurrently in a consistent manner, as if sharing a local file pointer. Shared file pointers have some excellent performance properties. For example, when a group of processes is reading from ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND92 -020, NAS Systems Division, NASA Ames, December 1992.

....file. In nCUBE system, the user can control the data storage on the disks by providing the stripe size of the declustering. The nCUBE system distributes the files over the disks in a round robin fashion. Intel iPSC 2 and iPSC 860 support the Concurrent File System or CFS [Int88, Pie89] Int89, Nit92, BR89] FPD93, FPD91a, Int90] The CFS provides three interfaces [AS89] standard,random sequential access and coordinated interleaving. This is carried out using distinct file access modes. User has the ability to control the disk storage of the file data. Similar file system is used for the ....

....maximum read write rates observed for different configurations. Therefore, the experiments try to saturate the I O system with the I O requests so as to obtain a peak performance. Similar performance measurements have been used in the study of Intel iPSC I O system [FPD91a, FPD91b, FPD91c, FPD91b, Nit92] 3.4.1 Single Compute Node The first part of the study aims at determining the maximum I O rates obtained for a single compute node. These studies are performed both for paged as well as burst I O modes. Paged I O performance is important for implementing and supporting node virtual memory to ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....of often distinct areas: file systems, programming environments, and database research. In this section, we discuss work from the three areas. 5. 1 Parallel File Systems High performance parallel file systems are abundant in the literature: PPFS [27] Galley [37] Vesta [16] Swift [10] CFS [38], SFS [33] and the SIO specification [6] However, most assumeperformance homogeneousdevices; thus, performance is dictated by the slowest component in the system. Further, devoid of a specific programming model, applications could be constructed in an single program, multiple data (SPMD) like ....

B. Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....[17] file systems use a memory mapped interface. The nCUBE [9] and Vesta [5] allow more user control over data layout by providing per process logical views of the data. In PIFS [11] the file system controls which processor handles which part of the file to exploit memory locality. Intel s CFS [14, 22], on the other hand, provides a Unixlike interface with a choice of four I O modes to coordinate parallel access. 2.3 The CM 5 The CM 5 is a scalable message passing multiprocessor. It may contain from tens to thousands of processing nodes (PNs) and a few Control Processors (CPs) Each PN has ....

B. Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....maximum read write rates observed for different configurations. Therefore, the experiments try to saturate the I O system with the I O requests so as to obtain a peak performance. Similar performance measurements have been used in the study of Intel iPSC I O system [FPD91a, FPD91b, FPD91c, FPD91b, Nit92] Northeast Parallel Architectures Center at Syracuse University To be appear in International Conference on Supercomputing 1993 Draft Table 1: Definitions of various terms used in the paper Term Definition F The size of the file distributed over the disks. N p The number of processors ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....2.2 Existing Parallel File Systems Existing parallel I O models are often closely tied to the machine architecture as well as to the programming model. Typically jobs can access files in different I O modes , which determine how a file pointer is shared among clients running on individual nodes [7, 4, 18, 14, 22]. The Hurricane [20] and KSR1 [17] file systems use a memorymapped interface. The nCUBE [9] and Vesta [5] file systems allow more user control over data layout by providing per process logical views of the data. In PIFS (Bridge) 11] the file system controls which processor handles which part of ....

B. Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....available on today s multiprocessors. When they find a multiprocessor that is configured with sufficient parallel I O hardware (unfortunately, many are not) they often discover that the file system software is not designed to meet their needs [CK93, KN94, PEK 95] or has poor performance [Nit92, KR94, FBD94] As a result, there are several proposals for new interfaces, run time libraries, compilers, languages, and file systems to support parallel applications on parallel computers. The focus of this paper is on a file system technique called disk directed I O, which can dramatically ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....that a large, contiguous, parallel file transfer is in progress is lost through this low level interface. A collective I O interface, in which all CPs cooperate to make a single, large request, retains this semantic information, making it easier to coordinate I O for better performance [dBC93, Nit92, PGK88] Collective I O need not involve matrices. Many out ofcore parallel algorithms do I O in memoryloads, that is, they repeatedly load some subset of the file into memory, process it, and write it out [CK93] Each transfer is a large, but not necessarily contiguous, set of data. ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....small pieces of data into larger messages for transferring between CPs and IOPs. 4 Performance Performance studies of parallel file systems tend to focus on the performance of large, sequential requests. Indeed, most do not even examine the performance of requests of fewer than many kilobytes [Nit92, BBH95, KR94] As discussed above, recent workload characterizations show that parallel file systems are frequently called upon to service many small requests. This disparity means that most performance studies actually fail to examine how a file system can be expected to perform when running ....

....also maintains a cache of the most recently used blocks from the disk it controls. For this study, the size of each cache was 34 megabytes, large enough to hold 1100 blocks. Galley does not attempt to prefetch data for two reasons. First, indiscriminate prefetching can cause the cache to thrash [Nit92] Second, prefetching is based on the assumption that the system can intelligently guess what an application is going to request next. Using the higher level requests described above, there is frequently no need for Galley to make guesses about an application s behavior; the application is able ....

[Article contains additional citation context not shown here]

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....of the first relation. When the second relation passes through the hash join modules, the join is performed, and the output relation is generated. 5. 1 Parallel File Systems High performance parallel file systems are abundant in the literature: PPFS [26] Galley [36] Vesta [15] Swift [9] CFS [37], SFS [32] and the SIO specification [6] However, most assume performancehomogeneous devices; thus, performance is dictated by the slowest component in the system. Further, devoid of a specific programming model, applications could be constructed in an single program, multiple data (SPMD) like ....

B. Nitzberg. Performance of the iPSC/860Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....that a large, contiguous, parallel file transfer is in progress is lost through this low level interface. A collective I O interface, in which all CPs cooperate to make a single, large request, retains this semantic information, making it easier to coordinate I O for better performance [dBC93, Nit92, PGK88] Collective I O need not involve matrices. Many out of core parallel algorithms do I O in memoryloads, that is, they repeatedly load some subset of the file into memory, process it, and write it out [CK93] Each transfer is a large, but not necessarily contiguous, set of data. ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND92 -020, NAS Systems Division, NASA Ames, December 1992.

....Feitelson et al. 10] studied the performance of the Vesta file system. Nieuwejaar and Kotz [16] present performance results for the Galley parallel file system. Several researchers have measured the performance of the Concurrent File System (CFS) on the Intel iPSC 2 and iPSC 860 hypercubes [5, 11, 18]. In an earlier work, we studied the I O characteristics of a different application on the SP and Paragon [20] For that study, we used a two dimensional astrophysics application that performs sequential I O (only processor 0 performs all I O) using the Unitree file system on the SP and the PFS ....

B. Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

.... studied the Intel Concurrent File System (CFS) and found that its caching improved performance, but they were not able to compare several policies [PFDJ89, FPD91] More recent studies found that, in general, despite its cache performance, CFS performance lags well behind hardware capabilities [Nit92, BCR93] Miller and Katz [MK91] used application traces from a Cray supercomputer to drive cache simulations, and found that there was not enough temporal locality for a file system cache to provide significant performance gains. When caching alone does not work well, prefetching can be useful ....

....not work well, prefetching can be useful for reads. French et al. studied prefetching in Intel CFS, in which each I O node prefetches several blocks ahead when the access patterns seem sequential [FPD91] This CFS strategy resulted in significant performance gains, but could also cause thrashing [Nit92] Towsley et al. found prefetching useful in multiprocessor file systems [Tow78, TCB78] Kotz designed more sophisticated prefetching policies to handle new parallel access patterns [Kot93] Compute processor read caches increased hit rate for a significant proportion of jobs in a trace driven ....

[Article contains additional citation context not shown here]

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

.... and is classified as compulsory, while periodic checkpointing and data staging is also reported [8] Subsequent characterization efforts that focused exclusively on scientific applications included either measurements from synthetic benchmarks that emulated the behavior of common scientific codes [11], or studies that focused on the behavior of individual applications and on ways to restructure individual applications by tuning their I O requirements to match the characteristics of the underlying file system [1, 18] Other studies concentrated on workload measurements at supercomputing centers ....

B. Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND92 -020, NAS Systems Division, NASA Ames, December 1992.

....user has more control over data layout. These systems provide a per process logical view of the data. In PIFS (Parallel Interleaved File System) Dib90] the file system controls which processor handles which part of the file to exploit memory locality. Intel s CFS (Concurrent File System) FPD93, Nit92, Pie89] on the other hand, provides a Unix like interface with a choice of four I O modes to coordinate parallel access. On the CM 5 the I O model depends on the programming model that is being used; one model provides a Unix like interface with different I O modes. 2.3 The CM 5 The CM 5 has a ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND92 -020, NAS Systems Division, NASA Ames, December 1992.

....to control file declustering, caching, and prefetching. The performance of Intel s CFS when reading or writing a two dimensional matrix, for example, depends heavily on the layout of the matrix across disks and across memories of the multiprocessor, and also on the order of requests [dBC93, BCR92, Nit92, GP91, GL91] del Rosario et al. dBC93] find that the nCUBE exhibits similar inefficiencies: when reading columns from a two dimensional matrix stored in row major order, read times increase by factors of 30 50. One solution is to transfer data from disk into memory and then permute it within ....

....the nCUBE exhibits similar inefficiencies: when reading columns from a two dimensional matrix stored in row major order, read times increase by factors of 30 50. One solution is to transfer data from disk into memory and then permute it within memory to its final destination [dBC93] Nitzberg [Nit92] shows that some layouts experience poor performance on CFS because of thrashing in the file system cache. His solution to this problem carefully schedules the processors accesses to the disks by reducing concurrency [Nit92] Each of these examples highlights the need for programs to organize ....

[Article contains additional citation context not shown here]

Bill Nitzberg. Performance of the iPSC/860 concurrent file system. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....on queries 10 and 11. 7 Related Work Many different parallel file systems have been developed over the past decade. While many of these were similar to the traditional Unix style file system, there have been also several more ambitious attempts. Intel s Concurrent File System (CFS) Pie89, Nit92] and its successor, PFS, are examples of parallel file systems that provide a linear file model with a Unix like interface. Support for parallel applications is limited to file pointers that may be shared by all the processes in the application. CFS and PFS provide several modes, each of which ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....high performance in parallel application domains that require manipulation of vast amounts of data. To improve performance, advances of I O hardware and file system parallelism are of principal importance. In the last few years, a wide variety of parallel I O systems have been proposed and built[11, 12, 4, 5, 20, 14, 17]. All these systems exploit parallel I O devices (i.e. partitioning data across This work was supported in part by the Advanced Research Projects Agency under ARPA contracts DABT63 94 C0049 (SIO Initiative) DAVT63 91 C 0029 and DABT63 93 C 0040, by the National Science Foundation under grant ....

Nitzberg, B. Performance of the iPSC/860 Concurrent File System. Tech. Rep. RND92 -020, NAS Systems Division, NASA Ames, December 1992.

.... caching and prefetching supplied with Intel s Concurrent File System (CFS) does improve performance [15] Recent studies have found that CFS caching and prefetching work well in limited situations, but that the throughput of CFS can be disappointing relative to the capabilities of the hardware [27, 3]. Miller and Katz drove a cache simulation using traces from a Cray supercomputer and found that access locality was not high enough for significant benefits to be realized from a file system cache [25] 2.4 Intel iPSC 860 and CFS The iPSC 860 is a distributed memory, messagepassing, MIMD ....

....node rather than directly to the hypercube interconnect. The I O nodes are based on the Intel i386 processor and each has a port for SCSI disk drives. There may also be one or more service nodes that handle as Ethernet connections or interactive shells [26] Intel s Concurrent File System (CFS) [30, 15, 27] provides a Unix like interface to the user with the addition of four I O modes to help the programmer coordinate parallel access to files. Mode 0 gives each process its own file pointer; mode 1 shares a single file pointer among all processes; mode 2 is like mode 1, but enforces a round robin ....

[Article contains additional citation context not shown here]

B. Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, Dec. 1992.

.... caching and prefetching supplied with Intel s Concurrent File System (CFS) does improve performance [FPD93] Recent studies have found that CFS caching and prefetching work well in limited situations, but that the throughput of CFS can be disappointing relative to the capabilities of the hardware [Nit92, BCR93] Miller and Katz drove a cache simulation using traces from a Cray supercomputer and found that access locality was not high enough for significant benefits to be realized from a file system cache [MK91] 2.4 Intel iPSC 860 and CFS The iPSC 860 is a distributed memory, message passing, ....

....than directly to the hypercube interconnect. The I O nodes are based on the Intel i386 processor and each has a port for SCSI disk drives. There may also be one or more service nodes that handle Ethernet connections or interactive shells [NAS93] Intel s Concurrent File System (CFS) Pie89, FPD93, Nit92] provides a Unix like interface to the user with the addition of four I O modes to help the programmer coordinate parallel access to files. Mode 0 gives each process its own file pointer; mode 1 shares a single file pointer among all processes; mode 2 is like mode 1, but enforces a round robin ....

[Article contains additional citation context not shown here]

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....We included these traces in our analysis to avoid bias due to choice of application domain. Previous research on the characterization the I O requirements of parallel applications has focused exclusively on scientific applications and has taken one of four approaches. French et al. [17] Nitzberg [31] and Bordawekar et al. [8] used synthetic benchmarks that are intended to emulate what the authors felt are common I O behaviors. This approach provides information about the I O capabilities of the machine that the experiments are being run on; does not provide information about application ....

B. Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, Dec 1992.

....to the hypercube interconnect. The I O nodes are based on the Intel i386 processor, and each has a single bus for SCSI disk drives. There may also be one or more service nodes that handle Ethernet connections or interactive shells [NAS93] Intel s Concurrent File System (CFS) Pie89, FPD93, Nit92] provides a Unix like interface to the user with the addition of four I O modes, as discussed in the previous chapter. CFS generally stripes each file across all I O nodes in 4 KB blocks. CFS allows users to specify that a file only be stored on a subset of the available I O nodes, but we found ....

....were for fewer than 100 bytes, but those reads transferred only about 2 of all data read. Similarly, 89 of all writes were for fewer than 100 bytes, but those writes transferred only about 2.5 of all data written. The number of small requests is surprising due to their poor performance in CFS [Nit92] The jump at 4 KB indicates that some users have optimized 1 As there was a large number of small files as well as a number of distinct peaks across the whole range of sizes, there was no constant granularity that captured the detail we felt was important in a histogram. We chose to plot the ....

[Article contains additional citation context not shown here]

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

....of I O operations from a single (SPMD or MIMD) parallel application run on many nodes, The application data is distributed among the nodes, and is read written to a single logical file, itself spread across nodes and disks. The significant optimizations required for efficiency (e.g. grouping [25], twophase I O [9] and disk directed I O [18] can only be implemented as part of a parallel I O environment if it supports a high level interface to describe the partitioning of file data among processes and a collective interface describing complete transfers of global data structures between ....

....the partitioning directly. Examples include Vesta [7] and the nCUBE system software [8] Recent studies show that various simple partitioning schemes do indeed account for most of observed parallel I O patterns [24] In addition to the commercial offerings (IBM SP2 PIOFS [6] Intel iPSC CFS [25, 27] and Paragon PFS [11, 28] nCUBE [8] and Thinking Machines CM 5 sfs [2, 20] there has been a recent flurry of activity in the research community. PIOUS [22, 23] and PETSc Chameleon I O [14] are both widely available nonproprietary portable parallel I O interfaces. PIOUS is a PVM based parallel ....

Bill Nitzberg. Performance of the iPSC/860 Concurrent File System. Technical Report RND-92-020, NAS Systems Division, NASA Ames, December 1992.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC