Results 1 - 10
of
41
PVFS: A Parallel File System For Linux Clusters
- IN PROCEEDINGS OF THE 4TH ANNUAL LINUX SHOWCASE AND CONFERENCE
, 2000
"... As Linux clusters have matured as platforms for low-cost, high-performance parallel computing, software packages to provide many key services have emerged, especially in areas such as message passing and networking. One area devoid of support, however, has been parallel file systems, which are criti ..."
Abstract
-
Cited by 425 (34 self)
- Add to MetaCart
As Linux clusters have matured as platforms for low-cost, high-performance parallel computing, software packages to provide many key services have emerged, especially in areas such as message passing and networking. One area devoid of support, however, has been parallel file systems, which are critical for high-performance I/O on such clusters. We have developed a parallel file system for Linux clusters, called the Parallel Virtual File System (PVFS). PVFS is intended both as a high-performance parallel file system that anyone can download and use and as a tool for pursuing further research in parallel I/O and parallel file systems for Linux clusters. In this paper, we describe the design and implementation of PVFS and present performance results on the Chiba City cluster at Argonne. We provide performance results for a workload of concurrent reads and writes for various numbers of compute nodes, I/O nodes, and I/O request sizes. We also present performance results for MPI-IO on PVFS, both for a concurrent read/write workload and for the BTIO benchmark. We compare the I/O performance when using a Myrinet network versus a fast-ethernet network for I/O-related communication in PVFS. We obtained read and write bandwidths as high as 700 Mbytes/sec with Myrinet and 225 Mbytes/sec with fast ethernet.
Decentralized Deduplication in SAN Cluster File Systems
"... File systems hosting virtual machines typically contain many duplicated blocks of data resulting in wasted storage space and increased storage array cache footprint. Deduplication addresses these problems by storing a single instance of each unique data block and sharing it between all original sour ..."
Abstract
-
Cited by 41 (2 self)
- Add to MetaCart
(Show Context)
File systems hosting virtual machines typically contain many duplicated blocks of data resulting in wasted storage space and increased storage array cache footprint. Deduplication addresses these problems by storing a single instance of each unique data block and sharing it between all original sources of that data. While deduplication is well understood for file systems with a centralized component, we investigate it in a decentralized cluster file system, specifically in the context of VM storage. We propose DEDE, a block-level deduplication system for live cluster file systems that does not require any central coordination, tolerates host failures, and takes advantage of the block layout policies of an existing cluster file system. In DEDE, hosts keep summaries of their own writes to the cluster file system in shared on-disk logs. Each host periodically and independently processes the summaries of its locked files, merges them with a shared index of blocks, and reclaims any duplicate blocks. DEDE manipulates metadata using general file system interfaces without knowledge of the file system implementation. We present the design, implementation, and evaluation of our techniques in the context of VMware ESX Server. Our results show an 80 % reduction in space with minor performance overhead for realistic workloads. 1
PVFS: A Parallel File System for Linux Clusters
- In Proceedings of the 4th Annual Linuix Showcase and Conference
, 2000
"... As Linux clusters have matured as platforms for low-cost, high-performance parallel computing, software packages to provide many key services have emerged, especially in areas such as message passing and net-working. One area devoid of support, however, has been parallel file systems, which are crit ..."
Abstract
-
Cited by 12 (0 self)
- Add to MetaCart
(Show Context)
As Linux clusters have matured as platforms for low-cost, high-performance parallel computing, software packages to provide many key services have emerged, especially in areas such as message passing and net-working. One area devoid of support, however, has been parallel file systems, which are critical for high-performance I/O on such clusters. We have developed a parallel file system for Linux clusters, called the Parallel Virtual File System (PVFS). PVFS is intended both as a high-performance parallel file system that anyone can download and use and as a tool for pursuing further re-search in parallel I/O and parallel file systems for Linux clusters. In this paper, we describe the design and implementa-tion of PVFS and present performance results on the Chiba City cluster at Argonne. We provide performance results for a workload of concurrent reads and writes for various numbers of compute nodes, I/O nodes, and I/O request sizes. We also present performance results for MPI-IO on PVFS, both for a concurrent read/write workload and for the BTIO benchmark. We compare the I/O performance when using a Myrinet network versus a fast-ethernet network for I/O-related communication in PVFS. We obtained read and write bandwidths as high as 700 Mbytes/sec with Myrinet and 225 Mbytes/sec with fast ethernet.
Handling Heterogeneity in Shared-Disk File Systems
- IN PROCEEDINGS OF THE 2003 ACM/IEEE CONFERENCE ON SUPERCOMPUTING (SC ’03
, 2003
"... We develop and evaluate a system for load management in shared-disk file systems built on clusters of heterogeneous computers. The system generalizes load balancing and server provisioning. It balances file metadata workload by moving file sets among cluster server nodes. It also responds to changi ..."
Abstract
-
Cited by 10 (2 self)
- Add to MetaCart
(Show Context)
We develop and evaluate a system for load management in shared-disk file systems built on clusters of heterogeneous computers. The system generalizes load balancing and server provisioning. It balances file metadata workload by moving file sets among cluster server nodes. It also responds to changing server resources that arise from failure and recovery and dynamically adding or removing servers. The system is adaptive and self-managing. It operates without any a-priori knowledge of workload properties or the capabilities of the servers. Rather, it continuously tunes load placement using a technique called adaptive, non-uniform (ANU) randomization. ANU randomization realizes the scalability and metadata reduction benefits of hash-based, randomized placement techniques. It also avoids hashing's drawbacks: load skew, inability to cope with heterogeneity, and lack of tunability. Simulation results show that our load-management algorithm performs comparably to a prescient algorithm.
Nache: design and implementation of a caching proxy for NFSv4
- In Proceedings of the 5th USENIX Conference on File and Storage Technologies (FAST ’07
"... In this paper, we present Nache, a caching proxy for NFSv4 that enables a consistent cache of a remote NFS server to be maintained and shared across multiple local NFS clients. Nache leverages the features of NFSv4 to improve the performance of file accesses in a widearea distributed setting by brin ..."
Abstract
-
Cited by 9 (0 self)
- Add to MetaCart
(Show Context)
In this paper, we present Nache, a caching proxy for NFSv4 that enables a consistent cache of a remote NFS server to be maintained and shared across multiple local NFS clients. Nache leverages the features of NFSv4 to improve the performance of file accesses in a widearea distributed setting by bringing the data closer to the client. Conceptually, Nache acts as an NFS server to the local clients and as an NFS client to the remote server. To provide cache consistency, Nache exploits the read and write delegations support in NFSv4. Nache enables the cache and the delegation to be shared among a set of local clients, thereby reducing conflicts and improving performance. We have implemented Nache in the Linux 2.6 kernel. Using Filebench workloads and other benchmarks, we present the evaluation of Nache and show that it can reduce the NFS operations at the server by 10-50%. 1
Adaptive disk striping for parallel input/output
- In Proceedings of the 7th NASA Goddard Conference on Mass Storage Systems and Technologies
, 1999
"... As disk capacities continue to rise more rapidly than transfer rates, adaptive, redundant striping smoothly trades capacity for higher performance. We devel-oped a fuzzy logic rule base for adaptive, redundant striping of les across multiple disks. This rule base is based on a queuing model of disk ..."
Abstract
-
Cited by 9 (1 self)
- Add to MetaCart
As disk capacities continue to rise more rapidly than transfer rates, adaptive, redundant striping smoothly trades capacity for higher performance. We devel-oped a fuzzy logic rule base for adaptive, redundant striping of les across multiple disks. This rule base is based on a queuing model of disk contention that includes le request sizes and disk hardware parame-ters. At low loads, the rule base stripes aggressively to minimize response time. As loads rise, it stripes less aggressively to maximize aggregate throughput. This adaptive striping rule base is incorporated into our second generation Portable Parallel File Sys-tem (PPFS II). Experimental results showed that the analytical models of disk striping are capable of accurately predicting le system behavior. Also, it is shown that, depending on the access pattern, adaptive striping can double the input/output per-formance compared to striping with xed distribution parameters. 1
Fault-tolerant replication management in large-scale distributed storage systems
- Proceedings 18th IEEE Symposium on Reliable Distributed Systems
, 1999
"... ..."
(Show Context)
Long Consistency and Locking for Distributing Updates to Web Servers Using a File System
- In Performance Evaluation Review, 28(2), ACM
, 2000
"... Distributed file systems are often used to replicate a Web site’s content among its many servers. However, for content that needs to be dynamically updated and distributed to many servers, file system locking protocols exhibit high latency and heavy network usage. Poor performance arises because the ..."
Abstract
-
Cited by 8 (0 self)
- Add to MetaCart
(Show Context)
Distributed file systems are often used to replicate a Web site’s content among its many servers. However, for content that needs to be dynamically updated and distributed to many servers, file system locking protocols exhibit high latency and heavy network usage. Poor performance arises because the Web-serving workload differs from the assumed workload. To address the shortcomings of file systems, we introduce the publish consistency model well suited to the Web-serving workload and implement it in the producer-consumer locking protocol. A comparison of this protocol against other file system protocols by simulation shows that producer-consumer locking removes almost all latency due to protocol overhead and significantly reduces network load. 1
Safe Caching in a Distributed File System for Network Attached Storage
- In Proceedings of the 14th International Parallel & Distributed Processing Symposium (IPDPS 2000). IEEE
, 2000
"... In a distributed file system built on network attached storage, client computers access data directly from shared storage, rather than submitting I/O requests through a server. Without a server marshaling access to data, if a computer fails or becomes isolated in a network partition while holding lo ..."
Abstract
-
Cited by 7 (1 self)
- Add to MetaCart
(Show Context)
In a distributed file system built on network attached storage, client computers access data directly from shared storage, rather than submitting I/O requests through a server. Without a server marshaling access to data, if a computer fails or becomes isolated in a network partition while holding locks on cached data objects, those objects become inaccessible to other computers until a locking authority can guarantee that the lock holder will not again directly access these data. We describe a server that acts as the locking authority and implements a lease-based protocol for revoking access to data objects locked by an isolated or failed computer. When a lease expires, the server can be assured that the client no longer acts on locked data, and can safely redistribute locks to other clients. During normal operation, this protocol invokes no message overhead, and uses no memory and performs no computation at the locking authority. 1. Introduction A distributed system provides an oper...
An Architecture for Scalable and Manageable File Services
, 2001
"... distributed file service, namespace, resource aggregation, manageability Monolithic file servers are limited by the power of an individual system. Cluster file servers are limited by resource sharing and recovery issues as the number of cluster nodes increases. DiFFS is a file service architecture t ..."
Abstract
-
Cited by 6 (3 self)
- Add to MetaCart
distributed file service, namespace, resource aggregation, manageability Monolithic file servers are limited by the power of an individual system. Cluster file servers are limited by resource sharing and recovery issues as the number of cluster nodes increases. DiFFS is a file service architecture that allows system resources to be added (or removed) dynamically, e.g., storage and processors. Resources are partitioned in such a way that contention is avoided, while maintaining a single namespace. Resources may be heterogeneous, and geographically dispersed. This architecture has several advantages. A file's physical location is decoupled from its location in the namespace. This decoupling enables a powerful and flexible mechanism for the placement of file system objects. For example, different types of files, e.g., text or video, may reside anywhere in the namespace while being hosted by servers best suited to handling their content type. DiFFS also provides lightweight protocols for online dynamic reconfiguration (volume reassignment and object migration) to address fluctuating demand and potentially mobile file system entities. A DiFFS prototype has been implemented in Linux. Performance results indicate that the architecture achieves its flexibility and scalability goals without sacrificing performance.
