Results 1  10
of
25
Rethinking Erasure Codes for Cloud File Systems: Minimizing I/O for Recovery and Degraded Reads,”
 Proc. 10th USENIX Conf. File and Storage Technologies,
, 2012
"... Abstract To reduce storage overhead, cloud file systems are transitioning from replication to erasure codes. This process has revealed new dimensions on which to evaluate the performance of different coding schemes: the amount of data used in recovery and when performing degraded reads. We present ..."
Abstract

Cited by 42 (4 self)
 Add to MetaCart
(Show Context)
Abstract To reduce storage overhead, cloud file systems are transitioning from replication to erasure codes. This process has revealed new dimensions on which to evaluate the performance of different coding schemes: the amount of data used in recovery and when performing degraded reads. We present an algorithm that finds the optimal number of codeword symbols needed for recovery for any XORbased erasure code and produces recovery schedules that use a minimum amount of data. We differentiate popular erasure codes based on this criterion and demonstrate that the differences improve I/O performance in practice for the large block sizes used in cloud file systems. Several cloud systems [Ford10,Calder11] have adopted ReedSolomon (RS) codes, because of their generality and their ability to tolerate larger numbers of failures. We define a new class of rotated ReedSolomon codes that perform degraded reads more efficiently than all known codes, but otherwise inherit the reliability and performance properties of ReedSolomon codes. The online home for this paper may be found at: http://web.eecs.utk.edu/~plank/plank/papers/ FAST2012 Abstract To reduce storage overhead, cloud file systems are transitioning from replication to erasure codes. This process has revealed new dimensions on which to evaluate the performance of different coding schemes: the amount of data used in recovery and when performing degraded reads. We present an algorithm that finds the optimal number of codeword symbols needed for recovery for any XORbased erasure code and produces recovery schedules that use a minimum amount of data. We differentiate popular erasure codes based on this criterion and demonstrate that the differences improve I/O performance in practice for the large block sizes used in cloud file systems. Several cloud systems
Repair optimal erasure codes through Hadamard designs
 THE 49TH ANNUAL ALLERTON CONFERENCE ON COMMUNICATION, CONTROL AND COMPUTATION
, 2011
"... In distributed storage systems that employ erasure coding, the issue of minimizing the total communication required to exactly rebuild a storage node after a failure arises. This repair bandwidth depends on the structure of the storage code and the repair strategies used to restore the lost data. De ..."
Abstract

Cited by 41 (5 self)
 Add to MetaCart
(Show Context)
In distributed storage systems that employ erasure coding, the issue of minimizing the total communication required to exactly rebuild a storage node after a failure arises. This repair bandwidth depends on the structure of the storage code and the repair strategies used to restore the lost data. Designing highrate maximumdistance separable (MDS) codes that achieve the optimum repair communication has been a wellknown open problem. In this work, we use Hadamard matrices to construct the first explicit 2parity MDS storage code with optimal repair properties for all single node failures, including the parities. Our construction relies on a novel method of achieving perfect interference alignment over finite fields with a finite file size, or number of extensions. We generalize this construction to design mparity MDS codes that achieve the optimum repair communication for single systematic node failures and show that there is an interesting connection between our mparity codes and the systematicrepair optimal permutationmatrix based codes of Tamo et al. [21] and Cadambe et al. [22], [23].
Simple Regenerating Codes: Network Coding for Cloud Storage
, 2011
"... Network codes designed specifically for distributed storage systems have the potential to provide dramatically higher storage efficiency for the same availability. One main challenge in the design of such codes is the exact repair problem: if a node storing encoded information fails, in order to ma ..."
Abstract

Cited by 39 (5 self)
 Add to MetaCart
Network codes designed specifically for distributed storage systems have the potential to provide dramatically higher storage efficiency for the same availability. One main challenge in the design of such codes is the exact repair problem: if a node storing encoded information fails, in order to maintain the same level of reliability we need to create encoded information at a new node. One of the main open problems in this emerging area has been the design of simple coding schemes that allow exact and low cost repair of failed nodes and have high data rates. In particular, all prior known explicit constructions have data rates bounded by 1/2. In this paper we introduce the first family of distributed storage codes that have simple lookup repair and can achieve arbitrarily high rates. Our constructions are very simple to implement and perform exact repair by simple XORing of packets. We experimentally evaluate the proposed codes in a realistic cloud storage simulator and show significant benefits in both performance and reliability compared to replication and standard ReedSolomon codes.
MDS array codes with optimal rebuilding
 CoRR
"... Abstract—MDS array codes are widely used in storage systems to protect data against erasures. We address the rebuilding ratio problem, namely, in the case of erasures, what is the the fraction of the remaining information that needs to be accessed in order to rebuild exactly the lost information? It ..."
Abstract

Cited by 30 (3 self)
 Add to MetaCart
(Show Context)
Abstract—MDS array codes are widely used in storage systems to protect data against erasures. We address the rebuilding ratio problem, namely, in the case of erasures, what is the the fraction of the remaining information that needs to be accessed in order to rebuild exactly the lost information? It is clear that when the number of erasures equals the maximum number of erasures that an MDS code can correct then the rebuilding ratio is 1 (access all the remaining information). However, the interesting (and more practical) case is when the number of erasures is smaller than the erasure correcting capability of the code. For example, consider an MDS code that can correct two erasures: What is the smallest amount of information that one needs to access in order to correct a single erasure? Previous work showed that the rebuilding ratio is bounded between 12 and 3 4, however, the exact value was left as an open problem. In this paper, we solve this open problem and prove that for the case of a single erasure with a 2erasure correcting code, the rebuilding ratio is 1 2. In general, we construct a new family of rerasure correcting MDS array codes that has optimal rebuilding ratio of 1r in the case of a single erasure. Our array codes have efficient encoding and decoding algorithms (for the case r = 2 they use a finite field of size 3) and an optimal update property. I.
Zigzag Codes: MDS Array Codes with Optimal Rebuilding
"... MDS array codes are widely used in storage systems to protect data against erasures. We address the rebuilding ratio problem, namely, in the case of erasures, what is the fraction of the remaining information that needs to be accessed in order to rebuild exactly the lost information? It is clear tha ..."
Abstract

Cited by 27 (4 self)
 Add to MetaCart
MDS array codes are widely used in storage systems to protect data against erasures. We address the rebuilding ratio problem, namely, in the case of erasures, what is the fraction of the remaining information that needs to be accessed in order to rebuild exactly the lost information? It is clear that when the number of erasures equals the maximum number of erasures that an MDS code can correct then the rebuilding ratio is 1 (access all the remaining information). However, the interesting and more practical case is when the number of erasures is smaller than the erasure correcting capability of the code. For example, consider an MDS code that can correct two erasures: What is the smallest amount of information that one needs to access in order to correct a single erasure? Previous work showed that the rebuilding ratio is bounded between 1 2 and 3 4, however, the exact value was left as an open problem. In this paper, we solve this open problem and prove that for the case of a single erasure with a 2erasure correcting code, the rebuilding ratio is 1 2. In general, we construct a new family of rerasure correcting MDS array codes that has optimal rebuilding ratio of e r in the case of e erasures, 1 ≤ e ≤ r. Our array codes have efficient encoding and decoding algorithms (for the case r = 2 they use a finite field of size 3) and an optimal update property.
Rebuilding for Array Codes in Distributed Storage Systems
"... Abstract—In distributed storage systems that use coding, the issue of minimizing the communication required to rebuild a storage node after a failure arises. We consider the problem of repairing an erased node in a distributed storage system that uses an EVENODD code. EVENODD codes are maximum dista ..."
Abstract

Cited by 24 (2 self)
 Add to MetaCart
Abstract—In distributed storage systems that use coding, the issue of minimizing the communication required to rebuild a storage node after a failure arises. We consider the problem of repairing an erased node in a distributed storage system that uses an EVENODD code. EVENODD codes are maximum distance separable (MDS) array codes that are used to protect against erasures, and only require XOR operations for encoding and decoding. We show that when there are two redundancy nodes, to rebuild one erased systematic node, only 3/4 of the information needs to be transmitted. Interestingly, in many cases, the required disk I/O is also minimized. I.
2011. In search of I/Ooptimal recovery from disk failures
 In Proceedings of the USENIX Workshop on Hot Topics in Storage and File Systems
"... We address the problem of minimizing the I/O needed to recover from disk failures in erasurecoded storage systems. The principal result is an algorithm that finds the optimal I/O recovery from an arbitrary number of disk failures for any XORbased erasure code. We also describe a family of codes wi ..."
Abstract

Cited by 21 (2 self)
 Add to MetaCart
(Show Context)
We address the problem of minimizing the I/O needed to recover from disk failures in erasurecoded storage systems. The principal result is an algorithm that finds the optimal I/O recovery from an arbitrary number of disk failures for any XORbased erasure code. We also describe a family of codes with highfault tolerance and low recovery I/O, e.g. one instance tolerates up to 11 failures and recovers a lost block in 4 I/Os. While we have determined I/O optimal recovery for any given code, it remains an open problem to identify codes with the best recovery properties. We describe our ongoing efforts toward characterizing space overhead versus recovery I/O tradeoffs and generating codes that realize these bounds. 1
A Solution to the Network Challenges of Data Recovery
 in Erasurecoded Distributed Storage Systems: A Study on the Facebook Warehouse Cluster”, in arXiv:1309.0186
, 2013
"... Erasure codes, such as ReedSolomon (RS) codes, are being increasingly employed in data centers to combat the cost of reliably storing large amounts of data. Although these codes provide optimal storage efficiency, they require significantly high network and disk usage during recovery of missing da ..."
Abstract

Cited by 20 (3 self)
 Add to MetaCart
(Show Context)
Erasure codes, such as ReedSolomon (RS) codes, are being increasingly employed in data centers to combat the cost of reliably storing large amounts of data. Although these codes provide optimal storage efficiency, they require significantly high network and disk usage during recovery of missing data. In this paper, we first present a study on the impact of recovery operations of erasurecoded data on the datacenter network, based on measurements from Facebook’s warehouse cluster in production. To the best of our knowledge, this is the first study of its kind available in the literature. Our study reveals that recovery of RScoded data results in a significant increase in network traffic, more than a hundred terabytes per day, in a cluster storing multiple petabytes of RScoded data. To address this issue, we present a new storage code using our recently proposed Piggybacking framework, that reduces the network and disk usage during recovery by 30 % in theory, while also being storage optimal and supporting arbitrary design parameters. The implementation of the proposed code in the Hadoop Distributed File System (HDFS) is underway. We use the measurements from the warehouse cluster to show that the proposed code would lead to a reduction of close to fifty terabytes of crossrack traffic per day. 1
Permutation Codes: Optimal ExactRepair of a Single Failed Node in MDS Code Based Distributed Storage Systems
"... It is well known that maximum distance separable (MDS) codes are an efficient means of storing data in distributed storage systems, since they provide optimal tolerance against failures (erasures) for a fixed amount of storage. A (n, k) code can be used in a distributed storage system with n disks, ..."
Abstract

Cited by 16 (4 self)
 Add to MetaCart
(Show Context)
It is well known that maximum distance separable (MDS) codes are an efficient means of storing data in distributed storage systems, since they provide optimal tolerance against failures (erasures) for a fixed amount of storage. A (n, k) code can be used in a distributed storage system with n disks, each having a storage capacity of 1 unit, to store k units of information, where n> k. If the code used is maximum distance separable, then the storage system can tolerate up to (n − k) disk failures (erasures). The focus of this paper is the design of a MDS code with the additional property that a single disk failure can be repaired with optimal repair bandwidth, i.e., the amount of data to be downloaded for recovery of the failed node. Previously, a lower bound of n−1 n−k units has been established by Dimakis et. al, on the repair bandwidth for a single node failure in an (n, k) MDS code based storage system, where each of the n disks store 1 unit. While the reference by Dimakis et. al., established a lower bound on the repair bandwidth, existence of codes achieving this lower bound for arbitrary (n, k) have only been established in an asymptotic manner, in the limit of arbitrarily large storage capacities. While finite code constructions achieving this lower bound have been provided previously for k ≤ max(n/2, 3), the question of existence of finite codes for arbitrary values of (n, k) achieving the lower bound remained open. In this paper, we provide the first known construction of a finite code for arbitrary (n, k), which can repair a single failed systematic node by downloading exactly n−1 n−k units of data. The codes, which are optimally efficient in terms of a metric known as repair bandwidth, are based on permutation matrices and hence termed Permutation Codes. We also show that our code has a simple repair property which enables efficiency, not only in terms of the amount of repair bandwidth, but also in terms of the amount of data accessed on the disk. I.