Results 1 - 10
of
338
Optimal Exact-Regenerating Codes for Distributed Storage at the MSR and MBR Points via a ProductMatrix Construction.
- IEEE Trans. on Information Theory,
, 2011
"... Abstract-Regenerating codes are a class of distributed storage codes that optimally trade the bandwidth needed for repair of a failed node with the amount of data stored per node of the network. Minimum Storage Regenerating (MSR) codes minimize first, the amount of data stored per node, and then th ..."
Abstract
-
Cited by 106 (5 self)
- Add to MetaCart
(Show Context)
Abstract-Regenerating codes are a class of distributed storage codes that optimally trade the bandwidth needed for repair of a failed node with the amount of data stored per node of the network. Minimum Storage Regenerating (MSR) codes minimize first, the amount of data stored per node, and then the repair bandwidth, while Minimum Bandwidth Regenerating (MBR) codes carry out the minimization in the reverse order. An [n, k, d] regenerating code permits the data to be recovered by connecting to any k of the n nodes in the network, while requiring that repair of a failed node be made possible by connecting (using links of lesser capacity) to any d nodes. Previous, explicit and general constructions of exact-regenerating codes have been confined to the case n = d + 1. In this paper, we present optimal, explicit constructions of MBR codes for all feasible values of [n, k, d] and MSR codes for all [n, k, d ≥ 2k−2], using a product-matrix framework. The particular product-matrix nature of the constructions is shown to significantly simplify system operation. To the best of our knowledge, these are the first constructions of exact-regenerating codes that allow the number n of nodes in the distributed storage network, to be chosen independent of the other parameters.
Pyramid codes: Flexible schemes to trade space for access efficiency in reliable data storage systems
- In Proceedings of the IEEE International Symposium on Network Computing and Applications. IEEE, Los Alamitos
"... We design flexible schemes to explore the tradeoffs between storage space and access efficiency in reliable data storage systems. Aiming at this goal, two new classes of erasure-resilient codes are introduced – Basic Pyramid Codes (BPC) and Generalized Pyramid Codes (GPC). Both schemes require sligh ..."
Abstract
-
Cited by 81 (9 self)
- Add to MetaCart
(Show Context)
We design flexible schemes to explore the tradeoffs between storage space and access efficiency in reliable data storage systems. Aiming at this goal, two new classes of erasure-resilient codes are introduced – Basic Pyramid Codes (BPC) and Generalized Pyramid Codes (GPC). Both schemes require slightly more storage space than conventional schemes, but significantly improve the critical performance of read during failures and unavailability. As a by-product, we establish a necessary matching condition to characterize the limit of failure recovery, that is, unless the matching condition is satisfied, a failure case is impossible to recover. In addition, we define a maximally recoverable (MR) property. For all ERC schemes holding the MR property, the matching condition becomes sufficient, that is, all failure cases satisfying the matching condition are indeed recoverable. We show that GPC is the first class of non-MDS schemes holding the MR property.
Reducing repair traffic for erasure coding-based storage via interference alignment
- In Information Theory Proceedings (ISIT), 2009 IEEE International Symposium on, 2009. [WDR07
, 2007
"... Abstract—We consider the problem of recovering from a single node failure in a storage system based on an (n, k) MDS code. In such a scenario, a straightforward solution is to perform a complete decoding, even though the data to be recovered only amount to 1/kth of the entire data. This paper presen ..."
Abstract
-
Cited by 74 (5 self)
- Add to MetaCart
(Show Context)
Abstract—We consider the problem of recovering from a single node failure in a storage system based on an (n, k) MDS code. In such a scenario, a straightforward solution is to perform a complete decoding, even though the data to be recovered only amount to 1/kth of the entire data. This paper presents techniques that can reduce the network traffic incurred. The techniques perform algebraic alignment so that the effective dimension of unwanted information is reduced. A A
XORing Elephants: Novel Erasure Codes for Big Data ⇤
"... Distributed storage systems for large clusters typically use replication to provide reliability. Recently, erasure codes have been used to reduce the large storage overhead of threereplicated systems. Reed-Solomon codes are the standard design choice and their high repair cost is often considered an ..."
Abstract
-
Cited by 59 (7 self)
- Add to MetaCart
Distributed storage systems for large clusters typically use replication to provide reliability. Recently, erasure codes have been used to reduce the large storage overhead of threereplicated systems. Reed-Solomon codes are the standard design choice and their high repair cost is often considered an unavoidable price to pay for high storage e ciency and high reliability. This paper shows how to overcome this limitation. We present a novel family of erasure codes that are e ciently repairable and o↵er higher reliability compared to Reed-Solomon codes. We show analytically that our codes are optimal on a recently identified tradeo ↵ between locality and minimum distance. We implement our new codes in Hadoop HDFS and compare
Distributed storage codes with repair-by-transfer and non-achievability of interior points on the storage-bandwidth tradeoff
- IEEE Trans. Inf. Theory
"... Abstract—Regenerating codes are a class of recently developed codes for distributed storage that, like Reed-Solomon codes, permit data recovery from any subset of nodes within the-node net-work. However, regenerating codes possess in addition, the ability to repair a failed node by connecting to an ..."
Abstract
-
Cited by 55 (14 self)
- Add to MetaCart
(Show Context)
Abstract—Regenerating codes are a class of recently developed codes for distributed storage that, like Reed-Solomon codes, permit data recovery from any subset of nodes within the-node net-work. However, regenerating codes possess in addition, the ability to repair a failed node by connecting to an arbitrary subset of nodes. It has been shown that for the case of functional repair, there is a tradeoff between the amount of data stored per node and the bandwidth required to repair a failed node. A special case of func-tional repair is exact repair where the replacement node is required to store data identical to that in the failed node. Exact repair is of interest as it greatly simplifies system implementation. The first result of this paper is an explicit, exact-repair code for the point on the storage-bandwidth tradeoff corresponding to the minimum possible repair bandwidth, for the case when . This code has a particularly simple graphical description, and most interest-ingly has the ability to carry out exact repair without any need to perform arithmetic operations. We term this ability of the code to perform repair through mere transfer of data as repair by transfer. The second result of this paper shows that the interior points on the storage-bandwidth tradeoff cannot be achieved under exact repair, thus pointing to the existence of a separate tradeoff under exact re-pair. Specifically, we identify a set of scenarios which we term as “helper node pooling, ” and show that it is the necessity to satisfy such scenarios that overconstrains the system. Index Terms—Distributed storage, minimum bandwidth, node repair, regenerating codes, storage versus repair-bandwidth tradeoff. I.
Self-repairing homomorphic codes for distributed storage systems
- in Proc. IEEE Infocom
, 2011
"... ar ..."
(Show Context)
Interference alignment in regenerating codes for distributed storage: Necessity and code constructions
- IEEE TRANS. INF. THEORY
, 2012
"... Regenerating codes are a class of recently developed codes for distributed storage that, like Reed-Solomon codes, permit data recovery from any arbitrary of nodes. However regenerating codes possess in addition, the ability to repair a failed node by connecting to any arbitrary nodes and downloadin ..."
Abstract
-
Cited by 49 (9 self)
- Add to MetaCart
(Show Context)
Regenerating codes are a class of recently developed codes for distributed storage that, like Reed-Solomon codes, permit data recovery from any arbitrary of nodes. However regenerating codes possess in addition, the ability to repair a failed node by connecting to any arbitrary nodes and downloading an amount of data that is typically far less than the size of the data file. This amount of download is termed the repair bandwidth. Minimum storage regenerating (MSR) codes are a subclass of regenerating codes that require the least amount of network storage; every such code is a maximum distance separable (MDS) code. Further, when a replacement node stores data identical to that in the failed node, the repair is termed as exact. The four principal results of the paper are (a) the explicit construction of a class of MDS codes for
On the locality of codeword symbols
- IEEE Trans. Inform. Theory
, 2012
"... Consider a linear [n, k, d]q code C. We say that that i-th coordinate of C has locality r, if the value at this coordinate can be recovered from accessing some other r coordinates of C. Data storage applications require codes with small redundancy, low locality for information coordinates, large dis ..."
Abstract
-
Cited by 49 (2 self)
- Add to MetaCart
(Show Context)
Consider a linear [n, k, d]q code C. We say that that i-th coordinate of C has locality r, if the value at this coordinate can be recovered from accessing some other r coordinates of C. Data storage applications require codes with small redundancy, low locality for information coordinates, large distance, and low locality for parity coordinates. In this paper we carry out an in-depth study of the relations between these parameters. We establish a tight bound for the redundancy n−k in terms of the message length, the distance, and the locality of information coordinates. We refer to codes attaining the bound as optimal. We prove some structure theorems about optimal codes, which are particularly strong for small distances. This gives a fairly complete picture of the tradeoffs between codewords length, worst-case distance and locality of information symbols. We then consider the locality of parity check symbols and erasure correction beyond worst case distance for optimal codes. Using our structure theorem, we obtain a tight bound for the locality of parity symbols possible in such codes for a broad class of parameter settings. We prove that there is a tradeoff between having good locality for parity checks and the ability to correct erasures beyond the minimum distance. 1
