Results 1 - 10
of
47
Compact Proofs of Retrievability
, 2008
"... In a proof-of-retrievability system, a data storage center must prove to a verifier that he is actually storing all of a client’s data. The central challenge is to build systems that are both efficient and provably secure — that is, it should be possible to extract the client’s data from any prover ..."
Abstract
-
Cited by 185 (1 self)
- Add to MetaCart
In a proof-of-retrievability system, a data storage center must prove to a verifier that he is actually storing all of a client’s data. The central challenge is to build systems that are both efficient and provably secure — that is, it should be possible to extract the client’s data from any prover that passes a verification check. All previous provably secure solutions require that a prover send O(l) authenticator values (i.e., MACs or signatures) to verify a file, for a total of O(l 2) bits of communication, where l is the security parameter. The extra cost over the ideal O(l) communication can be prohibitive in systems where a verifier needs to check many files. We create the first compact and provably secure proof of retrievability systems. Our solutions allow for compact proofs with just one authenticator value — in practice this can lead to proofs with as little as 40 bytes of communication. We present two solutions with similar structure. The first one is privately verifiable and builds elegantly on pseudorandom functions (PRFs); the second allows for publicly verifiable proofs and is built from the signature scheme of Boneh, Lynn, and Shacham in bilinear groups. Both solutions rely on homomorphic properties to aggregate a proof into one small authenticator value. 1
Pyramid codes: Flexible schemes to trade space for access efficiency in reliable data storage systems
- In Proceedings of the IEEE International Symposium on Network Computing and Applications. IEEE, Los Alamitos
"... We design flexible schemes to explore the tradeoffs between storage space and access efficiency in reliable data storage systems. Aiming at this goal, two new classes of erasure-resilient codes are introduced – Basic Pyramid Codes (BPC) and Generalized Pyramid Codes (GPC). Both schemes require sligh ..."
Abstract
-
Cited by 78 (8 self)
- Add to MetaCart
(Show Context)
We design flexible schemes to explore the tradeoffs between storage space and access efficiency in reliable data storage systems. Aiming at this goal, two new classes of erasure-resilient codes are introduced – Basic Pyramid Codes (BPC) and Generalized Pyramid Codes (GPC). Both schemes require slightly more storage space than conventional schemes, but significantly improve the critical performance of read during failures and unavailability. As a by-product, we establish a necessary matching condition to characterize the limit of failure recovery, that is, unless the matching condition is satisfied, a failure case is impossible to recover. In addition, we define a maximally recoverable (MR) property. For all ERC schemes holding the MR property, the matching condition becomes sufficient, that is, all failure cases satisfying the matching condition are indeed recoverable. We show that GPC is the first class of non-MDS schemes holding the MR property.
Optimal locally repairable codes and connections to matroid theory
- PROC. 2013 IEEE INTERNAT. SYMPOS. INFORM. THEORY
, 2013
"... Petabyte-scale distributed storage systems are cur-rently transitioning to erasure codes to achieve higher storage efficiency. Classical codes like Reed-Solomon are highly sub-optimal for distributed environments due to their high overhead in single-failure events. Locally Repairable Codes (LRCs) fo ..."
Abstract
-
Cited by 18 (3 self)
- Add to MetaCart
(Show Context)
Petabyte-scale distributed storage systems are cur-rently transitioning to erasure codes to achieve higher storage efficiency. Classical codes like Reed-Solomon are highly sub-optimal for distributed environments due to their high overhead in single-failure events. Locally Repairable Codes (LRCs) form a new family of codes that are repair efficient. In particular, LRCs minimize the number of nodes participating in single node repairs while generating small network traffic for repairs. Two large-scale distributed storage systems have already implemented different types of LRCs: Windows Azure Storage and the Hadoop Distributed File System RAID used by Facebook. The fundamental bounds for LRCs, namely the best possible distance for a given code locality, were recently discovered, but few explicit constructions exist. In this work, we present an explicit and simple to implement construction of optimal LRCs, for code parameters previously established only by existence results. For the analysis of the code’s optimality, we derive a new result on the matroid represented by the code’s generator matrix.
Codes with Local Regeneration
"... Regenerating codes and codes with locality are schemes recently proposed for a distributed storage network. While regenerating codes minimize data download for node repair, codes with locality minimize the number of nodes accessed during repair. In this paper, we provide constructions of codes with ..."
Abstract
-
Cited by 17 (1 self)
- Add to MetaCart
Regenerating codes and codes with locality are schemes recently proposed for a distributed storage network. While regenerating codes minimize data download for node repair, codes with locality minimize the number of nodes accessed during repair. In this paper, we provide constructions of codes with locality, in which the local codes are regenerating codes, thereby combining the advantages of both classes of codes. We also derive upper bounds on the minimum distance and code size for this class of codes and show that the proposed constructions achieve this bound. The constructions include both the cases when the local regenerating codes correspond to the MSR point as well as the MBR point on the storage repair-bandwidth tradeoff curve.
An Upper Bound On the Size of Locally Recoverable Codes
"... Abstract—In a locally recoverable or repairable code, any symbol of a codeword can be recovered by reading only a small (constant) number of other symbols. The notion of local recoverability is important in the area of distributed storage where a most frequent error-event is a single storage node fa ..."
Abstract
-
Cited by 9 (2 self)
- Add to MetaCart
Abstract—In a locally recoverable or repairable code, any symbol of a codeword can be recovered by reading only a small (constant) number of other symbols. The notion of local recoverability is important in the area of distributed storage where a most frequent error-event is a single storage node failure (erasure). A common objective is to repair the node by downloading data from as few other storage node as possible. In this paper, we bound the minimum distance of a code in terms of of its length, size and locality. Unlike previous bounds, our bound follows from a significantly simple analysis and depends on the size of the alphabet being used. I.
SD Codes: Erasure Codes Designed for How Storage Systems Really Fail
, 2013
"... Traditionally, when storage systems employ erasure codes, they are designed to tolerate the failures of entire disks. However, the most common types of failures are latent sector failures, which only affect individual disk sectors, and block failures which arise through wear on SSD’s. This paper int ..."
Abstract
-
Cited by 8 (3 self)
- Add to MetaCart
(Show Context)
Traditionally, when storage systems employ erasure codes, they are designed to tolerate the failures of entire disks. However, the most common types of failures are latent sector failures, which only affect individual disk sectors, and block failures which arise through wear on SSD’s. This paper introduces SD codes, which are designed to tolerate combinations of disk and sector failures. As such, they consume far less storage resources than traditional erasure codes. We specify the codes with enough detail for the storage practitioner to employ them, discuss their practical properties, and detail an open-source implementation.
Data secrecy in distributed storage systems under exact repair
- in Proc. IEEE NETCOD
, 2013
"... Abstract—The problem of securing data against eavesdropping in distributed storage systems is studied. The focus is on systems that use linear codes and implement exact repair to recover from node failures. The maximum file size that can be stored securely is determined for systems in which all the ..."
Abstract
-
Cited by 6 (0 self)
- Add to MetaCart
(Show Context)
Abstract—The problem of securing data against eavesdropping in distributed storage systems is studied. The focus is on systems that use linear codes and implement exact repair to recover from node failures. The maximum file size that can be stored securely is determined for systems in which all the available nodes help in repair (i.e., repair degree d = n − 1, where n is the total number of nodes) and for any number of compromised nodes. Similar results in the literature are restricted to the case of at most two compromised nodes. Moreover, new explicit upper bounds are given on the maximum secure file size for systems with d < n − 1. The key ingredients for the contribution of this paper are new results on subspace intersection for the data downloaded during repair. The new bounds imply the interesting fact that the maximum data that can be stored securely decreases exponentially with the number of compromised nodes. I.
Locally repairable codes with multiple repair alternatives
- In Proc. 2013 IEEE International Symposium on Information Theory (ISIT
, 2013
"... Abstract—Distributed storage systems need to store data re-dundantly in order to provide some fault-tolerance and guarantee system reliability. Different coding techniques have been proposed to provide the required redundancy more efficiently than tra-ditional replication schemes. However, compared ..."
Abstract
-
Cited by 6 (0 self)
- Add to MetaCart
(Show Context)
Abstract—Distributed storage systems need to store data re-dundantly in order to provide some fault-tolerance and guarantee system reliability. Different coding techniques have been proposed to provide the required redundancy more efficiently than tra-ditional replication schemes. However, compared to replication, coding techniques are less efficient for repairing lost redundancy, as they require retrieval of larger amounts of data from larger subsets of storage nodes. To mitigate these problems, several recent works have presented locally repairable codes designed to minimize the repair traffic and the number of nodes involved per repair. Unfortunately, existing methods often lead to codes where there is only one subset of nodes able to repair a piece of lost data, limiting the local repairability to the availability of the nodes in this subset. In this paper, we present a new family of locally repairable codes that allows different trade-offs between the number of contacted nodes per repair, and the number of different subsets of nodes that enable this repair. We show that slightly increasing the number of contacted nodes per repair allows to have repair alternatives, which in turn increases the probability of being able to perform efficient repairs. Finally, we present pg-BLRC, an explicit construction of locally repairable codes with multiple repair alternatives, constructed from partial geometries, in particular from Generalized Quad-rangles. We show how these codes can achieve practical lengths and high rates, while requiring a small number of nodes per repair, and providing multiple repair alternatives. I.
A hitchhiker’s guide to fast and efficient data reconstruction in erasure-coded data centers
- In Proceedings of the 2014 ACM conference on SIGCOMM (2014), ACM
"... Erasure codes such as Reed-Solomon (RS) codes are being extensively deployed in data centers since they offer signif-icantly higher reliability than data replication methods at much lower storage overheads. These codes however man-date much higher resources with respect to network band-width and dis ..."
Abstract
-
Cited by 5 (1 self)
- Add to MetaCart
Erasure codes such as Reed-Solomon (RS) codes are being extensively deployed in data centers since they offer signif-icantly higher reliability than data replication methods at much lower storage overheads. These codes however man-date much higher resources with respect to network band-width and disk IO during reconstruction of data that is miss-ing or otherwise unavailable. Existing solutions to this prob-lem either demand additional storage space or severely limit the choice of the system parameters. In this paper, we present Hitchhiker, a new erasure-coded storage system that reduces both network traffic and disk IO by around 25 % to 45 % during reconstruction of missing or otherwise unavailable data, with no additional storage, the same fault tolerance, and arbitrary flexibility in the choice of parameters, as compared to RS-based systems. Hitchhiker “rides ” on top of RS codes, and is based on novel encoding and decoding techniques that will be presented in this paper. We have implemented Hitchhiker in the Hadoop Distributed File System (HDFS). When evaluating various metrics on the data-warehouse cluster in production at Facebook with real-time traffic and workloads, during reconstruction, we observe a 36 % reduction in the computation time and a 32% reduction in the data read time, in addition to the 35 % re-duction in network traffic and disk IO. Hitchhiker can thus reduce the latency of degraded reads and perform faster re-covery from failed or decommissioned machines. 1.
