Citation Context

... VM synthesis with the alternative approach of assembling a large file from hash-addressed chunks. Variants of the alternative approach have been used in systems such as LBFS [12], Casper [19], Shark =-=[3]-=-, the Internet Suspend/Resume system [9], the Collective [14], and KeyChain [2]. All of these variants have a probabilistic character to them: chunks that are not available nearby (in the local cache,...

Citation Context

...ts. It has been used to evaluate a diverse set of planetary-scale network services, including content distribution [33, 8, 24], anycast [4, 9], DHTs [26], robust DNS [20, 25], large-file distribution =-=[19, 1]-=-, measurement and analysis [30], anomaly and fault diagnosis [35], and event notification [23]. It supports the design and evaluation of dozens of long-running services that transport an aggregate of ...

Citation Context

...or objects) and exchange these chunks with each other to form the complete file. The most widely used system of this type is BitTorrent [12], while related research systems include Bullet [20], Shark =-=[2]-=-, and FastReplica [9]. Using peer-to-peer systems makes sense when the window of interest in the content is short, or when the content provider cannot afford enough bandwidth or CDN hosting costs. How...

Citation Context

...t benchmark on 50 cluster nodes plus a remote server. file; not only does this approach minimize transfer latency, it effectively forms a multicast tree when multiple concurrent reads of a file occur =-=[1, 35]-=-. We examine a 3-phase benchmark that represents running an experiment: in phase 1 Disseminate, each node fetches 10MB of new executables and input data from the user’s home node; in phase 2 Process, ...

Citation Context

...so require a mapping between every unique chunk in the identical and similar files and their corresponding sources, i.e., O(N) mappings per object. Examples of per-chunk systems are CFS [9] and Shark =-=[3]-=-. In this paper, we present Similarity-Enhanced Transfer (SET)—a hybrid system that provides the best of both approaches, locating both identical and similar sources for data chunks using O(1) lookups...

Citation Context

... a file into chunks, which are cached at distributed nodes. Choffnes et al. [15] reduce cross-ISP traffic in peer-to-peer systems by harvesting data from existing CDNs for locality information. Shark =-=[3]-=- and ChunkCast [9] reduce client-perceived download latency via a structured overlay, and Coral [23] and Bamboo [50] assist clients in finding nearby copies of data. Antfarm similarly shifts cost to c...

Citation Context

...also raises the abstraction level of distributed computing; it focuses on protocols and does not support persistence or information security. Some distributed storage systems such as PAST [45], Shark =-=[2]-=-, CFS [15], and Boxwood [31] use distributed data structures to provide scalable file systems, but offer weak consistency and security guarantees for distributed computation. 9 Conclusions We have exp...

Citation Context

...to fixed [7] or variable length chunks. Variable-length chunking has been used to conserve bandwidth [6], to weed out near duplicates in large repositories [5], for wide-area distributed file systems =-=[12]-=-, and, to store and transfer large directory trees efficiently and with high reliability [13]. Previous Chunk Fixed Width Sliding Window Chunk Boundary Fig. 1. f mod D ≠ r f mod D = r Sliding Window T...

Citation Context

...tend object-based caching to subpacket granularity, and develop a protocol independent contentbased caching technique [57]. Since then, it has been adapted in many applications – network file systems =-=[7, 42]-=-, WAN acceleration [28, 50], Web caching [43, 49], and storage systems [20]. Recently, Anand et al. analyzed university and enterprise network traces, and show that 15-60% of the entire traffic is red...

Citation Context

...ne or two “false collisions” would not significantly alter our findings. Deduplication and similar technologies have already been used to reduce bandwidth and storage demands for network file systems =-=[17, 1, 11]-=-, reduce the storage demands created by VM checkpoints [18], store less data in backup environments [8, 29, 20], and reduce storage demands in archival storage [28, 27]. Using deduplication in an onli...