Citation Context

...a tract and uses deterministic placement to eliminate the need for an additional manager service to respond to requests for the location of individual files within the storage system. DHTs like Chord =-=[31]-=- use techniques such as consistent hashing [20] to eliminating the need for centralized coordination when locating data. However, under churn, a request within a DHT might be routed to several differe...

Citation Context

...ow does a writer know where to send data? How does a reader find data that has been previously written? Many systems solve this problem using a metadata server that stores the location of data blocks =-=[14, 30]-=-. Writers contact the metadata server to find out where to write a new block; the metadata server picks a data server, durably stores that decision and returns it to the writer. Readers contact the me...

Citation Context

...to accelerate RAID recovery, while FDS ensures that all disks participate in recovery by striping tracts among all disks in the cluster. Distributed file systems like Frangiapani [32], GPFS [28], AFS =-=[18]-=- and NFS [29] export a remote storage model across a shared, hierarchical name-space. These systems must contend with strong consistency guarantees, and the vagaries of remote, shared access to a POSI...

Citation Context

...aramount. Software developers accustomed to treating the network as an abstraction are forced to think in terms of “rack locality.” New programming models (e.g., MapReduce [13], Hadoop [1], and Dryad =-=[19]-=-) emerged to help exploit locality, preferentially moving computation to data rather than viceversa. They effectively expose a cluster’s aggregate disk throughput for tasks with high reduction factors...

Citation Context

...y servers to accelerate RAID recovery, while FDS ensures that all disks participate in recovery by striping tracts among all disks in the cluster. Distributed file systems like Frangiapani [32], GPFS =-=[28]-=-, AFS [18] and NFS [29] export a remote storage model across a shared, hierarchical name-space. These systems must contend with strong consistency guarantees, and the vagaries of remote, shared access...

Citation Context

...cts to stripe data across the cluster. Finally, FDS fully distributes the blob metadata into the cluster using metadata tracts, which further minimizes the need to centrally administer the store. xFS =-=[6]-=- also proposed distributing file system metadata among storage nodes through distributed metadata managers. A replicated manager map determined which manager was responsible for the location of a part...

Citation Context

...y constraint, itself rooted in the datacenter bandwidth shortage. When bandwidth was scarce, these sacrifices were necessary to achieve the best performance. However, recently developed CLOS networks =-=[16, 15, 24]-=-—large numbers of small commodity switches with redundant interconnections—have made it economical to build non-oversubscribed full bisection bandwidth networks at the scale of a datacenter for the fi...

Citation Context

...only on fast access to blob storage, FDS provides weak consistency guarantees with very high performance. Among many others, systems such as Swift [11], Zebra [17], GPFS [28], Panasas [35], and Petal =-=[21]-=- stripe files, blocks, or logs across file servers to improve read and write throughput for traditional hierarchical file systems. FDS follows in the footsteps of these systems by using the tract loca...

Citation Context

...ow does a writer know where to send data? How does a reader find data that has been previously written? Many systems solve this problem using a metadata server that stores the location of data blocks =-=[14, 30]-=-. Writers contact the metadata server to find out where to write a new block; the metadata server picks a data server, durably stores that decision and returns it to the writer. Readers contact the me...

Citation Context

... across many servers to accelerate RAID recovery, while FDS ensures that all disks participate in recovery by striping tracts among all disks in the cluster. Distributed file systems like Frangiapani =-=[32]-=-, GPFS [28], AFS [18] and NFS [29] export a remote storage model across a shared, hierarchical name-space. These systems must contend with strong consistency guarantees, and the vagaries of remote, sh...

Citation Context

...ss to a POSIX compliant API. By focusing only on fast access to blob storage, FDS provides weak consistency guarantees with very high performance. Among many others, systems such as Swift [11], Zebra =-=[17]-=-, GPFS [28], Panasas [35], and Petal [21] stripe files, blocks, or logs across file servers to improve read and write throughput for traditional hierarchical file systems. FDS follows in the footsteps...

Citation Context

.... One drawback of ECMP is that full bisection bandwidth is not guaranteed, but only stochastically likely across multiple flows. Long-lived, high-bandwidth flows are known to be problematic with ECMP =-=[3]-=-. FDS, however, was designed to use a large number of short-lived (tract-sized) flows to a large, pseudo-random sequence of destinations. This was done in part to satisfy the stochastic requirement of...

Citation Context

...ure recovery back to stable storage. Further, RAMCloud distributes data to disk purely for reasons of fault tolerance, while FDS replication is used both for fault tolerance and availability. Panasas =-=[35]-=- uses RAID 5 to stripe files, rather than blocks, across many servers to accelerate RAID recovery, while FDS ensures that all disks participate in recovery by striping tracts among all disks in the cl...

Citation Context

...common; for example, clients of the Google File System [14] must handle garbage entries in files. However, if strong consistency guarantees are desired, FDS could be modified to use chain replication =-=[33]-=- to provide strong consistency guarantees for all updates to individual tracts. Tractservers may also be inconsistent during failure recovery. A tractserver recently assigned to a TLT entry will not h...

Citation Context

...minate the need for an additional manager service to respond to requests for the location of individual files within the storage system. DHTs like Chord [31] use techniques such as consistent hashing =-=[20]-=- to eliminating the need for centralized coordination when locating data. However, under churn, a request within a DHT might be routed to several different servers before finding an up-to-date locatio...

Citation Context

... shared access to a POSIX compliant API. By focusing only on fast access to blob storage, FDS provides weak consistency guarantees with very high performance. Among many others, systems such as Swift =-=[11]-=-, Zebra [17], GPFS [28], Panasas [35], and Petal [21] stripe files, blocks, or logs across file servers to improve read and write throughput for traditional hierarchical file systems. FDS follows in t...

Citation Context

...k to worker can be done at very short timescales. This enables FDS to mitigate stragglers—a significant bottleneck in large systems because a task is not complete until its slowest worker is complete =-=[5]-=-. Hadoop- and MapReduce-style clusters that primarily process data locally are very sensitive to machines that are slow due to factors such as misbehaving hardware, jobs running concurrently, hotspots...

Citation Context

...0 disks recovers 92GB of data lost from a single disk in only 6.2s, and 655GB lost from 7 disks on a failed machine in 33.7s. Though the broad approach of FDS’ failure recovery is similar to RAMCloud =-=[26]-=-, RAMCloud recovers data to DRAM and uses replication only for fault tolerance. FDS uses replication both for availability and fault tolerance while recovering data back to stable storage. Such fast f...

Citation Context

...y constraint, itself rooted in the datacenter bandwidth shortage. When bandwidth was scarce, these sacrifices were necessary to achieve the best performance. However, recently developed CLOS networks =-=[16, 15, 24]-=-—large numbers of small commodity switches with redundant interconnections—have made it economical to build non-oversubscribed full bisection bandwidth networks at the scale of a datacenter for the fi...

Citation Context

...y constraint, itself rooted in the datacenter bandwidth shortage. When bandwidth was scarce, these sacrifices were necessary to achieve the best performance. However, recently developed CLOS networks =-=[16, 15, 24]-=-—large numbers of small commodity switches with redundant interconnections—have made it economical to build non-oversubscribed full bisection bandwidth networks at the scale of a datacenter for the fi...

Citation Context

...(OSDI ’12) USENIX AssociationSystem Computers Data Disks Sort Size Time Implied Disk Throughput MinuteSort—Daytona class (general purpose) FDS, 2012 256 1,033 1,401GB 59s 46MB/s Yahoo!, Hadoop, 2009 =-=[25]-=- 1,408 5,632 500GB 59s 3MB/s Yahoo!, Hadoop, 2009 [25] 1,408 5,632 1,000GB 62s 5.7MB/s (unofficial 1TB run) MinuteSort—Indy class (benchmark-specific optimizations allowed) FDS, 2012 256 1,033 1,470GB...

Citation Context

...adoop, 2009 [25] 1,408 5,632 1,000GB 62s 5.7MB/s (unofficial 1TB run) MinuteSort—Indy class (benchmark-specific optimizations allowed) FDS, 2012 256 1,033 1,470GB 59.4s 47.9MB/s UCSD TritonSort, 2011 =-=[27]-=- 66 1,056 1,353GB 59.2s 43.3MB/s Table 2: Comparison of FDS MinuteSort results with the previously standing records. In accordance with sort benchmark rules, all reported times are the median of 15 ru...

Citation Context

...scale full bisection bandwidth networks. Other full bisection networks exist, such as Infiniband [23], but at a cost and scale that limited them to supercomputing and HPC environments. Finally, River =-=[8]-=- used a distributed queue to dynamically adjust the assignment of work to applications nodes at run time in data flow computations. Similarly, FDS applications use dynamic work allocation to choose wh...

Citation Context

...data server becomes a centralized scaling and performance bottleneck. In a recent interview, Google architects described the GFS metadata server as a limiting factor in terms of scale and performance =-=[22]-=-. Additionally, a desire to reduce the size of a chunk from 64MB was limited by the proportional increase in the number of chunks in the system. FDS uses deterministic placement to eliminate the scali...

Citation Context

...ovide one, they come with a significant performance penalty because networks at datacenter scales have historically been oversubscribed. Individual machines were typically attached in a tree topology =-=[12]-=-; for cost efficiency, links near the root had significantly less capacity than the aggregate capacity below them. Core oversubscription ratios of hundreds to one were common, which meant communicatio...

Citation Context

... as assuming 100-byte records and uniformly distributed 10-byte keys. In April 2012, our FDS-based sort application set the world record for sort in both the Indy and Daytona categories of MinuteSort =-=[7]-=-, which measures the amount of data that can be sorted in 60 seconds. Using a cluster of 1,033 disks and 256 computers (136 for tractservers, 120 for the application), our Daytona-class app sorted 10 ...

Citation Context

...n seen during reads, high packet loss can occur as queues fill during bursts. The reaction of standard TCP to such losses can have a devastating effect on performance. This is sometimes called incast =-=[34]-=-. Schemes such as DCTCP [4] ameliorate incast in concert with routers’ explicit congestion notification (ECN). However, because our network has full bisection bandwidth, collisions mostly occur at the...

Citation Context

...ny big-data applications. Its load pattern is similar to other common tasks such as distributed database joins and large matrix operations. This has made it an important benchmark since at least 1985 =-=[9]-=-. A group informally sponsored by SIGMOD curates an annual disk-to-disk sort performance competition with divisions for speed, cost efficiency, and energy efficiency [2]. Each has sub-divisions for ge...

Citation Context

...ystem. At the scale of large, big-data clusters that routinely exceed thousands of computers, this “flat” model of storage is still highly desirable. While some blob storage systems such as Amazon S3 =-=[10]-=- provide one, they come with a significant performance penalty because networks at datacenter scales have historically been oversubscribed. Individual machines were typically attached in a tree topolo...

Citation Context

...hout loss of performance. PortLand [24] and VL2 [15] make it economically feasible to build datacenter-scale full bisection bandwidth networks. Other full bisection networks exist, such as Infiniband =-=[23]-=-, but at a cost and scale that limited them to supercomputing and HPC environments. Finally, River [8] used a distributed queue to dynamically adjust the assignment of work to applications nodes at ru...