Results 1  10
of
70
Beyond Bloom Filters: From Approximate Membership Checks to Approximate State Machines
 SIGCOMM '06
, 2006
"... Many networking applications require fast state lookups in a concurrent state machine, which tracks the state of a large number of flows simultaneously. We consider the question of how to compactly represent such concurrent state machines. To achieve compactness, we consider data structures for Appr ..."
Abstract

Cited by 51 (7 self)
 Add to MetaCart
(Show Context)
Many networking applications require fast state lookups in a concurrent state machine, which tracks the state of a large number of flows simultaneously. We consider the question of how to compactly represent such concurrent state machines. To achieve compactness, we consider data structures for Approximate Concurrent State Machines (ACSMs) that can return false positives, false negatives, or a “don’t know” response. We describe three techniques based on Bloom filters and hashing, and evaluate them using both theoretical analysis and simulation. Our analysis leads us to an extremely efficient hashingbased scheme with several parameters that can be chosen to trade off space, computation, and the impact of errors. Our hashing approach also yields a simple alternative structure with the same functionality as a counting Bloom filter that uses much less space. We show how ACSMs can be used for video congestion control. Using an ACSM, a router can implement sophisticated Active Queue Management (AQM) techniques for video traffic (without the need for standards changes to mark packets or change video formats), with a factor of four reduction in memory compared to fullstate schemes and with very little error. We also show that ACSMs show promise for realtime detection of P2P traffic.
Implementing signatures for transactional memory
 40th Intl. Symp. on Microarchitecture
, 2007
"... Transactional Memory (TM) systems must track the read and write sets—items read and written during a transaction—to detect conflicts among concurrent transactions. Several TMs use signatures, which summarize unbounded read/write sets in bounded hardware at a performance cost of false positives (conf ..."
Abstract

Cited by 49 (7 self)
 Add to MetaCart
(Show Context)
Transactional Memory (TM) systems must track the read and write sets—items read and written during a transaction—to detect conflicts among concurrent transactions. Several TMs use signatures, which summarize unbounded read/write sets in bounded hardware at a performance cost of false positives (conflicts detected when none exists). This paper examines different organizations to achieve hardwareefficient and accurate TM signatures. First, we find that implementing each signature with a single khashfunction Bloom filter (True Bloom signature) is inefficient, as it requires multiported SRAMs. Instead, we advocate using k singlehashfunction Bloom filters in parallel (Parallel Bloom signature), using areaefficient singleported SRAMs. Our formal analysis shows that both organizations perform equally well in theory and our simulationbased evaluation shows this to hold approximately in practice. We also show that by choosing highquality hash functions we can achieve signature designs noticeably more accurate than the previously proposed implementations. Finally, we adapt Pagh and Rodler’s cuckoo hashing to implement CuckooBloom signatures. While this representation does not support set intersection, it mitigates false positives for the common case of small read/write sets and performs like a Bloom filter for large sets. 1.
Theory and Practice of Bloom Filters for Distributed Systems
"... Many network solutions and overlay networks utilize probabilistic techniques to reduce information processing and networking costs. This survey article presents a number of frequently used and useful probabilistic techniques. Bloom filters and their variants are of prime importance, and they are h ..."
Abstract

Cited by 30 (0 self)
 Add to MetaCart
Many network solutions and overlay networks utilize probabilistic techniques to reduce information processing and networking costs. This survey article presents a number of frequently used and useful probabilistic techniques. Bloom filters and their variants are of prime importance, and they are heavily used in various distributed systems. This has been reflected in recent research and many new algorithms have been proposed for distributed systems that are either directly or indirectly based on Bloom filters. In this survey, we give an overview of the basic and advanced techniques, reviewing over 20 variants and discussing their application in distributed systems, in particular for caching, peertopeer systems, routing and forwarding, and measurement data summarization.
Compression of nextgeneration sequencing reads aided by highly efficient de novo assembly
 Nucleic Acids Res
, 2012
"... We present Quip, a lossless compression algorithm for nextgeneration sequencing data in the FASTQ and SAM/BAM formats. In addition to implementing referencebased compression, we have developed, to our knowledge, the first assemblybased compressor, using a novel de novo assembly algorithm. A prob ..."
Abstract

Cited by 22 (0 self)
 Add to MetaCart
(Show Context)
We present Quip, a lossless compression algorithm for nextgeneration sequencing data in the FASTQ and SAM/BAM formats. In addition to implementing referencebased compression, we have developed, to our knowledge, the first assemblybased compressor, using a novel de novo assembly algorithm. A probabilistic data structure is used to dramatically reduce the memory required by traditional de Bruijn graph assemblers, allowing millions of reads to be assembled very efficiently. Read sequences are then stored as positions within the assembled contigs. This is combined with statistical compression of read identifiers, quality scores, alignment information and sequences, effectively collapsing very large data sets to <15 % of their original size with no loss of information. Availability: Quip is freely available under the 3clause BSD license from
Finding complex concurrency bugs in large multithreaded applications
 In Proceedings of the European Conference on Computer Systems
, 2011
"... Parallel software is increasingly necessary to take advantage of multicore architectures, but it is also prone to concurrency bugs which are particularly hard to avoid, find, and fix, since their occurrence depends on specific thread interleavings. In this paper we propose a concurrency bug detecto ..."
Abstract

Cited by 19 (1 self)
 Add to MetaCart
Parallel software is increasingly necessary to take advantage of multicore architectures, but it is also prone to concurrency bugs which are particularly hard to avoid, find, and fix, since their occurrence depends on specific thread interleavings. In this paper we propose a concurrency bug detector that automatically identifies when an execution of a program triggers a concurrency bug. Unlike previous concurrency bug detectors, we are able to find two particularly hard classes of bugs. The first are bugs that manifest themselves by subtle violation of application semantics, such as returning an incorrect result. The second are latent bugs, which silently corrupt internal data structures, and are especially hard to detect because when these bugs are triggered they do not become immediately visible. PIKE detects these concurrency bugs by checking both the output and the internal state of the application for linearizability at the level of user requests. This paper presents this technique for finding concurrency bugs, its application in the context of a testing tool that systematically searches for such problems, and our experience in applying our approach to MySQL, a largescale complex multithreaded application. We were able to find several concurrency bugs in a stable version of the application, including subtle violations of application semantics, latent bugs, and incorrect error replies.
Spaceefficient straggler identification in roundtrip data streams via newton’s identities and invertible bloom filters
 In Workshop on Algorithms and Data Structures (WADS), volume 4619 of Lecture Notes Comput. Sci
, 2007
"... In this paper, we study the straggler identification problem, in which an algorithm must determine the identities of the remaining members of a set after it has had a large number of insertion and deletion operations performed on it, and now has relatively few remaining members. The goal is to do th ..."
Abstract

Cited by 18 (4 self)
 Add to MetaCart
(Show Context)
In this paper, we study the straggler identification problem, in which an algorithm must determine the identities of the remaining members of a set after it has had a large number of insertion and deletion operations performed on it, and now has relatively few remaining members. The goal is to do this in o(n) space, where n is the total number of identities. Straggler identification has applications, for example, in determining the unacknowledged packets in a highbandwidth multicast data stream. We provide a deterministic solution to the straggler identification problem that uses only O(d log n) bits, based on a novel application of Newton’s identities for symmetric polynomials. This solution can identify any subset of d stragglers from a set of n O(log n)bit identifiers, assuming that there are no false deletions of identities not already in the set. Indeed, we give a lower bound argument that shows that any smallspace deterministic solution to the straggler identification problem cannot be guaranteed to handle false deletions. Nevertheless, we provide a simple randomized solution using O(d log n log(1/ɛ)) bits that can maintain a multiset and solve the straggler identification problem, tolerating false deletions, where ɛ> 0 is a userdefined parameter bounding the probability of an incorrect response. This randomized solution is based on a new type of Bloom filter, which we call the invertible Bloom filter.
Invertible Bloom Lookup Tables
 In Proceedings of the 49th Allerton Conference
, 2011
"... We present a version of the Bloom filter data structure that supports not only the insertion, deletion, and lookup of keyvalue pairs, but also allows a complete listing of the pairs it contains with high probability, as long the number of keyvalue pairs is below a designed threshold. Our structure ..."
Abstract

Cited by 18 (5 self)
 Add to MetaCart
(Show Context)
We present a version of the Bloom filter data structure that supports not only the insertion, deletion, and lookup of keyvalue pairs, but also allows a complete listing of the pairs it contains with high probability, as long the number of keyvalue pairs is below a designed threshold. Our structure allows the number of keyvalue pairs to greatly exceed this threshold during normal operation. Exceeding the threshold simply temporarily prevents content listing and reduces the probability of a successful lookup. If entries are later deleted to return the structure below the threshold, everything again functions appropriately. We also show that simple variations of our structure are robust to certain standard errors, such as the deletion of a key without a corresponding insertion or the insertion of two distinct values for a key. The properties of our structure make it suitable for several applications, including database and networking applications that we highlight. 1
HashBased Techniques for HighSpeed Packet Processing
"...
Hashing is an extremely useful technique for a variety of highspeed packetprocessing applications in routers. In this chapter, we survey much of the recent work in this area, paying particular attention to the interaction between theoretical and applied research. We assume very little background ..."
Abstract

Cited by 15 (2 self)
 Add to MetaCart
(Show Context)
Hashing is an extremely useful technique for a variety of highspeed packetprocessing applications in routers. In this chapter, we survey much of the recent work in this area, paying particular attention to the interaction between theoretical and applied research. We assume very little background in either the theory or applications of hashing, reviewing the fundamentals as necessary.
Multilayer compressed counting bloom filters
 in INFOCOM 2008. The 27th Conference on Computer Communications. IEEE
"... Abstract—Bloom Filters are efficient randomized data structures for membership queries on a set with a certain known false positive probability. Counting Bloom Filters (CBFs) allow the same operation on dynamic sets that can be updated via insertions and deletions with larger memory requirements. T ..."
Abstract

Cited by 15 (1 self)
 Add to MetaCart
Abstract—Bloom Filters are efficient randomized data structures for membership queries on a set with a certain known false positive probability. Counting Bloom Filters (CBFs) allow the same operation on dynamic sets that can be updated via insertions and deletions with larger memory requirements. This paper first presents a new upper bound for counters overflow probability in CBFs. This bound is much tighter than that usually adopted in literature and it allows for designing more efficient CBFs. Three novel data structures are proposed, which introduce the idea of a hierarchical structure as well as the use of Huffman code. Our algorithms improve standard CBFs in terms of fast access and limited memory consumption (up to 50 % of memory saving): the target could be the implementation of the compressed data structures in the small (but fast) local memory or “onchip SRAM ” of devices such as Network Processors.
Rankindexed hashing: A compact construction of bloom filters and extra bits per counter (σ) lg(M/N
"... Abstract—Bloom filter and its variants have found widespread use in many networking applications. For these applications, minimizing storage cost is paramount as these filters often need to be implemented using scarce and costly (onchip) SRAM. Besides supporting membership queries, Bloom filters ha ..."
Abstract

Cited by 12 (1 self)
 Add to MetaCart
Abstract—Bloom filter and its variants have found widespread use in many networking applications. For these applications, minimizing storage cost is paramount as these filters often need to be implemented using scarce and costly (onchip) SRAM. Besides supporting membership queries, Bloom filters have been generalized to support deletions and the encoding of information. Although a standard Bloom filter construction has proven to be extremely spaceefficient, it is unnecessarily costly when generalized. Alternative constructions based on storing fingerprints in hash tables have been proposed that offer the same functionality as some Bloom filter variants, but using less space. In this paper, we propose a new fingerprint hash table construction called RankIndexed Hashing that can achieve very compact representations. A rankindexed hashing construction that offers the same functionality as a counting Bloom filter can be achieved with a factor of three or more in space savings even for a false positive probability of just 1%. Even for a basic Bloom filter function that only supports membership queries, a rankindexed hashing construction requires less space for a false positive probability as high as 0.1%, which is significant since a standard Bloom filter construction is widely regarded as extremely spaceefficient for approximate membership problems. I.