Y. Minsky, A. Trachtenberg, and R. Zippel. Set reconciliation with nearly optimal communication complexity. IEEE Transactions on Information Theory, 49(9):2212--2218, September 2003.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....have the same hash value as an element y of SA , in which case peer B will mistakenly believe x 2 SA . The miss probability can be made inversely polynomial in jSA j by setting h = poly(jSA j) in which case (jSA j log jSA j) bits are sent. Another approach is to use set discrepancy methods of [22]. If the discrepancy d = jSB SA j jSA SB j is known, then peer A can send a data collection of size only O(d log u) bits, or if hashing is done as pre processing, of size only O(d log h) bits. However, if d is not known, a reasonable upper bound on d must be determined through multiple rounds of ....

MINSKY, Y., TRACHTENBERG, A., AND ZIPPEL, R. Set reconciliation with nearly optimal communication complexity. In Proc. of IEEE Int'l Symp. on Information Theory (Washington, DC, June 2001).

....a file even in the event that every source node with the complete file has left the network. The second challenge of a multi source download algorithm is to make the reconciliation phase as bandwidth efficient as possible. This phase is an instance of the more general set reconciliation problem [10, 4, 11, 2]. Unfortunately, existing set reconciliation algorithms are not practical for multi source download algorithms. They are either too computationally costly, suboptimal in terms of message complexity, or simply too complicated to implement. In this paper, we propose an algorithm that combines ....

Y. Minsky, A. Trachtenberg, and R. Zippel. Set Reconciliation with Nearly Optimal Communication Complexity. In International Symposium on Information Theory, 2001.

....provide experimental results comparing the various methods proposed for approximate reconciliation. 1 Introduction The problem of nding set di erences is a fundamental problem in distributed computing, arising in protocols for gossiping, synchronization, and replication. Recent work such as [8, 10, 9] solves the problem of nding exact set di erences between two parties with near optimal communication complexity and presents various tradeo s between the number of rounds and the computational complexity. In contrast to previous work, we consider nding approximate set di erences, where the goal ....

....compelling. Previous to our e orts, work on set reconciliation generally focused on the exact case. Reconciliation can clearly be done by simple enumeration of set elements or, to save space, hashes of set elements. Exact reconciliation with near optimal communication complexity was considered in [8, 10, 9]. Their general approach involves each peer evaluating the characteristic polynomial of their set at several points and using this information to compute the ratio of the two characteristic polynomials and hence their set di erences. This necessitates interpolation and factorization of rational ....

Minsky, Y., Trachtenberg, A., and Zippel, R. Set reconciliation with nearly optimal communication complexity. In ISIT (Washington, DC, June 2001).

....have the same hash value as an element y of S# , in which case peer B will mistakenly believe x # S# . The miss probability can be made inversely polynomial in #S# # by setting h # poly##S# ##, in which case ###S# # ### #S# ## bits are sent. Another approach is to use set discrepancy methods of [22]. If the discrepancy d # #S# # S# # # #S# # S# # is known, then peer A can send a data collection of size only O#d ### u# bits, or if hashing is done as pre processing, of size only O#d ### h# bits. However, if d is not known, a reasonable upper bound on d must be determined through multiple ....

MINSKY,Y.,TRACHTENBERG,A., AND ZIPPEL, R. Set reconciliation with nearly optimal communication complexity. In Proc. of IEEE Int'l Symp. on Information Theory (Washington, DC, June 2001).

....will have the same hash value as an element y of S# , in which case peer B will mistakenly believe x # S# . The miss probability can be made inversely polynomial in n by setting h # poly##S# ##, in which case ###S# # ### #S# ## bits are sent. Another approach is to use set discrepancy methods of [19]. If the discrepancy d # #S# # S# # # #S# # S# # is known, then peer A can send a data collection of size only O#d ### u# bits, or if hashing is done as pre processing, of size only O#d ### h# bits. The preprocessing time, however, involves ##d#S# ## field operations in a field of size u (or h, if ....

MINSKY,Y.,TRACHTENBERG, A., AND ZIPPEL, R. Set reconciliation with nearly optimal communication complexity. In ISIT (Washington, DC, June 2001).

....will have the same hash value as an element y of SA , in which case peer B will mistakenly believe x 2 SA . The miss probability can be made inversely polynomial in n by setting h = poly(jSA j) in which case (jSA j log jSA j) bits are sent. Another approach is to use set discrepancy methods of [19]. If the discrepancy d = jSB SA j jSA SB j is known, then peer A can send a data collection of size only O(d log u) bits, or if hashing is done as pre processing, of size only O(d log h) bits. The preprocessing time, however, involves (djSA j) field operations in a field of size u (or h, if ....

MINSKY, Y., TRACHTENBERG, A., AND ZIPPEL, R. Set reconciliation with nearly optimal communication complexity. In ISIT (Washington, DC, June 2001).

No context found.

Y. Minsky, A. Trachtenberg, and R. Zippel. Set reconciliation with nearly optimal communication complexity. In International Symposium on Information Theory, page 232, 2001.

No context found.

Yaron Minsky, Ari Trachtenberg, and Richard Zippel, "Set reconciliation with nearly optimal communication complexity," IEEE Trans. Inf. Theory, 2000, submitted.

....that reconciliations occur with minimum communication, measured both by the number of transmitted bits and by the number of rounds of communication. When data are represented by sets, as can be reasonable modeled for the examples cited above, this problem is known as the set reconciliation problem [6, 7]. The data reconciliation problem is a natural generalization in which data is represented by multi sets rather than sets; the case where only a single message of communication is permitted is termed one way reconciliation. This paper examines the one way data reconciliation problem within a ....

....= E(x) for all integers k 1. Definition 5. A code C Z detects the error set G if ci G[cj] for all ci y cj C and corrects G if G[c] n G[c] 0 for all ci cj C. 2.3 Set and multi set reconciliation and verification. The traditional formalization of the set reconciliation problem is as follows [6, 7]: given a pair of hosts A and B, each with a set (St and B respectively) of length b bit strings and no a priori knowledge of the other host s set, how can each host determine the mutual difference of the two sets with a minimal amount of communication. In general, we may consider data ....

[Article contains additional citation context not shown here]

Y. Minsky, A. Trachtenberg, and R. Zippel, "Set reconciliation with nearly optimal communication complexity," in International Symposium on Information Theory, June 2001, p. 232.

....that reconciliations occur with minimum communication, measured both by the number of transmitted bits and by the number of rounds of communication. When data are represented by sets, as can be reasonable modeled for the examples cited above, this problem is known as the set reconciliation problem [6, 7]. The data reconciliation problem is a natural generalization in which data is represented by multi sets rather than sets; the case where only a single message of communication is permitted is termed one way reconciliation. This paper examines the one way data reconciliation problem within a ....

....= E(x) for all integers k 1. Definition 5. A code C Z detects the error set G if ci G[cj] for all ci y cj C and corrects G if G[c] n G[c] 0 for all ci cj C. 2.3 Set and multi set reconciliation and verification. The traditional formalization of the set reconciliation problem is as follows [6, 7]: given a pair of hosts A and B, each with a set (St and B respectively) of length b bit strings and no a priori knowledge of the other host s set, how can each host determine the mutual difference of the two sets with a minimal amount of communication. In general, we may consider data ....

[Article contains additional citation context not shown here]

Y. Minsky, A. Trachtenberg, and R. Zippel, "Set reconciliation with nearly optimal communication complexity," Tech. Rep. TR1999-1778, TR2000-1796,TR2000-1813, Cornell University, 2000.

....number of errors in a transmitted vector. Transferred to the parlance of set reconciliation, this means that the decoding time depends mostly on the number of differences between sets (rather than on the overall sizes of the sets) The solution provided by Reed Solomon codes is related to CPIsync [15] and provides a good means of data synchronization. III. CHARACTERISTIC POLYNOMIAL INTERPOLATION BASED SYNCHRONIZATION The key challenge to efficient PDA synchronization is a synchronization protocol whose communication complexity depends only on the number of differences between synchronizing ....

....performance equivalent to fast sync in a general setting. C. Probabilistic scheme with an unknown upper bound In the general setting where no knowledge on a upper bound is provided, it is impossible to reconcile sets to a theoretical certainty without performing the equivalence of a slow sync [15, 22]. Fortunately, a probabilistic scheme can synchronize, with arbitrarily low probability of error, much more efficiently than the deterministic optimum given by slow sync. Specifically, the scheme in [2] suggests guessing such a bound # and subsequently verifying if the guess was correct. If the ....

Y. Minsky, A. Trachtenberg, and R. Zippel, "Set reconciliation with nearly optimal communication complexity," IEEE Trans. on Info. Theory, 2000, submitted.

....PDA. Indeed, the typical case is where handheld devices and desktops regularly synchronize with each other so that few changes are made between synchronizations. We consider the application of a near optimal synchronization methodology, based on recent research advances in fast set reconciliation [2, 3], in order to minimize the waste of network resources. Broadly speaking, given a PDA and a PC with data sets # and #, this new scheme can synchronize the hosts using one message in each direction of length ## (i.e. essentially independent of the size of the data sets ###) Thus, two ....

....held by # but not #,orheldby# but not #) using a minimal amount of communication. Within this context, only the elements of the sets are important and not their relative positions within the set. Note also that the integer sets being synchronized can generically encode all types of data. In [2, 3] this formalization is called the set reconciliation problem. Natural examples of set reconciliation include synchronization of bibliographic data [16] resource availability [17, 18] data within gossip protocols [19, 20] or memos and address books. On the other hand, synchronization of edited ....

[Article contains additional citation context not shown here]

Y. Minsky, A. Trachtenberg, and R. Zippel, "Set reconciliation with nearly optimal communication complexity," in International Symposium on Information Theory, June 2001, p. 232.

....PDA. Indeed, the typical case is where handheld devices and desktops regularly synchronize with each other so that few changes are made between synchronizations. We consider the application of a near optimal synchronization methodology, based on recent research advances in fast set reconciliation [2, 3], in order to minimize the waste of network resources. Broadly speaking, given a PDA and a PC with data sets # and #, this new scheme can synchronize the hosts using one message in each direction of length ## (i.e. essentially independent of the size of the data sets ###) Thus, two ....

....large, making CPIsync the protocol of choice for many synchronization applications. Another complication of CPIsync is that it requires a good a priori bound on the number of differences between two synchronizing sets. We, thus, propose to make use and implement a prob2 abilistic technique from [2] for testing the correctness of a guessed upper bound. If one guess turns out to be incorrect, then the guess can be modified in a second attempted synchronization, and so forth. The error of this probabilistic technique can be made arbitrarily small. We show that the communication and time used ....

[Article contains additional citation context not shown here]

Y. Minsky, A. Trachtenberg, and R. Zippel, "Set reconciliation with nearly optimal communication complexity," Tech. Rep. TR1999-1778, TR2000-1796,TR2000-1813, Cornell University, 2000.

....on the PDA. Indeed, the typical case is where handheld devices and desktops regularly synchronize with each other so that few changes are made between synchronizations. We propose to apply a near optimal synchronization methodology based on recent research advances in fast set reconciliation [2, 3], in order to minimize the waste of network resources. Broadly speaking, given a PDA and a PC with data sets # and #, this new scheme can synchronize the hosts using one message in each direction of length ## ### # ## ### (i.e. essentially independent of the size of the data sets # and #) Thus, ....

....we have implemented on a PDA as described in Section IV, requires a nearly minimal communication complexity and operates with a reasonable computational complexity. A. Deterministic scheme with a known upper bound The key to the set reconciliation algorithm of Minsky, Trachtenberg, and Zippel [2, 3] is a translation of data sets into polyFig. 5. Inefficiency of timestamps. Hosts # and # first synchronize so that each has item # (left figure) Thereafter hosts # and # independently synchronize with host #, noting the addition of items ## ## ## and #. When hosts # and # then re synchronize, ....

[Article contains additional citation context not shown here]

Y. Minsky, A. Trachtenberg, and R. Zippel, "Set reconciliation with nearly optimal communication complexity," in International Symposium on Information Theory, June 2001, p. 232.

....on the PDA. Indeed, the typical case is where handheld devices and desktops regularly synchronize with each other so that few changes are made between synchronizations. We propose to apply a near optimal synchronization methodology based on recent research advances in fast set reconciliation [2, 3], in order to minimize the waste of network resources. Broadly speaking, given a PDA and a PC with data sets # and #, this new scheme can synchronize the hosts using one message in each direction of length ## ### # ## ### (i.e. essentially independent of the size of the data sets # and #) Thus, ....

....practical approaches for determining such a bound. In the first case, we propose a simple method that performs well for the synchronization of a small number of hosts (e.g. a PDA with two different PCs, one at work and one at home) In the second case, we make use of a probabilistic technique from [2] for testing the correctness of a guessed upper bound. If one guess turns out to be incorrect, then it can be modified in a second attempted synchronization, and so forth. The error of this probabilistic technique can be made arbitrarily small. We also show that the communication and time used by ....

[Article contains additional citation context not shown here]

Y. Minsky, A. Trachtenberg, and R. Zippel, "Set reconciliation with nearly optimal communication complexity," Tech. Rep. TR1999-1778, TR2000-1796,TR2000-1813, Cornell University, 2000.

....Periodic synchronization of data between various devices is an enabling technology for this computing paradigm and an essential feature of any effective mobile and heterogeneous network architecture. We propose a peer to peer synchronization algorithm based on recent information theoretic advances [3] and demonstrate an implementation of that algorithm through an improved PC PDA synchronizer. PalmPilot PDAs (Personal Digital Assistants) synchronize their data with that on a host PC either locally over a USB or serial connection, or remotely through a TCP IP network using a protocol known as ....

....costly as data storage capacity outpaces the increase in network bandwidth. To address this scalability issue, we have developed and analyzed an algorithm termed CPISync (Characteristic Polynomial Interpolation Synchronization) based on a recent solution to the set reconciliation problem given in [3]. The CPISync algorithm uses interpolation of a rational function whose components can be factored to identify the differences between data sets stored on any two synchronizing computers. Since the number of sample points needed to correctly perform the interpolation depends only on the number of ....

Y. Minsky, A. Trachtenberg and R. Zippel, Set Reconciliation with Nearly Optimal Communication Complexity, International Symposium on Information Theory, Washington D.C., June 2001.

....on the PDA. Indeed, the typical case is where handheld devices and desktops regularly synchronize with each other so that few changes are made between synchronizations. We propose to apply a near optimal synchronization methodology based on recent research advances in fast set reconciliation [2, 3], in order to minimize the waste of network resources. Broadly speaking, given a PDA and a PC with data sets A and B, this new scheme can synchronize the hosts using one message in each direction of length A B B A (i.e. essentially independent of the size of the data sets A and B) ....

....we have implemented on a PDA as described in Section IV, requires a nearly minimal communication complexity and operates with a reasonable computational complexity. A. Deterministic scheme with a known upper bound The key to the set reconciliation algorithm of Minsky, Trachtenberg, and Zippel [2, 3] is a translation of data sets into polynomials designed specifically for efficient reconciliation. To this end, 2] makes use of a characteristic polynomial # S (Z) of a set S = x 1 ,x 2 , x n , defined to be: S (Z) Z x 3 ) Z x n ) 1) If we define the sets of missing ....

[Article contains additional citation context not shown here]

Y. Minsky, A. Trachtenberg, and R. Zippel, "Set reconciliation with nearly optimal communication complexity," in International Symposium on Information Theory, June 2001, p. 232.

....on the PDA. Indeed, the typical case is where handheld devices and desktops regularly synchronize with each other so that few changes are made between synchronizations. We propose to apply a near optimal synchronization methodology based on recent research advances in fast set reconciliation [2, 3], in order to minimize the waste of network resources. Broadly speaking, given a PDA and a PC with data sets A and B, this new scheme can synchronize the hosts using one message in each direction of length A B B A (i.e. essentially independent of the size of the data sets A and B) ....

....practical approaches for determining such a bound. In the first case, we propose a simple method that performs well for the synchronization of a small number of hosts (e.g. a PDA with two different PCs, one at work and one at home) In the second case, we make use of a probabilistic technique from [2] for testing the correctness of a guessed upper bound. If one guess turns out to be incorrect, then it can be modified in a second attempted synchronization, and so forth. The error of this probabilistic technique can be made arbitrarily small. We also show that the communication and time used by ....

[Article contains additional citation context not shown here]

Y. Minsky, A. Trachtenberg, and R. Zippel, "Set reconciliation with nearly optimal communication complexity," Tech. Rep. TR1999.

....that timestamp protocols, such as SyncML, adapt poorly to dynamic situations where devices are frequently added or removed from the network. This is because adding or removing a device from the network entails an update to every other device in the network. 3. 4 CPISync The CPISync algorithm [8, 9], which is an abbreviation for Characteristic Polynomial Interpolation Synchronization, is based on an algebraic solution to the problem of reconciling two remote sets of information. This algorithm has been shown [2] to be significantly more e#cient than Slow Sync in the most common scenario of ....

Y. Minsky, A. Trachtenberg, and R. Zippel. Set reconciliation with nearly optimal communication complexity. Technical Report TR1999.

....that timestamp protocols, such as SyncML, adapt poorly to dynamic situations where devices are frequently added or removed from the network. This is because adding or removing a device from the network entails an update to every other device in the network. 3. 4 CPISync The CPISync algorithm [8, 9], which is an abbreviation for Characteristic Polynomial Interpolation Synchronization, is based on an algebraic solution to the problem of reconciling two remote sets of information. This algorithm has been shown [2] to be signi cantly more ecient than Slow Sync in the most common scenario of ....

Y. Minsky, A. Trachtenberg, and R. Zippel. Set reconciliation with nearly optimal communication complexity. Technical Report TR1999.

....of databases is inecient when there are few actual di erences between the databases. This situation is, in fact, common in gossip protocols, particularly in situations where information is introduced into the system at a low rate. Set reconciliation. The set reconciliation problem was proposed in [10, 11] in an attempt to improve the performance of anti entropy. Consider a pair of hosts A and B, each holding a set SA and SB of b bit bitstrings. The goal of set reconciliation is for A and B to each compute SA [ SB with a minimum of communication. Set reconciliation is connected to graph coloring ....

....a number of applications outside of gossip protocols. In particular, it has been applied to PDA synchronization and routing table maintenance [8, 13, 14] More generally, set reconciliation can be used in a variety of systems where distributed information needs to be reconciled. Earlier work in [10, 11] presented algorithms for solving set reconciliation with near optimal communication complexity by interpolating rational functions over a nite eld. These algorithms require sending an amount of data on the order of the size of the symmetric di erence between SA and SB . Though very ecient from ....

[Article contains additional citation context not shown here]

Minsky, Y., Trachtenberg, A., and Zippel, R. Set reconciliation with nearly optimal communication complexity. In International Symposium on Information Theory (June 2001), p. 232.

....of databases is inecient when there are few actual di erences between the databases. This situation is, in fact, common in gossip protocols, particularly in situations where information is introduced into the system at a low rate. Set reconciliation. The set reconciliation problem was proposed in [10, 11] in an attempt to improve the performance of anti entropy. Consider a pair of hosts A and B, each holding a set SA and SB of b bit bitstrings. The goal of set reconciliation is for A and B to each compute SA [ SB with a minimum of communication. Set reconciliation is connected to graph coloring ....

....a number of applications outside of gossip protocols. In particular, it has been applied to PDA synchronization and routing table maintenance [8, 13, 14] More generally, set reconciliation can be used in a variety of systems where distributed information needs to be reconciled. Earlier work in [10, 11] presented algorithms for solving set reconciliation with near optimal communication complexity by interpolating rational functions over a nite eld. These algorithms require sending an amount of data on the order of the size of the symmetric di erence between SA and SB . Though very ecient from ....

[Article contains additional citation context not shown here]

Minsky, Y., Trachtenberg, A., and Zippel, R. Set reconciliation with nearly optimal communication complexity. Tech. Rep. TR

....that reconciliations occur with minimum communication, measured both by the number of transmitted bits and by the number of rounds of communication. When data are represented by sets, as can be reasonable modeled for the examples cited above, this problem is known as the set reconciliation problem [6, 7]. The data reconciliation problem is a natural generalization in which data is represented by multi sets rather than sets. This paper examines the data reconciliation problem within a generalized framework in which di erences between multi sets correspond to evaluations of arbitrary error ....

....C 2Z n q detects the error set E if c i 6 2 E(c j ) 8c i 6= c j 2C: De nition 6 A code C 2Z n q corrects the error set E if E(c i ) E(c j ) 8c i 6= c j 2C: Set and multi set reconciliation and veri cation. The traditional formalization of the set reconciliation problem is as follows [6, 7]: given a pair of hosts A and B, each with a set of length b bit strings (denoted SA and SB respectively) and no a priori knowledge of the other host s set, how can each host determine the mutual di erence of the two sets with a minimal amount of communication. In general, we may consider data ....

[Article contains additional citation context not shown here]

Yaron Minsky, Ari Trachtenberg, and Richard Zippel, \Set reconciliation with nearly optimal communication complexity," in International Symposium on Information Theory, 2001, to appear.

....that reconciliations occur with minimum communication, measured both by the number of transmitted bits and by the number of rounds of communication. When data are represented by sets, as can be reasonable modeled for the examples cited above, this problem is known as the set reconciliation problem [6, 7]. The data reconciliation problem is a natural generalization in which data is represented by multi sets rather than sets. This paper examines the data reconciliation problem within a generalized framework in which di erences between multi sets correspond to evaluations of arbitrary error ....

....C 2Z n q detects the error set E if c i 6 2 E(c j ) 8c i 6= c j 2C: De nition 6 A code C 2Z n q corrects the error set E if E(c i ) E(c j ) 8c i 6= c j 2C: Set and multi set reconciliation and veri cation. The traditional formalization of the set reconciliation problem is as follows [6, 7]: given a pair of hosts A and B, each with a set of length b bit strings (denoted SA and SB respectively) and no a priori knowledge of the other host s set, how can each host determine the mutual di erence of the two sets with a minimal amount of communication. In general, we may consider data ....

Yaron Minsky, Ari Trachtenberg, and Richard Zippel, \Set reconciliation with nearly optimal communication complexity," Tech. Rep. TR

....union of the two sets with a minimal amount of communication both with respect to the number of exchanges between the two hosts and with respect to the number of bits of information exchanged. Within this context, only the contents of the sets is important, but not their actual organization. In [2] thisformalization iscalled the ### ############## problem. Thesimplest solution to the set reconciliation problem, and one that is often implemented in practice [3] is for both hosts to exchange hashes of their integers and then to individually request integers for the hashes they aremissing. ....

....this scheme cannot be directly applied to the case where both ## and ## are changing over time or where data isbothinserted and deleted from ## . Two recent solutions to the set reconciliation problem, one based on Reed Solomon decoding and one based on rational function interpolation [2], show promise for signi cant performance improvement in synchronization schemes. Both of them approach an information theoretic lower bound of ## # # log # bits of communication needed to reconcile two sets whose contents di er in # # bit integers. In the next subsectionswegiveahigh level ....

[Article contains additional citation context not shown here]

Yaron Minsky, Ari Trachtenberg, and Richard Zippel, \Set reconciliation with nearlyoptimal communication complexity," #### ###### #### ######, 2000, submitted.

No context found.

Y. Minsky, A. Trachtenberg, and R. Zippel. Set reconciliation with nearly optimal communication complexity. IEEE Transactions on Information Theory, 49(9):2212--2218, September 2003.

No context found.

Y. Minsky, A. Trachtenberg, and R. Zippel. Set reconciliation with nearly optimal communication complexity. In Int. Symp. on Information Theory, page 232, June 2001.

No context found.

Y. Minsky, A. Trachtenberg, R. Zippel, Set reconciliation with nearly optimal communications complexity, in: IEEE Intl. Symp. on Information Theory, 2001.

No context found.

Yaron Minsky, Ari Trachtenberg, and Richard Zippel. Set reconciliation with nearly optimal communications complexity. In ISIT, 2001.

No context found.

Y. Minsky, A. Trachtenberg and R. Zippel. Set reconciliation with nearly optimal communication complexity. Technical Report 2000-1796, Cornell Univ.

No context found.

Y. Minsky, A. Trachtenberg, and R. Zippel. Set Reconciliation with Nearly Optimal Communication Complexity. In International Symposium on Information Theory, 2001.

No context found.

Y. Minsky, A. Trachtenberg and R. Zippel. Set reconciliation with nearly optimal communication complexity. Technical Report 2000-1796, Cornell Univ.

No context found.

Y. Minsky, A. Trachtenberg, and R. Zippel, "Set reconciliation with nearly optimal communications complexity," in IEEE International Symposium on Information Theory, Washington DC, June 2001.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC