Citation Context

...cations that require detecting similar pairs of records where the records contain string or setbased data. A list of possible applications includes: detecting near duplicate web-pages in web crawling =-=[14]-=-, document clustering[5], plagiarism detection [15], master data management, making recommendations to users based on their similarity to other users in query refinement [22], mining in social network...

Citation Context

...matics, and pattern recognition. Similarity join is also adopted in the industry solutions. For example, Google adopts both approximate and exact similarity join for near duplicate Web page detection =-=[18]-=-, query log mining, and collaborative filtering [3]. Microsoft researchers proposed the SSJoin primitive operator [12] to support similarity join and it has been used in the Data Debugger project [11]...

Citation Context

...oaches for filtering good shingles, while Büttcher and Clarke [9] focus on informationtheoretic measures such as Kullback-Leibler divergence in the more general context of search. Recently, Henzinger =-=[16]-=- combined two algorithms for detecting near-duplicate Web pages, namely Broder’s DSC and Charikar’s [10] random projection algorithm. Henzinger improved on precision compared to using the constituent ...

Citation Context

...ds returns pairs of records from two sets such that their similarities are no less than a given threshold. Similar joins have a wide range of applications, including near duplicate Web page detection =-=[1]-=-, data integration [2], record linkage [3], and data mining [4]. Consequently, there has been much interest in developing efficient algorithms for this fundamental operation. However, the traditional ...

Citation Context

... family are used to compare objects that are naturally sequenced, e.g., documents that consist of a sequence of words. For example, shingling [5] is frequently used to detect near-duplicate web pages =-=[15]-=-. Because sequence comparison algorithms are efficient and can operate on large inputs, we want to consider them as candidates for our problem. Thus, here we use a sequence comparison scheme, shinglin...

Citation Context

...rithm generating new fingerprints (super-shingles) by hashing sequences of fingerprints again. Charikar [5] introduced a hashing algorithm based on random projections of words in documents. Henzinger =-=[10]-=- empirically compared a variant of Broder et al’s algorithm and Charikar’s algorithm on a large scale Web collection. Chowdhury et al. [6] and Bernstein and Zobel [1] proposed filtration algorithms fo...

Citation Context

...e less relevant in the setting of this paper, where text reuse is detected based on a query, duplicate detection has been shown to be useful for related tasks such as finding near-duplicate web pages =-=[15]-=-, detection of passage-level text reuse in blogs [28], spam web pages detection [12], and tracking the evolution of textual content on the web [4]. Prior work that examined similarity measures for tex...

Citation Context

... shapes of the distributions, the distance measure is proportional to the effort to transform one pile into the other. 7 [16] has reported the bias factor grows linearly with |A|. In another analysis =-=[17]-=-, Henzinger reported that the algorithm in [9] is more accurate than the algorithm in [5, 6] on the application of detecting near-duplicate web pages when using the same fingerprint size. That is attr...

Citation Context

...China crawling is discussed, to automatically fetch hidden Web pagesswhich are typically only accessible by submitting queries to asdatabase. Another related topic is near-duplicate page detections[6]=-=[17]-=-[19], to remove redundant pages and reduce indexingsoverhead. And based on the ideas from [7], the Sitemaps Protocols[3] was recently proposed in industry to allow a webmaster tosinform search engines...