DMCA
Efficient similarity joins for near duplicate detection (2008)
Cached
Download Links
- [www.cse.unsw.edu.au]
- [www.cse.unsw.edu.au]
- DBLP
Other Repositories/Bibliography
| Venue: | In WWW |
| Citations: | 103 - 9 self |
BibTeX
@INPROCEEDINGS{Xiao08efficientsimilarity,
author = {Chuan Xiao and New South Wales and Wei Wang},
title = {Efficient similarity joins for near duplicate detection},
booktitle = {In WWW},
year = {2008}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
With the increasing amount of data and the need to integrate data from multiple data sources, one of the challenging issues is to identify near duplicate records efficiently. In this paper, we focus on efficient algorithms to find pair of records such that their similarities are no less than a given threshold. Several existing algorithms rely on the prefix filtering principle to avoid computing similarity values for all possible pairs of records. We propose new filtering techniques by exploiting the token ordering information; they are integrated into the existing methods and drastically reduce the candidate sizes and hence improve the efficiency. We have also studied the implementation of our proposed algorithm in stand-alone and RDBMSbased settings. Experimental results show our proposed algorithms can outperforms previous algorithms on several real datasets.
