Abstract

An ever-increasing amount of information on the Web today is available only through search interfaces: the users have to type in a set of keywords in a search form in order to access the pages from certain Web sites. These pages are often referred to as the Hidden Web or the Deep Web. Since there are no static links to the Hidden Web pages, search engines cannot discover and index such pages and thus do not return them in the results. However, according to recent studies, the content provided by many Hidden Web sites is often of very high quality and can be extremely valuable to many users. In this paper, we study how we can build an effective Hidden Web crawler that can autonomously discover and download pages from the Hidden Web. Since the only “entry point ” to a Hidden Web site is a query interface, the main challenge that a Hidden Web crawler has to face is how to automatically generate meaningful queries to issue to the site. Here, we provide a theoretical framework to investigate the query generation problem for the Hidden Web and we propose effective policies for generating queries automatically. Our policies proceed iteratively, issuing a different query in every iteration. We experimentally evaluate the effectiveness of these policies on 4 real Hidden Web sites and our results are very promising. For instance, in one experiment, one of our policies downloaded more than 90 % of a Hidden Web site (that contains 14 million documents) after issuing fewer than 100 queries.

Citations

1159	Fractal Geometry of Nature - Mandelbrot - 1982
787	Human Behavior and the Principle of Least-Effort - Zipf - 1949
346	Syntactic clustering of the web - Broder, Glassman, et al. - 1997
253	Introduction to Algorithms. 2nd edition - Cormen, Leiserson, et al. - 1990
221	Database techniques for the WorldWide Web: a survey - Florescu, Levy, et al. - 1998
173	Crawling The Hidden Web - Raghavan, Garcia - 2001
148	A technique for measuring the relative size and overlap of public web search engines - Bharat, Broder - 1998
134	Query-based sampling of text databases - Callan, Connell, et al. - 1999
130	What’s new on the Web? The evolution of the Web from a search engine perspective - Ntoulas, Cho, et al. - 2004
107	chen-chuan chang; “Statistical schema matching across web query interfaces - He, Kevin - 2003
104	Automatic discovery of language models for text databases - Callan, Connell, et al. - 1999
85	Distributed search over the hidden web: Hierarchical database sampling and selection - Ipeirotis, Gravano - 2002
60	Finding replicated web collections - Cho, Shivakumar, et al.
50	Structured databases on the web: observations and implications - Chang, He, et al. - 2004
43	The open archive initiative: Building a low-barrier interoperability framework - LAGOZE, SOMPEL - 2001
41	M.: Probe, count, and classify: Categorizing hidden web databases - Ipeirotis, Gravano, et al. - 2001
37	Querying text databases for efficient information extraction - Agichtein, Gravano
35	Siphoning hidden-web data through keyword-based interfaces - Barbosa, Freire - 2004
28	Learning to query the web - Cohen, Singer - 1996
15	Automated Discovery of Search Interfaces on the web - Cope, Craswell, et al. - 2003
12	Modeling query-based access to text databases - Agichtein, Ipeirotis, et al. - 2003
7	Does search engine’s power threaten Web’s independence - Olsen - 2002
3	Dpro: A probabilistic approach for hidden web database selection using dynamic probing - Liu, and, et al. - 2004
3	Downloading hidden web content - Ntoulas, Zerfos, et al. - 2004
2	The deep web: Surfacing hidden value,http - Bergman - 2001
2	DP9-An OAI Gateway Service for Web Crawlers - Liu, Maly, et al. - 2002