Abstract Search Engines and Web Dynamics
Abstract:
In this paper we study several dimensions of web dynamics in the context of large-scale Internet search engines. Both growth and update dynamics clearly represent big challenges for search engines. We show how the problems arise in all components of a reference search engine model. Furthermore, we use the FAST Search Engine architecture as a case study for showing some possible solutions for web dynamics and search engines. The focus is to demonstrate solutions that work in practice for real systems. The service is running live at www.alltheweb.com and major portals worldwide with more than 30 million queries a day, about 700 million full-text documents, a crawl base of 1.8 billion documents, updated every 11 days, at a rate of 400 documents/second. We discuss future evolution of the web, and some important issues for search engines will be scheduling and query execution as well as increasingly heterogeneous architectures to handle the dynamic web. 1
Citations
| 1632 | The anatomy of a large-scale hypertextual (Web) search engine – Brin, Page - 1998 |
| 130 | The Evolution of the Web and Implications for an Incremental Crawler – Cho, Garica-Molina - 2000 |
| 123 | Synchronizing a Database to Improve Freshness – Cho, García-Molina - 2000 |
| 98 | Mercator: A scalable, extensible web crawler – Heydon, Najork - 1999 |
| 98 | GENVL and WWWW: Tools for Taming the Web – McBryan |
| 95 | Measuring the Web – Bray |
| 91 | How Dynamic is the Web – Brewington, Cybenko - 2000 |
| 66 | WebBase: A Repository of Web Pages – Hirai, Raghavan, et al. |
| 53 | An adaptive model for optimizing performance of an incremental Web crawler – Edwards, McCurley, et al. - 2002 |
| 33 | Growth dynamics of the world wide web – ADAMIC, A - 1999 |
| 29 | A Standard for Robot Exclusion – Koster - 1994 |
| 13 | Searching the World Wide – Lawrence, Giles - 1998 |
| 8 | et al. Graph Structure in the Web – Broder - 2000 |
| 4 | The AltaVista Search Revolution. Osborne McGraw-Hill – Ray, Ray, et al. - 1998 |
