The rapid growth of the World-Wide Web poses unprecedented scaling challenges for general-purpose crawlers and search engines. In this paper we describe a new hypertext resource discovery system called a Focused Crawler. The goal of a focused crawler is to selectively seek out pages

This paper presents an overview of resource discovery services currently available on the Internet. First, we survey a number of existing Internet discovery services. Then, we present a taxonomy of design decisions and characteristics of tools for the Internet resource discovery problem [30

In large distributed networks of computers, it is often the case that a subset of machines wants to cooperate to perform a task. Before they can do so, these machines need to learn of the existence of each other. In this paper we are interested in distributed algorithms whereby machines in a network learn of other machines in the network by making queries to machines they already know. The algorithms should be efficient both in terms of the time required and in terms of the total network communication required until all machines have discovered all other machines. We propose a very simple algorithm called Name-Dropper whereby all machines learn about each other within O(log 2 n) rounds, where n is the number of machines in the network. The total number of connections required is O(n log 2 n), and the total number of pointers which must be communicated is O(n 2 log 2 n). Each of the preceding bounds is optimal to within polylogarithmic factors. Contact author: Mor Harc...

Abstract. Resource discovery systems become more and more important as distributed systems grow and as their pool of resources becomes more variable. As such, an increasing amount of networked systems provide a discovery service. This paper provides a taxonomy for resource discovery systems

efficient resource discovery

The resource discovery problem was introduced by Harchol-Balter, Leighton and Lewin.

information without any user intervention. Resource discovery will play an integral role in the future of pervasive systems but there are many significant problems yet to be overcome. This paper investigates and analyses the deficiencies of the current resource discovery protocols; identifies the key

Consider a dynamic, large-scale communication infrastructure (e.g., the Internet) where nodes (e.g., in a peer to peer system) can communicate only with nodes whose id (e.g., IP address) are known to them. One of the basic building blocks of such a distributed system is resource discovery

This paper reports on a simple---yet effective---resource discovery tool called babyOIL which is a query interface to a number of information sources. The babyOIL system provides a unified interface to common resource discovery information sources and utilises current Internet technologies

This paper reports on a simple---yet effective---resource discovery tool called babyOIL which is a query interface to a number of information sources. The babyOIL system provides a unified interface to common resource discovery information sources and utilises current Internet technologies