Citation Context

... the index of a Web search engine, or to archive them for future reference. 2.1. Web Crawling Descriptions of early versions of Google’s and Internet Archive’s large-scale crawler systems appeared in =-=[9,10]-=-, respectively. However, one of the first detailed descriptions of a scalable Web crawler is that of Mercator by Heydon and Najork [11], who provide information on the various modules of the crawler a...

Citation Context

...spects of data collection from the Web [23], by selectively crawling pages that are relevant to a set of topics, defined as a set of keywords [24], by example documents mapped to a taxonomy of topics =-=[25]-=-, or by ontologies [26,27]. Recent approaches also address the crawling of information for specific geographical locations [28,29]. The main challenges in focused crawling relate to the prioritization...

Citation Context

...er crawling billions of Web pages. As the Web evolves, and Web pages are created, modified, or deleted [17,18], effective crawling approaches are needed to handle these changes. Cho and Garcia Molina =-=[19]-=- describe an incremental crawler for optimizing the average freshness of crawled Web data. Olston and Pandey [20] describe re-crawling strategies to optimize freshness based on the longevity of inform...

Citation Context

...4,26], hyperlink distance-based limits [30,31], or combinations of text and hyperlink analysis with Latent Semantic Indexing (LSI) [32]. Machine learning approaches, including naïve Bayes classifiers =-=[25,33]-=-, Hidden Markov Models [34], reinforcement learning [35], genetic algorithms [36], and neural networks [37], have also been applied to prioritize the unvisited URLs. Focused crawlers and crawlers in g...

Citation Context

...as been seen previously. The use of DRUM allows IRLBot to maintain a high crawling rate, even after crawling billions of Web pages. As the Web evolves, and Web pages are created, modified, or deleted =-=[17,18]-=-, effective crawling approaches are needed to handle these changes. Cho and Garcia Molina [19] describe an incremental crawler for optimizing the average freshness of crawled Web data. Olston and Pand...

Citation Context

...as been seen previously. The use of DRUM allows IRLBot to maintain a high crawling rate, even after crawling billions of Web pages. As the Web evolves, and Web pages are created, modified, or deleted =-=[17,18]-=-, effective crawling approaches are needed to handle these changes. Cho and Garcia Molina [19] describe an incremental crawler for optimizing the average freshness of crawled Web data. Olston and Pand...

Citation Context

...e crawler, developed at Future Internet 2014, 6 521 the Internet Archive. In Section 4.4 we will describe how we have adapted Heritrix in order to fit in ARCOMEM’s crawling architecture. Boldi et al. =-=[15]-=- describe UBICrawler, a distributed Web crawler, implemented in Java, which operates in a decentralized way and uses consistent hashing to partition the domains to crawl across the crawling servers. L...

Citation Context

...and Internet Archive’s large-scale crawler systems appeared in [9,10], respectively. However, one of the first detailed descriptions of a scalable Web crawler is that of Mercator by Heydon and Najork =-=[11]-=-, who provide information on the various modules of the crawler and the design options. Najork and Heydon also describe a distributed crawler based on Mercator in [12]. Shkapenyuk and Suel [13] introd...

Citation Context

...Najork [11], who provide information on the various modules of the crawler and the design options. Najork and Heydon also describe a distributed crawler based on Mercator in [12]. Shkapenyuk and Suel =-=[13]-=- introduce a distributed and robust crawler, managing the failure of individual servers. Heritrix [14] is an archival-quality and modular open source crawler, developed at Future Internet 2014, 6 521 ...

Citation Context

...effective way to balance the cost, coverage, and quality aspects of data collection from the Web [23], by selectively crawling pages that are relevant to a set of topics, defined as a set of keywords =-=[24]-=-, by example documents mapped to a taxonomy of topics [25], or by ontologies [26,27]. Recent approaches also address the crawling of information for specific geographical locations [28,29]. The main c...

Citation Context

...cal locations [28,29]. The main challenges in focused crawling relate to the prioritization of URLs not yet visited, which may be based on similarity measures [24,26], hyperlink distance-based limits =-=[30,31]-=-, or combinations of text and hyperlink analysis with Latent Semantic Indexing (LSI) [32]. Machine learning approaches, including naïve Bayes classifiers [25,33], Hidden Markov Models [34], reinforcem...

Citation Context

...ML forms [38]. Such forms are easy to complete by human users. Automatic deep-Web crawlers, however, need to complete HTML forms and retrieve results from the underlying databases. Barbosa and Freire =-=[39]-=- develop mechanisms for generating simple keyword queries that cover the underlying database through unstructured simple search forms. Madhavan et al. [40] handle structured forms by automatically com...

Citation Context

... the index of a Web search engine, or to archive them for future reference. 2.1. Web Crawling Descriptions of early versions of Google’s and Internet Archive’s large-scale crawler systems appeared in =-=[9,10]-=-, respectively. However, one of the first detailed descriptions of a scalable Web crawler is that of Mercator by Heydon and Najork [11], who provide information on the various modules of the crawler a...

Citation Context

...cal locations [28,29]. The main challenges in focused crawling relate to the prioritization of URLs not yet visited, which may be based on similarity measures [24,26], hyperlink distance-based limits =-=[30,31]-=-, or combinations of text and hyperlink analysis with Latent Semantic Indexing (LSI) [32]. Machine learning approaches, including naïve Bayes classifiers [25,33], Hidden Markov Models [34], reinforcem...

Citation Context

...of Mercator by Heydon and Najork [11], who provide information on the various modules of the crawler and the design options. Najork and Heydon also describe a distributed crawler based on Mercator in =-=[12]-=-. Shkapenyuk and Suel [13] introduce a distributed and robust crawler, managing the failure of individual servers. Heritrix [14] is an archival-quality and modular open source crawler, developed at Fu...

Citation Context

...on from the Web [23], by selectively crawling pages that are relevant to a set of topics, defined as a set of keywords [24], by example documents mapped to a taxonomy of topics [25], or by ontologies =-=[26,27]-=-. Recent approaches also address the crawling of information for specific geographical locations [28,29]. The main challenges in focused crawling relate to the prioritization of URLs not yet visited, ...

Citation Context

...ork and Heydon also describe a distributed crawler based on Mercator in [12]. Shkapenyuk and Suel [13] introduce a distributed and robust crawler, managing the failure of individual servers. Heritrix =-=[14]-=- is an archival-quality and modular open source crawler, developed at Future Internet 2014, 6 521 the Internet Archive. In Section 4.4 we will describe how we have adapted Heritrix in order to fit in ...

Citation Context

...ffective crawling approaches are needed to handle these changes. Cho and Garcia Molina [19] describe an incremental crawler for optimizing the average freshness of crawled Web data. Olston and Pandey =-=[20]-=- describe re-crawling strategies to optimize freshness based on the longevity of information on Web pages. Pandey and Olston [21] also introduce a parameterized algorithm for monitoring Web resources ...

Citation Context

...acquisition 1. Introduction The World Wide Web is the largest information repository. However, this information is very volatile: the typical half-life of content referenced by URLs is of a few years =-=[1]-=-; this trend is even aggravated in social media, where social networking APIs sometimes only extend to a week’s worth of content [2]. Web archiving [3] deals with the collection, enrichment, curation,...

Citation Context

...ptimizing the average freshness of crawled Web data. Olston and Pandey [20] describe re-crawling strategies to optimize freshness based on the longevity of information on Web pages. Pandey and Olston =-=[21]-=- also introduce a parameterized algorithm for monitoring Web resources for updates and optimizing timeliness or completeness depending on application-specific requirements. 2.2. Focused and Deep-Web C...

Citation Context

...on from the Web [23], by selectively crawling pages that are relevant to a set of topics, defined as a set of keywords [24], by example documents mapped to a taxonomy of topics [25], or by ontologies =-=[26,27]-=-. Recent approaches also address the crawling of information for specific geographical locations [28,29]. The main challenges in focused crawling relate to the prioritization of URLs not yet visited, ...

Citation Context

...of URLs not yet visited, which may be based on similarity measures [24,26], hyperlink distance-based limits [30,31], or combinations of text and hyperlink analysis with Latent Semantic Indexing (LSI) =-=[32]-=-. Machine learning approaches, including naïve Bayes classifiers [25,33], Hidden Markov Models [34], reinforcement learning [35], genetic algorithms [36], and neural networks [37], have also been appl...

Citation Context

...ers and crawlers in general can harvest data from the publicly indexable Web by following hyperlinks between Web pages. However, there is a very large part of the Web that is hidden behind HTML forms =-=[38]-=-. Such forms are easy to complete by human users. Automatic deep-Web crawlers, however, need to complete HTML forms and retrieve results from the underlying databases. Barbosa and Freire [39] develop ...

Citation Context

...ing complete snapshots of a domain taken at regular intervals. A drawback of this approach is the lack of knowledge about changes of Web pages between crawls and the consistency of the collected data =-=[44]-=-. The latter approach results in higher quality collections restricted only to selected Web sites. Denev et al. [45] introduce a framework for assessing the quality of archives and tune the crawling s...

Citation Context

...k analysis with Latent Semantic Indexing (LSI) [32]. Machine learning approaches, including naïve Bayes classifiers [25,33], Hidden Markov Models [34], reinforcement learning [35], genetic algorithms =-=[36]-=-, and neural networks [37], have also been applied to prioritize the unvisited URLs. Focused crawlers and crawlers in general can harvest data from the publicly indexable Web by following hyperlinks b...

Citation Context

...icted only to selected Web sites. Denev et al. [45] introduce a framework for assessing the quality of archives and tune the crawling strategies to optimize quality with given resources. Gomes et al. =-=[46]-=- provide a survey of Web archiving initiatives. Focused crawlers, as described above, can be used for creating focused Web archives, by relying on a selective content acquisition approach. The crawlin...

Citation Context

...tive crawls require a lot of manual work for the crawl preparation, crawler control, and quality assurance. On the technical level, current-day archiving crawlers, such as Internet Archive’s Heritrix =-=[4]-=-, crawl the Web in a conceptually simple manner (See Figure 1). They start from a seed list of URLs (typically provided by a Web archivist) to be stored in a queue. Web pages are then fetched from thi...

Citation Context

...about changes of Web pages between crawls and the consistency of the collected data [44]. The latter approach results in higher quality collections restricted only to selected Web sites. Denev et al. =-=[45]-=- introduce a framework for assessing the quality of archives and tune the crawling strategies to optimize quality with given resources. Gomes et al. [46] provide a survey of Web archiving initiatives....

Citation Context

...-specific requirements. 2.2. Focused and Deep-Web Crawling Focused or topical crawlers [22] provide an effective way to balance the cost, coverage, and quality aspects of data collection from the Web =-=[23]-=-, by selectively crawling pages that are relevant to a set of topics, defined as a set of keywords [24], by example documents mapped to a taxonomy of topics [25], or by ontologies [26,27]. Recent appr...

Citation Context

...tion and quality assurance. It is the aim of the ARCOMEM project [5] to support the selective crawling on the technical level by leveraging social media and semantics to build meaningful Web archives =-=[6]-=-. This requires, in particular, a change of paradigm in how content is collected technically via Web crawling, which is the topic of the present article. This traditional processing chain of a Web cra...

Citation Context

...ons of Web objects that have been collected over time and can cover several crawls. Analysis implemented exemplary on this level within the ARCOMEM system is used to recognize Named Entity Evolutions =-=[47]-=- and to analyze the evolutions of associations between interesting terms and tweets (Twitter Dynamics) [48]. 3.5. Applications For the interaction with the crawler and exploration of the content a num...

Citation Context

...et of keywords [24], by example documents mapped to a taxonomy of topics [25], or by ontologies [26,27]. Recent approaches also address the crawling of information for specific geographical locations =-=[28,29]-=-. The main challenges in focused crawling relate to the prioritization of URLs not yet visited, which may be based on similarity measures [24,26], hyperlink distance-based limits [30,31], or combinati...

Citation Context

...ces in the process. Application-aware crawling also helps adding semantic information to the ARCOMEM database. More details about the functioning and independent evaluation of the AAH are provided in =-=[49,50]-=-. 4.2. Online Analysis Within the online analysis, several modules analyze crawled Web objects in order to guide the crawler. The purpose of this process is to provide scores for detected URLs. These ...

Citation Context

...d limits [30,31], or combinations of text and hyperlink analysis with Latent Semantic Indexing (LSI) [32]. Machine learning approaches, including naïve Bayes classifiers [25,33], Hidden Markov Models =-=[34]-=-, reinforcement learning [35], genetic algorithms [36], and neural networks [37], have also been applied to prioritize the unvisited URLs. Focused crawlers and crawlers in general can harvest data fro...

Citation Context

...UBICrawler, a distributed Web crawler, implemented in Java, which operates in a decentralized way and uses consistent hashing to partition the domains to crawl across the crawling servers. Lee et al. =-=[16]-=- describe the architecture and main data structures of IRLBot, a crawler which implements DRUM (Disk Repository with Update Management) for checking whether a URL has been seen previously. The use of ...

Citation Context

...et of keywords [24], by example documents mapped to a taxonomy of topics [25], or by ontologies [26,27]. Recent approaches also address the crawling of information for specific geographical locations =-=[28,29]-=-. The main challenges in focused crawling relate to the prioritization of URLs not yet visited, which may be based on similarity measures [24,26], hyperlink distance-based limits [30,31], or combinati...

Citation Context

...ze and dynamics, there have been several national initiatives for preserving the Web of a country, based on full crawls in Sweden [41] and on a selective collection of Web pages in the United Kingdom =-=[42]-=- and Australia [43]. The former Future Internet 2014, 6 522 approach aims at providing complete snapshots of a domain taken at regular intervals. A drawback of this approach is the lack of knowledge a...

Citation Context

...antic Indexing (LSI) [32]. Machine learning approaches, including naïve Bayes classifiers [25,33], Hidden Markov Models [34], reinforcement learning [35], genetic algorithms [36], and neural networks =-=[37]-=-, have also been applied to prioritize the unvisited URLs. Focused crawlers and crawlers in general can harvest data from the publicly indexable Web by following hyperlinks between Web pages. However,...

Citation Context

...ere have been several national initiatives for preserving the Web of a country, based on full crawls in Sweden [41] and on a selective collection of Web pages in the United Kingdom [42] and Australia =-=[43]-=-. The former Future Internet 2014, 6 522 approach aims at providing complete snapshots of a domain taken at regular intervals. A drawback of this approach is the lack of knowledge about changes of Web...

Citation Context

...ions of text and hyperlink analysis with Latent Semantic Indexing (LSI) [32]. Machine learning approaches, including naïve Bayes classifiers [25,33], Hidden Markov Models [34], reinforcement learning =-=[35]-=-, genetic algorithms [36], and neural networks [37], have also been applied to prioritize the unvisited URLs. Focused crawlers and crawlers in general can harvest data from the publicly indexable Web ...

Citation Context

...rithm for monitoring Web resources for updates and optimizing timeliness or completeness depending on application-specific requirements. 2.2. Focused and Deep-Web Crawling Focused or topical crawlers =-=[22]-=- provide an effective way to balance the cost, coverage, and quality aspects of data collection from the Web [23], by selectively crawling pages that are relevant to a set of topics, defined as a set ...

Citation Context

...Since archiving the whole Web is a very challenging task due to its size and dynamics, there have been several national initiatives for preserving the Web of a country, based on full crawls in Sweden =-=[41]-=- and on a selective collection of Web pages in the United Kingdom [42] and Australia [43]. The former Future Internet 2014, 6 522 approach aims at providing complete snapshots of a domain taken at reg...

Citation Context

... of content referenced by URLs is of a few years [1]; this trend is even aggravated in social media, where social networking APIs sometimes only extend to a week’s worth of content [2]. Web archiving =-=[3]-=- deals with the collection, enrichment, curation, and preservation of today’s volatile Web content in an archive that remains accessible to tomorrow’s historians. Different strategies for Web archivin...

Citation Context

...s article is to present an overview of this crawling architecture, and of its performance (both in terms of efficiency and of quality of the archive obtained) on real-Web crawls. This article extends =-=[7]-=-. The remainder of this work is organized as follows. We first discuss in Section 2 the related work. Then we present in Section 3 a high-level view of the ARCOMEM architecture, before reviewing indiv...

Citation Context

...emplary on this level within the ARCOMEM system is used to recognize Named Entity Evolutions [47] and to analyze the evolutions of associations between interesting terms and tweets (Twitter Dynamics) =-=[48]-=-. 3.5. Applications For the interaction with the crawler and exploration of the content a number of applications are used around the ARCOMEM core system. The crawler cockpit is used to create the craw...

Citation Context

...ces in the process. Application-aware crawling also helps adding semantic information to the ARCOMEM database. More details about the functioning and independent evaluation of the AAH are provided in =-=[49,50]-=-. 4.2. Online Analysis Within the online analysis, several modules analyze crawled Web objects in order to guide the crawler. The purpose of this process is to provide scores for detected URLs. These ...