Package org.apache.nutch.crawl
package org.apache.nutch.crawl
Crawl control code and tools to run the crawler.
-
ClassDescriptionThis class provides common methods for implementations of
FetchSchedule.This class implements an adaptive re-fetch algorithm.A Comparator optimized for CrawlDatum.This class takes the output of the fetcher and updates the crawldb accordingly.This class provides a way to separate the URL normalization and filtering steps from the rest of CrawlDb manipulation code.This tool merges several CrawlDb-s into one, optionally filtering URLs through the current URLFilters, to skip prohibited pages.Read utility for the CrawlDB.Merge new page entries with existing entries.Generic deduplicator which groups fetched URLs with the same digest and marks all of them as duplicate except the one with the highest score (based on the score in the crawldb, which is not necessarily the same as the score indexed).DeduplicationJob.DedupReducer<K extends Writable>Combine multiple new entries for a url.This class implements the default re-fetch schedule.This interface defines the contract for implementations that manipulate fetch times and re-fetch intervals.Creates and caches aFetchScheduleimplementation.Generates a subset of a CrawlDb to fetch.Update the CrawlDB so that the next generate won't include the same URLs.Sort fetch lists by hash of URL.Selects entries due for fetch.Select and invert subset due for fetch.Collect until limit is reached.Injector takes a flat text file of URLs (or a folder containing text files) and merges ("injects") these URLs into the CrawlDb.InjectMapper reads the CrawlDb seeds are injected into the plain-text seed files and parses each line into the URL and metadata.Combine multiple new entries for a url.An incoming link to a page.A list ofInlinks.Maintains an inverted link map, listing incoming links for each url.This class provides a way to separate the URL normalization and filtering steps from the rest of LinkDb manipulation code.This tool merges several LinkDb-s into one, optionally filtering URLs through the current URLFilters, to skip prohibited URLs and links.Read utility for the LinkDb.Default implementation of a page signature.Extension of @see AdaptiveFetchSchedule that allows for more flexible configuration of DEC and INC factors for various MIME-types.Factory class, which instantiates a Signature implementation according to the current Configuration configuration.Implementation of a page signature.An implementation of a page signature.Partition urls by host, domain name or IP depending on the value of the parameter 'partition.url.mode' which can be 'byHost', 'byDomain' or 'byIP'