Package org.apache.nutch.util
package org.apache.nutch.util
Miscellaneous utility classes.
-
ClassesClassDescriptionScaffolding class for the various Checker implementations.Extracts some simple crawl completion stats from the crawldb Stats will be sorted by host/domain and will be of the form: 1 www.spitzer.caltech.edu FETCHED 50 www.spitzer.caltech.edu UNFETCHEDA collection of utility methods for working on deflated data.Extracts some very basic statistics about domains from the crawldbA simple class for detecting character encodings.Utility methods for common filesystem operations.A generic Writable wrapper that can inject Configuration to
ConfigurablesA collection of utility methods for working on GZIPed data.Utility methods for handling JEXL expressions used in crawl and index pipelines.Utility methods for handling application-level locking.This is a facade class to insulate Nutch from its underlying Mime Type substrate library, Apache Tika.A utility class that allows the walking of any DOM tree using a stack instead of recursion.Utility to create HadoopConfigurations that include Nutch-specific resources.AJobfor Nutch jobs.A class for efficiently matchingStrings against a set of prefixes.Extracts protocol status code information from the crawl database.Performs sitemap processing by fetching sitemap links, parsing the content and merging the URLs from sitemaps (with the metadata) into the CrawlDb.A collection of String processing utility methods.A class for efficiently matchingStrings against a set of suffixes.TrieStringMatcher is a base class for simple tree-based string matching.Utility class for URL analysis