Uses of Interface
org.apache.nutch.plugin.Pluggable
Packages that use Pluggable
Package
Description
Text document language identifier.
Subcollection is a subset of an index.
Control code for exchange component, which acts in indexing job and decides to
which index writer a document should be routed, based on plugins behavior.
Plugin of Exchange component based on JEXL expressions.
Index content, configure and run indexing and cleaning jobs to
add, update, and delete documents from an index.
An indexing plugin for inbound anchor text.
Indexing filter to add document arbitrary data to the index
from the output of a user-specified class.
A basic indexing plugin, adds basic fields: url, host, title, content, etc.
Indexing filter to index meta data from RSS feeds.
This plugin implements an indexing filter which takes
advantage of the
GeoIP2-java API.
This plugin implements a dynamic indexing filter which uses JEXL
expressions to allow filtering based on the page's metadata
Indexing filter to add document metadata to the index.
A more indexing plugin, adds "more" index fields:last modified
date, MIME type, content length.
Indexing filter to allow pattern replacements on metadata.
A simple plugin called at indexing that adds fields with static data.
Indexing filter to assign documents to subcollections.
Top Level Domain Indexing plugin.
URL Meta Tag Indexing Plugin
Index writer plugin to write a plain CSV file.
Index writer plugin for debugging, writes pairs of <action, url> to a
text file, action is one of "add", "update", or "delete".
Index writer plugin for Elasticsearch.
Index writer plugin to produce JSON messages to Kafka.
Index writer plugin for OpenSearch.
Index writer plugin for Apache Solr.
A microformats Rel-Tag
Parser/Indexer/Querier plugin.
Web-related interfaces: URL
filters
and normalizers.The
Parse interface and related classes.Parse wrapper to run external command to do the parsing.
Parse RSS feeds.
Parse filter to extract headings (h1, h2, etc.) from DOM parse tree.
An HTML document parsing plugin.
Parser and parse filter plugin to extract all (possible) links
from JavaScript files and embedded JavaScript code snippets.
Parse filter to extract meta tags: keywords, description, etc.
Parse various document formats with help of
Apache Tika.
Parse ZIP files: embedded files are recursively passed to appropriate parsers.
Adds serialized DOM to parse data, useful for debugging, to understand how
the parser implementation interprets a document (not only HTML).
Html Parse filter that classifies the outlinks from the parseresult as
relevant or irrelevant based on the parseText's relevancy (using a training
file where you can give positive and negative example texts see the
description of parsefilter.naivebayes.trainfile) and if found irrelevent
it gives the link a second chance if it contains any of the words from the
list given in parsefilter.naivebayes.wordlist.
RegexParseFilter.
Classes related to the
Protocol interface,
see also org.apache.nutch.net.protocols.Protocol plugin which supports retrieving local file resources.
Protocol plugin which supports retrieving documents via the ftp protocol.
Protocol plugin which supports retrieving documents via HTTP/HTTPS using
Selenium and the
HtmlUnitDriver web
driver for the for the
HtmlUnit headless browser.
Protocol plugin which supports retrieving documents via the http protocol.
Common API used by HTTP plugins (
http,
httpclient, etc.)Protocol plugin which supports retrieving documents via the HTTP andHTTPS
protocols, optionally with Basic, Digest and NTLM authentication schemes for
web server as well as proxy server.
Protocol plugin which supports retrieving documents using and interacting
with Selenium.
Protocol plugin for HTTP/HTTPS based on
okhttp, supports HTTP 1.1
and/or http/2.
Protocol plugin which supports retrieving documents via
Selenium.
Publisher package to implement queues
The
ScoringFilter interface.Scoring filter to stop crawling at a configurable depth
(number of "hops" from seed URLs).
Scoring filter used in conjunction with
WebGraph.Metadata Scoring Plugin
Scoring filter implementing a variant of the Online Page Importance Computation
(OPIC) algorithm.
Scoring filter to modify score or status of orphaned pages (no inlinks found
for a configurable amount of time).
URL Meta Tag Scoring Plugin
Generic
URL filter library,
abstracting away from regular expression implementations.URL filter plugin based on
dk.brics.automaton Finite-State
Automata for JavaTM.
URL filter plugin to include only URLs which match an element in a given list of
domain suffixes, domain names, and/or host names.
URL filter plugin to exclude URLs by domain suffixes, domain names, and/or host names.
URL filter plugin that first does fast exact suffix matches on host/domain
names before applying regular expressions to the path component of a URL.
URL filter plugin which identifies exemptions to external urls when
when external urls are set to ignore.
URL filter plugin to include only URLs which match one of a given list of URL prefixes.
URL filter plugin to include and/or exclude URLs matching Java regular expressions.
URL filter plugin to either exclude or include only URLs which match
one of the given (path) suffixes.
URL filter plugin that validates given urls.
Sample plugins that parse and index Creative Commons metadata.
-
Uses of Pluggable in org.apache.nutch.analysis.lang
Classes in org.apache.nutch.analysis.lang that implement PluggableModifier and TypeClassDescriptionclassclassAnIndexingFilterthat add alang(language) field to the document. -
Uses of Pluggable in org.apache.nutch.collection
Classes in org.apache.nutch.collection that implement PluggableModifier and TypeClassDescriptionclassSubCollection represents a subset of index, you can define url patterns that will indicate that particular page (url) is part of SubCollection. -
Uses of Pluggable in org.apache.nutch.exchange
Subinterfaces of Pluggable in org.apache.nutch.exchange -
Uses of Pluggable in org.apache.nutch.exchange.jexl
Classes in org.apache.nutch.exchange.jexl that implement Pluggable -
Uses of Pluggable in org.apache.nutch.indexer
Subinterfaces of Pluggable in org.apache.nutch.indexerModifier and TypeInterfaceDescriptioninterfaceExtension point for indexing.interface -
Uses of Pluggable in org.apache.nutch.indexer.anchor
Classes in org.apache.nutch.indexer.anchor that implement PluggableModifier and TypeClassDescriptionclassIndexing filter that offers an option to either index all inbound anchor text for a document or deduplicate anchors. -
Uses of Pluggable in org.apache.nutch.indexer.arbitrary
Classes in org.apache.nutch.indexer.arbitrary that implement PluggableModifier and TypeClassDescriptionclassAdds arbitrary searchable fields to a document from the class and method the user identifies in the config. -
Uses of Pluggable in org.apache.nutch.indexer.basic
Classes in org.apache.nutch.indexer.basic that implement PluggableModifier and TypeClassDescriptionclassAdds basic searchable fields to a document. -
Uses of Pluggable in org.apache.nutch.indexer.feed
Classes in org.apache.nutch.indexer.feed that implement Pluggable -
Uses of Pluggable in org.apache.nutch.indexer.filter
Classes in org.apache.nutch.indexer.filter that implement PluggableModifier and TypeClassDescriptionclassAnIndexingFilterthat allows filtering of documents based on the MIME Type detected by Tika -
Uses of Pluggable in org.apache.nutch.indexer.geoip
Classes in org.apache.nutch.indexer.geoip that implement PluggableModifier and TypeClassDescriptionclassThis plugin implements an indexing filter which takes advantage of the GeoIP2-java API. -
Uses of Pluggable in org.apache.nutch.indexer.jexl
Classes in org.apache.nutch.indexer.jexl that implement PluggableModifier and TypeClassDescriptionclassAnIndexingFilterthat allows filtering of documents based on a JEXL expression. -
Uses of Pluggable in org.apache.nutch.indexer.links
Classes in org.apache.nutch.indexer.links that implement PluggableModifier and TypeClassDescriptionclass -
Uses of Pluggable in org.apache.nutch.indexer.metadata
Classes in org.apache.nutch.indexer.metadata that implement PluggableModifier and TypeClassDescriptionclassIndexer which can be configured to extract metadata from the crawldb, parse metadata or content metadata. -
Uses of Pluggable in org.apache.nutch.indexer.more
Classes in org.apache.nutch.indexer.more that implement PluggableModifier and TypeClassDescriptionclassAdd (or reset) a few metaData properties as respective fields (if they are available), so that they can be accurately used within the search index. -
Uses of Pluggable in org.apache.nutch.indexer.replace
Classes in org.apache.nutch.indexer.replace that implement PluggableModifier and TypeClassDescriptionclassDo pattern replacements on selected field contents prior to indexing. -
Uses of Pluggable in org.apache.nutch.indexer.staticfield
Classes in org.apache.nutch.indexer.staticfield that implement PluggableModifier and TypeClassDescriptionclassA simple plugin called at indexing that adds fields with static data. -
Uses of Pluggable in org.apache.nutch.indexer.subcollection
Classes in org.apache.nutch.indexer.subcollection that implement Pluggable -
Uses of Pluggable in org.apache.nutch.indexer.tld
Classes in org.apache.nutch.indexer.tld that implement Pluggable -
Uses of Pluggable in org.apache.nutch.indexer.urlmeta
Classes in org.apache.nutch.indexer.urlmeta that implement PluggableModifier and TypeClassDescriptionclassThis is part of the URL Meta plugin. -
Uses of Pluggable in org.apache.nutch.indexwriter.cloudsearch
Classes in org.apache.nutch.indexwriter.cloudsearch that implement Pluggable -
Uses of Pluggable in org.apache.nutch.indexwriter.csv
Classes in org.apache.nutch.indexwriter.csv that implement PluggableModifier and TypeClassDescriptionclassWrite Nutch documents to a CSV file (comma separated values), i.e., dump index as CSV or tab-separated plain text table. -
Uses of Pluggable in org.apache.nutch.indexwriter.dummy
Classes in org.apache.nutch.indexwriter.dummy that implement Pluggable -
Uses of Pluggable in org.apache.nutch.indexwriter.elastic
Classes in org.apache.nutch.indexwriter.elastic that implement PluggableModifier and TypeClassDescriptionclassSends NutchDocuments to a configured Elasticsearch index. -
Uses of Pluggable in org.apache.nutch.indexwriter.kafka
Classes in org.apache.nutch.indexwriter.kafka that implement PluggableModifier and TypeClassDescriptionclassSends Nutch documents to a configured Kafka Cluster -
Uses of Pluggable in org.apache.nutch.indexwriter.opensearch1x
Classes in org.apache.nutch.indexwriter.opensearch1x that implement PluggableModifier and TypeClassDescriptionclassSends NutchDocuments to a configured OpenSearch index. -
Uses of Pluggable in org.apache.nutch.indexwriter.rabbit
Classes in org.apache.nutch.indexwriter.rabbit that implement Pluggable -
Uses of Pluggable in org.apache.nutch.indexwriter.solr
Classes in org.apache.nutch.indexwriter.solr that implement Pluggable -
Uses of Pluggable in org.apache.nutch.microformats.reltag
Classes in org.apache.nutch.microformats.reltag that implement PluggableModifier and TypeClassDescriptionclassAnIndexingFilterthat addtagfield(s) to the document.classAdds microformat rel-tags of document if found. -
Uses of Pluggable in org.apache.nutch.net
Subinterfaces of Pluggable in org.apache.nutch.netModifier and TypeInterfaceDescriptioninterfaceInterface used to allow exemptions to external domain resources by overridingdb.ignore.external.links.interfaceInterface used to limit which URLs enter Nutch. -
Uses of Pluggable in org.apache.nutch.parse
Subinterfaces of Pluggable in org.apache.nutch.parseModifier and TypeInterfaceDescriptioninterfaceExtension point for DOM-based HTML parsers.interfaceA parser for content generated by aProtocolimplementation. -
Uses of Pluggable in org.apache.nutch.parse.ext
Classes in org.apache.nutch.parse.ext that implement PluggableModifier and TypeClassDescriptionclassA wrapper that invokes external command to do real parsing job. -
Uses of Pluggable in org.apache.nutch.parse.feed
Classes in org.apache.nutch.parse.feed that implement PluggableModifier and TypeClassDescriptionclassA RSS/ATOM FeedParserthat rapidly parses all referenced links and content present in the feed. -
Uses of Pluggable in org.apache.nutch.parse.headings
Classes in org.apache.nutch.parse.headings that implement PluggableModifier and TypeClassDescriptionclassHtmlParseFilter to retrieve h1 and h2 values from the DOM. -
Uses of Pluggable in org.apache.nutch.parse.html
Classes in org.apache.nutch.parse.html that implement Pluggable -
Uses of Pluggable in org.apache.nutch.parse.js
Classes in org.apache.nutch.parse.js that implement PluggableModifier and TypeClassDescriptionclassThis class is a heuristic link extractor for JavaScript files and code snippets. -
Uses of Pluggable in org.apache.nutch.parse.metatags
Classes in org.apache.nutch.parse.metatags that implement PluggableModifier and TypeClassDescriptionclassParse HTML meta tags (keywords, description) and store them in the parse metadata so that they can be indexed with the index-metadata plugin with the prefix 'metatag.'. -
Uses of Pluggable in org.apache.nutch.parse.tika
Classes in org.apache.nutch.parse.tika that implement Pluggable -
Uses of Pluggable in org.apache.nutch.parse.zip
Classes in org.apache.nutch.parse.zip that implement PluggableModifier and TypeClassDescriptionclassZipParser class based on MSPowerPointParser class by Stephan Strittmatter. -
Uses of Pluggable in org.apache.nutch.parsefilter.debug
Classes in org.apache.nutch.parsefilter.debug that implement PluggableModifier and TypeClassDescriptionclassAdds serialized DOM to parse data, useful for debugging, to understand how the parser implementation interprets a document (not only HTML). -
Uses of Pluggable in org.apache.nutch.parsefilter.naivebayes
Classes in org.apache.nutch.parsefilter.naivebayes that implement PluggableModifier and TypeClassDescriptionclassHtml Parse filter that classifies the outlinks from the parseresult as relevant or irrelevant based on the parseText's relevancy (using a training file where you can give positive and negative example texts see the description of parsefilter.naivebayes.trainfile) and if found irrelevant it gives the link a second chance if it contains any of the words from the list given in parsefilter.naivebayes.wordlist. -
Uses of Pluggable in org.apache.nutch.parsefilter.regex
Classes in org.apache.nutch.parsefilter.regex that implement Pluggable -
Uses of Pluggable in org.apache.nutch.protocol
Subinterfaces of Pluggable in org.apache.nutch.protocol -
Uses of Pluggable in org.apache.nutch.protocol.file
Classes in org.apache.nutch.protocol.file that implement PluggableModifier and TypeClassDescriptionclassThis class is a protocol plugin used for file: scheme. -
Uses of Pluggable in org.apache.nutch.protocol.ftp
Classes in org.apache.nutch.protocol.ftp that implement Pluggable -
Uses of Pluggable in org.apache.nutch.protocol.htmlunit
Classes in org.apache.nutch.protocol.htmlunit that implement Pluggable -
Uses of Pluggable in org.apache.nutch.protocol.http
Classes in org.apache.nutch.protocol.http that implement Pluggable -
Uses of Pluggable in org.apache.nutch.protocol.http.api
Classes in org.apache.nutch.protocol.http.api that implement Pluggable -
Uses of Pluggable in org.apache.nutch.protocol.httpclient
Classes in org.apache.nutch.protocol.httpclient that implement PluggableModifier and TypeClassDescriptionclassThis class is a protocol plugin that configures an HTTP client for Basic, Digest and NTLM authentication schemes for web server as well as proxy server. -
Uses of Pluggable in org.apache.nutch.protocol.interactiveselenium
Classes in org.apache.nutch.protocol.interactiveselenium that implement Pluggable -
Uses of Pluggable in org.apache.nutch.protocol.okhttp
Classes in org.apache.nutch.protocol.okhttp that implement Pluggable -
Uses of Pluggable in org.apache.nutch.protocol.selenium
Classes in org.apache.nutch.protocol.selenium that implement Pluggable -
Uses of Pluggable in org.apache.nutch.publisher
Subinterfaces of Pluggable in org.apache.nutch.publisherModifier and TypeInterfaceDescriptioninterfaceAll publisher subscriber model implementations should implement this interface.Classes in org.apache.nutch.publisher that implement Pluggable -
Uses of Pluggable in org.apache.nutch.publisher.rabbitmq
Classes in org.apache.nutch.publisher.rabbitmq that implement Pluggable -
Uses of Pluggable in org.apache.nutch.scoring
Subinterfaces of Pluggable in org.apache.nutch.scoringModifier and TypeInterfaceDescriptioninterfaceA contract defining behavior of scoring plugins.Classes in org.apache.nutch.scoring that implement PluggableModifier and TypeClassDescriptionclassclassCreates and cachesScoringFilterimplementing plugins. -
Uses of Pluggable in org.apache.nutch.scoring.depth
Classes in org.apache.nutch.scoring.depth that implement PluggableModifier and TypeClassDescriptionclassThis scoring filter limits the number of hops from the initial seed urls. -
Uses of Pluggable in org.apache.nutch.scoring.link
Classes in org.apache.nutch.scoring.link that implement Pluggable -
Uses of Pluggable in org.apache.nutch.scoring.metadata
Classes in org.apache.nutch.scoring.metadata that implement Pluggable -
Uses of Pluggable in org.apache.nutch.scoring.opic
Classes in org.apache.nutch.scoring.opic that implement PluggableModifier and TypeClassDescriptionclassThis plugin implements a variant of an Online Page Importance Computation (OPIC) score, described in this paper: Abiteboul, Serge and Preda, Mihai and Cobena, Gregory (2003), Adaptive On-Line Page Importance Computation. -
Uses of Pluggable in org.apache.nutch.scoring.orphan
Classes in org.apache.nutch.scoring.orphan that implement PluggableModifier and TypeClassDescriptionclassOrphan scoring filter that determines whether a page has become orphaned, e.g. -
Uses of Pluggable in org.apache.nutch.scoring.similarity
Classes in org.apache.nutch.scoring.similarity that implement Pluggable -
Uses of Pluggable in org.apache.nutch.scoring.urlmeta
Classes in org.apache.nutch.scoring.urlmeta that implement Pluggable -
Uses of Pluggable in org.apache.nutch.urlfilter.api
Classes in org.apache.nutch.urlfilter.api that implement Pluggable -
Uses of Pluggable in org.apache.nutch.urlfilter.automaton
Classes in org.apache.nutch.urlfilter.automaton that implement PluggableModifier and TypeClassDescriptionclassRegexURLFilterBase implementation based on the dk.brics.automaton Finite-State Automata for JavaTM. -
Uses of Pluggable in org.apache.nutch.urlfilter.domain
Classes in org.apache.nutch.urlfilter.domain that implement PluggableModifier and TypeClassDescriptionclassFilters URLs based on a file containing domain suffixes, domain names, and hostnames. -
Uses of Pluggable in org.apache.nutch.urlfilter.domaindenylist
Classes in org.apache.nutch.urlfilter.domaindenylist that implement PluggableModifier and TypeClassDescriptionclassFilters URLs based on a file containing domain suffixes, domain names, and hostnames. -
Uses of Pluggable in org.apache.nutch.urlfilter.fast
Classes in org.apache.nutch.urlfilter.fast that implement PluggableModifier and TypeClassDescriptionclassFilters URLs based on a file of regular expressions using host/domains matching first. -
Uses of Pluggable in org.apache.nutch.urlfilter.ignoreexempt
Classes in org.apache.nutch.urlfilter.ignoreexempt that implement PluggableModifier and TypeClassDescriptionclassThis implementation ofURLExemptionFilteruses regex configuration to check if URL is eligible for exemption from thedb.ignore.external.linksconfiguration property. -
Uses of Pluggable in org.apache.nutch.urlfilter.prefix
Classes in org.apache.nutch.urlfilter.prefix that implement PluggableModifier and TypeClassDescriptionclassFilters URLs based on a file of URL prefixes. -
Uses of Pluggable in org.apache.nutch.urlfilter.regex
Classes in org.apache.nutch.urlfilter.regex that implement PluggableModifier and TypeClassDescriptionclassFilters URLs based on a file of regular expressions using theJava Regex implementation. -
Uses of Pluggable in org.apache.nutch.urlfilter.suffix
Classes in org.apache.nutch.urlfilter.suffix that implement PluggableModifier and TypeClassDescriptionclassFilters URLs based on a file of URL suffixes. -
Uses of Pluggable in org.apache.nutch.urlfilter.validator
Classes in org.apache.nutch.urlfilter.validator that implement Pluggable -
Uses of Pluggable in org.creativecommons.nutch
Classes in org.creativecommons.nutch that implement PluggableModifier and TypeClassDescriptionclassAdds basic searchable fields to a document.classAdds metadata identifying the Creative Commons license used, if any.