Package org.apache.nutch.protocol

Interface Protocol

All Superinterfaces:
Configurable, Pluggable
All Known Implementing Classes:
File, Ftp, Http, Http, Http, Http, Http, HttpBase, OkHttp
public interface Protocol extends Pluggable, Configurable
A retriever of url content. Implemented by protocol extensions.

  • Field Details

    • X_POINT_ID

      static final String X_POINT_ID
      The name of the extension point.

  • Method Details

    • getProtocolOutput

      ProtocolOutput getProtocolOutput(Text url, CrawlDatum datum)
      Get the ProtocolOutput for a given url and crawldatum
      Parameters:
      url - canonical url
      datum - associated CrawlDatum
      Returns:
      the ProtocolOutput

    • resolveUrl

      default URL resolveUrl(URL base, String relative) throws MalformedURLException
      Resolve a relative URL against a base URL using the protocol's URL library.
      Parameters:
      base - the base URL the relative URL is resolved against
      relative - the relative URL string (typically a Location: header value)
      Returns:
      resolved absolute URL
      Throws:
      MalformedURLException - if the URL is malformed

    • getRobotRules

      crawlercommons.robots.BaseRobotRules getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)
      Retrieve robot rules applicable for this URL.
      Parameters:
      url - URL to check
      datum - page datum
      robotsTxtContent - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response Content is appended to the passed list. If null is passed nothing is stored.
      Returns:
      robot rules (specific for this URL or default), never null

    • getRobotRules

      default crawlercommons.robots.BaseRobotRules getRobotRules(URL url, CrawlDatum datum, List<Content> robotsTxtContent)
      Retrieve robot rules applicable for this URL.
      Parameters:
      url - URL to check
      datum - page datum
      robotsTxtContent - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response Content is appended to the passed list. If null is passed nothing is stored.
      Returns:
      robot rules (specific for this URL or default), never null