Package org.apache.nutch.segment

Class SegmentChecker

java.lang.Object
org.apache.nutch.segment.SegmentChecker
public class SegmentChecker extends Object
Checks whether a segment is valid, or has a certain status (generated, fetched, parsed), or can be used safely for a certain processing step (e.g., indexing).

  • Constructor Details

    • SegmentChecker

      public SegmentChecker()

  • Method Details

    • isIndexable

      public static boolean isIndexable(Path segmentPath, FileSystem fs) throws IOException
      Check if the segment is indexable. May add new check methods here.
      Parameters:
      segmentPath - path to an individual segment on disk
      fs - the FileSystem that the segment resides on
      Returns:
      true if the checks pass and the segment can be indexed, false otherwise
      Throws:
      IOException - if there is an I/O error locating or checking either the segment contents or locating it on the filesystem

    • checkSegmentDir

      public static boolean checkSegmentDir(Path segmentPath, FileSystem fs) throws IOException
      Check the segment to see if it is valid based on the sub directories.
      Parameters:
      segmentPath - path to an individual segment on disk
      fs - the FileSystem that the segment resides on
      Returns:
      true if the checks pass false otherwise
      Throws:
      IOException - if there is an I/O error locating or checking either the segment contents or locating it on the filesystem

    • isParsed

      public static boolean isParsed(Path segment, FileSystem fs) throws IOException
      Check the segment to see if it is has been parsed before.
      Parameters:
      segment - path to an individual segment on disk
      fs - the FileSystem that the segment resides on
      Returns:
      true if the checks pass and the segment has been parsed, false otherwise
      Throws:
      IOException - if there is an I/O error locating or checking either the segment contents or locating it on the filesystem