Package org.apache.nutch.crawl

Class DeduplicationJob.DedupReducer<K extends Writable>

java.lang.Object

org.apache.hadoop.mapreduce.Reducer<K,CrawlDatum,Text,CrawlDatum>

org.apache.nutch.crawl.DeduplicationJob.DedupReducer<K>

Enclosing class:

DeduplicationJob

public static class DeduplicationJob.DedupReducer<K extends Writable>
extends Reducer<K,CrawlDatum,Text,CrawlDatum>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Reducer

Reducer.Context

Field Summary

Fields

Modifier and Type	Field	Description
protected String[]	compareOrder

Constructor Summary

Constructors

Constructor	Description
DedupReducer()

Method Summary

Modifier and Type	Method	Description
protected CrawlDatum	getDuplicate(CrawlDatum existingDoc, CrawlDatum newDoc)
void	reduce(K key, Iterable<CrawlDatum> values, Reducer<K,CrawlDatum,Text,CrawlDatum>.Context context)
void	setup(Reducer<K,CrawlDatum,Text,CrawlDatum>.Context context)
protected void	writeOutAsDuplicate(CrawlDatum datum, Reducer<K,CrawlDatum,Text,CrawlDatum>.Context context)

Methods inherited from class org.apache.hadoop.mapreduce.Reducer

cleanup, run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

compareOrder

protected String[] compareOrder

Constructor Details

DedupReducer

public DedupReducer()

Method Details

setup

public void setup(Reducer<K,CrawlDatum,Text,CrawlDatum>.Context context)

Overrides:

setup in class Reducer<K extends Writable,CrawlDatum,Text,CrawlDatum>

writeOutAsDuplicate

protected void writeOutAsDuplicate(CrawlDatum datum,
 Reducer<K,CrawlDatum,Text,CrawlDatum>.Context context)
                            throws IOException,
InterruptedException

Throws:

IOException

InterruptedException

reduce

public void reduce(K key,
 Iterable<CrawlDatum> values,
 Reducer<K,CrawlDatum,Text,CrawlDatum>.Context context)
            throws IOException,
InterruptedException

Overrides:

reduce in class Reducer<K extends Writable,CrawlDatum,Text,CrawlDatum>

Throws:

IOException

InterruptedException

getDuplicate

protected CrawlDatum getDuplicate(CrawlDatum existingDoc,
 CrawlDatum newDoc)