Package org.apache.nutch.net.urlnormalizer.basic

Class BasicURLNormalizer

java.lang.Object
org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer
All Implemented Interfaces:
Configurable, URLNormalizer
public class BasicURLNormalizer extends Object implements URLNormalizer
Converts URLs to a normal form:
  • remove dot segments in path: /./ or /../
  • remove default ports, e.g. 80 for protocol http://
  • normalize percent-encoding in URL paths
  • normalize the host name if it is an Internationalized Domain Name (IDN) to ASCII or Unicode, depending on the configuration properties urlnormalizer.basic.host.idn and urlnormalizer.basic.host.idna2008
  • remove a trailing dot in the host name (if the property urlnormalizer.basic.host.trim-trailing-dot is true)