opennlp.tools.tokenize

## Class TokenizerFactory

- java.lang.Object
- - opennlp.tools.util.BaseToolFactory
  - - opennlp.tools.tokenize.TokenizerFactory

-     public class TokenizerFactory
      extends BaseToolFactory

  The factory that provides `Tokenizer` default implementations and resources. Users can extend this class if their application requires overriding the `TokenContextGenerator`, `Dictionary` etc.

- - ### Constructor Summary

    Constructors
    Constructor and Description
    `TokenizerFactory()` Creates a `TokenizerFactory` that provides the default implementation of the resources.
    `TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern)` Creates a `TokenizerFactory`.

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `static TokenizerFactory` | `create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern)` Factory method the framework uses create a new `TokenizerFactory`.
    `Map<String,Object>` | `createArtifactMap()` Creates a `Map` with pairs of keys and objects.
    `Map<String,String>` | `createManifestEntries()` Creates the manifest entries that will be added to the model manifest
    `Dictionary` | `getAbbreviationDictionary()` Gets the abbreviation dictionary
    `Pattern` | `getAlphaNumericPattern()` Gets the alpha numeric pattern.
    `TokenContextGenerator` | `getContextGenerator()` Gets the context generator
    `String` | `getLanguageCode()` Retrieves the language code.
    `boolean` | `isUseAlphaNumericOptmization()` Gets whether to use alphanumeric optimization.
    `void` | `validateArtifactMap()` Validates the parsed artifacts.

    - ### Methods inherited from class opennlp.tools.util.BaseToolFactory

      `create, create, createArtifactSerializersMap`

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### TokenizerFactory

          public TokenizerFactory()

      Creates a `TokenizerFactory` that provides the default implementation of the resources.

    - #### TokenizerFactory

          public TokenizerFactory(String languageCode,
                                  Dictionary abbreviationDictionary,
                                  boolean useAlphaNumericOptimization,
                                  Pattern alphaNumericPattern)

      Creates a `TokenizerFactory`. Use this constructor to programmatically create a factory.
      Parameters:
        `languageCode` - the language of the natural text
        `abbreviationDictionary` - an abbreviations dictionary
        `useAlphaNumericOptimization` - if true alpha numerics are skipped
        `alphaNumericPattern` - null or a custom alphanumeric pattern (default is: "^[A-Za-z0-9]+$", provided by `Factory.DEFAULT_ALPHANUMERIC`

  - ### Method Detail

    - #### validateArtifactMap

          public void validateArtifactMap()
                                   throws InvalidFormatException

      Description copied from class: `BaseToolFactory`
      Validates the parsed artifacts. If something is not valid subclasses should throw an `InvalidFormatException`. Note: Subclasses should generally invoke super.validateArtifactMap at the beginning of this method.
      Specified by:
        `validateArtifactMap` in class `BaseToolFactory`
      Throws:
        `InvalidFormatException`

    - #### createArtifactMap

          public Map<String,Object> createArtifactMap()

      Description copied from class: `BaseToolFactory`
      Creates a `Map` with pairs of keys and objects. The models implementation should call this constructor that creates a model programmatically.

      The base implementation will return a `HashMap` that should be populated by sub-classes.

      Overrides:
        `createArtifactMap` in class `BaseToolFactory`

    - #### createManifestEntries

          public Map<String,String> createManifestEntries()

      Description copied from class: `BaseToolFactory`
      Creates the manifest entries that will be added to the model manifest
      Overrides:
        `createManifestEntries` in class `BaseToolFactory`
      Returns:
        the manifest entries to added to the model manifest

    - #### create

          public static TokenizerFactory create(String subclassName,
                                                String languageCode,
                                                Dictionary abbreviationDictionary,
                                                boolean useAlphaNumericOptimization,
                                                Pattern alphaNumericPattern)
                                         throws InvalidFormatException

      Factory method the framework uses create a new `TokenizerFactory`.
      Parameters:
        `subclassName` - the name of the class implementing the `TokenizerFactory`
        `languageCode` - the language code the tokenizer should use
        `abbreviationDictionary` - an optional dictionary containing abbreviations, or null if not present
        `useAlphaNumericOptimization` - indicate if the alpha numeric optimization should be enabled or disabled
        `alphaNumericPattern` - the pattern the alpha numeric optimization should use
      Returns:
        the instance of the Tokenizer Factory
      Throws:
        `InvalidFormatException` - if once of the input parameters doesn't comply if the expected format

    - #### getAlphaNumericPattern

          public Pattern getAlphaNumericPattern()

      Gets the alpha numeric pattern.
      Returns:
        the user specified alpha numeric pattern or a default.

    - #### isUseAlphaNumericOptmization

          public boolean isUseAlphaNumericOptmization()

      Gets whether to use alphanumeric optimization.
      Returns:
        true if the alpha numeric optimization is enabled, otherwise false

    - #### getAbbreviationDictionary

          public Dictionary getAbbreviationDictionary()

      Gets the abbreviation dictionary
      Returns:
        null or the abbreviation dictionary

    - #### getLanguageCode

          public String getLanguageCode()

      Retrieves the language code.
      Returns:
        the language code

    - #### getContextGenerator

          public TokenContextGenerator getContextGenerator()

      Gets the context generator
      Returns:
        a new instance of the context generator

Copyright © 2026 The Apache Software Foundation. All rights reserved.