opennlp.tools.tokenize

## Class TokenizerModel

- java.lang.Object
- - opennlp.tools.util.model.BaseModel
  - - opennlp.tools.tokenize.TokenizerModel

- All Implemented Interfaces:
    Serializable, ArtifactProvider

      public final class TokenizerModel
      extends BaseModel

  The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  See Also:
    `TokenizerME`, Serialized Form

- - ### Field Summary

    - ### Fields inherited from class opennlp.tools.util.model.BaseModel

      `TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `TokenizerModel(File modelFile)` Initializes the current instance.
    `TokenizerModel(InputStream in)` Initializes the current instance.
    `TokenizerModel(MaxentModel tokenizerModel, Map<String,String> manifestInfoEntries, TokenizerFactory tokenizerFactory)` Initializes the current instance.
    `TokenizerModel(Path modelPath)`
    `TokenizerModel(URL modelURL)` Initializes the current instance.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `Dictionary` | `getAbbreviations()`
    `TokenizerFactory` | `getFactory()`
    `MaxentModel` | `getMaxentModel()`
    `boolean` | `useAlphaNumericOptimization()`

    - ### Methods inherited from class opennlp.tools.util.model.BaseModel

      `getArtifact, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, serialize, serialize`

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### TokenizerModel

          public TokenizerModel(MaxentModel tokenizerModel,
                                Map<String,String> manifestInfoEntries,
                                TokenizerFactory tokenizerFactory)

      Initializes the current instance.
      Parameters:
        `tokenizerModel` - the model
        `manifestInfoEntries` - the manifest
        `tokenizerFactory` - the factory

    - #### TokenizerModel

          public TokenizerModel(InputStream in)
                         throws IOException

      Initializes the current instance.
      Parameters:
        `in` - the Input Stream to load the model from
      Throws:
        `IOException` - if reading from the stream fails in anyway
        `InvalidFormatException` - if the stream doesn't have the expected format

    - #### TokenizerModel

          public TokenizerModel(File modelFile)
                         throws IOException

      Initializes the current instance.
      Parameters:
        `modelFile` - the file containing the tokenizer model
      Throws:
        `IOException` - if reading from the stream fails in anyway

    - #### TokenizerModel

          public TokenizerModel(Path modelPath)
                         throws IOException

      Throws:
        `IOException`

    - #### TokenizerModel

          public TokenizerModel(URL modelURL)
                         throws IOException

      Initializes the current instance.
      Parameters:
        `modelURL` - the URL pointing to the tokenizer model
      Throws:
        `IOException` - if reading from the stream fails in anyway

  - ### Method Detail

    - #### getFactory

          public TokenizerFactory getFactory()

    - #### getMaxentModel

          public MaxentModel getMaxentModel()

    - #### getAbbreviations

          public Dictionary getAbbreviations()

    - #### useAlphaNumericOptimization

          public boolean useAlphaNumericOptimization()

Copyright © 2026 The Apache Software Foundation. All rights reserved.