# Uses of Package
opennlp.tools.tokenize

- Packages that use opennlp.tools.tokenize
  Package | Description
  --- | ---
  opennlp.tools.cmdline.parser |
  opennlp.tools.cmdline.tokenizer |
  opennlp.tools.formats | Experimental package related to converting various corpora to OpenNLP Format.
  opennlp.tools.formats.ad |
  opennlp.tools.formats.brat |
  opennlp.tools.formats.conllu |
  opennlp.tools.formats.convert |
  opennlp.tools.formats.irishsentencebank |
  opennlp.tools.formats.letsmt |
  opennlp.tools.formats.muc |
  opennlp.tools.sentdetect | Package related to identifying sentece boundries.
  opennlp.tools.tokenize | Contains classes related to finding token or words in a string.
  opennlp.tools.tokenize.lang |
  opennlp.tools.util.featuregen | This package contains classes for generating sequence features.

- Classes in opennlp.tools.tokenize used by opennlp.tools.cmdline.parser
  Class and Description
  Tokenizer The interface for tokenizers, which segment a string into its tokens.

- Classes in opennlp.tools.tokenize used by opennlp.tools.cmdline.tokenizer
  Class and Description
  TokenizerEvaluationMonitor
  TokenizerModel The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  TokenSample A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats
  Class and Description
  Detokenizer A Detokenizer merges tokens back to their untokenized representation.
  TokenSample A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.ad
  Class and Description
  TokenSample A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.brat
  Class and Description
  Tokenizer The interface for tokenizers, which segment a string into its tokens.
  TokenizerModel The `TokenizerModel` is the model used by a learnable `Tokenizer`.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.conllu
  Class and Description
  TokenSample A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.convert
  Class and Description
  Detokenizer A Detokenizer merges tokens back to their untokenized representation.
  TokenSample A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.irishsentencebank
  Class and Description
  TokenSample A `TokenSample` is text with token spans.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.letsmt
  Class and Description
  Detokenizer A Detokenizer merges tokens back to their untokenized representation.

- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.muc
  Class and Description
  Tokenizer The interface for tokenizers, which segment a string into its tokens.

- Classes in opennlp.tools.tokenize used by opennlp.tools.sentdetect
  Class and Description
  Detokenizer A Detokenizer merges tokens back to their untokenized representation.

- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize
  Class and Description
  DetokenizationDictionary
  DetokenizationDictionary.Operation
  Detokenizer A Detokenizer merges tokens back to their untokenized representation.
  Detokenizer.DetokenizationOperation This enum contains an operation for every token to merge the tokens together to their detokenized form.
  SimpleTokenizer Performs tokenization using character classes.
  TokenContextGenerator Interface for `TokenizerME` context generators.
  Tokenizer The interface for tokenizers, which segment a string into its tokens.
  TokenizerEvaluationMonitor
  TokenizerFactory The factory that provides `Tokenizer` default implementations and resources.
  TokenizerModel The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  TokenSample A `TokenSample` is text with token spans.
  WhitespaceTokenizer This tokenizer uses white spaces to tokenize the input text.

- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize.lang
  Class and Description
  TokenContextGenerator Interface for `TokenizerME` context generators.

- Classes in opennlp.tools.tokenize used by opennlp.tools.util.featuregen
  Class and Description
  Tokenizer The interface for tokenizers, which segment a string into its tokens.

Copyright © 2026 The Apache Software Foundation. All rights reserved.