opennlp.tools.tokenize

## Class SimpleTokenizer

- java.lang.Object
- - opennlp.tools.tokenize.SimpleTokenizer

- All Implemented Interfaces:
    Tokenizer

      public class SimpleTokenizer
      extends Object

  Performs tokenization using character classes.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static SimpleTokenizer` | `INSTANCE`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `SimpleTokenizer()` Deprecated. Use INSTANCE field instead to obtain an instance, constructor will be made private in the future.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `String[]` | `tokenize(String s)` Splits a string into its atomic parts
    `Span[]` | `tokenizePos(String s)` Finds the boundaries of atomic parts in a string.

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### INSTANCE

          public static final SimpleTokenizer INSTANCE

  - ### Constructor Detail

    - #### SimpleTokenizer

          @Deprecated
          public SimpleTokenizer()

      Deprecated. Use INSTANCE field instead to obtain an instance, constructor will be made private in the future.

  - ### Method Detail

    - #### tokenizePos

          public Span[] tokenizePos(String s)

      Description copied from interface: `Tokenizer`
      Finds the boundaries of atomic parts in a string.
      Parameters:
        `s` - The string to be tokenized.
      Returns:
        The Span[] with the spans (offsets into s) for each token as the individuals array elements.

    - #### tokenize

          public String[] tokenize(String s)

      Description copied from interface: `Tokenizer`
      Splits a string into its atomic parts
      Specified by:
        `tokenize` in interface `Tokenizer`
      Parameters:
        `s` - The string to be tokenized.
      Returns:
        The String[] with the individual tokens as the array elements.

Copyright © 2026 The Apache Software Foundation. All rights reserved.