opennlp.tools.tokenize

## Class TokenSample

- java.lang.Object
- - opennlp.tools.tokenize.TokenSample

- All Implemented Interfaces:
    Serializable

      public class TokenSample
      extends Object
      implements Serializable

  A `TokenSample` is text with token spans.
  See Also:
    Serialized Form

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `DEFAULT_SEPARATOR_CHARS`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `TokenSample(Detokenizer detokenizer, String[] tokens)`
    `TokenSample(String text, Span[] tokenSpans)` Initializes the current instance.

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `boolean` | `equals(Object obj)`
    `String` | `getText()` Retrieves the text.
    `Span[]` | `getTokenSpans()` Retrieves the token spans.
    `int` | `hashCode()`
    `static TokenSample` | `parse(String sampleString, String separatorChars)`
    `String` | `toString()`

    - ### Methods inherited from class java.lang.Object

      `getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### DEFAULT_SEPARATOR_CHARS

          public static final String DEFAULT_SEPARATOR_CHARS

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### TokenSample

          public TokenSample(String text,
                             Span[] tokenSpans)

      Initializes the current instance.
      Parameters:
        `text` - the text which contains the tokens.
        `tokenSpans` - the spans which mark the begin and end of the tokens.

    - #### TokenSample

          public TokenSample(Detokenizer detokenizer,
                             String[] tokens)

  - ### Method Detail

    - #### getText

          public String getText()

      Retrieves the text.

    - #### getTokenSpans

          public Span[] getTokenSpans()

      Retrieves the token spans.

    - #### toString

          public String toString()

      Overrides:
        `toString` in class `Object`

    - #### parse

          public static TokenSample parse(String sampleString,
                                          String separatorChars)

    - #### hashCode

          public int hashCode()

      Overrides:
        `hashCode` in class `Object`

    - #### equals

          public boolean equals(Object obj)

      Overrides:
        `equals` in class `Object`

Copyright © 2026 The Apache Software Foundation. All rights reserved.