opennlp.tools.tokenize

## Class DictionaryDetokenizer

- java.lang.Object
- - opennlp.tools.tokenize.DictionaryDetokenizer

- All Implemented Interfaces:
    Detokenizer

      public class DictionaryDetokenizer
      extends Object
      implements Detokenizer

  A rule based detokenizer. Simple rules which indicate in which direction a token should be moved are looked up in a `DetokenizationDictionary` object.
  See Also:
    `Detokenizer`, `DetokenizationDictionary`

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from interface opennlp.tools.tokenize.Detokenizer

      `Detokenizer.DetokenizationOperation`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `DictionaryDetokenizer(DetokenizationDictionary dict)`

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `Detokenizer.DetokenizationOperation[]` | `detokenize(String[] tokens)` Detokenize the input tokens.
    `String` | `detokenize(String[] tokens, String splitMarker)` Detokenize the input tokens into a String.

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### DictionaryDetokenizer

          public DictionaryDetokenizer(DetokenizationDictionary dict)

  - ### Method Detail

    - #### detokenize

          public Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)

      Description copied from interface: `Detokenizer`
      Detokenize the input tokens.
      Specified by:
        `detokenize` in interface `Detokenizer`
      Parameters:
        `tokens` - the tokens to detokenize.
      Returns:
        the merge operations to detokenize the input tokens.

    - #### detokenize

          public String detokenize(String[] tokens,
                                   String splitMarker)

      Description copied from interface: `Detokenizer`
      Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.
      Specified by:
        `detokenize` in interface `Detokenizer`
      Parameters:
        `tokens` - the token which should be concatenated
        `splitMarker` - the split marker or null
      Returns:
        the concatenated tokens

Copyright © 2026 The Apache Software Foundation. All rights reserved.