opennlp.tools.tokenize

## Interface Detokenizer

- All Known Implementing Classes:
    DictionaryDetokenizer

      public interface Detokenizer

  A Detokenizer merges tokens back to their untokenized representation.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Interface and Description
    --- | ---
    `static class` | `Detokenizer.DetokenizationOperation` This enum contains an operation for every token to merge the tokens together to their detokenized form.

  - ### Method Summary

    All Methods Instance Methods Abstract Methods
    Modifier and Type | Method and Description
    --- | ---
    `Detokenizer.DetokenizationOperation[]` | `detokenize(String[] tokens)` Detokenize the input tokens.
    `String` | `detokenize(String[] tokens, String splitMarker)` Detokenize the input tokens into a String.

- - ### Method Detail

    - #### detokenize

          Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)

      Detokenize the input tokens.
      Parameters:
        `tokens` - the tokens to detokenize.
      Returns:
        the merge operations to detokenize the input tokens.

    - #### detokenize

          String detokenize(String[] tokens,
                            String splitMarker)

      Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.
      Parameters:
        `tokens` - the token which should be concatenated
        `splitMarker` - the split marker or null
      Returns:
        the concatenated tokens

Copyright © 2026 The Apache Software Foundation. All rights reserved.