## Uses of Class
opennlp.tools.tokenize.TokenSample

- Packages that use TokenSample
  Package | Description
  --- | ---
  opennlp.tools.cmdline.tokenizer |
  opennlp.tools.formats | Experimental package related to converting various corpora to OpenNLP Format.
  opennlp.tools.formats.ad |
  opennlp.tools.formats.conllu |
  opennlp.tools.formats.convert |
  opennlp.tools.formats.irishsentencebank |
  opennlp.tools.tokenize | Contains classes related to finding token or words in a string.
  opennlp.tools.tokenize.lang.en |

- - ### Uses of TokenSample in opennlp.tools.cmdline.tokenizer

    Methods in opennlp.tools.cmdline.tokenizer with parameters of type TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `void` | DetokenEvaluationErrorListener.`missclassified(TokenSample reference, TokenSample prediction)`
    `void` | TokenEvaluationErrorListener.`missclassified(TokenSample reference, TokenSample prediction)`

  - ### Uses of TokenSample in opennlp.tools.formats

    Methods in opennlp.tools.formats that return types with arguments of type TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `ObjectStream<TokenSample>` | TokenSampleStreamFactory.`create(String[] args)`
    `ObjectStream<TokenSample>` | ConllXTokenSampleStreamFactory.`create(String[] args)`

  - ### Uses of TokenSample in opennlp.tools.formats.ad

    Methods in opennlp.tools.formats.ad that return types with arguments of type TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `ObjectStream<TokenSample>` | ADTokenSampleStreamFactory.`create(String[] args)`

  - ### Uses of TokenSample in opennlp.tools.formats.conllu

    Methods in opennlp.tools.formats.conllu that return TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `TokenSample` | ConlluTokenSampleStream.`read()`

    Methods in opennlp.tools.formats.conllu that return types with arguments of type TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `ObjectStream<TokenSample>` | ConlluTokenSampleStreamFactory.`create(String[] args)`

  - ### Uses of TokenSample in opennlp.tools.formats.convert

    Methods in opennlp.tools.formats.convert that return TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `TokenSample` | POSToTokenSampleStream.`read()`
    `TokenSample` | NameToTokenSampleStream.`read()`

    Methods in opennlp.tools.formats.convert that return types with arguments of type TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `ObjectStream<TokenSample>` | ParseToTokenSampleStreamFactory.`create(String[] args)`
    `ObjectStream<TokenSample>` | POSToTokenSampleStreamFactory.`create(String[] args)`
    `ObjectStream<TokenSample>` | NameToTokenSampleStreamFactory.`create(String[] args)`

  - ### Uses of TokenSample in opennlp.tools.formats.irishsentencebank

    Methods in opennlp.tools.formats.irishsentencebank that return TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `TokenSample` | IrishSentenceBankDocument.IrishSentenceBankSentence.`getTokenSample()`

    Methods in opennlp.tools.formats.irishsentencebank that return types with arguments of type TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `ObjectStream<TokenSample>` | IrishSentenceBankTokenSampleStreamFactory.`create(String[] args)`

  - ### Uses of TokenSample in opennlp.tools.tokenize

    Methods in opennlp.tools.tokenize that return TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `static TokenSample` | TokenSample.`parse(String sampleString, String separatorChars)`
    `protected TokenSample` | DetokenizerEvaluator.`processSample(TokenSample reference)`
    `protected TokenSample` | TokenizerEvaluator.`processSample(TokenSample reference)`
    `TokenSample` | TokenizerStream.`read()`
    `TokenSample` | TokenSampleStream.`read()`

    Methods in opennlp.tools.tokenize with parameters of type TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `protected Iterator<Event>` | TokSpanEventStream.`createEvents(TokenSample tokenSample)` Adds training events to the event stream for each of the specified tokens.
    `protected TokenSample` | DetokenizerEvaluator.`processSample(TokenSample reference)`
    `protected TokenSample` | TokenizerEvaluator.`processSample(TokenSample reference)`

    Method parameters in opennlp.tools.tokenize with type arguments of type TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `void` | TokenizerCrossValidator.`evaluate(ObjectStream<TokenSample> samples, int nFolds)` Starts the evaluation.
    `static TokenizerModel` | TokenizerME.`train(ObjectStream<TokenSample> samples, TokenizerFactory factory, TrainingParameters mlParams)` Trains a model for the `TokenizerME`.

    Constructor parameters in opennlp.tools.tokenize with type arguments of type TokenSample
    Constructor and Description
    `TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics)` Initializes the current instance.
    `TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, Pattern alphaNumeric, TokenContextGenerator cg)` Initializes the current instance.
    `TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenContextGenerator cg)` Initializes the current instance.
    `WhitespaceTokenStream(ObjectStream<TokenSample> tokens)`

  - ### Uses of TokenSample in opennlp.tools.tokenize.lang.en

    Methods in opennlp.tools.tokenize.lang.en that return TokenSample
    Modifier and Type | Method and Description
    --- | ---
    `TokenSample` | TokenSampleStream.`next()`

Copyright © 2026 The Apache Software Foundation. All rights reserved.