| Interface | Description |
|---|---|
| Token |
Token containing some text that is either words, symbols or whitespace.
|
| Tokenizer |
Tokenizer used to tokenize input into
Tokens. |
| TokenizerFactory |
Factory for instances of
Tokenizer working on CharSequences. |
| TokenMatcher |
Matcher created from a
TokenPattern that will match the sequence the
pattern describes. |
| TokenPattern |
Pattern for matching a sequence of tokens.
|
| Class | Description |
|---|---|
| ImmutableToken |
Implementation of
Token that is immutable. |
| MutableToken |
Implementation of
Token that is mutable. |
| OffsetTokenizer |
Tokenizer that tokenizes using
OffsetLocations. |
| TextTokenization |
Utilities to help with some text tokenization tasks.
|
| TokenProperty<T> | |
| Tokens |
Utilities for working with
Tokens. |
| Enum | Description |
|---|---|
| TokenType |
Enumeration for the different types of tokens extracted from text.
|
Copyright © 2018. All rights reserved.