Package se.l4.lect.tokens

Interface Summary

Interface	Description
Token	Token containing some text that is either words, symbols or whitespace.
Tokenizer	Tokenizer used to tokenize input into Tokens.
TokenizerFactory	Factory for instances of Tokenizer working on CharSequences.
TokenMatcher	Matcher created from a TokenPattern that will match the sequence the pattern describes.
TokenPattern	Pattern for matching a sequence of tokens.

Class Summary

Class	Description
ImmutableToken	Implementation of Token that is immutable.
MutableToken	Implementation of Token that is mutable.
OffsetTokenizer	Tokenizer that tokenizes using OffsetLocations.
TextTokenization	Utilities to help with some text tokenization tasks.
TokenProperty<T>	Extended language data for a Token that a LanguageParser or Handler may enhance a token with.
Tokens	Utilities for working with Tokens.

Enum Summary

Enum	Description
TokenType	Enumeration for the different types of tokens extracted from text.