Uses of Interface
se.l4.lect.tokens.Token

Packages that use Token

Package	Description
se.l4.lect
se.l4.lect.handlers
se.l4.lect.internal.tokens
se.l4.lect.tokens

Uses of Token in se.l4.lect

Methods in se.l4.lect with parameters of type Token

Modifier and Type	Method and Description
void	Handler.token(Token token) Token has been found, can either be whitespace, symbol or word.
void	LanguageEncounter.token(Token token) A token has been encountered, either within a sentence or outside a sentence.

Uses of Token in se.l4.lect.handlers

Methods in se.l4.lect.handlers with parameters of type Token

Modifier and Type	Method and Description
protected void	DefaultHandler.special(Token token) Special token has been found.
protected void	DefaultHandler.symbol(Token token) A sequence of one or more symbols has been found in a sentence.
void	DefaultHandler.token(Token token)
void	DelegatingMultiStageHandler.token(Token token)
protected void	DefaultHandler.unknown(Token token) Unknown token has been found, uncommon case as most language implementations will not emit unknown tokens.
protected void	DefaultHandler.whitespace(Token token) Whitespace either within our outside a sentence.
protected void	DefaultHandler.word(Token token) A word has been found in a sentence.

Uses of Token in se.l4.lect.internal.tokens

Fields in se.l4.lect.internal.tokens declared as Token

Modifier and Type	Field and Description
static Token	TokenMatcherImpl.END
static Token	TokenMatcherImpl.START

Methods in se.l4.lect.internal.tokens with parameters of type Token

Modifier and Type	Method and Description
boolean	TokenMatcherImpl.add(Token token)

Constructors in se.l4.lect.internal.tokens with parameters of type Token

Constructor and Description
ExactTokenNode(Token token)

Uses of Token in se.l4.lect.tokens

Classes in se.l4.lect.tokens that implement Token

Modifier and Type	Class and Description
class	ImmutableToken Implementation of Token that is immutable.
class	MutableToken Implementation of Token that is mutable.

Methods in se.l4.lect.tokens that return Token

Modifier and Type	Method and Description
Token	Token.copy() Get a copy of this token as an ImmutableToken.
Token	MutableToken.copy()
Token	ImmutableToken.copy()
static Token	Tokens.merge(Iterable<Token> tokens) Merge the given tokens into a new token.
static Token	Tokens.merge(List<Token> tokens, int startIdx, int endIdx)
Token	Tokenizer.next() Get the next available token.
Token	OffsetTokenizer.next()
default Token[]	Tokenizer.toArray() Get these tokens as an array.

Methods in se.l4.lect.tokens that return types with arguments of type Token

Modifier and Type	Method and Description
default java.util.stream.Stream<Token>	Tokenizer.stream() Stream these tokens.
default List<Token>	Tokenizer.toList() Get the tokens as a List.

Methods in se.l4.lect.tokens with parameters of type Token

Modifier and Type	Method and Description
boolean	TokenMatcher.add(Token token) Add a token to this matcher returning whether a match has now been found.
void	MutableToken.copyFrom(Token other) Copy data from another token into this one.
default Tokenizer	TokenizerFactory.create(Token token) Create a new Tokenizer for the given input token.

Method parameters in se.l4.lect.tokens with type arguments of type Token

Modifier and Type	Method and Description
static Token	Tokens.merge(Iterable<Token> tokens) Merge the given tokens into a new token.
static Token	Tokens.merge(List<Token> tokens, int startIdx, int endIdx)