se.l4.lect.tokens

Interface Tokenizer

All Superinterfaces:

Iterator<Token>

All Known Implementing Classes:

OffsetTokenizer

public interface Tokenizer
extends Iterator<Token>

Tokenizer used to tokenize input into Tokens.

Author:

Andreas Holstenson

Method Summary

Modifier and Type	Method and Description
boolean	hasNext() Get if any more tokens are available.
Token	next() Get the next available token.
default java.util.stream.Stream<Token>	stream() Stream these tokens.
default Token[]	toArray() Get these tokens as an array.
default List<Token>	toList() Get the tokens as a List.

Methods inherited from interface java.util.Iterator

forEachRemaining, remove

Method Detail

hasNext

boolean hasNext()

Get if any more tokens are available.

Specified by:

hasNext in interface Iterator<Token>

Returns:

next

Token next()

Get the next available token.

Specified by:

next in interface Iterator<Token>

Returns:

stream

default java.util.stream.Stream<Token> stream()

Stream these tokens.

Returns:

toList

default List<Token> toList()

Get the tokens as a List.

Returns:

toArray

default Token[] toArray()

Get these tokens as an array.

Returns: