se.l4.lect.tokens

Class OffsetTokenizer

java.lang.Object

se.l4.lect.tokens.OffsetTokenizer

All Implemented Interfaces:

Iterator<Token>, Tokenizer

public abstract class OffsetTokenizer
extends Object
implements Tokenizer

Tokenizer that tokenizes using OffsetLocations.

Author:

Andreas Holstenson

Constructor Summary

Constructors

Constructor and Description
OffsetTokenizer()

Method Summary

Modifier and Type	Method and Description
protected <T> void	enhanceNextToken(TokenProperty<T> property, T value) Enhance the next token.
protected abstract boolean	findNextToken(int previousEnd) Find the next token.
boolean	hasNext() Get if any more tokens are available.
Token	next() Get the next available token.
protected MutableToken	nextToken() Get the token that has been set by the last call to setNextToken(TokenType, int, CharSequence).
protected void	setNextToken(TokenType type, int start, CharSequence text) Set the next token.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface se.l4.lect.tokens.Tokenizer

stream, toArray, toList

Methods inherited from interface java.util.Iterator

forEachRemaining, remove

Constructor Detail

OffsetTokenizer

public OffsetTokenizer()

Method Detail

hasNext

public boolean hasNext()

Description copied from interface: Tokenizer

Get if any more tokens are available.

Specified by:

hasNext in interface Iterator<Token>

Specified by:

hasNext in interface Tokenizer

next

public Token next()

Description copied from interface: Tokenizer

Get the next available token.

Specified by:

next in interface Iterator<Token>

Specified by:

next in interface Tokenizer

findNextToken

protected abstract boolean findNextToken(int previousEnd)

Find the next token.

Parameters:

previousEnd - the offset at which the previous token ended

Returns:

true if a token was found, false if no more tokens are available.

nextToken

protected MutableToken nextToken()

Get the token that has been set by the last call to setNextToken(TokenType, int, CharSequence).

Returns:

setNextToken

protected void setNextToken(TokenType type,
                            int start,
                            CharSequence text)

Set the next token. Should be called by findNextToken(int) to update information about the token that has been found.

enhanceNextToken

protected <T> void enhanceNextToken(TokenProperty<T> property,
                                    T value)

Enhance the next token. Can be called after setNextToken(TokenType, int, CharSequence) to enhance the token with properties.