se.l4.lect

Class TokenizingLanguage

java.lang.Object

se.l4.lect.ChunkedLanguageParser

se.l4.lect.TokenizingLanguage

All Implemented Interfaces:

LanguageParser

public class TokenizingLanguage
extends ChunkedLanguageParser

Implementation of LanguageParser that uses Tokenizers to split text into sentences and tokens. The tokenizers used should use OffsetLocation for its tokens so that the language can correctly map the locations of the tokens to the actual location in the TextSource.

Author:

Andreas Holstenson

Field Summary

Fields inherited from class se.l4.lect.ChunkedLanguageParser

encounter

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	TokenizingLanguage(Locale locale, TokenizerFactory paragraphTokenizers, TokenizerFactory sentenceTokenizers, LanguageEncounter encounter)

Method Summary

Modifier and Type	Method and Description
static LanguageFactory	create(Locale locale, TokenizerFactory paragraphTokenizers, TokenizerFactory sentenceTokenizers) Create a Function that can create a parser for the given locale.
static LanguageParser	create(Locale locale, TokenizerFactory paragraphTokenizers, TokenizerFactory sentenceTokenizers, LanguageEncounter encounter) Create an instance for the given locale and tokenizers.
protected void	handleChunk(CharSequence sequence) Handle the given sequence of characters.
Locale	locale() Get the locale of this parser.

Methods inherited from class se.l4.lect.ChunkedLanguageParser

emitToken, emitToken, emitToken, endSentence, flush, startSentence, text, text

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizingLanguage

protected TokenizingLanguage(Locale locale,
                             TokenizerFactory paragraphTokenizers,
                             TokenizerFactory sentenceTokenizers,
                             LanguageEncounter encounter)

Method Detail

locale

public Locale locale()

Description copied from interface: LanguageParser

Get the locale of this parser.

Returns:

handleChunk

protected void handleChunk(CharSequence sequence)

Description copied from class: ChunkedLanguageParser

Handle the given sequence of characters. This method should go through the sequence and in order emit events via ChunkedLanguageParser.startSentence(int), #emitToken(int, se.l4.lect.Token.TokenType, String) and ChunkedLanguageParser.endSentence(int).

Specified by:

handleChunk in class ChunkedLanguageParser

create

public static LanguageParser create(Locale locale,
                                    TokenizerFactory paragraphTokenizers,
                                    TokenizerFactory sentenceTokenizers,
                                    LanguageEncounter encounter)

Create an instance for the given locale and tokenizers.

Parameters:

locale - the Locale that the parser is for

paragraphTokenizers - the tokenizer used to tokenize a paragraph into sentences. The tokenizer must use OffsetLocation for its tokens.

sentenceTokenizers - the tokenizer used to tokenize a sentence into individual tokens. The tokenizer must use OffsetLocation for its tokens.

encounter - the encounter that should receive results

Returns:

instance of TokenizingLanguage

create

public static LanguageFactory create(Locale locale,
                                     TokenizerFactory paragraphTokenizers,
                                     TokenizerFactory sentenceTokenizers)

Create a Function that can create a parser for the given locale.

Parameters:

locale - the Locale that the parser is for

paragraphTokenizers - the tokenizer used to tokenize a paragraph into sentences. The tokenizer must use OffsetLocation for its tokens.

sentenceTokenizers - the tokenizer used to tokenize a sentence into individual tokens. The tokenizer must use OffsetLocation for its tokens.

Returns:

function that creates a TokenizingLanguage for a given LanguageEncounter