se.lth.cs.srl.preprocessor.tokenization

Interface Tokenizer

All Known Implementing Classes:

ExnerSwedishTokenizer, OpenNLPToolsTokenizerWrapper, StanfordChineseSegmenterWrapper, StanfordFrenchTokenizer, StanfordPTBTokenizer, WhiteSpaceTokenizer

public interface Tokenizer

Method Summary

Modifier and Type	Method and Description
String[]	tokenize(String sentence) Tokenize a sentence.

Method Detail

tokenize

String[] tokenize(String sentence)

Tokenize a sentence. The returned array contains a root-token

Parameters:

sentence - The sentence to tokenize

Returns:

a root token, followed by the forms