eus.ixa.ixa.pipe.ml.tok

Interface Tokenizer

All Known Implementing Classes:

RuleBasedTokenizer

public interface Tokenizer

Method Summary

Modifier and Type	Method and Description
List<List<Token>>	tokenize(String[] sentence)

Method Detail

tokenize

List<List<Token>> tokenize(String[] sentence)