Uses of Class
eus.ixa.ixa.pipe.tok.Token

Packages that use Token

Package	Description
eus.ixa.ixa.pipe.tok

Uses of Token in eus.ixa.ixa.pipe.tok

Methods in eus.ixa.ixa.pipe.tok that return Token

Modifier and Type	Method and Description
Token	TokenFactory.createToken(String tokenString, int offset, int length) Constructs a Token as a String with corresponding offsets and length from which to calculate start and end position of the Token.

Methods in eus.ixa.ixa.pipe.tok that return types with arguments of type Token

Modifier and Type	Method and Description
List<List<Token>>	Tokenizer.tokenize(String[] sentence)
List<List<Token>>	RuleBasedTokenizer.tokenize(String[] sentences)

Method parameters in eus.ixa.ixa.pipe.tok with type arguments of type Token

Modifier and Type	Method and Description
static void	Normalizer.convertNonCanonicalStrings(List<Token> sentence, String lang) Converts non-unicode and other strings into their unicode counterparts.
void	TokenizerEvaluator.evaluate(List<Token> referenceList, List<Token> predictedList) Evaluates the given reference Token list wrt to the predicted Token list.
void	TokenizerEvaluator.evaluate(List<Token> referenceList, List<Token> predictedList) Evaluates the given reference Token list wrt to the predicted Token list.
static String	StringUtils.getStringFromTokens(List<Token> tokens) Get a List of Strings and convert it into a string separated by spaces.
static void	Normalizer.normalizeDoubleQuotes(List<Token> sentence, String lang) Normalizes double and ambiguous quotes according to language and corpus.
static void	Normalizer.normalizeQuotes(List<Token> sentence, String lang) Normalizes non-ambiguous quotes according to language and corpus.
static void	RuleBasedTokenizer.normalizeTokens(List<List<Token>> tokens, String lang) Set as value of the token its normalized counterpart.