Package org.languagetool.tokenizers.ca

Class CatalanWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.ca.CatalanWordTokenizer

All Implemented Interfaces:

Tokenizer

public class CatalanWordTokenizer
extends WordTokenizer

Tokenizes a sentence into words. Punctuation and whitespace gets its own token. Special treatment for hyphens and apostrophes in Catalan.

Author:

Jaume Ortolà

Field Summary

Fields

Modifier and Type	Field	Description
static final CatalanWordTokenizer	INSTANCE

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
CatalanWordTokenizer()

Method Summary

Modifier and Type	Method	Description
List<String>	tokenize(String text)

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, getTokenizingCharacters, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

INSTANCE

public static final CatalanWordTokenizer INSTANCE

Constructor Details

CatalanWordTokenizer

public CatalanWordTokenizer()

Method Details

tokenize

public List<String> tokenize(String text)

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - Text to tokenize

Returns:

List of tokens. Note: a special string xxCA_APOSxx is used to replace apostrophes, and xxCA_HYPHENxx to replace hyphens.