Package org.languagetool.tokenizers.ru

Class RussianWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.ru.RussianWordTokenizer

All Implemented Interfaces:

Tokenizer

public class RussianWordTokenizer
extends WordTokenizer

Since:

6.1

Field Summary

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
RussianWordTokenizer()

Method Summary

Modifier and Type	Method	Description
String	getTokenizingCharacters()
List<String>	tokenize(String text)

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

RussianWordTokenizer

public RussianWordTokenizer()

Method Details

getTokenizingCharacters

public String getTokenizingCharacters()

Overrides:

getTokenizingCharacters in class WordTokenizer

tokenize

public List<String> tokenize(String text)

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer