Package org.languagetool.tokenizers.nl

Class DutchWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.nl.DutchWordTokenizer

All Implemented Interfaces:

Tokenizer

public class DutchWordTokenizer
extends WordTokenizer

Field Summary

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
DutchWordTokenizer()

Method Summary

Modifier and Type	Method	Description
String	getTokenizingCharacters()
List<String>	tokenize(String text)	Tokenizes just like WordTokenizer with the exception for words such as "oma's" that contain an apostrophe in their middle.

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DutchWordTokenizer

public DutchWordTokenizer()

Method Details

tokenize

public List<String> tokenize(String text)

Tokenizes just like WordTokenizer with the exception for words such as "oma's" that contain an apostrophe in their middle.

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - Text to tokenize

Returns:

List of tokens

getTokenizingCharacters

public String getTokenizingCharacters()

Overrides:

getTokenizingCharacters in class WordTokenizer