| Class | Description |
|---|---|
| CaseSensitiveAllTokenAnalyzer |
all tokens are token characters, tokens are not converted
|
| InsensitivePunctAnalyzer |
Analyzer that separates tokens on whitespace and punctuation, converts tokens to lowercase and ascii
|
| InsensitiveSemicolonAnalyzer |
Analyzer that separates tokens on semicolon, converts tokens to lowercase and ascii
|
| InsensitiveWhitespaceAnalyzer |
Analyzer that separates tokens on whitespace, converts tokens to lowercase and ascii
|
| NoPunctuationTokenizer |
For this tokenizer every character is a tokenChar except whitespace and .
|
| NoSemiColonTokenizer |
For this tokenizer "; " or ";" are token separators, and therefore no token characters.
|
| NoSeparatorTokenizer |
For this tokenizer every character is a tokenChar
|
| SensitivePunctAnalyzer |
Analyzer that separates tokens on whitespace and punctuations
|
| SensitiveSemicolonAnalyzer |
Analyzer that separates tokens on semicolon and does not convert tokens
|
| SensitiveWhitespaceAnalyzer |
Analyzer that separates tokens on whitespace and does not convert tokens
|
Copyright © 2020–2022 Fryske Akademy. All rights reserved.