RuleBasedTokenizer (ixa-pipe-tok 1.8.1 API)

java.lang.Object
- eus.ixa.ixa.pipe.tok.RuleBasedTokenizer

All Implemented Interfaces:

Tokenizer
```
public class RuleBasedTokenizer
extends Object
implements Tokenizer
```
This class provides a multilingual rule based tokenizer. The input of the tokenizer must be a list of already segmented sentences. Additionally, the tokenizer normalizes according to language and corpus.

Version:

2015-04-14

Author:

ragerri

Field Summary

Fields
Modifier and Type	Field and Description
`static Pattern`	`AlphaAposAlpha` Alphabetic apostrophe and alphabetic.
`static Pattern`	`alphaAposNonAlpha` Alphabetic apostrophe and non alpha.
`static Pattern`	`asciiHex` Non printable control characters.
`static Pattern`	`beginLink` Re-tokenize beginning of link.
`static Pattern`	`commaNoDigit` Comma and no digit.
`static Pattern`	`detokenParagraphs` De-tokenize paragraph marks.
`static Pattern`	`digitCommaNoDigit` Digit comma and non digit.
`static Pattern`	`dotmultiDot` Multi dot pattern and extra dot.
`static Pattern`	`dotmultiDotAny` Dot multi pattern followed by anything.
`static Pattern`	`doubleSpaces`
`static Pattern`	`endLink`
`static Pattern`	`endOfSentenceApos` Tokenize apostrophes occurring at the end of the string.
`static Pattern`	`englishApos` Split English apostrophes.
`static Pattern`	`multiDots` Multidots.
`static Pattern`	`noAlphaAposNoAlpha` No alphabetic apostrophe and no alphabetic.
`static Pattern`	`noAlphaDigitAposAlpha` Non alpha, digit, apostrophe and alpha.
`static Pattern`	`noDigitComma` No digit comma.
`static Pattern`	`noDigitCommaDigit` Non digit comma and digit.
`static Pattern`	`qexc` Question and exclamation marks (do not separate if multiple).
`static Pattern`	`replacement`
`static Pattern`	`spaceDashSpace` Dashes or slashes preceded or followed by space.
`static Pattern`	`specials` Tokenize everything but these characters.
`static String`	`TLD` Top level domains for stopping the wrongLink pattern below.
`static Pattern`	`wrongLink` Detect wrongly tokenized links.
`static Pattern`	`yearApos` Digit apostrophe and s (for 1990's).

Constructor Summary

Constructors
Constructor and Description

RuleBasedTokenizer(String text, Properties properties)
Construct a rule based tokenizer.

Constructors
Constructor and Description
`RuleBasedTokenizer(String text, Properties properties)` Construct a rule based tokenizer.

Method Summary

Methods
Modifier and Type	Method and Description
`static void`	`normalizeTokens(List<List<Token>> tokens, String lang)` Set as value of the token its normalized counterpart.
`List<List<Token>>`	`tokenize(String[] sentences)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - replacement
```
public static Pattern replacement
```
  - doubleSpaces
```
public static Pattern doubleSpaces
```
  - asciiHex
```
public static Pattern asciiHex
```
    Non printable control characters.
  - specials
```
public static Pattern specials
```
    Tokenize everything but these characters.
  - qexc
```
public static Pattern qexc
```
    Question and exclamation marks (do not separate if multiple).
  - spaceDashSpace
```
public static Pattern spaceDashSpace
```
    Dashes or slashes preceded or followed by space.
  - multiDots
```
public static Pattern multiDots
```
    Multidots.
  - dotmultiDot
```
public static Pattern dotmultiDot
```
    Multi dot pattern and extra dot.
  - dotmultiDotAny
```
public static Pattern dotmultiDotAny
```
    Dot multi pattern followed by anything.
  - noDigitComma
```
public static Pattern noDigitComma
```
    No digit comma.
  - commaNoDigit
```
public static Pattern commaNoDigit
```
    Comma and no digit.
  - digitCommaNoDigit
```
public static Pattern digitCommaNoDigit
```
    Digit comma and non digit.
  - noDigitCommaDigit
```
public static Pattern noDigitCommaDigit
```
    Non digit comma and digit.
  - TLD
```
public static final String TLD
```
    Top level domains for stopping the wrongLink pattern below.
    
    See Also:
    Constant Field Values
  - wrongLink
```
public static Pattern wrongLink
```
    Detect wrongly tokenized links.
  - beginLink
```
public static Pattern beginLink
```
    Re-tokenize beginning of link.
  - endLink
```
public static Pattern endLink
```
  - noAlphaAposNoAlpha
```
public static Pattern noAlphaAposNoAlpha
```
    No alphabetic apostrophe and no alphabetic.
  - noAlphaDigitAposAlpha
```
public static Pattern noAlphaDigitAposAlpha
```
    Non alpha, digit, apostrophe and alpha.
  - alphaAposNonAlpha
```
public static Pattern alphaAposNonAlpha
```
    Alphabetic apostrophe and non alpha.
  - AlphaAposAlpha
```
public static Pattern AlphaAposAlpha
```
    Alphabetic apostrophe and alphabetic. Mostly for romance languages separation.
  - englishApos
```
public static Pattern englishApos
```
    Split English apostrophes.
  - yearApos
```
public static Pattern yearApos
```
    Digit apostrophe and s (for 1990's).
  - endOfSentenceApos
```
public static Pattern endOfSentenceApos
```
    Tokenize apostrophes occurring at the end of the string.
  - detokenParagraphs
```
public static Pattern detokenParagraphs
```
    De-tokenize paragraph marks.
- Constructor Detail
  - RuleBasedTokenizer
```
public RuleBasedTokenizer(String text,
                  Properties properties)
```
    Construct a rule based tokenizer.
    
    Parameters:
    text - the text used for offset calculation
    properties - the options
- Method Detail
  - tokenize
```
public List<List<Token>> tokenize(String[] sentences)
```
    Specified by:
    
    tokenize in interface Tokenizer
  - normalizeTokens
```
public static void normalizeTokens(List<List<Token>> tokens,
                   String lang)
```
    Set as value of the token its normalized counterpart. Normalization is done following languages and corpora (Penn TreeBank, Ancora, Tiger, Tutpenn, etc.) conventions.
    
    Parameters:
    tokens - the tokens
    lang - the language

Class RuleBasedTokenizer

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

replacement

doubleSpaces

asciiHex

specials

qexc

spaceDashSpace

multiDots

dotmultiDot

dotmultiDotAny

noDigitComma

commaNoDigit

digitCommaNoDigit

noDigitCommaDigit

TLD

wrongLink

beginLink

endLink

noAlphaAposNoAlpha

noAlphaDigitAposAlpha

alphaAposNonAlpha

AlphaAposAlpha

englishApos

yearApos

endOfSentenceApos

detokenParagraphs

Constructor Detail

RuleBasedTokenizer

Method Detail

tokenize

normalizeTokens