RuleBasedTokenizer (ixa-pipe-ml 0.0.2 API)

java.lang.Object
- eus.ixa.ixa.pipe.ml.tok.RuleBasedTokenizer

All Implemented Interfaces:

Tokenizer
```
public class RuleBasedTokenizer
extends Object
implements Tokenizer
```
This class provides a multilingual rule based tokenizer. The input of the tokenizer must be a list of already segmented sentences. Additionally, the tokenizer normalizes according to language and corpus.

Version:

2015-04-14

Author:

ragerri

Field Summary

Fields
Modifier and Type	Field and Description
`static Pattern`	`AlphaAposAlpha` Alphabetic apostrophe and alphabetic.
`static Pattern`	`alphaAposNonAlpha` Alphabetic apostrophe and non alpha.
`static Pattern`	`asciiHex` Non printable control characters.
`static Pattern`	`beginLink` Re-tokenize beginning of link.
`static Pattern`	`commaNoDigit` Comma and no digit.
`static Pattern`	`deTokenEnglishNegation` Detokenize wrongly tokenize n't English contractions.
`static Pattern`	`detokenParagraphs` De-tokenize paragraph marks.
`static Pattern`	`digitCommaNoDigit` Digit comma and non digit.
`static Pattern`	`dotmultiDot` Multi dot pattern and extra dot.
`static Pattern`	`dotmultiDotAny` Dot multi pattern followed by anything.
`static Pattern`	`doubleSpaces` One or more spaces.
`static Pattern`	`endLink`
`static Pattern`	`endOfSentenceApos` Tokenize apostrophes occurring at the end of the string.
`static Pattern`	`englishApos` Split English apostrophes.
`static Pattern`	`englishNegations` Split English negative contractions.
`static Pattern`	`generalBlankPunctuation` Non printable punctuation characters.
`static Pattern`	`multiDots` Multidots.
`static Pattern`	`noAlphaAposNoAlpha` No alphabetic apostrophe and no alphabetic.
`static Pattern`	`noAlphaDigitAposAlpha` Non alpha, digit, apostrophe and alpha.
`static Pattern`	`noDigitComma` No digit comma.
`static Pattern`	`noDigitCommaDigit` Non digit comma and digit.
`static Pattern`	`qexc` Question and exclamation marks (do not separate if multiple).
`static Pattern`	`replacement` Unicode replacement character.
`static Pattern`	`spaceDashSpace` Dashes or slashes preceded or followed by space.
`static Pattern`	`specials` Tokenize everything but these characters.
`static String`	`TLD` Top level domains for stopping the wrongLink pattern below.
`static Pattern`	`wrongLink` Detect wrongly tokenized links.
`static Pattern`	`yearApos` Digit apostrophe and s (for 1990's).

Constructor Summary

Constructors
Constructor and Description

RuleBasedTokenizer(String text, Properties properties)
Construct a rule based tokenizer.

Constructors
Constructor and Description
`RuleBasedTokenizer(String text, Properties properties)` Construct a rule based tokenizer.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`static void`	`normalizeTokens(List<List<Token>> tokens, String lang)` Set as value of the token its normalized counterpart.
`List<List<Token>>`	`tokenize(String[] sentences)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - replacement
```
public static Pattern replacement
```
    Unicode replacement character.
  - asciiHex
```
public static Pattern asciiHex
```
    Non printable control characters.
  - generalBlankPunctuation
```
public static Pattern generalBlankPunctuation
```
    Non printable punctuation characters.
  - doubleSpaces
```
public static Pattern doubleSpaces
```
    One or more spaces.
  - specials
```
public static Pattern specials
```
    Tokenize everything but these characters.
  - qexc
```
public static Pattern qexc
```
    Question and exclamation marks (do not separate if multiple).
  - spaceDashSpace
```
public static Pattern spaceDashSpace
```
    Dashes or slashes preceded or followed by space.
  - multiDots
```
public static Pattern multiDots
```
    Multidots.
  - dotmultiDot
```
public static Pattern dotmultiDot
```
    Multi dot pattern and extra dot.
  - dotmultiDotAny
```
public static Pattern dotmultiDotAny
```
    Dot multi pattern followed by anything.
  - noDigitComma
```
public static Pattern noDigitComma
```
    No digit comma.
  - commaNoDigit
```
public static Pattern commaNoDigit
```
    Comma and no digit.
  - digitCommaNoDigit
```
public static Pattern digitCommaNoDigit
```
    Digit comma and non digit.
  - noDigitCommaDigit
```
public static Pattern noDigitCommaDigit
```
    Non digit comma and digit.
  - TLD
```
public static final String TLD
```
    Top level domains for stopping the wrongLink pattern below.
    
    See Also:
    
    Constant Field Values
  - wrongLink
```
public static Pattern wrongLink
```
    Detect wrongly tokenized links.
  - beginLink
```
public static Pattern beginLink
```
    Re-tokenize beginning of link.
  - endLink
```
public static Pattern endLink
```
  - noAlphaAposNoAlpha
```
public static Pattern noAlphaAposNoAlpha
```
    No alphabetic apostrophe and no alphabetic.
  - noAlphaDigitAposAlpha
```
public static Pattern noAlphaDigitAposAlpha
```
    Non alpha, digit, apostrophe and alpha.
  - alphaAposNonAlpha
```
public static Pattern alphaAposNonAlpha
```
    Alphabetic apostrophe and non alpha.
  - AlphaAposAlpha
```
public static Pattern AlphaAposAlpha
```
    Alphabetic apostrophe and alphabetic. Mostly for romance languages separation.
  - englishNegations
```
public static Pattern englishNegations
```
    Split English negative contractions.
  - englishApos
```
public static Pattern englishApos
```
    Split English apostrophes.
  - yearApos
```
public static Pattern yearApos
```
    Digit apostrophe and s (for 1990's).
  - endOfSentenceApos
```
public static Pattern endOfSentenceApos
```
    Tokenize apostrophes occurring at the end of the string.
  - deTokenEnglishNegation
```
public static Pattern deTokenEnglishNegation
```
    Detokenize wrongly tokenize n't English contractions.
  - detokenParagraphs
```
public static Pattern detokenParagraphs
```
    De-tokenize paragraph marks.
- Constructor Detail
  - RuleBasedTokenizer
```
public RuleBasedTokenizer(String text,
                          Properties properties)
```
    Construct a rule based tokenizer.
    
    Parameters:
    
    text - the the breader used for offset calculation
    
    properties - the options
- Method Detail
  - tokenize
```
public List<List<Token>> tokenize(String[] sentences)
```
    Specified by:
    
    tokenize in interface Tokenizer
  - normalizeTokens
```
public static void normalizeTokens(List<List<Token>> tokens,
                                   String lang)
```
    Set as value of the token its normalized counterpart. Normalization is done following languages and corpora (Penn TreeBank, Ancora, Tiger, Tutpenn, etc.) conventions.
    
    Parameters:
    
    tokens - the tokens
    
    lang - the language

Class RuleBasedTokenizer

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

replacement

asciiHex

generalBlankPunctuation

doubleSpaces

specials

qexc

spaceDashSpace

multiDots

dotmultiDot

dotmultiDotAny

noDigitComma

commaNoDigit

digitCommaNoDigit

noDigitCommaDigit

TLD

wrongLink

beginLink

endLink

noAlphaAposNoAlpha

noAlphaDigitAposAlpha

alphaAposNonAlpha

AlphaAposAlpha

englishNegations

englishApos

yearApos

endOfSentenceApos

deTokenEnglishNegation

detokenParagraphs

Constructor Detail

RuleBasedTokenizer

Method Detail

tokenize

normalizeTokens