eus.ixa.ixa.pipe.ml.tok

Class Normalizer

java.lang.Object

eus.ixa.ixa.pipe.ml.tok.Normalizer

public class Normalizer
extends Object

Normalizer class for converting punctuation mostly following various corpora conventions such as Penn TreeBank, Ancora, Tutpenn, Tiger and CTAG.

Version:

2015-04-13

Author:

ragerri

Field Summary

Fields

Modifier and Type	Field and Description
static Pattern	apostrophe
static Pattern	doubleAsciiQuote
static Pattern	doubleAsciiQuoteAlphaNumeric
static Pattern	ellipsis
static Pattern	invertSingleAsciiQuote
static Pattern	leftDoubleQuote
static Pattern	leftSingleQuote
static Pattern	longDash
static Pattern	oneFourth
static Pattern	oneHalf
static Pattern	oneThird
static Pattern	rightDoubleQuote
static Pattern	rightSingleQuote
static Pattern	singleAsciiQuote
static String	THREE_DOTS
static Pattern	threeQuarters
static String	TO_ASCII_SINGLE_QUOTE
static Pattern	toAsciiDoubleQuote
static Pattern	toAsciiSingleQuote
static Pattern	twoThirds

Method Summary

Modifier and Type	Method and Description
static void	convertNonCanonicalStrings(List<Token> sentence, String lang) Converts non-unicode and other strings into their unicode counterparts.
static void	normalizeDoubleQuotes(List<Token> sentence, String lang) Normalizes double and ambiguous quotes according to language and corpus.
static void	normalizeQuotes(List<Token> sentence, String lang) Normalizes non-ambiguous quotes according to language and corpus.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

THREE_DOTS

public static final String THREE_DOTS

See Also:

Constant Field Values

ellipsis

public static final Pattern ellipsis

longDash

public static final Pattern longDash

oneFourth

public static final Pattern oneFourth

oneThird

public static final Pattern oneThird

oneHalf

public static final Pattern oneHalf

twoThirds

public static final Pattern twoThirds

threeQuarters

public static final Pattern threeQuarters

apostrophe

public static final Pattern apostrophe

leftSingleQuote

public static final Pattern leftSingleQuote

rightSingleQuote

public static final Pattern rightSingleQuote

leftDoubleQuote

public static final Pattern leftDoubleQuote

rightDoubleQuote

public static final Pattern rightDoubleQuote

singleAsciiQuote

public static final Pattern singleAsciiQuote

invertSingleAsciiQuote

public static final Pattern invertSingleAsciiQuote

doubleAsciiQuote

public static final Pattern doubleAsciiQuote

doubleAsciiQuoteAlphaNumeric

public static final Pattern doubleAsciiQuoteAlphaNumeric

TO_ASCII_SINGLE_QUOTE

public static final String TO_ASCII_SINGLE_QUOTE

See Also:

Constant Field Values

toAsciiSingleQuote

public static final Pattern toAsciiSingleQuote

toAsciiDoubleQuote

public static final Pattern toAsciiDoubleQuote

Method Detail

convertNonCanonicalStrings

public static void convertNonCanonicalStrings(List<Token> sentence,
                                              String lang)

Converts non-unicode and other strings into their unicode counterparts.

Parameters:

sentence - the list of tokens

lang - the language

normalizeQuotes

public static void normalizeQuotes(List<Token> sentence,
                                   String lang)

Normalizes non-ambiguous quotes according to language and corpus.

Parameters:

sentence - the list of tokens

lang - the language

normalizeDoubleQuotes

public static void normalizeDoubleQuotes(List<Token> sentence,
                                         String lang)

Normalizes double and ambiguous quotes according to language and corpus.

Parameters:

sentence - the list of tokens

lang - the language