eus.ixa.ixa.pipe.ml.tok

Class NonPeriodBreaker

java.lang.Object

eus.ixa.ixa.pipe.ml.tok.NonPeriodBreaker

public class NonPeriodBreaker
extends Object

This class implements exceptions for periods as sentence breakers and tokens. It decides when a period induces a new sentence or a new token and when it does not.

Version:

2015-04-04

Author:

ragerri

Field Summary

Fields

Modifier and Type	Field and Description
static Pattern	acronym General acronyms.
static Pattern	alphabetic Any alphabetic character.
static Pattern	nextCandidateWord Next word wrt to the candidate to indicate sentence breaker.
static String	NON_BREAKER_DIGITS Do not split dot after these words if followed by number.
static Pattern	nonSegmentedWords Non segmented words, candidates for sentence breaking.
static Pattern	numbers Do not segment numbers like 11.1.
static Pattern	startDigit Starts with a digit.
static Pattern	startLower Starts with a lowercase.
static Pattern	startPunct Starts with punctuation that is not beginning of sentence marker.
static Pattern	wordDot Any non white space followed by a period.

Constructor Summary

Constructors

Constructor and Description
NonPeriodBreaker(Properties properties) This constructor reads some non breaking prefixes files in resources to create exceptions of segmentation and tokenization.

Method Summary

Modifier and Type	Method and Description
String[]	segmenterExceptions(String[] lines) Segment the rest of the text taking into account some exceptions for periods as sentence breakers.
String	TokenizerNonBreaker(String line) It decides when periods do not need to be tokenized.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

nonSegmentedWords

public static Pattern nonSegmentedWords

Non segmented words, candidates for sentence breaking.

nextCandidateWord

public static Pattern nextCandidateWord

Next word wrt to the candidate to indicate sentence breaker.

NON_BREAKER_DIGITS

public static String NON_BREAKER_DIGITS

Do not split dot after these words if followed by number.

acronym

public static Pattern acronym

General acronyms.

numbers

public static Pattern numbers

Do not segment numbers like 11.1.

wordDot

public static Pattern wordDot

Any non white space followed by a period.

alphabetic

public static Pattern alphabetic

Any alphabetic character.

startLower

public static Pattern startLower

Starts with a lowercase.

startPunct

public static Pattern startPunct

Starts with punctuation that is not beginning of sentence marker.

startDigit

public static Pattern startDigit

Starts with a digit.

Constructor Detail

NonPeriodBreaker

public NonPeriodBreaker(Properties properties)

This constructor reads some non breaking prefixes files in resources to create exceptions of segmentation and tokenization.

Parameters:

properties - the options

Method Detail

segmenterExceptions

public String[] segmenterExceptions(String[] lines)

Segment the rest of the text taking into account some exceptions for periods as sentence breakers. It decides when a period marks an end of sentence.

Parameters:

lines - the segmented sentences so far

Returns:

all the segmented sentences

TokenizerNonBreaker

public String TokenizerNonBreaker(String line)

It decides when periods do not need to be tokenized.

Parameters:

line - the sentence to be tokenized

Returns:

line