eus.ixa.ixa.pipe.tok

Class NonPeriodBreaker

java.lang.Object

eus.ixa.ixa.pipe.tok.NonPeriodBreaker

public class NonPeriodBreaker
extends Object

This class implements exceptions for periods as sentence breakers and tokens. It decides when a period induces a new sentence or a new token and when it does not.

Version:

2015-04-04

Author:

ragerri

Field Summary

Fields

Modifier and Type	Field and Description
static Pattern	acronym General acronyms.
static String	NON_BREAKER_DIGITS Do not split dot after these words if followed by number.
static Pattern	nonBreakerDigits Re-attach segmented dots after non breaker digits.
static Pattern	numbers Do not segment numbers like 11.1.
static Pattern	section
static String	SECTION
static Pattern	segmentAll Segment everything not segmented in the SentenceSegmenter.
static Pattern	startDigit Starts with a digit.
static Pattern	startLower Starts with a lowercase.
static Pattern	startPunct Starts with punctuation that is not beginning of sentence marker.
static Pattern	wordDot Any non white space followed by a period.

Constructor Summary

Constructors

Constructor and Description
NonPeriodBreaker(Properties properties) This constructor reads some non breaking prefixes files in resources to create exceptions of segmentation and tokenization.

Method Summary

Modifier and Type	Method and Description
static String	deSegmentAcronyms(String line) Removes wrongly introduce SECTION marks in acronyms.
String	SegmenterNonBreaker(String line) This function implements exceptions for periods as sentence breakers.
String	TokenizerNonBreaker(String line) It decides when periods do not need to be tokenized.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SECTION

public static String SECTION

section

public static Pattern section

segmentAll

public static Pattern segmentAll

Segment everything not segmented in the SentenceSegmenter.

NON_BREAKER_DIGITS

public static String NON_BREAKER_DIGITS

Do not split dot after these words if followed by number.

nonBreakerDigits

public static Pattern nonBreakerDigits

Re-attach segmented dots after non breaker digits.

acronym

public static Pattern acronym

General acronyms.

numbers

public static Pattern numbers

Do not segment numbers like 11.1.

wordDot

public static Pattern wordDot

Any non white space followed by a period.

startLower

public static Pattern startLower

Starts with a lowercase.

startPunct

public static Pattern startPunct

Starts with punctuation that is not beginning of sentence marker.

startDigit

public static Pattern startDigit

Starts with a digit.

Constructor Detail

NonPeriodBreaker

public NonPeriodBreaker(Properties properties)

This constructor reads some non breaking prefixes files in resources to create exceptions of segmentation and tokenization.

Parameters:

properties - the options

Method Detail

SegmenterNonBreaker

public String SegmenterNonBreaker(String line)

This function implements exceptions for periods as sentence breakers. It decides when a period induces a new sentence or not.

Parameters:

line - the text to be processed

Returns:

segmented text (with newlines included)

deSegmentAcronyms

public static String deSegmentAcronyms(String line)

Removes wrongly introduce SECTION marks in acronyms.

Parameters:

line - the text

Returns:

the segmented text

TokenizerNonBreaker

public String TokenizerNonBreaker(String line)

It decides when periods do not need to be tokenized.

Parameters:

line - the sentence to be tokenized

Returns:

line