org.corpus_tools.salt.common.tokenizer

Class Tokenizer

java.lang.Object

org.corpus_tools.salt.common.tokenizer.Tokenizer

public class Tokenizer
extends Object

The general task of this class is to tokenize a given text in the same order as the tool TreeTagger will do. A list of tokenized text is returned with the text anchor (start and end position) in original text. Reimplemented in Java with permission from the original TreeTagger tokenizer in Perl by Helmut Schmid (see http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/). This implementation uses sets of abbreviations to detect tokens, which are abbreviations in a specific language. Therefore you can set a file containing abbreviations, to take others than the default ones. Because of abbreviations are language dependend, you can set a language, to use only a specific set of abbreviations. The current version of the Tokenizer supports abbreviations for english, french, italian and german language. If no language is set, all available abbreviations will be used.

Author:

Amir Zeldes, Florian Zipser

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	F_CHAR
protected static String	P_CHAR

Constructor Summary

Constructors

Constructor and Description
Tokenizer() Initializes a new TTokenizer object.

Method Summary

Modifier and Type	Method and Description
void	addAbbreviation(com.neovisionaries.i18n.LanguageCode language, File abbreviationFile) Adds the content of given file as a list of abbreviation to the internal map corresponding to given language.
void	addAbbreviation(com.neovisionaries.i18n.LanguageCode language, HashSet<String> abbreviations) Adds the given list of abbreviation to the internal map corresponding to given language.
void	addClitics(com.neovisionaries.i18n.LanguageCode language, Clitics clitics) Adds the given clitics to the internal map corresponding to given language.
void	addClitics(com.neovisionaries.i18n.LanguageCode language, File cliticsFile) Adds the content of given file as a set of clitics to the internal map corresponding to given language.
static com.neovisionaries.i18n.LanguageCode	checkLanguage(String text) Tries to detect language and returns ISO 639-2 language code
HashSet<String>	getAbbreviations(com.neovisionaries.i18n.LanguageCode language) Returns a list of abbreviations corresponding to the given language.
Clitics	getClitics(com.neovisionaries.i18n.LanguageCode language) Returns a list of abbreviations corresponding to the given language.
SDocumentGraph	getDocumentGraph()
static com.neovisionaries.i18n.LanguageCode	mapISOLanguageCode(String language) Maps the knallgrau TextCategorizer language description codes to ISO 639 codes.
void	setsDocumentGraph(SDocumentGraph sDocumentGraph)
List<SToken>	tokenize(STextualDS sTextualDSs) Sets the STextualDS to be tokenized.
List<SToken>	tokenize(STextualDS sTextualDSs, com.neovisionaries.i18n.LanguageCode language) Sets the STextualDS to be tokenized and the language of the text.
List<SToken>	tokenize(STextualDS sTextualDS, com.neovisionaries.i18n.LanguageCode language, Integer startPos, Integer endPos) Sets the STextualDS to be tokenized and the language of the text.
List<String>	tokenizeToString(String strInput, com.neovisionaries.i18n.LanguageCode language) The general task of this class is to tokenize a given text in the same order as the tool TreeTagger will do.
List<SToken>	tokenizeToToken(STextualDS sTextualDS, com.neovisionaries.i18n.LanguageCode language, Integer startPos, Integer endPos) The general task of this class is to tokenize a given text in the same order as the tool TreeTagger will do.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

P_CHAR

protected static final String P_CHAR

See Also:

Constant Field Values

F_CHAR

protected static final String F_CHAR

See Also:

Constant Field Values

Constructor Detail

Tokenizer

public Tokenizer()

Initializes a new TTokenizer object.

Method Detail

setsDocumentGraph

public void setsDocumentGraph(SDocumentGraph sDocumentGraph)

getDocumentGraph

public SDocumentGraph getDocumentGraph()

tokenize

public List<SToken> tokenize(STextualDS sTextualDSs)

Sets the STextualDS to be tokenized. Its language will be detected automatically if possible.

Parameters:

sTextualDSs -

tokenize

public List<SToken> tokenize(STextualDS sTextualDSs,
                             com.neovisionaries.i18n.LanguageCode language)

Sets the STextualDS to be tokenized and the language of the text. If language is null, it will be detected automatically if possible.

Parameters:

sTextualDSs -

tokenize

public List<SToken> tokenize(STextualDS sTextualDS,
                             com.neovisionaries.i18n.LanguageCode language,
                             Integer startPos,
                             Integer endPos)

Sets the STextualDS to be tokenized and the language of the text. If language is null, it will be detected automatically if possible.

Parameters:

sTextualDSs - STextualDS object containing the text to be tokenized

language - language of text, if null, language will be detected automatically

startPos - start position, if text to be tokenized is subset (0 assumed if set to null)

startPos - end position, if text to be tokenized is subset (length of text assumed if set to null)

checkLanguage

public static com.neovisionaries.i18n.LanguageCode checkLanguage(String text)

Tries to detect language and returns ISO 639-2 language code

Parameters:

text -

Returns:

mapISOLanguageCode

public static com.neovisionaries.i18n.LanguageCode mapISOLanguageCode(String language)

Maps the knallgrau TextCategorizer language description codes to ISO 639 codes.

Returns:

addAbbreviation

public void addAbbreviation(com.neovisionaries.i18n.LanguageCode language,
                            HashSet<String> abbreviations)

Adds the given list of abbreviation to the internal map corresponding to given language.

Parameters:

language -

abbreviations -

addAbbreviation

public void addAbbreviation(com.neovisionaries.i18n.LanguageCode language,
                            File abbreviationFile)

Adds the content of given file as a list of abbreviation to the internal map corresponding to given language. Form of the file: Adm.
Ala.
Ariz.
Ark.
Aug.
Ave.
Bancorp.

Parameters:

language -

abbreviations -

getAbbreviations

public HashSet<String> getAbbreviations(com.neovisionaries.i18n.LanguageCode language)

Returns a list of abbreviations corresponding to the given language.

Parameters:

language -

Returns:

addClitics

public void addClitics(com.neovisionaries.i18n.LanguageCode language,
                       Clitics clitics)

Adds the given clitics to the internal map corresponding to given language.

Parameters:

language -

clitics -

addClitics

public void addClitics(com.neovisionaries.i18n.LanguageCode language,
                       File cliticsFile)

Adds the content of given file as a set of clitics to the internal map corresponding to given language. Form of the file: Adm.

The file must be structured so that the first line contains the regex for proclitics, and the second line the regex for enclitics, e.g.:

([dcjlmnstDCJLNMST]'|[Qq]u'|[Jj]usqu'|[Ll]orsqu') (-t-elles?|-t-ils?|-t-on|-ce|-elles?|-ils?|-je|-la|-les?|-leur|-lui|-mêmes?|-m'|-moi|-nous|-on|-toi|-tu|-t'|-vous|-en|-y|-ci|-là)

Parameters:

language -

cliticsFile -

getClitics

public Clitics getClitics(com.neovisionaries.i18n.LanguageCode language)

Returns a list of abbreviations corresponding to the given language.

Parameters:

language -

Returns:

tokenizeToToken

public List<SToken> tokenizeToToken(STextualDS sTextualDS,
                                    com.neovisionaries.i18n.LanguageCode language,
                                    Integer startPos,
                                    Integer endPos)

The general task of this class is to tokenize a given text in the same order as the tool TreeTagger will do. A list of tokenized text is returned with the text anchor (start and end position) in original text. If the SDocumentGraph already contains tokens, the tokens will be preserved, if they overlap the same textual range as the new one. Otherwise a SSpan is created covering corresponding to the existing token. The span than overlaps all new tokens and contains all annotations the old token did. In case, the span would overlaps the same textual range as the old token did, no span is created.

Parameters:

strInput - original text

Returns:

tokenized text fragments and their position in the original text

tokenizeToString

public List<String> tokenizeToString(String strInput,
                                     com.neovisionaries.i18n.LanguageCode language)

The general task of this class is to tokenize a given text in the same order as the tool TreeTagger will do. Returns a list of tokenized text.

Parameters:

strInput - original text

Returns:

tokeized text fragments