Package org.corpus_tools.salt.common.tokenizer

Class Summary

Class	Description
AbbreviationDE	This offers a set of german abbreviations.
AbbreviationEN	This offers a set of english abbreviations.
AbbreviationFR	This offers a set of french abbreviations.
AbbreviationIT	This offers a set of italian abbreviations.
SimpleTokenizer	This class is a very simple implementation of a tokenizer, which just splits a primary text by a given list of characters. e.g. a blank.
Tokenizer	The general task of this class is to tokenize a given text in the same order as the tool TreeTagger will do.