org.corpus_tools.salt.common.tokenizer

Class SimpleTokenizer

java.lang.Object

org.corpus_tools.salt.common.tokenizer.SimpleTokenizer

public class SimpleTokenizer
extends Object

This class is a very simple implementation of a tokenizer, which just splits a primary text by a given list of characters. e.g. a blank.

Author:

Florian Zipser

Constructor Summary

Constructors

Constructor and Description
SimpleTokenizer() Initializes a new TTokenizer object.

Method Summary

Methods

Modifier and Type	Method and Description
SDocumentGraph	getDocumentGraph()
void	setDocumentGraph(SDocumentGraph documentGraph)
List<SToken>	tokenize(STextualDS textualDSs, Character... separator) Sets the STextualDS to be tokenized.
List<SToken>	tokenize(STextualDS textualDS, Integer startPos, Integer endPos, Character... separator) Sets the STextualDS to be tokenized and the language of the text.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleTokenizer

public SimpleTokenizer()

Initializes a new TTokenizer object.

Method Detail

setDocumentGraph

public void setDocumentGraph(SDocumentGraph documentGraph)

getDocumentGraph

public SDocumentGraph getDocumentGraph()

tokenize

public List<SToken> tokenize(STextualDS textualDSs,
                    Character... separator)

Sets the STextualDS to be tokenized. Its language will be detected automatically if possible.

Parameters:

textualDSs -

tokenize

public List<SToken> tokenize(STextualDS textualDS,
                    Integer startPos,
                    Integer endPos,
                    Character... separator)

Sets the STextualDS to be tokenized and the language of the text. If language is null, it will be detected automatically if possible.

Parameters:

sTextualDSs - STextualDS object containing the text to be tokenized

startPos - start position, if text to be tokenized is subset (0 assumed if set to null)

startPos - end position, if text to be tokenized is subset (length of text assumed if set to null)