Constructor Detail

• Tokenizer_ImplBase

  public Tokenizer_ImplBase()

Method Detail

• getTokens

  public List<Token> getTokens(String text)

  performs tokenization on the input text and returns the character offsets corresponding to the begining and end of each token w.r.t. the input text.

  Specified by:

  getTokens in interface Tokenizer

  Parameters:

  text - the text that you want tokenized.

  Returns:

  offsets corresponding to the begining and end of each token. The dimensions of the returned array will be number of tokens returned by getTokens by 2 (one for token begin and one for token end).

   
            int[][] offsets = getTokenOffsets("example text"); int
           beginFirstToken = offsets[0][0]; int endFirstToken =
           offsets[0][1]; int begin4thToken = offsets[3][0]; int end4thToken
           = offsets[3][1];
   
   

• getTokenTexts

  public abstract String[] getTokenTexts(String text)