public abstract class Tokenizer_ImplBase extends Object implements Tokenizer
| Constructor and Description |
|---|
Tokenizer_ImplBase() |
public Tokenizer_ImplBase()
public List<Token> getTokens(String text)
getTokens in interface Tokenizertext - the text that you want tokenized.
int[][] offsets = getTokenOffsets("example text"); int
beginFirstToken = offsets[0][0]; int endFirstToken =
offsets[0][1]; int begin4thToken = offsets[3][0]; int end4thToken
= offsets[3][1];
public abstract String[] getTokenTexts(String text)
Copyright © 2014. All rights reserved.