Tokenizer (JCoRe Token Annotator 2.5.0 API)

java.lang.Object
- de.julielab.jcore.ae.jtbd.Tokenizer

```
public class Tokenizer
extends Object
```

Constructor Summary

Constructors
Constructor Description

Tokenizer()

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`cc.mallet.fst.CRF`	`getModel()`
`ArrayList<Unit>`	`predict(String sentence)`	do the prediction
`void`	`readModel(File file)`	load a previously trained FeatureSubsetModel (CRF4+Properties) which was stored as serialized object to disk.
`void`	`readModel(InputStream is)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - Tokenizer
```
public Tokenizer()
```
- Method Detail
  - getModel
```
public cc.mallet.fst.CRF getModel()
```
  - predict
```
public ArrayList<Unit> predict(String sentence)
```
    do the prediction
    
    Parameters:
    
    original - sentence
    
    Returns:
    
    an ArrayList of Unit objects containing the predicted label
  - readModel
```
public void readModel(File file)
               throws IOException,
                      FileNotFoundException,
                      ClassNotFoundException
```
    load a previously trained FeatureSubsetModel (CRF4+Properties) which was stored as serialized object to disk.
    
    Parameters:
    
    filename - where to find the serialized featureSubsetModel (full path!)
    
    Throws:
    
    IOException
    
    FileNotFoundException
    
    ClassNotFoundException
  - readModel
```
public void readModel(InputStream is)
               throws IOException,
                      ClassNotFoundException
```
    Throws:
    
    IOException
    
    ClassNotFoundException