Tokenizer (JCoRe Token Annotator 2.4.0 API)

java.lang.Object
- de.julielab.jcore.ae.jtbd.Tokenizer

public class Tokenizer
extends java.lang.Object

Constructor Summary

Constructors
Constructor Description

Tokenizer()

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`cc.mallet.fst.CRF`	`getModel()`
`java.util.ArrayList<Unit>`	`predict(java.lang.String sentence)`	do the prediction
`void`	`readModel(java.io.File file)`	load a previously trained FeatureSubsetModel (CRF4+Properties) which was stored as serialized object to disk.
`void`	`readModel(java.io.InputStream is)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - Tokenizer
```
public Tokenizer()
```
- Method Detail
  - getModel
```
public cc.mallet.fst.CRF getModel()
```
  - predict
```
public java.util.ArrayList<Unit> predict(java.lang.String sentence)
```
    do the prediction
    
    Parameters:
    
    original - sentence
    
    Returns:
    
    an ArrayList of Unit objects containing the predicted label
  - readModel
```
public void readModel(java.io.File file)
               throws java.io.IOException,
                      java.io.FileNotFoundException,
                      java.lang.ClassNotFoundException
```
    load a previously trained FeatureSubsetModel (CRF4+Properties) which was stored as serialized object to disk.
    
    Parameters:
    
    filename - where to find the serialized featureSubsetModel (full path!)
    
    Throws:
    
    java.io.IOException
    
    java.io.FileNotFoundException
    
    java.lang.ClassNotFoundException
  - readModel
```
public void readModel(java.io.InputStream is)
               throws java.io.IOException,
                      java.lang.ClassNotFoundException
```
    Throws:
    
    java.io.IOException
    
    java.lang.ClassNotFoundException