Evaluation_ImplBase (ClearTK TimeML 2.0.0 API)

java.lang.Object
- org.cleartk.eval.Evaluation_ImplBase<ITEM_TYPE,STATS_TYPE>

Type Parameters:
ITEM_TYPE - The type of items that define the train and test data in trainAndTest(List, List) and crossValidation(List, int), and that are used to create CollectionReaders in getCollectionReader(List). A common choice for this parameter when working with data on a filesystem is File.
STATS_TYPE - The type of statistics object that will be returned by testing methods. A common choice for this parameter is AnnotationStatistics.

Direct Known Subclasses:

TempEval2010Evaluation, TempEval2013Evaluation
```
public abstract class Evaluation_ImplBase<ITEM_TYPE,STATS_TYPE>
extends Object
```
A base class for training, testing and cross-validation of models.
Copyright (c) 2012, Regents of the University of Colorado
All rights reserved.

Author:

Steven Bethard

Field Summary

Fields
Modifier and Type Field and Description

protected File baseDirectory

Fields
Modifier and Type	Field and Description
`protected File`	`baseDirectory`

Constructor Summary

Constructors
Constructor and Description

Evaluation_ImplBase(File baseDirectory)
Create an evaluation that will write all auxiliary files to the given directory.

Constructors
Constructor and Description
`Evaluation_ImplBase(File baseDirectory)` Create an evaluation that will write all auxiliary files to the given directory.

Method Summary

Methods
Modifier and Type	Method and Description
`List<STATS_TYPE>`	`crossValidation(List<ITEM_TYPE> items, int nFolds)` Run a cross-validation.
`protected abstract CollectionReader`	`getCollectionReader(List<ITEM_TYPE> items)` Creates a `CollectionReader` from the given items.
`protected List<ITEM_TYPE>`	`selectFoldTestItems(List<ITEM_TYPE> items, int nFolds, int fold)` Determines which items should be used for testing in one fold of a cross-validation.
`protected List<ITEM_TYPE>`	`selectFoldTrainItems(List<ITEM_TYPE> items, int nFolds, int fold)` Determines which items should be used for training in one fold of a cross-validation.
`protected abstract STATS_TYPE`	`test(CollectionReader collectionReader, File directory)` Evaluates a model on a set of testing data.
`protected abstract void`	`train(CollectionReader collectionReader, File directory)` Trains a model on a set of training data.
`STATS_TYPE`	`trainAndTest(List<ITEM_TYPE> trainItems, List<ITEM_TYPE> testItems)` Train a model on one set of items and test it on another.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - baseDirectory
```
protected File baseDirectory
```
- Constructor Detail
  - Evaluation_ImplBase
```
public Evaluation_ImplBase(File baseDirectory)
```
    Create an evaluation that will write all auxiliary files to the given directory.
    
    Parameters:
    baseDirectory - The directory for all evaluation files.
- Method Detail
  - crossValidation
```
public List<STATS_TYPE> crossValidation(List<ITEM_TYPE> items,
                               int nFolds)
                                 throws Exception
```
    Run a cross-validation. Splits the items into nFolds approximately equal subsets, and for each subset, training a classifier on all the remaining data and testing on the subset.
    
    Parameters:
    items - The items on which to train and test. Good machine learning practice requires that these items come only from the training data, and not from the test data.
    nFolds - The number of subsets into which the items should be split. Note that the number of folds may not be larger than the number of items.
    
    Returns:
    The statistics that result from testing the model, one for each fold.
    
    Throws:
    
    Exception
  - getCollectionReader
```
protected abstract CollectionReader getCollectionReader(List<ITEM_TYPE> items)
                                                 throws Exception
```
    Creates a CollectionReader from the given items. This method is called in trainAndTest(List, List) and crossValidation(List, int) to create readers both for the training data and for the testing data.
    
    Parameters:
    items - Items from the training, test or cross-validation sets.
    
    Returns:
    A CollectionReader that produces CASes for the items.
    
    Throws:
    
    Exception
  - selectFoldTestItems
```
protected List<ITEM_TYPE> selectFoldTestItems(List<ITEM_TYPE> items,
                                  int nFolds,
                                  int fold)
```
    Determines which items should be used for testing in one fold of a cross-validation. The default implementation includes every (nFolds)th item, but this may be overridden in subclasses.
    
    Parameters:
    items - The full list of training items.
    nFolds - The total number of folds in this cross validation.
    fold - The index of the fold (0 <= fold < nFolds) whose test items are to be selected.
    
    Returns:
    The items that should be used for testing.
  - selectFoldTrainItems
```
protected List<ITEM_TYPE> selectFoldTrainItems(List<ITEM_TYPE> items,
                                   int nFolds,
                                   int fold)
```
    Determines which items should be used for training in one fold of a cross-validation. The default implementation includes all items except for every (nFolds)th item, but this may be overridden in subclasses.
    
    Parameters:
    items - The full list of training items.
    nFolds - The total number of folds in this cross validation.
    fold - The index of the fold (0 <= fold < nFolds) whose training items are to be selected.
    
    Returns:
    The items that should be used for training.
  - test
```
protected abstract STATS_TYPE test(CollectionReader collectionReader,
              File directory)
                            throws Exception
```
    Evaluates a model on a set of testing data.
    
    Parameters:
    collectionReader - The data on which the model should be tested.
    directory - The directory in which any model files should be written. This method may safely assume that train(CollectionReader, File) was called on this same directory before test(CollectionReader, File) was called.
    
    Returns:
    The statistics that result from testing the model
    
    Throws:
    
    Exception
  - train
```
protected abstract void train(CollectionReader collectionReader,
         File directory)
                       throws Exception
```
    Trains a model on a set of training data.
    
    Parameters:
    collectionReader - The data on which the model should be trained.
    directory - The directory in which any model files should be written.
    
    Throws:
    
    Exception
  - trainAndTest
```
public STATS_TYPE trainAndTest(List<ITEM_TYPE> trainItems,
                      List<ITEM_TYPE> testItems)
                        throws Exception
```
    Train a model on one set of items and test it on another.
    
    Parameters:
    trainItems - The items on which to train.
    testItems - The items on which to test.
    
    Returns:
    The statistics that result from testing the model.
    
    Throws:
    
    Exception

Class Evaluation_ImplBase<ITEM_TYPE,STATS_TYPE>

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

baseDirectory

Constructor Detail

Evaluation_ImplBase

Method Detail

crossValidation

getCollectionReader

selectFoldTestItems

selectFoldTrainItems

test

train

trainAndTest