org.maochen.nlp.ml.util

Class TrainingDataUtils

java.lang.Object

org.maochen.nlp.ml.util.TrainingDataUtils

public class TrainingDataUtils
extends Object

Created by Maochen on 8/10/15.

Constructor Summary

Constructors

Constructor and Description
TrainingDataUtils()

Method Summary

Modifier and Type	Method and Description
static List<Tuple>	createBalancedTrainingData(List<Tuple> trainingData) Regardless of the label, just consider isPosExample and !isPosExample
static List<SequenceTuple>	readSeqFile(InputStream trainingFile, String delimiter, int tagCol)
static void	reduceDimension(List<Tuple> trainingData) reduce the dimension of training data.
static org.apache.commons.lang3.tuple.Pair<List<Tuple>,List<Tuple>>	splitData(List<Tuple> trainingData, double proportion) Shuffle the data and split by proportion

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TrainingDataUtils

public TrainingDataUtils()

Method Detail

createBalancedTrainingData

public static List<Tuple> createBalancedTrainingData(List<Tuple> trainingData)

Regardless of the label, just consider isPosExample and !isPosExample

Parameters:

trainingData - input training data.

Returns:

List of balanced tuples.

reduceDimension

public static void reduceDimension(List<Tuple> trainingData)

reduce the dimension of training data. Remove feats that has the same value across all examples.

Parameters:

trainingData - inputdata.

splitData

public static org.apache.commons.lang3.tuple.Pair<List<Tuple>,List<Tuple>> splitData(List<Tuple> trainingData,
                                                                                     double proportion)

Shuffle the data and split by proportion

Parameters:

trainingData - whole data collection.

proportion - scale from 0.1 - 1.0.

Returns:

Left- small chunk. Right - large chunk.

readSeqFile

public static List<SequenceTuple> readSeqFile(InputStream trainingFile,
                                              String delimiter,
                                              int tagCol)