Package ml.shifu.guagua.example.nn

Class NNWorker

java.lang.Object

ml.shifu.guagua.worker.AbstractWorkerComputable<NNParams,NNParams,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.Text>>

ml.shifu.guagua.example.nn.NNWorker

All Implemented Interfaces:

ml.shifu.guagua.worker.WorkerComputable<NNParams,NNParams>

public class NNWorker
extends ml.shifu.guagua.worker.AbstractWorkerComputable<NNParams,NNParams,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.Text>>

NNWorker is used to compute NN model according to splits assigned. The result will be sent to master for accumulation.

Gradients in each worker will be sent to master to update weights of model in worker, which follows Encog's multi-core implementation.

Constructor Summary

Constructors

Constructor	Description
NNWorker()

Method Summary

Modifier and Type	Method	Description
NNParams	doCompute(ml.shifu.guagua.worker.WorkerContext<NNParams,NNParams> context)
org.encog.ml.data.MLDataSet	getTestingData()
org.encog.ml.data.MLDataSet	getTrainingData()
void	init(ml.shifu.guagua.worker.WorkerContext<NNParams,NNParams> context)
void	initRecordReader(ml.shifu.guagua.io.GuaguaFileSplit fileSplit)
void	load(ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable> currentKey, ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.Text> currentValue, ml.shifu.guagua.worker.WorkerContext<NNParams,NNParams> workerContext)
protected void	postLoad(ml.shifu.guagua.worker.WorkerContext<NNParams,NNParams> workerContext)
void	setTestingData(org.encog.ml.data.MLDataSet testingData)
void	setTrainingData(org.encog.ml.data.MLDataSet trainingData)

Methods inherited from class ml.shifu.guagua.worker.AbstractWorkerComputable

compute, getRecordReader, preLoad, setRecordReader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NNWorker

public NNWorker()

Method Detail

init

public void init(ml.shifu.guagua.worker.WorkerContext<NNParams,NNParams> context)

Specified by:

init in class ml.shifu.guagua.worker.AbstractWorkerComputable<NNParams,NNParams,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.Text>>

doCompute

public NNParams doCompute(ml.shifu.guagua.worker.WorkerContext<NNParams,NNParams> context)

Specified by:

doCompute in class ml.shifu.guagua.worker.AbstractWorkerComputable<NNParams,NNParams,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.Text>>

postLoad

protected void postLoad(ml.shifu.guagua.worker.WorkerContext<NNParams,NNParams> workerContext)

Overrides:

postLoad in class ml.shifu.guagua.worker.AbstractWorkerComputable<NNParams,NNParams,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.Text>>

load

public void load(ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable> currentKey,
                 ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.Text> currentValue,
                 ml.shifu.guagua.worker.WorkerContext<NNParams,NNParams> workerContext)

Specified by:

load in class ml.shifu.guagua.worker.AbstractWorkerComputable<NNParams,NNParams,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.Text>>

initRecordReader

public void initRecordReader(ml.shifu.guagua.io.GuaguaFileSplit fileSplit)
                      throws IOException

Specified by:

initRecordReader in class ml.shifu.guagua.worker.AbstractWorkerComputable<NNParams,NNParams,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,ml.shifu.guagua.hadoop.io.GuaguaWritableAdapter<org.apache.hadoop.io.Text>>

Throws:

IOException

getTrainingData

public org.encog.ml.data.MLDataSet getTrainingData()

setTrainingData

public void setTrainingData(org.encog.ml.data.MLDataSet trainingData)

getTestingData

public org.encog.ml.data.MLDataSet getTestingData()

setTestingData

public void setTestingData(org.encog.ml.data.MLDataSet testingData)