ml.shifu.guagua.worker

Class AbstractWorkerComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable,KEY extends Bytable,VALUE extends Bytable>

java.lang.Object

ml.shifu.guagua.worker.AbstractWorkerComputable<MASTER_RESULT,WORKER_RESULT,KEY,VALUE>

Type Parameters:

MASTER_RESULT - master result for computation in each iteration.

WORKER_RESULT - worker result for computation in each iteration.

KEY - key type for each record

VALUE - value type for each record

All Implemented Interfaces:

WorkerComputable<MASTER_RESULT,WORKER_RESULT>

public abstract class AbstractWorkerComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable,KEY extends Bytable,VALUE extends Bytable>
extends Object
implements WorkerComputable<MASTER_RESULT,WORKER_RESULT>

Abstract WorkerComputable implementation to load data one by one and only in the very 1st iteration.

To load data successfully, make sure GuaguaRecordReader is initialized firstly.

After data is loaded in the first iteration, one can store the data into collections (meomory or disk) to do later iteration logic.

TODO add multi-thread version to load data.

Constructor Summary

Constructors

Constructor and Description
AbstractWorkerComputable()

Method Summary

Modifier and Type	Method and Description
WORKER_RESULT	compute(WorkerContext<MASTER_RESULT,WORKER_RESULT> context) Worker computation for each iteration.
abstract WORKER_RESULT	doCompute(WorkerContext<MASTER_RESULT,WORKER_RESULT> context) Real computation logic after data loading.
GuaguaRecordReader<KEY,VALUE>	getRecordReader()
abstract void	init(WorkerContext<MASTER_RESULT,WORKER_RESULT> context) Initialization work for the whole computation
abstract void	initRecordReader(GuaguaFileSplit fileSplit) Each GuaguaFileSplit must be initialized before loading data.
abstract void	load(KEY currentKey, VALUE currentValue, WorkerContext<MASTER_RESULT,WORKER_RESULT> context) Load data one by one before computation.
protected void	postLoad(WorkerContext<MASTER_RESULT,WORKER_RESULT> context) Do some post work before loading data.
protected void	preLoad(WorkerContext<MASTER_RESULT,WORKER_RESULT> context) Do some pre work before loading data.
void	setRecordReader(GuaguaRecordReader<KEY,VALUE> recordReader)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbstractWorkerComputable

public AbstractWorkerComputable()

Method Detail

compute

public WORKER_RESULT compute(WorkerContext<MASTER_RESULT,WORKER_RESULT> context)
                      throws IOException

Description copied from interface: WorkerComputable

Worker computation for each iteration.

Specified by:

compute in interface WorkerComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

Parameters:

context - the worker context instance which includes worker info, master result of last iteration or other useful into for each iteration.

Returns:

the worker result of each iteration.

Throws:

IOException - if any io exception in computation, for example, IOException in reading data.

preLoad

protected void preLoad(WorkerContext<MASTER_RESULT,WORKER_RESULT> context)

Do some pre work before loading data.

postLoad

protected void postLoad(WorkerContext<MASTER_RESULT,WORKER_RESULT> context)

Do some post work before loading data.

initRecordReader

public abstract void initRecordReader(GuaguaFileSplit fileSplit)
                               throws IOException

Each GuaguaFileSplit must be initialized before loading data.

Throws:

IOException

init

public abstract void init(WorkerContext<MASTER_RESULT,WORKER_RESULT> context)

Initialization work for the whole computation

doCompute

public abstract WORKER_RESULT doCompute(WorkerContext<MASTER_RESULT,WORKER_RESULT> context)

Real computation logic after data loading.

load

public abstract void load(KEY currentKey,
                          VALUE currentValue,
                          WorkerContext<MASTER_RESULT,WORKER_RESULT> context)

Load data one by one before computation.

getRecordReader

public GuaguaRecordReader<KEY,VALUE> getRecordReader()

setRecordReader

public void setRecordReader(GuaguaRecordReader<KEY,VALUE> recordReader)