ml.shifu.guagua.worker

Interface WorkerComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

Type Parameters:

MASTER_RESULT - master result for computation in each iteration.

WORKER_RESULT - worker result for computation in each iteration.

All Known Implementing Classes:

AbstractCombineWorkerComputable, AbstractWorkerComputable, HookWorkerComputable

public interface WorkerComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

WorkerComputable defines worker computation for any guagua application. All guagua application should define its own worker logic and to configure it by using guagua client configuration.

Any context parameters like current iteration, master result in last iteration can be got from WorkerContext.

To get global properties like Hadoop and YARN configuration properties, WorkerContext.getProps() is a wrapper for Hadoop Configuration. Any configuration by using '-D' in command line or by internal Hadoop/YARN filling can be got from these properties.

WorkerContext.getLastMasterResult() is the master result from last iteration. For first iteration (current iteration equals 1), it is null.

Iteration starts from 1, ends with WorkerContext.getTotalIteration().Total iteration number can be set in command line through '-c' parameter. It is the same for master and worker total iteration number setting.

WorkerContext.getAppId() denotes the job ID for a map-reduce job in Hadoop, or YARN application ID for a YARN application. WorkerContext.getContainerId() denotes current worker container. It is task partition index in Hadoop map-reduce job or split index in YARN guagua application. Container ID is unique for each worker task, if current worker task is restarted after failure, it should use the same container id with the failed worker task to make sure guagua knows who it is.

In worker computation logic, sometimes data loading is needed for only the first iteration. An abstract AbstractWorkerComputable class which wraps data loading in first iteration (iteration 1) to help load data and do worker computation logic easier.

See Also:

AbstractWorkerComputable

Method Summary

Methods

Modifier and Type	Method and Description
WORKER_RESULT	compute(WorkerContext<MASTER_RESULT,WORKER_RESULT> context) Worker computation for each iteration.

Method Detail

compute

WORKER_RESULT compute(WorkerContext<MASTER_RESULT,WORKER_RESULT> context)
                                      throws IOException

Worker computation for each iteration.

Parameters:

context - the worker context instance which includes worker info, master result of last iteration or other useful into for each iteration.

Returns:

the worker result of each iteration.

Throws:

IOException - if any io exception in computation, for example, IOException in reading data.