ml.shifu.guagua.worker

Class NettyWorkerCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

java.lang.Object

ml.shifu.guagua.BasicCoordinator<MASTER_RESULT,WORKER_RESULT>

ml.shifu.guagua.worker.AbstractWorkerCoordinator<MASTER_RESULT,WORKER_RESULT>

ml.shifu.guagua.worker.NettyWorkerCoordinator<MASTER_RESULT,WORKER_RESULT>

All Implemented Interfaces:

WorkerInterceptor<MASTER_RESULT,WORKER_RESULT>, org.apache.zookeeper.Watcher

public class NettyWorkerCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>
extends AbstractWorkerCoordinator<MASTER_RESULT,WORKER_RESULT>

Worker coordinator coordinates master with Netty client channel.

Worker results and iteration info is not stored in to znode like SyncWorkerCoordinator. For a big task with much more workers, this can decrease pressure on zookeeper. To leverage Netty, fast worker and master coordination is expected.

Master results are still stored into zookeeper for fail-over. Only one master per each job, this shouldn't be a burden to zookeeper.

Nested Class Summary

Nested classes/interfaces inherited from class ml.shifu.guagua.worker.AbstractWorkerCoordinator

AbstractWorkerCoordinator.FailOverCoordinatorCommand

Nested classes/interfaces inherited from class ml.shifu.guagua.BasicCoordinator

BasicCoordinator.BasicCoordinatorCommand, BasicCoordinator.CoordinatorCommand, BasicCoordinator.GetSplitBytes, BasicCoordinator.RetryCoordinatorCommand, BasicCoordinator.SaveResultToZookeeper

Nested classes/interfaces inherited from interface org.apache.zookeeper.Watcher

org.apache.zookeeper.Watcher.Event

Field Summary

Fields inherited from class ml.shifu.guagua.BasicCoordinator

WAIT_SLOT_MILLS

Constructor Summary

Constructors

Constructor and Description
NettyWorkerCoordinator()

Method Summary

Modifier and Type	Method and Description
void	postApplication(WorkerContext<MASTER_RESULT,WORKER_RESULT> context) Send stop message to master and then clean resources.
void	postIteration(WorkerContext<MASTER_RESULT,WORKER_RESULT> context) Send worker results to master; wait for current master stop; get current master result.
void	preApplication(WorkerContext<MASTER_RESULT,WORKER_RESULT> context) Worker coordinator initialization.
void	preIteration(WorkerContext<MASTER_RESULT,WORKER_RESULT> context) The hook point before computation of each iteration.

Methods inherited from class ml.shifu.guagua.worker.AbstractWorkerCoordinator

setMasterResult

Methods inherited from class ml.shifu.guagua.BasicCoordinator

checkAndSetZooKeeper, close, getAppNode, getBaseMasterElectionNode, getBytesFromZNode, getCurrentMasterNode, getCurrentMasterSplitNode, getCurrentWorkerNode, getCurrentWorkerSplitNode, getLastMasterNode, getLastWorkerNode, getMasterBaseNode, getMasterElectionNode, getMasterNode, getMasterSerializer, getRootNode, getSleepTime, getWorkerBaseNode, getWorkerBaseNode, getWorkerNode, getWorkerSerializer, getZkConnLatch, getZooKeeper, initialize, isFixedTime, process, setBytesToZNode, setFixedTime, setMasterSerializer, setSleepTime, setWorkerSerializer, setZooKeeper, startHeartbeat, stopHeartBeat

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NettyWorkerCoordinator

public NettyWorkerCoordinator()

Method Detail

preApplication

public void preApplication(WorkerContext<MASTER_RESULT,WORKER_RESULT> context)

Worker coordinator initialization.

• 1. Initialize Zookeeper instance to connect to zookeeper ensemble;
• 2. check fail over to recover from last failure point;
• 3. wait for master initialization and get master address from master initialization znode;
• 4. Recover last master result from master znode if fail-over task.

Parameters:

context - the worker context instance which includes master result and other useful parameters.

preIteration

public void preIteration(WorkerContext<MASTER_RESULT,WORKER_RESULT> context)

Description copied from interface: WorkerInterceptor

The hook point before computation of each iteration.

Specified by:

preIteration in interface WorkerInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

Overrides:

preIteration in class AbstractWorkerCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

Parameters:

context - the worker context instance which includes master result and other useful parameters.

postIteration

public void postIteration(WorkerContext<MASTER_RESULT,WORKER_RESULT> context)

Send worker results to master; wait for current master stop; get current master result.

Parameters:

context - the worker context instance which includes master result and other useful parameters.

postApplication

public void postApplication(WorkerContext<MASTER_RESULT,WORKER_RESULT> context)

Send stop message to master and then clean resources.

Specified by:

postApplication in interface WorkerInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

Overrides:

postApplication in class AbstractWorkerCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

Parameters:

context - the worker context instance which includes master result and other useful parameters.