ml.shifu.guagua.master

Class NettyMasterCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

java.lang.Object

ml.shifu.guagua.BasicCoordinator<MASTER_RESULT,WORKER_RESULT>

ml.shifu.guagua.master.AbstractMasterCoordinator<MASTER_RESULT,WORKER_RESULT>

ml.shifu.guagua.master.NettyMasterCoordinator<MASTER_RESULT,WORKER_RESULT>

All Implemented Interfaces:

MasterInterceptor<MASTER_RESULT,WORKER_RESULT>, org.apache.zookeeper.Watcher

public class NettyMasterCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>
extends AbstractMasterCoordinator<MASTER_RESULT,WORKER_RESULT>

A master coordinator to coordinate with workers through Netty server.

Master still updates results to ZooKeeper znodes for fail-over. While workers sends results to master through Netty server connection.

Worker results are persisted into MemoryDiskList, the reason is that for big model, limited memory may not be enough to store all worker results in memory.

Nested Class Summary

Nested classes/interfaces inherited from class ml.shifu.guagua.master.AbstractMasterCoordinator

AbstractMasterCoordinator.FailOverCommand, AbstractMasterCoordinator.MasterElectionCommand

Nested classes/interfaces inherited from class ml.shifu.guagua.BasicCoordinator

BasicCoordinator.BasicCoordinatorCommand, BasicCoordinator.CoordinatorCommand, BasicCoordinator.GetSplitBytes, BasicCoordinator.RetryCoordinatorCommand, BasicCoordinator.SaveResultToZookeeper

Nested classes/interfaces inherited from interface org.apache.zookeeper.Watcher

org.apache.zookeeper.Watcher.Event

Field Summary

Fields inherited from class ml.shifu.guagua.BasicCoordinator

WAIT_SLOT_MILLS

Constructor Summary

Constructors

Constructor and Description
NettyMasterCoordinator()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	initialize(Properties props) Coordinator initialization.
void	postApplication(MasterContext<MASTER_RESULT,WORKER_RESULT> context) Wait for unregister message for all workers and then clean all znodes existing for this job.
void	postIteration(MasterContext<MASTER_RESULT,WORKER_RESULT> context) Update master computable result to master znode.
void	preApplication(MasterContext<MASTER_RESULT,WORKER_RESULT> context) Do initialization and fail-over checking before all iterations.
void	preIteration(MasterContext<MASTER_RESULT,WORKER_RESULT> context) Wait for all workers done in current iteration.

Methods inherited from class ml.shifu.guagua.master.AbstractMasterCoordinator

getMyBid, isAllWorkersHalt, setMasterResult, setMyBid, setWorkerResults, updateMasterHaltStatus

Methods inherited from class ml.shifu.guagua.BasicCoordinator

checkAndSetZooKeeper, close, getAppNode, getBaseMasterElectionNode, getBytesFromZNode, getCurrentMasterNode, getCurrentMasterSplitNode, getCurrentWorkerNode, getCurrentWorkerSplitNode, getLastMasterNode, getLastWorkerNode, getMasterBaseNode, getMasterElectionNode, getMasterNode, getMasterSerializer, getRootNode, getSleepTime, getWorkerBaseNode, getWorkerBaseNode, getWorkerNode, getWorkerSerializer, getZkConnLatch, getZooKeeper, isFixedTime, process, setBytesToZNode, setFixedTime, setMasterSerializer, setSleepTime, setWorkerSerializer, setZooKeeper, startHeartbeat, stopHeartBeat

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NettyMasterCoordinator

public NettyMasterCoordinator()

Method Detail

initialize

protected void initialize(Properties props)

Description copied from class: BasicCoordinator

Coordinator initialization.

Overrides:

initialize in class BasicCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

preApplication

public void preApplication(MasterContext<MASTER_RESULT,WORKER_RESULT> context)

Do initialization and fail-over checking before all iterations.

Parameters:

context - the master context instance which includes worker results and other useful parameters.

preIteration

public void preIteration(MasterContext<MASTER_RESULT,WORKER_RESULT> context)

Wait for all workers done in current iteration.

Parameters:

context - the master context instance which includes worker results and other useful parameters.

postIteration

public void postIteration(MasterContext<MASTER_RESULT,WORKER_RESULT> context)

Update master computable result to master znode. At the same time clean znodes for old iterations. Iteration 0 and last iteration will not be removed for fail over.

Specified by:

postIteration in interface MasterInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

Overrides:

postIteration in class AbstractMasterCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

Parameters:

context - the master context instance which includes worker results and other useful parameters.

postApplication

public void postApplication(MasterContext<MASTER_RESULT,WORKER_RESULT> context)

Wait for unregister message for all workers and then clean all znodes existing for this job.

Specified by:

postApplication in interface MasterInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

Overrides:

postApplication in class AbstractMasterCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

Parameters:

context - the master context instance which includes worker results and other useful parameters.