ml.shifu.guagua

Class BasicCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>

java.lang.Object

ml.shifu.guagua.BasicCoordinator<MASTER_RESULT,WORKER_RESULT>

Type Parameters:

MASTER_RESULT - master result for computation in each iteration.

WORKER_RESULT - worker result for computation in each iteration.

All Implemented Interfaces:

org.apache.zookeeper.Watcher

Direct Known Subclasses:

AbstractMasterCoordinator, AbstractWorkerCoordinator

public class BasicCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable>
extends Object
implements org.apache.zookeeper.Watcher

BasicCoordinator is a basic implementation for both SyncWorkerCoordinator and SyncMasterCoordinator:

• 1. A Watcher to monitor zookeeper znodes.
• 2. Help functions to construct master znodes and worker znodes.
• 3. Basic functions to convert object to bytes or bytes to object.
• 4. Heart beat used to avoid ZooKeeperSessionExpiredException.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BasicCoordinator.BasicCoordinatorCommand BasicCoordinator.BasicCoordinatorCommand is to process exceptions for zookeeper operations.
static interface	BasicCoordinator.CoordinatorCommand BasicCoordinator.CoordinatorCommand is used for consistent process of zookeeper coordination.
static class	BasicCoordinator.GetSplitBytes
static class	BasicCoordinator.RetryCoordinatorCommand BasicCoordinator.RetryCoordinatorCommand is used to wrap retry logic.
class	BasicCoordinator.SaveResultToZookeeper

Nested classes/interfaces inherited from interface org.apache.zookeeper.Watcher

org.apache.zookeeper.Watcher.Event

Field Summary

Fields

Modifier and Type	Field and Description
protected static int	WAIT_SLOT_MILLS Default waiting time to check master or worker progress from zookeeper servers.

Constructor Summary

Constructors

Constructor and Description
BasicCoordinator()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	checkAndSetZooKeeper(Properties props) Set up connection with given zookeeper settings.
protected void	close() Close resources like zookeeper, thread pool
protected StringBuilder	getAppNode(String appId)
protected StringBuilder	getBaseMasterElectionNode(String appId)
protected byte[]	getBytesFromZNode(String znode, String splitZnode) This is reverse method to setBytesToZNode(String, String, byte[], CreateMode).
protected StringBuilder	getCurrentMasterNode(String appId, int iteration)
protected StringBuilder	getCurrentMasterSplitNode(String appId, int iteration)
protected StringBuilder	getCurrentWorkerNode(String appId, String containerId, int iteration)
protected StringBuilder	getCurrentWorkerSplitNode(String appId, String containerId, int iteration)
protected StringBuilder	getLastMasterNode(String appId, int iteration)
protected StringBuilder	getLastWorkerNode(String appId, String containerId, int iteration)
protected StringBuilder	getMasterBaseNode(String appId)
protected StringBuilder	getMasterElectionNode(String appId, long sessionId)
protected StringBuilder	getMasterNode(String appId, int iteration)
Serializer<MASTER_RESULT>	getMasterSerializer()
protected StringBuilder	getRootNode()
long	getSleepTime()
protected StringBuilder	getWorkerBaseNode(String appId)
protected StringBuilder	getWorkerBaseNode(String appId, int iteration)
protected StringBuilder	getWorkerNode(String appId, String containerId, int iteration)
Serializer<WORKER_RESULT>	getWorkerSerializer()
CountDownLatch	getZkConnLatch()
GuaguaZooKeeper	getZooKeeper()
protected void	initialize(Properties props) Coordinator initialization.
boolean	isFixedTime()
void	process(org.apache.zookeeper.WatchedEvent event)
protected boolean	setBytesToZNode(String znode, String splitZnode, byte[] bytes, org.apache.zookeeper.CreateMode createNode) Set bytes to znode, if bytes is over zookeeper data limit(1MB), use children znodes to store each part.
void	setFixedTime(boolean isFixedTime)
void	setMasterSerializer(Serializer<MASTER_RESULT> masterSerializer)
void	setSleepTime(long sleepTime)
void	setWorkerSerializer(Serializer<WORKER_RESULT> workerSerializer)
void	setZooKeeper(GuaguaZooKeeper zooKeeper)
protected void	startHeartbeat()
protected void	stopHeartBeat()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WAIT_SLOT_MILLS

protected static final int WAIT_SLOT_MILLS

Default waiting time to check master or worker progress from zookeeper servers.

See Also:

Constant Field Values

Constructor Detail

BasicCoordinator

public BasicCoordinator()

Method Detail

getMasterBaseNode

protected StringBuilder getMasterBaseNode(String appId)

getMasterNode

protected StringBuilder getMasterNode(String appId,
                          int iteration)

getCurrentMasterNode

protected StringBuilder getCurrentMasterNode(String appId,
                                 int iteration)

getCurrentMasterSplitNode

protected StringBuilder getCurrentMasterSplitNode(String appId,
                                      int iteration)

getLastMasterNode

protected StringBuilder getLastMasterNode(String appId,
                              int iteration)

getRootNode

protected StringBuilder getRootNode()

getBaseMasterElectionNode

protected StringBuilder getBaseMasterElectionNode(String appId)

getMasterElectionNode

protected StringBuilder getMasterElectionNode(String appId,
                                  long sessionId)

getAppNode

protected StringBuilder getAppNode(String appId)

getWorkerBaseNode

protected StringBuilder getWorkerBaseNode(String appId)

getWorkerBaseNode

protected StringBuilder getWorkerBaseNode(String appId,
                              int iteration)

getWorkerNode

protected StringBuilder getWorkerNode(String appId,
                          String containerId,
                          int iteration)

getCurrentWorkerNode

protected StringBuilder getCurrentWorkerNode(String appId,
                                 String containerId,
                                 int iteration)

getCurrentWorkerSplitNode

protected StringBuilder getCurrentWorkerSplitNode(String appId,
                                      String containerId,
                                      int iteration)

getLastWorkerNode

protected StringBuilder getLastWorkerNode(String appId,
                              String containerId,
                              int iteration)

initialize

protected void initialize(Properties props)

Coordinator initialization.

checkAndSetZooKeeper

protected void checkAndSetZooKeeper(Properties props)

Set up connection with given zookeeper settings.

close

protected void close()
              throws InterruptedException

Close resources like zookeeper, thread pool

Throws:

InterruptedException

startHeartbeat

protected void startHeartbeat()

stopHeartBeat

protected void stopHeartBeat()
                      throws InterruptedException

Throws:

InterruptedException

process

public void process(org.apache.zookeeper.WatchedEvent event)

Specified by:

process in interface org.apache.zookeeper.Watcher

setBytesToZNode

protected boolean setBytesToZNode(String znode,
                      String splitZnode,
                      byte[] bytes,
                      org.apache.zookeeper.CreateMode createNode)
                           throws org.apache.zookeeper.KeeperException,
                                  InterruptedException

Set bytes to znode, if bytes is over zookeeper data limit(1MB), use children znodes to store each part.

Returns:

if result is split.

Throws:

org.apache.zookeeper.KeeperException

InterruptedException

getBytesFromZNode

protected byte[] getBytesFromZNode(String znode,
                       String splitZnode)
                            throws org.apache.zookeeper.KeeperException,
                                   InterruptedException

This is reverse method to setBytesToZNode(String, String, byte[], CreateMode). Firstly get data from znode. If data is empty, get data from its children.

Throws:

org.apache.zookeeper.KeeperException

InterruptedException

getZooKeeper

public GuaguaZooKeeper getZooKeeper()

setZooKeeper

public void setZooKeeper(GuaguaZooKeeper zooKeeper)

getSleepTime

public long getSleepTime()

setSleepTime

public void setSleepTime(long sleepTime)

isFixedTime

public boolean isFixedTime()

setFixedTime

public void setFixedTime(boolean isFixedTime)

getWorkerSerializer

public Serializer<WORKER_RESULT> getWorkerSerializer()

setWorkerSerializer

public void setWorkerSerializer(Serializer<WORKER_RESULT> workerSerializer)

getMasterSerializer

public Serializer<MASTER_RESULT> getMasterSerializer()

setMasterSerializer

public void setMasterSerializer(Serializer<MASTER_RESULT> masterSerializer)

getZkConnLatch

public CountDownLatch getZkConnLatch()