A B C D E F G H I J L M N O P R S T U V W Z

A

AbstractCombineWorkerComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable,KEY extends Bytable,VALUE extends Bytable> - Class in ml.shifu.guagua.worker

A high-effective implementation to load data and do computation.

AbstractCombineWorkerComputable() - Constructor for class ml.shifu.guagua.worker.AbstractCombineWorkerComputable

AbstractCombineWorkerComputable(boolean) - Constructor for class ml.shifu.guagua.worker.AbstractCombineWorkerComputable

AbstractMasterCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

AbstractMasterCoordinator has some common implementation for both async and sync worker coordinator.

AbstractMasterCoordinator() - Constructor for class ml.shifu.guagua.master.AbstractMasterCoordinator

AbstractMasterCoordinator.FailOverCommand - Class in ml.shifu.guagua.master

AbstractMasterCoordinator.FailOverCommand is used to read last iteration before task failed.

AbstractMasterCoordinator.FailOverCommand(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Constructor for class ml.shifu.guagua.master.AbstractMasterCoordinator.FailOverCommand

AbstractMasterCoordinator.MasterElectionCommand - Class in ml.shifu.guagua.master

Elect master from several backup masters.

AbstractMasterCoordinator.MasterElectionCommand(String) - Constructor for class ml.shifu.guagua.master.AbstractMasterCoordinator.MasterElectionCommand

AbstractWorkerComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable,KEY extends Bytable,VALUE extends Bytable> - Class in ml.shifu.guagua.worker

Abstract WorkerComputable implementation to load data one by one and only in the very 1st iteration.

AbstractWorkerComputable() - Constructor for class ml.shifu.guagua.worker.AbstractWorkerComputable

AbstractWorkerCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

AbstractWorkerCoordinator has some common implementation for both async and sync worker coordinator.

AbstractWorkerCoordinator() - Constructor for class ml.shifu.guagua.worker.AbstractWorkerCoordinator

AbstractWorkerCoordinator.FailOverCoordinatorCommand - Class in ml.shifu.guagua.worker

AbstractWorkerCoordinator.FailOverCoordinatorCommand(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Constructor for class ml.shifu.guagua.worker.AbstractWorkerCoordinator.FailOverCoordinatorCommand

addCompletionCallBack(MasterContext.MasterCompletionCallBack<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MasterContext

Add callback function.

addCompletionCallBack(WorkerContext.WorkerCompletionCallBack<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.WorkerContext

Add callback function.

addMasterInterceptors(MasterInterceptor<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.GuaguaMasterService

append(T) - Method in interface ml.shifu.guagua.util.AppendList

Append object to this list

append(T) - Method in class ml.shifu.guagua.util.BytableDiskList

append(T) - Method in class ml.shifu.guagua.util.BytableMemoryDiskList

append(T) - Method in class ml.shifu.guagua.util.DiskList

append(T) - Method in class ml.shifu.guagua.util.MemoryDiskList

append(T) - Method in class ml.shifu.guagua.util.MemoryLimitedList

AppendList<T> - Interface in ml.shifu.guagua.util

Appendable list interface definition.

AppendList.State - Enum in ml.shifu.guagua.util

Inner state definition.

ARGS - Static variable in class ml.shifu.guagua.util.JMap

Arguments to pass in to command

AsyncCoordinatorFactory<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua

Async coordinator factory to create async coordinators for master and workers.

AsyncCoordinatorFactory() - Constructor for class ml.shifu.guagua.AsyncCoordinatorFactory

AsyncMasterCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

AsyncMasterCoordinator is used to as a barrier for each iteration.

AsyncMasterCoordinator() - Constructor for class ml.shifu.guagua.master.AsyncMasterCoordinator

AsyncWorkerCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

AsyncWorkerCoordinator is used to as a worker barrier for each iteration.

AsyncWorkerCoordinator() - Constructor for class ml.shifu.guagua.worker.AsyncWorkerCoordinator

awaitMaster(int) - Method in class ml.shifu.guagua.InMemoryCoordinator

Workers wait for master on each iteration condition

awaitWorkers(int) - Method in class ml.shifu.guagua.InMemoryCoordinator

Master waits for workers on each iteration condition

B

BasicCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua

BasicCoordinator is a basic implementation for both SyncWorkerCoordinator and SyncMasterCoordinator: 1.

BasicCoordinator() - Constructor for class ml.shifu.guagua.BasicCoordinator

BasicCoordinator.BasicCoordinatorCommand - Class in ml.shifu.guagua

BasicCoordinator.BasicCoordinatorCommand is to process exceptions for zookeeper operations.

BasicCoordinator.BasicCoordinatorCommand() - Constructor for class ml.shifu.guagua.BasicCoordinator.BasicCoordinatorCommand

BasicCoordinator.CoordinatorCommand - Interface in ml.shifu.guagua

BasicCoordinator.CoordinatorCommand is used for consistent process of zookeeper coordination.

BasicCoordinator.RetryCoordinatorCommand - Class in ml.shifu.guagua

BasicCoordinator.RetryCoordinatorCommand is used to wrap retry logic.

BasicCoordinator.RetryCoordinatorCommand(boolean, long) - Constructor for class ml.shifu.guagua.BasicCoordinator.RetryCoordinatorCommand

BasicMasterInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

Empty MasterInterceptor implementation for user to choose which function should be override.

BasicMasterInterceptor() - Constructor for class ml.shifu.guagua.master.BasicMasterInterceptor

BasicWorkerInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

Empty WorkerInterceptor implementation for user to choose which function should be override.

BasicWorkerInterceptor() - Constructor for class ml.shifu.guagua.worker.BasicWorkerInterceptor

Bytable - Interface in ml.shifu.guagua.io

Bytable defines master and worker results.

BytableDiskList<T extends Bytable> - Class in ml.shifu.guagua.util

A list wrapper to implement store Bytable data into disk.

BytableDiskList(String, String) - Constructor for class ml.shifu.guagua.util.BytableDiskList

Constructor with file name in current working dir and instance class name.

BytableDiskList(String, String, BytableSerializer<T>) - Constructor for class ml.shifu.guagua.util.BytableDiskList

Constructor with file name in current working dir, class name and serializer instance.

BytableMemoryDiskList<T extends Bytable> - Class in ml.shifu.guagua.util

A list to store Bytable data firstly into memory then into disk if over threshold.

BytableMemoryDiskList(long, List<T>, String) - Constructor for class ml.shifu.guagua.util.BytableMemoryDiskList

Constructor with max bytes size of memory and BytableMemoryDiskList.delegationList.

BytableMemoryDiskList(List<T>, String) - Constructor for class ml.shifu.guagua.util.BytableMemoryDiskList

Constructor with only memory BytableMemoryDiskList.delegationList.

BytableMemoryDiskList(long, List<T>, String, String) - Constructor for class ml.shifu.guagua.util.BytableMemoryDiskList

Constructor with max bytes size of memory and BytableMemoryDiskList.delegationList and disk file name in current working dir.

BytableMemoryDiskList(long, String, String) - Constructor for class ml.shifu.guagua.util.BytableMemoryDiskList

Constructor with max bytes size of memory and disk file name in current working dir.

BytableMemoryDiskList(long, BytableDiskList<T>) - Constructor for class ml.shifu.guagua.util.BytableMemoryDiskList

Constructor with memory bytes size limit and bytableDiskList setting.

BytableMemoryDiskList(long, String) - Constructor for class ml.shifu.guagua.util.BytableMemoryDiskList

Constructor with only memory bytes size limit and className.

BytableMemoryDiskList(String) - Constructor for class ml.shifu.guagua.util.BytableMemoryDiskList

Default constructor.

BytableSerializer<RESULT extends Bytable> - Class in ml.shifu.guagua.io

BytableSerializer is used to write Bytable object to bytes and read bytes to Bytable object.

BytableSerializer() - Constructor for class ml.shifu.guagua.io.BytableSerializer

BytableWrapper - Class in ml.shifu.guagua.io

A Bytable wrapper to wrap some useful information to netty server and client communication.

BytableWrapper() - Constructor for class ml.shifu.guagua.io.BytableWrapper

bytesToObject(byte[], String) - Method in class ml.shifu.guagua.io.BytableSerializer

De-serialize from bytes to object.

bytesToObject(byte[], String) - Method in class ml.shifu.guagua.io.Bzip2BytableSerializer

De-serialize from bytes to object.

bytesToObject(byte[], String) - Method in class ml.shifu.guagua.io.GZIPBytableSerializer

De-serialize from bytes to object.

bytesToObject(byte[], String) - Method in interface ml.shifu.guagua.io.Serializer

Transform byte array to object.

Bzip2BytableSerializer<RESULT extends Bytable> - Class in ml.shifu.guagua.io

Bzip2BytableSerializer is using bzip2 compression algorithm for serialization.

Bzip2BytableSerializer() - Constructor for class ml.shifu.guagua.io.Bzip2BytableSerializer

C

callback(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.master.MasterContext.MasterCompletionCallBack

Callback function which will be called at the end of last iteration of master computation.

callback(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.worker.WorkerContext.WorkerCompletionCallBack

Callback function which will be called at the end of last iteration of worker computation.

checkAndSetZooKeeper(Properties) - Method in class ml.shifu.guagua.BasicCoordinator

Set up connection with given zookeeper settings.

checkIfEmbedZooKeeperStarted(int) - Static method in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Wait for a time period until zookeeper server is started.

checkServers(String) - Static method in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Check zookeeper servers, if one of them is alive, return true;

cleanDirectory(File) - Static method in class ml.shifu.guagua.util.FileUtils

Cleans a directory without deleting it.

cleanup(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.HookMasterComputable

Cleanup method for master computation.

cleanup(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.HookWorkerComputable

Cleanup method for master computation.

clear() - Method in interface ml.shifu.guagua.util.AppendList

Clear all elements.

clear() - Method in class ml.shifu.guagua.util.BytableDiskList

Delete file to store elements.

clear() - Method in class ml.shifu.guagua.util.BytableMemoryDiskList

clear() - Method in class ml.shifu.guagua.util.DiskList

clear() - Method in class ml.shifu.guagua.util.MemoryDiskList

clear() - Method in class ml.shifu.guagua.util.MemoryLimitedList

close() - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Close this client object.

close() - Method in interface ml.shifu.guagua.io.GuaguaRecordReader

Close resources at last, for example file descriptors.

close() - Method in class ml.shifu.guagua.util.BytableDiskList

This method should be called at the end of BytableDiskList usage to release file descriptors.

close() - Method in class ml.shifu.guagua.util.BytableMemoryDiskList

Close disk stream.

close() - Method in class ml.shifu.guagua.util.DiskList

This method should be called at the end of DiskList usage to release file descriptors.

close() - Method in class ml.shifu.guagua.util.MemoryDiskList

Close disk stream.

closeZooKeeper() - Method in class ml.shifu.guagua.BasicCoordinator

CMD - Static variable in class ml.shifu.guagua.util.JMap

The command to run

ComputableMonitor - Annotation Type in ml.shifu.guagua

Master and Worker computable maximal time out setting.

compute(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.HookMasterComputable

compute(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.master.MasterComputable

Master computation for each iteration.

compute(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractCombineWorkerComputable

compute(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractWorkerComputable

compute(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.HookWorkerComputable

compute(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.worker.WorkerComputable

Worker computation for each iteration.

CoordinatorFactory<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Interface in ml.shifu.guagua

Coordinator factory to create sync or async coordinator.

createExt(String, byte[], List<ACL>, CreateMode, boolean) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Provides a possibility of a creating a path consisting of more than one znode (not atomic).

createMasterCoordinator() - Method in class ml.shifu.guagua.AsyncCoordinatorFactory

createMasterCoordinator() - Method in interface ml.shifu.guagua.CoordinatorFactory

Create coordinator instance for master.

createMasterCoordinator() - Method in class ml.shifu.guagua.SyncCoordinatorFactory

createOnceExt(String, byte[], List<ACL>, CreateMode, boolean) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Create a znode if there is no other znode there

createOrSetExt(String, byte[], List<ACL>, CreateMode, boolean, int) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Create a znode.

createWorkerCoordinator() - Method in class ml.shifu.guagua.AsyncCoordinatorFactory

createWorkerCoordinator() - Method in interface ml.shifu.guagua.CoordinatorFactory

Create coordinator instance for worker.

createWorkerCoordinator() - Method in class ml.shifu.guagua.SyncCoordinatorFactory

D

decode(ChannelHandlerContext, Channel, ChannelBuffer) - Method in class ml.shifu.guagua.io.NettyBytableDecoder

DEFAULT_IO_BUFFER_SIZE - Static variable in class ml.shifu.guagua.GuaguaConstants

DEFAULT_MASTER_NUMBER - Static variable in class ml.shifu.guagua.GuaguaConstants

deleteDirectory(File) - Static method in class ml.shifu.guagua.util.FileUtils

Deletes a directory recursively.

deleteExt(String, int, boolean) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Delete a path recursively.

deleteQuietly(File) - Static method in class ml.shifu.guagua.util.FileUtils

Deletes a file, never throwing an exception.

deserialize(byte[], Class<?>) - Method in class ml.shifu.guagua.util.JavaObjectSerializer

deserialize(byte[], Class<?>) - Method in interface ml.shifu.guagua.util.ObjectSerializer

De-serialize byte array to an instance.

DiskList<T extends Serializable> - Class in ml.shifu.guagua.util

A list wrapper to implement store data into disk.

DiskList(String) - Constructor for class ml.shifu.guagua.util.DiskList

Constructor with file name in current working dir.

doCompute(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.HookMasterComputable

Real computation logic excluding some setup and cleanup hooks.

doCompute(KEY, VALUE, WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractCombineWorkerComputable

Computation by each record, all update can be set to WORKER_RESULT by context.setCurrentWorkerResult(WORKER_RESULT);

doCompute(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractWorkerComputable

Real computation logic after data loading.

doCompute(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.HookWorkerComputable

Real computation logic excluding some setup and cleanup hooks.

doExecute() - Method in class ml.shifu.guagua.BasicCoordinator.BasicCoordinatorCommand

Real method to do coordinator operation.

doExecute() - Method in class ml.shifu.guagua.BasicCoordinator.RetryCoordinatorCommand

doExecute() - Method in class ml.shifu.guagua.master.AbstractMasterCoordinator.FailOverCommand

doExecute() - Method in class ml.shifu.guagua.master.AbstractMasterCoordinator.MasterElectionCommand

doExecute() - Method in class ml.shifu.guagua.worker.AbstractWorkerCoordinator.FailOverCoordinatorCommand

doReadFields(DataInput) - Method in class ml.shifu.guagua.io.HaltBytable

Read fields from in stream.

doRun() - Method in class ml.shifu.guagua.unit.GuaguaUnitDriver

Real running logic

doWrite(DataOutput) - Method in class ml.shifu.guagua.io.HaltBytable

Write fields to out stream.

E

encode(ChannelHandlerContext, Channel, Object) - Method in class ml.shifu.guagua.io.NettyBytableEncoder

estimate(Object) - Static method in class ml.shifu.guagua.util.SizeEstimator

Estimate size of one object instance.

execute() - Method in class ml.shifu.guagua.BasicCoordinator.BasicCoordinatorCommand

execute() - Method in interface ml.shifu.guagua.BasicCoordinator.CoordinatorCommand

Command method.

execute() - Method in interface ml.shifu.guagua.coordinator.zk.GuaguaZooKeeperOperation

Performs the operation - which may be involved multiple times if the connection to ZooKeeper closes during this operation.

exists(String, boolean) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Return the stat of the node of the given path.

exists(String, Watcher) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Return the stat of the node of the given path.

extractFrame(ChannelBuffer, int, int) - Method in class ml.shifu.guagua.io.NettyBytableDecoder

F

FileUtils - Class in ml.shifu.guagua.util

Copied from apache common-io source code.

filter(String) - Method in interface ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper.Filter

findContainingJar(Class) - Static method in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Find a jar that contains a class of the same name, if any.

forceDelete(File) - Static method in class ml.shifu.guagua.util.FileUtils

Deletes a file.

forceDeleteOnExit(File) - Static method in class ml.shifu.guagua.util.FileUtils

Schedules a file to be deleted when JVM exits.

freeMemoryMB() - Static method in class ml.shifu.guagua.util.MemoryUtils

Get free memory in megabytes

G

GcMasterInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

GcMasterInterceptor is used to log gc time of preXXXX of interceptors, master computation and postXXXX of interceptors.

GcMasterInterceptor() - Constructor for class ml.shifu.guagua.master.GcMasterInterceptor

GcWorkerInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

GcWorkerInterceptor is used to log gc time of preXXXX of interceptors, worker computation and postXXXX of interceptors.

GcWorkerInterceptor() - Constructor for class ml.shifu.guagua.worker.GcWorkerInterceptor

generateWorkerSplits(String) - Method in class ml.shifu.guagua.unit.GuaguaUnitDriver

Generate file splits according to inputs.

get(String) - Static method in class ml.shifu.guagua.util.StringUtils

Return string object.

get(String, boolean) - Static method in class ml.shifu.guagua.util.StringUtils

Return string object.

get(String, String) - Static method in class ml.shifu.guagua.util.StringUtils

Return string object.

get(String, boolean, String) - Static method in class ml.shifu.guagua.util.StringUtils

Return string object.

getAppId() - Method in class ml.shifu.guagua.master.AsyncMasterCoordinator

getAppId() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getAppId() - Method in class ml.shifu.guagua.master.MasterContext

getAppId() - Method in class ml.shifu.guagua.worker.AsyncWorkerCoordinator

getAppId() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

getAppId() - Method in class ml.shifu.guagua.worker.WorkerContext

getAppNode(String) - Method in class ml.shifu.guagua.BasicCoordinator

getAttachment() - Method in class ml.shifu.guagua.master.MasterContext

getAttachment() - Method in class ml.shifu.guagua.worker.WorkerContext

getBaseMasterElectionNode(String) - Method in class ml.shifu.guagua.BasicCoordinator

getBytes() - Method in class ml.shifu.guagua.io.BytableWrapper

getBytesFromZNode(String, String) - Method in class ml.shifu.guagua.BasicCoordinator

This is reverse method to BasicCoordinator.setBytesToZNode(String, String, byte[], CreateMode).

getCallBackList() - Method in class ml.shifu.guagua.master.MasterContext

getCallBackList() - Method in class ml.shifu.guagua.worker.WorkerContext

getChildrenExt(String, boolean, boolean, Comparator<String>) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Get the children of the path with extensions.

getChildrenExt(String, boolean, boolean, boolean) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Get the children of the path with extensions.

getChildrenExt(String, boolean, boolean, boolean, GuaguaZooKeeper.Filter) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Get the children of the path with extensions.

getContainerId() - Method in class ml.shifu.guagua.io.BytableWrapper

getContainerId() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getContainerId() - Method in class ml.shifu.guagua.master.MasterContext

getContainerId() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

getContainerId() - Method in class ml.shifu.guagua.worker.WorkerContext

getContext() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getCoordinator() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getCoordinator() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

getCurrentIteration() - Method in class ml.shifu.guagua.io.BytableWrapper

getCurrentIteration() - Method in class ml.shifu.guagua.master.AsyncMasterCoordinator

getCurrentIteration() - Method in class ml.shifu.guagua.master.MasterContext

getCurrentIteration() - Method in class ml.shifu.guagua.worker.AsyncWorkerCoordinator

getCurrentIteration() - Method in class ml.shifu.guagua.worker.WorkerContext

getCurrentKey() - Method in interface ml.shifu.guagua.io.GuaguaRecordReader

Tmp we only support LongWritable key for byte offset in each line, follow LineRecordReader in hadoop.

getCurrentMasterNode(String, int) - Method in class ml.shifu.guagua.BasicCoordinator

getCurrentMasterSplitNode(String, int) - Method in class ml.shifu.guagua.BasicCoordinator

getCurrentValue() - Method in interface ml.shifu.guagua.io.GuaguaRecordReader

Tmp we only support Text value for whole content in each line, follow LineRecordReader in hadoop.

getCurrentWorkerNode(String, String, int) - Method in class ml.shifu.guagua.BasicCoordinator

getCurrentWorkerSplitNode(String, String, int) - Method in class ml.shifu.guagua.BasicCoordinator

getData(String, Watcher, Stat) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Return the data and the stat of the node of the given path.

getData(String, boolean, Stat) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Return the data and the stat of the node of the given path.

getDouble(String, double) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to double type.

getDouble(String) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to double type.

getDouble(String, boolean) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to double type.

getDouble(String, boolean, double) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to double type.

getElapsedTime() - Method in class ml.shifu.guagua.BasicCoordinator.RetryCoordinatorCommand

getFileSplits() - Method in class ml.shifu.guagua.worker.WorkerContext

getFloat(String, float) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to float type.

getFloat(String) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to float type.

getFloat(String, boolean) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to float type.

getFloat(String, boolean, float) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to float type.

getInt(String, int) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to int type.

getInt(String) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to int type.

getInt(String, boolean) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to int type.

getInt(String, boolean, int) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to int type.

getLastMasterNode(String, int) - Method in class ml.shifu.guagua.BasicCoordinator

getLastMasterResult() - Method in class ml.shifu.guagua.worker.WorkerContext

getLastWorkerNode(String, String, int) - Method in class ml.shifu.guagua.BasicCoordinator

getLength() - Method in class ml.shifu.guagua.io.GuaguaFileSplit

getLong(String, long) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to long type.

getLong(String) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to long type.

getLong(String, boolean) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to long type.

getLong(String, boolean, long) - Static method in class ml.shifu.guagua.util.NumberFormatUtils

Parse string object to long type.

getMasterBaseNode(String) - Method in class ml.shifu.guagua.BasicCoordinator

getMasterComputable() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getMasterElectionNode(String, long) - Method in class ml.shifu.guagua.BasicCoordinator

getMasterInterceptors() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getMasterNode(String, int) - Method in class ml.shifu.guagua.BasicCoordinator

getMasterResult() - Method in class ml.shifu.guagua.InMemoryCoordinator

getMasterResult() - Method in class ml.shifu.guagua.master.MasterContext

getMasterResultClassName() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getMasterResultClassName() - Method in class ml.shifu.guagua.master.MasterContext

getMasterResultClassName() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

getMasterResultClassName() - Method in class ml.shifu.guagua.worker.WorkerContext

getMasterSerializer() - Method in class ml.shifu.guagua.BasicCoordinator

getMaxRetryAttempts() - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

getMethod(Class<?>, String) - Static method in class ml.shifu.guagua.util.ReflectionUtils

Just to check if has method with methodName.

getMinWorkersRatio() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getMinWorkersRatio() - Method in class ml.shifu.guagua.master.MasterContext

getMinWorkersTimeOut() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getMinWorkersTimeOut() - Method in class ml.shifu.guagua.master.MasterContext

getMyBid() - Method in class ml.shifu.guagua.master.AbstractMasterCoordinator

getOffset() - Method in class ml.shifu.guagua.io.GuaguaFileSplit

getPath() - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper.PathStat

Get the path of the created znode if it was created.

getPath() - Method in class ml.shifu.guagua.io.GuaguaFileSplit

getProcessId() - Static method in class ml.shifu.guagua.util.JMap

Get the process ID of the current running process

getProps() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getProps() - Method in class ml.shifu.guagua.master.MasterContext

getProps() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

getProps() - Method in class ml.shifu.guagua.worker.WorkerContext

getRecordReader() - Method in class ml.shifu.guagua.worker.AbstractCombineWorkerComputable

getRecordReader() - Method in class ml.shifu.guagua.worker.AbstractWorkerComputable

getRetryWaitMsecs() - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

getRootNode() - Method in class ml.shifu.guagua.BasicCoordinator

getRuntimeMemoryStats() - Static method in class ml.shifu.guagua.util.MemoryUtils

Get runtime memory stats

getSerializer() - Method in class ml.shifu.guagua.util.BytableDiskList

getSerializer() - Method in class ml.shifu.guagua.util.DiskList

getSleepTime() - Method in class ml.shifu.guagua.BasicCoordinator

getSplits() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

getStat() - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper.PathStat

Get the stat of the set znode if set

getTempDirectory() - Static method in class ml.shifu.guagua.util.FileUtils

Returns a File representing the system temporary directory.

getTempDirectoryPath() - Static method in class ml.shifu.guagua.util.FileUtils

Returns the path to the system temporary directory.

getTotalIteration() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getTotalIteration() - Method in class ml.shifu.guagua.master.MasterContext

getTotalIteration() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

getTotalIteration() - Method in class ml.shifu.guagua.worker.WorkerContext

getUserDirectory() - Static method in class ml.shifu.guagua.util.FileUtils

Returns a File representing the user's home directory.

getUserDirectoryPath() - Static method in class ml.shifu.guagua.util.FileUtils

Returns the path to the user's home directory.

getValidServerPort(int) - Static method in class ml.shifu.guagua.util.NetworkUtils

Get valid TCP server port.

getValidZooKeeperPort() - Static method in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Check from DEFAULT_ZK_PORT to (DEFAULT_ZK_PORT + TRY_PORT_COUNT) to see if there is valid port to launch embed zookeeper server.

getWorkerBaseNode(String) - Method in class ml.shifu.guagua.BasicCoordinator

getWorkerBaseNode(String, int) - Method in class ml.shifu.guagua.BasicCoordinator

getWorkerComputable() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

getWorkerInterceptors() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

getWorkerNode(String, String, int) - Method in class ml.shifu.guagua.BasicCoordinator

getWorkerResult() - Method in class ml.shifu.guagua.worker.WorkerContext

getWorkerResultClassName() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getWorkerResultClassName() - Method in class ml.shifu.guagua.master.MasterContext

getWorkerResultClassName() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

getWorkerResultClassName() - Method in class ml.shifu.guagua.worker.WorkerContext

getWorkerResults() - Method in class ml.shifu.guagua.InMemoryCoordinator

getWorkerResults() - Method in class ml.shifu.guagua.master.MasterContext

getWorkers() - Method in class ml.shifu.guagua.master.GuaguaMasterService

getWorkers() - Method in class ml.shifu.guagua.master.MasterContext

getWorkerSerializer() - Method in class ml.shifu.guagua.BasicCoordinator

getZkConnLatch() - Method in class ml.shifu.guagua.BasicCoordinator

getZooKeeper() - Method in class ml.shifu.guagua.BasicCoordinator

getZooKeeper() - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

GUAGUA - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_CHILD_DEFAULT_ZKSERVER_OPTS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_CHILD_ZKSERVER_OPTS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_CLEANUP_INTERVAL - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_CLUSTER_ZOOKEEPER_SERVER_FILE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_COMPUTATION_TIME_THRESHOLD - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_COORDINATOR_FIXED_SLEEP - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_COORDINATOR_FIXED_SLEEP_ENABLE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_COORDINATOR_SLEEP_UNIT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_DEFAULT_CLEANUP_INTERVAL - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_DEFAULT_COMPUTATION_TIME_THRESHOLD - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_DEFAULT_ITERATION_COUNT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_DEFAULT_MIN_WORKERS_RATIO - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_DEFAULT_MIN_WORKERS_TIMEOUT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_FIRST_ITERATION - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_INIT_DONE_NODE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_INIT_STEP - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_INPUT_DIR - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_INTERCEPTOR_SEPARATOR - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_IO_DEFAULT_SERIALIZER - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ITERATION_COUNT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MASTER_DEFAULT_SYSTEM_INTERCEPTERS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MASTER_ELECTION - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MASTER_INTERCEPTERS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MASTER_IO_SERIALIZER - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MASTER_NUMBER - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MASTER_RESULT_CLASS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MASTER_SYSTEM_INTERCEPTERS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MASTER_WORKERESULTS_DEFAULT_MEMORY_FRACTION - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MASTER_WORKERESULTS_MEMORY_FRACTION - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MIN_WORKERS_RATIO - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_MIN_WORKERS_TIMEOUT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_NETTY_SERVER_DEFAULT_THREAD_COUNT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_NETTY_SEVER_DEFAULT_PORT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_NETTY_SEVER_PORT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_SITE_FILE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_SPLIT_COMBINABLE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_SPLIT_MAX_COMBINED_SPLIT_SIZE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_STRAGGLER_ITERATORS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_WORKER_DEFAULT_HALT_ENABLE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_WORKER_DEFAULT_SYSTEM_INTERCEPTERS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_WORKER_HALT_ENABLE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_WORKER_INTERCEPTERS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_WORKER_IO_SERIALIZER - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_WORKER_NUMBER - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_WORKER_RESULT_CLASS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_WORKER_SYSTEM_INTERCEPTERS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_CLEANUP_ENABLE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_CLUSTER_SERVER_FOLDER - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_DATA_LIMIT - Static variable in class ml.shifu.guagua.GuaguaConstants

Zookeeper data limit is 1M, use 1023K for safety.

GUAGUA_ZK_DEFAULT_CLEANUP_VALUE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_DEFAULT_MAX_ATTEMPTS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_DEFAULT_RETRY_WAIT_MILLS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_EMBEDBED_IS_IN_CLIENT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_HEARTBEAT_ENABLED - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_MASTER_NODE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_MAX_ATTEMPTS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_RETRY_WAIT_MILLS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_ROOT_NODE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_SERVERS - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_SESSION_TIMEOUT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_SESSON_DEFAULT_TIMEOUT - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_SPLIT_NODE - Static variable in class ml.shifu.guagua.GuaguaConstants

GUAGUA_ZK_WORKERS_NODE - Static variable in class ml.shifu.guagua.GuaguaConstants

GuaguaConstants - Class in ml.shifu.guagua

GuaguaException - Exception in ml.shifu.guagua

GuaguaException is checked exception in guagua framework.

GuaguaException() - Constructor for exception ml.shifu.guagua.GuaguaException

Constructs an GuaguaException with null as its error detail message.

GuaguaException(String) - Constructor for exception ml.shifu.guagua.GuaguaException

Constructs an GuaguaException with the specified detail message.

GuaguaException(String, Throwable) - Constructor for exception ml.shifu.guagua.GuaguaException

Constructs an GuaguaException with the specified detail message and cause.

GuaguaException(Throwable) - Constructor for exception ml.shifu.guagua.GuaguaException

Constructs an GuaguaException with the specified cause and a detail message of (cause==null ? null : cause.toString()) (which typically contains the class and detail message of cause).

GuaguaFileSplit - Class in ml.shifu.guagua.io

GuaguaFileSplit is used for WorkerComputable to read data.

GuaguaFileSplit() - Constructor for class ml.shifu.guagua.io.GuaguaFileSplit

Default constructor.

GuaguaFileSplit(String, long, long) - Constructor for class ml.shifu.guagua.io.GuaguaFileSplit

Constructor which is used to build full GuaguaFileSplit.

GuaguaMasterService<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

GuaguaMasterService is the basic implementation as a master for whole guagua application process.

GuaguaMasterService() - Constructor for class ml.shifu.guagua.master.GuaguaMasterService

GuaguaRecordReader<KEY extends Bytable,VALUE extends Bytable> - Interface in ml.shifu.guagua.io

GuaguaRecordReader is used for consistent interface to iterate data through FileSplit provided.

GuaguaRuntimeException - Exception in ml.shifu.guagua

GuaguaRuntimeException which is used to wrap all RuntimeExceptions in guagua framework.

GuaguaRuntimeException() - Constructor for exception ml.shifu.guagua.GuaguaRuntimeException

Constructs a new guagua runtime exception with null as its detail message.

GuaguaRuntimeException(String) - Constructor for exception ml.shifu.guagua.GuaguaRuntimeException

Constructs a new guagua runtime exception with the specified detail message.

GuaguaRuntimeException(String, Throwable) - Constructor for exception ml.shifu.guagua.GuaguaRuntimeException

Constructs a new guagua runtime exception with the specified detail message and cause.

GuaguaRuntimeException(Throwable) - Constructor for exception ml.shifu.guagua.GuaguaRuntimeException

Constructs a new guagua runtime exception with the specified cause and a detail message of (cause==null ? null : cause.toString()) (which typically contains the class and detail message of cause).

GuaguaService - Interface in ml.shifu.guagua

GuaguaService is a common service interface of both master and worker service implementations.

GuaguaUnitDriver<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.unit

GuaguaUnitDriver is a helper class to run master, worker and intercepters in one jvm instance.

GuaguaUnitDriver(Properties) - Constructor for class ml.shifu.guagua.unit.GuaguaUnitDriver

Constructor with GuaguaUnitDriver.props setting.

GuaguaWorkerService<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

GuaguaWorkerService is the basic implementation as a worker for whole application process.

GuaguaWorkerService() - Constructor for class ml.shifu.guagua.worker.GuaguaWorkerService

GuaguaZooKeeper - Class in ml.shifu.guagua.coordinator.zk

ZooKeeper provides only atomic operations.

GuaguaZooKeeper(String, int, int, int, Watcher) - Constructor for class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Constructor to connect to ZooKeeper, make progress

GuaguaZooKeeper.Filter - Interface in ml.shifu.guagua.coordinator.zk

Filter path out GuaguaZooKeeper.Filter.filter(String) return true.

GuaguaZooKeeper.PathStat - Class in ml.shifu.guagua.coordinator.zk

Data structure for handling the output of createOrSet()

GuaguaZooKeeper.PathStat(String, Stat) - Constructor for class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper.PathStat

Put in results from createOrSet()

GuaguaZooKeeperOperation<T> - Interface in ml.shifu.guagua.coordinator.zk

GuaguaZooKeeperOperation is used as command to process consistent logic like re-trying for all zookeeper operations (create, get, ...).

GZIPBytableSerializer<RESULT extends Bytable> - Class in ml.shifu.guagua.io

GZIPBytableSerializer is using gzip compression algorithm for serialization.

GZIPBytableSerializer() - Constructor for class ml.shifu.guagua.io.GZIPBytableSerializer

H

HaltBytable - Class in ml.shifu.guagua.io

Abstract Bytable implementation to read and store halt status.

HaltBytable() - Constructor for class ml.shifu.guagua.io.HaltBytable

hasEmptyParameterConstructor(Class<?>) - Static method in class ml.shifu.guagua.util.ReflectionUtils

Check if there is empty-parameter constructor in one clazz.

heapHistogramDump(int) - Static method in class ml.shifu.guagua.util.JMap

Run jmap, print numLines of output from it to stderr.

heapHistogramDump(int, PrintStream) - Static method in class ml.shifu.guagua.util.JMap

Run jmap, print numLines of output from it to stream passed in.

HookMasterComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

Common MasterComputable abstraction to define HookMasterComputable.setup(MasterContext) and HookMasterComputable.cleanup(MasterContext) hooks.

HookMasterComputable() - Constructor for class ml.shifu.guagua.master.HookMasterComputable

HookWorkerComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

Common MasterComputable abstraction to define HookWorkerComputable.setup(WorkerContext) and HookWorkerComputable.cleanup(WorkerContext) hooks.

HookWorkerComputable() - Constructor for class ml.shifu.guagua.worker.HookWorkerComputable

I

init(Properties) - Method in interface ml.shifu.guagua.GuaguaService

Service initialization.

init(Properties) - Method in class ml.shifu.guagua.master.GuaguaMasterService

Parameter initialization, such as set result class name, iteration number and so on.

init(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractCombineWorkerComputable

Initialization work for the whole computation

init(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractWorkerComputable

Initialization work for the whole computation

init(Properties) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

Parameter initialization, such as set result class name, iteration number and so on.

INITAL_ZK_PORT - Static variable in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Initial zookeeper port used for check valid zookeeper port.

initialize(Properties) - Method in class ml.shifu.guagua.BasicCoordinator

Coordinator initialization.

initialize(GuaguaFileSplit) - Method in interface ml.shifu.guagua.io.GuaguaRecordReader

Initialize file split for user to create relative reader instance.

initialize(Properties) - Method in class ml.shifu.guagua.master.NettyMasterCoordinator

initRecordReader(GuaguaFileSplit) - Method in class ml.shifu.guagua.worker.AbstractCombineWorkerComputable

Each GuaguaFileSplit must be initialized before loading data.

initRecordReader(GuaguaFileSplit) - Method in class ml.shifu.guagua.worker.AbstractWorkerComputable

Each GuaguaFileSplit must be initialized before loading data.

InMemoryCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua

InMemoryCoordinator is a helper for in-memory master and worker coordination.

InMemoryCoordinator(int, int) - Constructor for class ml.shifu.guagua.InMemoryCoordinator

isAllWorkersHalt(Iterable<WORKER_RESULT>) - Method in class ml.shifu.guagua.master.AbstractMasterCoordinator

Check whether all workers are halted.

isFirstIteration() - Method in class ml.shifu.guagua.master.MasterContext

Whether is in first iteration, default first iteration is 1.

isFirstIteration() - Method in class ml.shifu.guagua.worker.WorkerContext

Whether is in first iteration, default first iteration is 1.

isFixedTime() - Method in class ml.shifu.guagua.BasicCoordinator

isHalt() - Method in class ml.shifu.guagua.io.HaltBytable

Return halt status.

isInitIteration() - Method in class ml.shifu.guagua.master.MasterContext

Whether is in initiate iteration, default initiate iteration is 1.

isInitIteration() - Method in class ml.shifu.guagua.worker.WorkerContext

Whether is in initiate iteration, default initiate iteration is 1.

isServerAlive(InetAddress, int) - Static method in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Check whether a server is alive.

isServerAlive(String, int) - Static method in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Check whether a server is alive.

isServerAlive(InetAddress, int) - Static method in class ml.shifu.guagua.util.NetworkUtils

Check whether a server is alive.

isStopMessage() - Method in class ml.shifu.guagua.io.BytableWrapper

isSymlink(File) - Static method in class ml.shifu.guagua.util.FileUtils

Determines whether the specified file is a Symbolic Link rather than an actual file.

isTerminated(int, int, double, long) - Method in class ml.shifu.guagua.BasicCoordinator.RetryCoordinatorCommand

Return true for BasicCoordinator.RetryCoordinatorCommand.retryExecution() if in minWorkersTimeout time get (int) (workers * minWorkersRatio)) workers completed.

iterate(MasterContext<MASTER_RESULT, WORKER_RESULT>, int, Progressable) - Method in class ml.shifu.guagua.master.GuaguaMasterService

Call each iteration computation and preIteration, postIteration in interceptors.

iterate(WorkerContext<MASTER_RESULT, WORKER_RESULT>, int, Progressable) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

Call each iteration computation and preIteration, postIteration in interceptors.

iterator() - Method in class ml.shifu.guagua.util.BytableDiskList

iterator() - Method in class ml.shifu.guagua.util.BytableMemoryDiskList

iterator() - Method in class ml.shifu.guagua.util.DiskList

iterator() - Method in class ml.shifu.guagua.util.MemoryDiskList

iterator() - Method in class ml.shifu.guagua.util.MemoryLimitedList

J

JavaObjectSerializer<T> - Class in ml.shifu.guagua.util

Default object serialization by using java serialization implementation.

JavaObjectSerializer() - Constructor for class ml.shifu.guagua.util.JavaObjectSerializer

JMap - Class in ml.shifu.guagua.util

Helper to run jmap and print the output.

JMap() - Constructor for class ml.shifu.guagua.util.JMap

Do not construct

L

load(KEY, VALUE, WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractWorkerComputable

Load data one by one before computation.

LocalMasterCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

LocalMasterCoordinator is local coordinator implementation in one jvm instance.

LocalMasterCoordinator() - Constructor for class ml.shifu.guagua.master.LocalMasterCoordinator

LocalWorkerCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

LocalWorkerCoordinator is local coordinator implementation in one jvm instance.

LocalWorkerCoordinator() - Constructor for class ml.shifu.guagua.worker.LocalWorkerCoordinator

M

main(String[]) - Static method in class ml.shifu.guagua.util.JMap

MASTER_COMPUTABLE_CLASS - Static variable in class ml.shifu.guagua.GuaguaConstants

MasterComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Interface in ml.shifu.guagua.master

MasterComputable defines master computation for any guagua application.

MasterComputableTimer<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

MasterComputableTimer is used to print execution time for master computation.

MasterComputableTimer() - Constructor for class ml.shifu.guagua.master.MasterComputableTimer

MasterContext<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

MasterContext is a context object which contains all useful info used in master computation.

MasterContext(int, int, Properties, String, String, String, String, double, long) - Constructor for class ml.shifu.guagua.master.MasterContext

MasterContext.MasterCompletionCallBack<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Interface in ml.shifu.guagua.master

Define callback for compeletion of master computation.

masterInitLock - Variable in class ml.shifu.guagua.worker.AsyncWorkerCoordinator

Lock is used to check register info from master.

MasterInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Interface in ml.shifu.guagua.master

MasterInterceptor defines hook points for each guagua application.

masterIterationLock - Variable in class ml.shifu.guagua.worker.AsyncWorkerCoordinator

Lock is used to check iteration info from master.

MasterTimer<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

MasterTimer is used to log execution time of preXXXX of interceptors, master computation and postXXXX of interceptors.

MasterTimer() - Constructor for class ml.shifu.guagua.master.MasterTimer

maxMemoryMB() - Static method in class ml.shifu.guagua.util.MemoryUtils

Get maximum memory in megabytes

MemoryDiskList<T extends Serializable> - Class in ml.shifu.guagua.util

A list to store data firstly into memory then into disk if over threshold.

MemoryDiskList(long, List<T>) - Constructor for class ml.shifu.guagua.util.MemoryDiskList

Constructor with max bytes size of memory and MemoryDiskList.delegationList.

MemoryDiskList(List<T>) - Constructor for class ml.shifu.guagua.util.MemoryDiskList

Constructor with only memory MemoryDiskList.delegationList.

MemoryDiskList(long, List<T>, String) - Constructor for class ml.shifu.guagua.util.MemoryDiskList

Constructor with max bytes size of memory and MemoryDiskList.delegationList and disk file name in current working dir.

MemoryDiskList(long, String) - Constructor for class ml.shifu.guagua.util.MemoryDiskList

Constructor with max bytes size of memory and disk file name in current working dir.

MemoryDiskList(long) - Constructor for class ml.shifu.guagua.util.MemoryDiskList

Constructor with only memory bytes size limit.

MemoryDiskList() - Constructor for class ml.shifu.guagua.util.MemoryDiskList

Default constructor.

MemoryLimitedList<T> - Class in ml.shifu.guagua.util

A simple wrapper list with limited byte size.

MemoryLimitedList(long, List<T>) - Constructor for class ml.shifu.guagua.util.MemoryLimitedList

Constructor with max bytes size limit and delegation list.

MemoryLimitedList(List<T>) - Constructor for class ml.shifu.guagua.util.MemoryLimitedList

Constructor with delegation list

MemoryStatsMasterInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

MemoryStatsMasterInterceptor is used to print memory usage for each master checkpoint.

MemoryStatsMasterInterceptor() - Constructor for class ml.shifu.guagua.master.MemoryStatsMasterInterceptor

MemoryStatsWorkerInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

MemoryStatsWorkerInterceptor is used to print memory usage for each worker checkpoint.

MemoryStatsWorkerInterceptor() - Constructor for class ml.shifu.guagua.worker.MemoryStatsWorkerInterceptor

MemoryUtils - Class in ml.shifu.guagua.util

Helper static methods for tracking memory usage.

ml.shifu.guagua - package ml.shifu.guagua

This package contains some common classes for both master and worker.

ml.shifu.guagua.coordinator.zk - package ml.shifu.guagua.coordinator.zk

This package contains zookeeper wrapper to talk with ZooKeeper servers.

ml.shifu.guagua.io - package ml.shifu.guagua.io

This package contains guagua own io settings.

ml.shifu.guagua.master - package ml.shifu.guagua.master

This package contains guagua master implementation.

ml.shifu.guagua.unit - package ml.shifu.guagua.unit

This package is used to run guagua master-workers in one jvm instance.

ml.shifu.guagua.util - package ml.shifu.guagua.util

This package contains some useful utility functions to help guagua-core.

ml.shifu.guagua.worker - package ml.shifu.guagua.worker

This package contains guagua worker implementation.

N

NettyBytableDecoder - Class in ml.shifu.guagua.io

A decoder to decode bytes array to Bytable object.

NettyBytableDecoder() - Constructor for class ml.shifu.guagua.io.NettyBytableDecoder

Default constructor with max object size 64M.

NettyBytableDecoder(int) - Constructor for class ml.shifu.guagua.io.NettyBytableDecoder

Creates a new decoder with the specified maximum object size.

NettyBytableEncoder - Class in ml.shifu.guagua.io

A encoder to encode Bytable instance to coded bytes array for netty pipeline.

NettyBytableEncoder() - Constructor for class ml.shifu.guagua.io.NettyBytableEncoder

Creates a new encoder with the estimated length of 64k bytes.

NettyBytableEncoder(int) - Constructor for class ml.shifu.guagua.io.NettyBytableEncoder

Creates a new encoder.

NettyMasterCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

A master coordinator to coordinate with workers through Netty server.

NettyMasterCoordinator() - Constructor for class ml.shifu.guagua.master.NettyMasterCoordinator

NettyWorkerCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

Worker coordinator coordinates master with Netty client channel.

NettyWorkerCoordinator() - Constructor for class ml.shifu.guagua.worker.NettyWorkerCoordinator

NetworkUtils - Class in ml.shifu.guagua.util

NetworkUtils is used as TCP server or port helper functions.

newInstance(Class<T>) - Static method in class ml.shifu.guagua.util.ReflectionUtils

Create an object for the given class.

newInstance(String) - Static method in class ml.shifu.guagua.util.ReflectionUtils

Create an object for the given class.

nextKeyValue() - Method in interface ml.shifu.guagua.io.GuaguaRecordReader

Cursor shift to next and set current key value.

NumberFormatUtils - Class in ml.shifu.guagua.util

Util class to parse int, long or any other type.

O

ObjectSerializer<T> - Interface in ml.shifu.guagua.util

To serialize object to byte array and de-serialize bytes to an instance.

objectToBytes(RESULT) - Method in class ml.shifu.guagua.io.BytableSerializer

Serialize from object to bytes.

objectToBytes(RESULT) - Method in class ml.shifu.guagua.io.Bzip2BytableSerializer

Serialize from object to bytes.

objectToBytes(RESULT) - Method in class ml.shifu.guagua.io.GZIPBytableSerializer

Serialize from object to bytes.

objectToBytes(RESULT) - Method in interface ml.shifu.guagua.io.Serializer

Transform object to byte array.

openInputStream(File) - Static method in class ml.shifu.guagua.util.FileUtils

Opens a FileInputStream for the specified file, providing better error messages than simply calling new FileInputStream(file).

P

populateZooKeeperConfFile(String, Map<String, String>) - Static method in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Create zoo.cfg file to launch a embed zookeeper server.

postApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.AbstractMasterCoordinator

postApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.BasicMasterInterceptor

postApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.GcMasterInterceptor

postApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MasterComputableTimer

postApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.master.MasterInterceptor

The hook point after any computation logic.

postApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MasterTimer

postApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MemoryStatsMasterInterceptor

postApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.NettyMasterCoordinator

Wait for unregister message for all workers and then clean all znodes existing for this job.

postApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.TraceMasterInterceptor

postApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractWorkerCoordinator

postApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.BasicWorkerInterceptor

postApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.GcWorkerInterceptor

postApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.MemoryStatsWorkerInterceptor

postApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.NettyWorkerCoordinator

Send stop message to master and then clean resources.

postApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.TraceWorkerInterceptor

postApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.WorkerComputableTimer

postApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.worker.WorkerInterceptor

The hook point after any computation logic.

postApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.WorkerTimer

postIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.AbstractMasterCoordinator

postIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.BasicMasterInterceptor

postIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.GcMasterInterceptor

postIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.LocalMasterCoordinator

postIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MasterComputableTimer

postIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.master.MasterInterceptor

The hook point after computation of each iteration.

postIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MasterTimer

postIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MemoryStatsMasterInterceptor

postIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.NettyMasterCoordinator

Update master computable result to master znode.

postIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.TraceMasterInterceptor

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AsyncWorkerCoordinator

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.BasicWorkerInterceptor

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.GcWorkerInterceptor

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.LocalWorkerCoordinator

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.MemoryStatsWorkerInterceptor

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.NettyWorkerCoordinator

Send worker results to master; wait for current master stop; get current master result.

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.SyncWorkerCoordinator

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.TraceWorkerInterceptor

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.WorkerComputableTimer

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.worker.WorkerInterceptor

The hook point after computation of each iteration.

postIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.WorkerTimer

postLoad(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractCombineWorkerComputable

Do some post work after loading data.

postLoad(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractWorkerComputable

Do some post work before loading data.

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.AsyncMasterCoordinator

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.BasicMasterInterceptor

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.GcMasterInterceptor

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.LocalMasterCoordinator

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MasterComputableTimer

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.master.MasterInterceptor

The hook point before any computation logic.

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MasterTimer

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MemoryStatsMasterInterceptor

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.NettyMasterCoordinator

Do initialization and fail-over checking before all iterations.

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.SyncMasterCoordinator

preApplication(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.TraceMasterInterceptor

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AsyncWorkerCoordinator

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.BasicWorkerInterceptor

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.GcWorkerInterceptor

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.LocalWorkerCoordinator

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.MemoryStatsWorkerInterceptor

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.NettyWorkerCoordinator

Worker coordinator initialization.

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.SyncWorkerCoordinator

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.TraceWorkerInterceptor

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.WorkerComputableTimer

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.worker.WorkerInterceptor

The hook point before any computation logic.

preApplication(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.WorkerTimer

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.AsyncMasterCoordinator

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.BasicMasterInterceptor

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.GcMasterInterceptor

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.LocalMasterCoordinator

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MasterComputableTimer

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.master.MasterInterceptor

The hook point before computation of each iteration.

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MasterTimer

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MemoryStatsMasterInterceptor

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.NettyMasterCoordinator

Wait for all workers done in current iteration.

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.SyncMasterCoordinator

preIteration(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.TraceMasterInterceptor

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractWorkerCoordinator

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AsyncWorkerCoordinator

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.BasicWorkerInterceptor

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.GcWorkerInterceptor

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.LocalWorkerCoordinator

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.MemoryStatsWorkerInterceptor

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.NettyWorkerCoordinator

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.TraceWorkerInterceptor

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.WorkerComputableTimer

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in interface ml.shifu.guagua.worker.WorkerInterceptor

The hook point before computation of each iteration.

preIteration(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.WorkerTimer

preLoad(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractCombineWorkerComputable

Do some pre work before loading data.

preLoad(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.AbstractWorkerComputable

Do some pre work before loading data.

prepZooKeeperConf(String, String) - Static method in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Create zookeeper file with specified name and client port setting.

process(WatchedEvent) - Method in class ml.shifu.guagua.BasicCoordinator

process(WatchedEvent) - Method in class ml.shifu.guagua.master.AsyncMasterCoordinator

process(WatchedEvent) - Method in class ml.shifu.guagua.worker.AsyncWorkerCoordinator

progress(int, int, String, boolean, boolean) - Method in interface ml.shifu.guagua.util.Progressable

Report progress to caller.

Progressable - Interface in ml.shifu.guagua.util

A facility for reporting progress.

ProgressLock - Class in ml.shifu.guagua.util

A lock with a predicate that was be used to synchronize events and keep the job updated while waiting.

ProgressLock() - Constructor for class ml.shifu.guagua.util.ProgressLock

Constructor with default values.

ProgressLock(int) - Constructor for class ml.shifu.guagua.util.ProgressLock

Constructor.

R

readFields(DataInput) - Method in interface ml.shifu.guagua.io.Bytable

De-serialize the fields of this object from in.

readFields(DataInput) - Method in class ml.shifu.guagua.io.BytableWrapper

readFields(DataInput) - Method in class ml.shifu.guagua.io.HaltBytable

readLines(File, String) - Static method in class ml.shifu.guagua.util.FileUtils

Reads the contents of a file line by line to a List of Strings.

readLines(File) - Static method in class ml.shifu.guagua.util.FileUtils

Reads the contents of a file line by line to a List of Strings using the default encoding for the VM.

readLines(InputStream) - Static method in class ml.shifu.guagua.util.FileUtils

Get the contents of an InputStream as a list of Strings, one entry per line, using the default character encoding of the platform.

readLines(InputStream, String) - Static method in class ml.shifu.guagua.util.FileUtils

Get the contents of an InputStream as a list of Strings, one entry per line, using the specified character encoding.

readLines(Reader) - Static method in class ml.shifu.guagua.util.FileUtils

Get the contents of a Reader as a list of Strings, one entry per line.

ReflectionUtils - Class in ml.shifu.guagua.util

ReflectionUtils is used to get instance from java reflection mechanism.

reOpen() - Method in class ml.shifu.guagua.util.BytableDiskList

Re-open stream for iterators.

reOpen() - Method in class ml.shifu.guagua.util.BytableMemoryDiskList

Reopen stream for iteration.

reOpen() - Method in class ml.shifu.guagua.util.DiskList

Re-open stream for iterators.

reOpen() - Method in class ml.shifu.guagua.util.MemoryDiskList

Reopen stream for iteration.

reset() - Method in class ml.shifu.guagua.util.ProgressLock

RETRY_COUNT - Static variable in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Client connection retry count

RETRY_COUNT - Static variable in class ml.shifu.guagua.util.NetworkUtils

Client connection retry count

retryDelay(int) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Performs a retry delay if this is not the first attempt

retryExecution() - Method in class ml.shifu.guagua.BasicCoordinator.RetryCoordinatorCommand

retryOperation(GuaguaZooKeeperOperation<T>) - Method in class ml.shifu.guagua.coordinator.zk.GuaguaZooKeeper

Perform the given operation, retrying if the connection fails

run(Progressable) - Method in interface ml.shifu.guagua.GuaguaService

Real logic implementation, for example, master and worker iteration logic.

run(Progressable) - Method in class ml.shifu.guagua.master.GuaguaMasterService

Iterate MasterComputable.compute(MasterContext), and set hook point before and after each iteration.

run() - Method in class ml.shifu.guagua.unit.GuaguaUnitDriver

To run master-workers iteration.

run(Progressable) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

Iterate WorkerComputable.compute(WorkerContext), and set hook point before and after each iteration.

S

serialize(T) - Method in class ml.shifu.guagua.util.JavaObjectSerializer

serialize(T) - Method in interface ml.shifu.guagua.util.ObjectSerializer

Serialize an instance to byte array.

Serializer<RESULT extends Bytable> - Interface in ml.shifu.guagua.io

Serializer is used to serialize Bytable object to bytes and de-serialize bytes to Bytable object.

setAppId(String) - Method in interface ml.shifu.guagua.GuaguaService

App id for whole application.

setAppId(String) - Method in class ml.shifu.guagua.master.AsyncMasterCoordinator

setAppId(String) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setAppId(String) - Method in class ml.shifu.guagua.worker.AsyncWorkerCoordinator

setAppId(String) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

setAttachment(Object) - Method in class ml.shifu.guagua.master.MasterContext

setAttachment(Object) - Method in class ml.shifu.guagua.worker.WorkerContext

setBytes(byte[]) - Method in class ml.shifu.guagua.io.BytableWrapper

setBytesToZNode(String, String, byte[], CreateMode) - Method in class ml.shifu.guagua.BasicCoordinator

Set bytes to znode, if bytes is over zookeeper data limit(1MB), use children znodes to store each part.

setContainerId(String) - Method in interface ml.shifu.guagua.GuaguaService

Set the unique container id for master or worker.

setContainerId(String) - Method in class ml.shifu.guagua.io.BytableWrapper

setContainerId(String) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setContainerId(String) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

setCoordinator(InMemoryCoordinator<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setCoordinator(InMemoryCoordinator<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.LocalMasterCoordinator

setCoordinator(InMemoryCoordinator<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

setCoordinator(InMemoryCoordinator<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.LocalWorkerCoordinator

setCurrentIteration(int) - Method in class ml.shifu.guagua.io.BytableWrapper

setCurrentIteration(int) - Method in class ml.shifu.guagua.master.AsyncMasterCoordinator

setCurrentIteration(int) - Method in class ml.shifu.guagua.master.MasterContext

setCurrentIteration(int) - Method in class ml.shifu.guagua.worker.AsyncWorkerCoordinator

setCurrentIteration(int) - Method in class ml.shifu.guagua.worker.WorkerContext

setFixedTime(boolean) - Method in class ml.shifu.guagua.BasicCoordinator

setHalt(boolean) - Method in class ml.shifu.guagua.io.HaltBytable

Set status to halt if isHalt is true.

setLastMasterResult(MASTER_RESULT) - Method in class ml.shifu.guagua.worker.WorkerContext

setLength(long) - Method in class ml.shifu.guagua.io.GuaguaFileSplit

setMasterComputable(MasterComputable<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setMasterInterceptors(List<MasterInterceptor<MASTER_RESULT, WORKER_RESULT>>) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setMasterResult(MASTER_RESULT) - Method in class ml.shifu.guagua.master.MasterContext

setMasterResult(WorkerContext<MASTER_RESULT, WORKER_RESULT>, String, String) - Method in class ml.shifu.guagua.worker.AbstractWorkerCoordinator

setMasterResultClassName(String) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setMasterResultClassName(String) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

setMasterSerializer(Serializer<MASTER_RESULT>) - Method in class ml.shifu.guagua.BasicCoordinator

setMinWorkersRatio(double) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setMinWorkersTimeOut(long) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setMyBid(String) - Method in class ml.shifu.guagua.master.AbstractMasterCoordinator

setOffset(long) - Method in class ml.shifu.guagua.io.GuaguaFileSplit

setPath(String) - Method in class ml.shifu.guagua.io.GuaguaFileSplit

setProps(Properties) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setProps(Properties) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

setRecordReader(GuaguaRecordReader<KEY, VALUE>) - Method in class ml.shifu.guagua.worker.AbstractCombineWorkerComputable

setRecordReader(GuaguaRecordReader<KEY, VALUE>) - Method in class ml.shifu.guagua.worker.AbstractWorkerComputable

setSerializer(BytableSerializer<T>) - Method in class ml.shifu.guagua.util.BytableDiskList

setSerializer(ObjectSerializer<T>) - Method in class ml.shifu.guagua.util.DiskList

setSleepTime(long) - Method in class ml.shifu.guagua.BasicCoordinator

setSplits(List<GuaguaFileSplit>) - Method in interface ml.shifu.guagua.GuaguaService

Assign splits to each worker to make them load that data.

setSplits(List<GuaguaFileSplit>) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setSplits(List<GuaguaFileSplit>) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

setStopMessage(boolean) - Method in class ml.shifu.guagua.io.BytableWrapper

setTotalIteration(int) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setTotalIteration(int) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

setup(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.HookMasterComputable

Setup method for master computation.

setUp() - Method in class ml.shifu.guagua.unit.GuaguaUnitDriver

setup(WorkerContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.HookWorkerComputable

Setup method for master computation.

setWorkerComputable(WorkerComputable<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

setWorkerInterceptors(List<WorkerInterceptor<MASTER_RESULT, WORKER_RESULT>>) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

setWorkerResultClassName(String) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setWorkerResultClassName(String) - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

setWorkerResults(MasterContext<MASTER_RESULT, WORKER_RESULT>, String, String, int) - Method in class ml.shifu.guagua.master.AbstractMasterCoordinator

Set worker results from znodes.

setWorkerResults(Iterable<WORKER_RESULT>) - Method in class ml.shifu.guagua.master.MasterContext

setWorkers(int) - Method in class ml.shifu.guagua.master.GuaguaMasterService

setWorkerSerializer(Serializer<WORKER_RESULT>) - Method in class ml.shifu.guagua.BasicCoordinator

setZooKeeper(GuaguaZooKeeper) - Method in class ml.shifu.guagua.BasicCoordinator

signal() - Method in class ml.shifu.guagua.util.ProgressLock

signalMaster(int, int, WORKER_RESULT) - Method in class ml.shifu.guagua.InMemoryCoordinator

Signal master with result setting.

signalWorkers(int, MASTER_RESULT) - Method in class ml.shifu.guagua.InMemoryCoordinator

Master signals workers with result setting.

size() - Method in interface ml.shifu.guagua.util.AppendList

Return size of this list.

size() - Method in class ml.shifu.guagua.util.BytableDiskList

size() - Method in class ml.shifu.guagua.util.BytableMemoryDiskList

size() - Method in class ml.shifu.guagua.util.DiskList

size() - Method in class ml.shifu.guagua.util.MemoryDiskList

size() - Method in class ml.shifu.guagua.util.MemoryLimitedList

SizeEstimator - Class in ml.shifu.guagua.util

Copied from Apache Spark project to estimate size of a Java instance.

SizeEstimator() - Constructor for class ml.shifu.guagua.util.SizeEstimator

start() - Method in interface ml.shifu.guagua.GuaguaService

Start point for the service.

start() - Method in class ml.shifu.guagua.master.GuaguaMasterService

start() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

Start services from interceptors, all services started logic should be wrapperd in WorkerInterceptor.preApplication(WorkerContext);

startChildZooKeeperProcess(String) - Static method in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Start embed zookeeper server in a child process.

startEmbedZooKeeper() - Static method in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

Start embed zookeeper server in a daemon thread.

startHeartbeat() - Method in class ml.shifu.guagua.BasicCoordinator

stop() - Method in interface ml.shifu.guagua.GuaguaService

Stop service, resources cleaning should be added in this function.

stop() - Method in class ml.shifu.guagua.master.GuaguaMasterService

Stop services from interceptors which is used for resource cleaning or servers shutting down.

stop() - Method in class ml.shifu.guagua.worker.GuaguaWorkerService

Stop services from interceptors which is used for resource cleaning or servers shutting down.

stopHeartBeat() - Method in class ml.shifu.guagua.BasicCoordinator

StringUtils - Class in ml.shifu.guagua.util

StringUtils class for string operation.

switchState() - Method in interface ml.shifu.guagua.util.AppendList

After appending, switch state from WRITE to READ to allow iterating.

switchState() - Method in class ml.shifu.guagua.util.BytableDiskList

Switch state from WRITE to READ

switchState() - Method in class ml.shifu.guagua.util.BytableMemoryDiskList

switchState() - Method in class ml.shifu.guagua.util.DiskList

Switch state from WRITE to READ

switchState() - Method in class ml.shifu.guagua.util.MemoryDiskList

switchState() - Method in class ml.shifu.guagua.util.MemoryLimitedList

SyncCoordinatorFactory<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua

Sync coordinator factory to create sync coordinators for master and workers.

SyncCoordinatorFactory() - Constructor for class ml.shifu.guagua.SyncCoordinatorFactory

SyncMasterCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

SyncMasterCoordinator is used to as a barrier for each iteration.

SyncMasterCoordinator() - Constructor for class ml.shifu.guagua.master.SyncMasterCoordinator

SyncWorkerCoordinator<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

SyncWorkerCoordinator is used to as a worker barrier for each iteration.

SyncWorkerCoordinator() - Constructor for class ml.shifu.guagua.worker.SyncWorkerCoordinator

SYSTEM_SEPARATOR - Static variable in class ml.shifu.guagua.util.FileUtils

The system separator character.

T

tearDown() - Method in class ml.shifu.guagua.unit.GuaguaUnitDriver

toString() - Method in class ml.shifu.guagua.io.BytableWrapper

toString() - Method in class ml.shifu.guagua.io.GuaguaFileSplit

toString() - Method in class ml.shifu.guagua.master.MasterContext

toString() - Method in class ml.shifu.guagua.worker.WorkerContext

totalMemoryMB() - Static method in class ml.shifu.guagua.util.MemoryUtils

Get total memory in megabytes

TraceMasterInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.master

TraceMasterInterceptor is used to trace whole application for master.

TraceMasterInterceptor() - Constructor for class ml.shifu.guagua.master.TraceMasterInterceptor

TraceWorkerInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

TraceWorkerInterceptor is used to trace whole application process.

TraceWorkerInterceptor() - Constructor for class ml.shifu.guagua.worker.TraceWorkerInterceptor

TRY_PORT_COUNT - Static variable in class ml.shifu.guagua.coordinator.zk.ZooKeeperUtils

How many ports will be used to launch embed zookeeper server.

TRY_PORT_COUNT - Static variable in class ml.shifu.guagua.util.NetworkUtils

How many ports will be used to launch embed zookeeper server.

U

UNIX_SEPARATOR - Static variable in class ml.shifu.guagua.util.FileUtils

The Unix separator character.

updateMasterHaltStatus(MasterContext<MASTER_RESULT, WORKER_RESULT>) - Method in class ml.shifu.guagua.master.AbstractMasterCoordinator

Check whether GuaguaConstants.GUAGUA_WORKER_HALT_ENABLE) is enabled, if yes, check whether all workers are halted and update master status.

usedMemoryMB() - Static method in class ml.shifu.guagua.util.MemoryUtils

Get used memory in megabytes

V

valueOf(String) - Static method in enum ml.shifu.guagua.util.AppendList.State

Returns the enum constant of this type with the specified name.

values() - Static method in enum ml.shifu.guagua.util.AppendList.State

Returns an array containing the constants of this enum type, in the order they are declared.

W

WAIT_SLOT_MILLS - Static variable in class ml.shifu.guagua.BasicCoordinator

Default waiting time to check master or worker progress from zookeeper servers.

waitForever() - Method in class ml.shifu.guagua.util.ProgressLock

waitMsecs(int) - Method in class ml.shifu.guagua.util.ProgressLock

WINDOWS_SEPARATOR - Static variable in class ml.shifu.guagua.util.FileUtils

The Windows separator character.

WORKER_COMPUTABLE_CLASS - Static variable in class ml.shifu.guagua.GuaguaConstants

WorkerComputable<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Interface in ml.shifu.guagua.worker

WorkerComputable defines worker computation for any guagua application.

WorkerComputableTimer<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

WorkerComputableTimer is used to print execution time for master computation function.

WorkerComputableTimer() - Constructor for class ml.shifu.guagua.worker.WorkerComputableTimer

WorkerContext<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

WorkerContext is a context to contain all useful info which can be used in worker computation.

WorkerContext(int, String, Properties, String, List<GuaguaFileSplit>, String, String) - Constructor for class ml.shifu.guagua.worker.WorkerContext

WorkerContext.WorkerCompletionCallBack<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Interface in ml.shifu.guagua.worker

Define callback for completion of worker computation.

workerInitLock - Variable in class ml.shifu.guagua.master.AsyncMasterCoordinator

Lock is used to check register info from all workers.

WorkerInterceptor<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Interface in ml.shifu.guagua.worker

WorkerInterceptor defines hook points for each guagua application.

workerIterationLock - Variable in class ml.shifu.guagua.master.AsyncMasterCoordinator

Lock is used to check iteration info from all workers.

WorkerTimer<MASTER_RESULT extends Bytable,WORKER_RESULT extends Bytable> - Class in ml.shifu.guagua.worker

WorkerTimer is used to log execution time of preXXXX of interceptors, worker computation and postXXXX of interceptors.

WorkerTimer() - Constructor for class ml.shifu.guagua.worker.WorkerTimer

write(DataOutput) - Method in interface ml.shifu.guagua.io.Bytable

Serialize the fields of this object to out.

write(DataOutput) - Method in class ml.shifu.guagua.io.BytableWrapper

write(DataOutput) - Method in class ml.shifu.guagua.io.HaltBytable

Z

ZOOKEEPER_SEPARATOR - Static variable in class ml.shifu.guagua.GuaguaConstants

ZooKeeperUtils - Class in ml.shifu.guagua.coordinator.zk

ZooKeeperUtils is a helper used to start embed zookeeper server in CLI host.

A B C D E F G H I J L M N O P R S T U V W Z