ml.shifu.guagua.hadoop

Class GuaguaMRUnitDriver<MASTER_RESULT extends ml.shifu.guagua.io.Bytable,WORKER_RESULT extends ml.shifu.guagua.io.Bytable>

java.lang.Object

ml.shifu.guagua.unit.GuaguaUnitDriver<MASTER_RESULT,WORKER_RESULT>

ml.shifu.guagua.hadoop.GuaguaMRUnitDriver<MASTER_RESULT,WORKER_RESULT>

Type Parameters:

MASTER_RESULT - master result for computation in each iteration.

WORKER_RESULT - worker result for computation in each iteration.

public class GuaguaMRUnitDriver<MASTER_RESULT extends ml.shifu.guagua.io.Bytable,WORKER_RESULT extends ml.shifu.guagua.io.Bytable>
extends ml.shifu.guagua.unit.GuaguaUnitDriver<MASTER_RESULT,WORKER_RESULT>

GuaguaMRUnitDriver is used to run in-memory guagua application by using hadoop MapReduce splits.

See Also:

in guagua-mapreduce-examples project.

Constructor Summary

Constructors

Constructor and Description
GuaguaMRUnitDriver(Properties props) A only constructor here for local in-memory guagua job.

Method Summary

Modifier and Type	Method and Description
List<ml.shifu.guagua.io.GuaguaFileSplit[]>	generateWorkerSplits(String inputs)
protected List<org.apache.hadoop.fs.FileStatus>	listStatus(org.apache.hadoop.conf.Configuration conf, String input) List input directories.

Methods inherited from class ml.shifu.guagua.unit.GuaguaUnitDriver

doRun, run, setUp, tearDown

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GuaguaMRUnitDriver

public GuaguaMRUnitDriver(Properties props)

A only constructor here for local in-memory guagua job.

Parameters:

props - set all the configurations like input, output and ..

See Also:

in guagua-mapreduce-examples project.

Method Detail

generateWorkerSplits

public List<ml.shifu.guagua.io.GuaguaFileSplit[]> generateWorkerSplits(String inputs)
                                                                throws IOException

Specified by:

generateWorkerSplits in class ml.shifu.guagua.unit.GuaguaUnitDriver<MASTER_RESULT extends ml.shifu.guagua.io.Bytable,WORKER_RESULT extends ml.shifu.guagua.io.Bytable>

Throws:

IOException

listStatus

protected List<org.apache.hadoop.fs.FileStatus> listStatus(org.apache.hadoop.conf.Configuration conf,
                                                           String input)
                                                    throws IOException

List input directories. Subclasses may override to, e.g., select only files matching a regular expression.

Parameters:

job - the job to list input paths for

Returns:

array of FileStatus objects

Throws:

IOException - if zero items.

org.apache.hadoop.mapreduce.lib.input.InvalidInputException - If any IOException for input files.