Package ml.shifu.guagua.hadoop

Class GuaguaMRUnitDriver<MASTER_RESULT extends ml.shifu.guagua.io.Bytable,WORKER_RESULT extends ml.shifu.guagua.io.Bytable>

java.lang.Object

ml.shifu.guagua.unit.GuaguaUnitDriver<MASTER_RESULT,WORKER_RESULT>

ml.shifu.guagua.hadoop.GuaguaMRUnitDriver<MASTER_RESULT,WORKER_RESULT>

Type Parameters:

MASTER_RESULT - master result for computation in each iteration.

WORKER_RESULT - worker result for computation in each iteration.

public class GuaguaMRUnitDriver<MASTER_RESULT extends ml.shifu.guagua.io.Bytable,WORKER_RESULT extends ml.shifu.guagua.io.Bytable>
extends ml.shifu.guagua.unit.GuaguaUnitDriver<MASTER_RESULT,WORKER_RESULT>

GuaguaMRUnitDriver is used to run in-memory guagua application by using hadoop MapReduce splits. Please check ml.shifu.guagua.mapreduce.example.sum.SumTest in guagua-mapreduce-examples project.

Constructor Summary

Constructors

Constructor	Description
GuaguaMRUnitDriver(Properties props)	A only constructor here for local in-memory guagua job.

Method Summary

Modifier and Type	Method	Description
List<ml.shifu.guagua.io.GuaguaFileSplit[]>	generateWorkerSplits(String inputs)
protected List<org.apache.hadoop.fs.FileStatus>	listStatus(org.apache.hadoop.conf.Configuration conf, String input)	List input directories.

Methods inherited from class ml.shifu.guagua.unit.GuaguaUnitDriver

doRun, run, setUp, tearDown

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GuaguaMRUnitDriver

public GuaguaMRUnitDriver(Properties props)

A only constructor here for local in-memory guagua job. Please check ml.shifu.guagua.mapreduce.example.sum.SumTest in guagua-mapreduce-examples project.

Parameters:

props - set all the configurations like input, output and ..

Method Detail

generateWorkerSplits

public List<ml.shifu.guagua.io.GuaguaFileSplit[]> generateWorkerSplits(String inputs)
                                                                throws IOException

Specified by:

generateWorkerSplits in class ml.shifu.guagua.unit.GuaguaUnitDriver<MASTER_RESULT extends ml.shifu.guagua.io.Bytable,WORKER_RESULT extends ml.shifu.guagua.io.Bytable>

Throws:

IOException

listStatus

protected List<org.apache.hadoop.fs.FileStatus> listStatus(org.apache.hadoop.conf.Configuration conf,
                                                           String input)
                                                    throws IOException

List input directories. Subclasses may override to, e.g., select only files matching a regular expression.

Parameters:

conf - the job configuration

input - the input file path

Returns:

array of FileStatus objects

Throws:

IOException - if zero items.

org.apache.hadoop.mapreduce.lib.input.InvalidInputException - If any IOException for input files.