ml.shifu.guagua.hadoop.io

Class GuaguaInputSplit

java.lang.Object

org.apache.hadoop.mapreduce.InputSplit

ml.shifu.guagua.hadoop.io.GuaguaInputSplit

All Implemented Interfaces:

org.apache.hadoop.io.Writable

public class GuaguaInputSplit
extends org.apache.hadoop.mapreduce.InputSplit
implements org.apache.hadoop.io.Writable

InputSplit implementation in guagua for Hadoop MapReduce job.

If mapper with isMaster true means it is master, for master so far fileSplits is null.

For worker, input fileSplits are included, here FileSplit array is used to make guagua support combining FileSplits in one task.

Constructor Summary

Constructors

Constructor and Description
GuaguaInputSplit() Default constructor without any setting.
GuaguaInputSplit(boolean isMaster, org.apache.hadoop.mapreduce.lib.input.FileSplit... fileSplits) Constructor with isMaster and fileSplits settings.
GuaguaInputSplit(boolean isMaster, org.apache.hadoop.mapreduce.lib.input.FileSplit fileSplit) Constructor with isMaster and one FileSplit settings.

Method Summary

Modifier and Type	Method and Description
Object[]	getExtensions()
org.apache.hadoop.mapreduce.lib.input.FileSplit[]	getFileSplits()
long	getLength() For master split, use Long.MAX_VALUE as its length to make it is the first task for Hadoop job.
String[]	getLocations() Data locality functions, return all hosts for all file splits.
boolean	isMaster()
void	readFields(DataInput in)
void	setExtensions(Object[] extensions)
void	setFileSplits(org.apache.hadoop.mapreduce.lib.input.FileSplit[] fileSplits)
void	setMaster(boolean isMaster)
String	toString()
void	write(DataOutput out)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

GuaguaInputSplit

public GuaguaInputSplit()

Default constructor without any setting.

GuaguaInputSplit

public GuaguaInputSplit(boolean isMaster,
                        org.apache.hadoop.mapreduce.lib.input.FileSplit... fileSplits)

Constructor with isMaster and fileSplits settings.

Parameters:

isMaster - Whether the input split is master split.

fileSplits - File splits used for mapper task.

GuaguaInputSplit

public GuaguaInputSplit(boolean isMaster,
                        org.apache.hadoop.mapreduce.lib.input.FileSplit fileSplit)

Constructor with isMaster and one FileSplit settings.

Parameters:

isMaster - Whether the input split is master split.

fileSplit - File split used for mapper task.

Method Detail

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Throws:

IOException

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Throws:

IOException

getLength

public long getLength()
               throws IOException,
                      InterruptedException

For master split, use Long.MAX_VALUE as its length to make it is the first task for Hadoop job. It is convenient for users to check master in Hadoop UI.

Specified by:

getLength in class org.apache.hadoop.mapreduce.InputSplit

Throws:

IOException

InterruptedException

getLocations

public String[] getLocations()
                      throws IOException,
                             InterruptedException

Data locality functions, return all hosts for all file splits.

Specified by:

getLocations in class org.apache.hadoop.mapreduce.InputSplit

Throws:

IOException

InterruptedException

isMaster

public boolean isMaster()

setMaster

public void setMaster(boolean isMaster)

getFileSplits

public org.apache.hadoop.mapreduce.lib.input.FileSplit[] getFileSplits()

setFileSplits

public void setFileSplits(org.apache.hadoop.mapreduce.lib.input.FileSplit[] fileSplits)

getExtensions

public Object[] getExtensions()

setExtensions

public void setExtensions(Object[] extensions)

toString

public String toString()

Overrides:

toString in class Object