org.apache.hadoop.tools.mapred.lib

Class DynamicInputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.tools.mapred.lib.DynamicInputFormat<K,V>

public class DynamicInputFormat<K,V>
extends org.apache.hadoop.mapreduce.InputFormat<K,V>

DynamicInputFormat implements the "Worker pattern" for DistCp. Rather than to split up the copy-list into a set of static splits, the DynamicInputFormat does the following: 1. Splits the copy-list into small chunks on the DFS. 2. Creates a set of empty "dynamic" splits, that each consume as many chunks as it can. This arrangement ensures that a single slow mapper won't slow down the entire job (since the slack will be picked up by other mappers, who consume more chunks.) By varying the split-ratio, one can vary chunk sizes to achieve different performance characteristics.

Constructor Summary

Constructors

Constructor and Description
DynamicInputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<K,V>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext) Implementation of Inputformat::createRecordReader().
org.apache.hadoop.tools.mapred.lib.DynamicInputChunkContext<K,V>	getChunkContext(org.apache.hadoop.conf.Configuration configuration)
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext jobContext) Implementation of InputFormat::getSplits().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DynamicInputFormat

public DynamicInputFormat()

Method Detail

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext jobContext)
                                                       throws IOException,
                                                              InterruptedException

Implementation of InputFormat::getSplits(). This method splits up the copy-listing file into chunks, and assigns the first batch to different tasks.

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<K,V>

Parameters:

jobContext - JobContext for the map job.

Returns:

The list of (empty) dynamic input-splits.

Throws:

IOException

InterruptedException

getChunkContext

public org.apache.hadoop.tools.mapred.lib.DynamicInputChunkContext<K,V> getChunkContext(org.apache.hadoop.conf.Configuration configuration)
                                                                                 throws IOException

Throws:

IOException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<K,V> createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit,
                                                                        org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
                                                                 throws IOException,
                                                                        InterruptedException

Implementation of Inputformat::createRecordReader().

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<K,V>

Parameters:

inputSplit - The split for which the RecordReader is required.

taskAttemptContext - TaskAttemptContext for the current attempt.

Returns:

DynamicRecordReader instance.

Throws:

IOException

InterruptedException