ml.shifu.guagua.yarn.util

Class InputSplitUtils

java.lang.Object

ml.shifu.guagua.yarn.util.InputSplitUtils

public final class InputSplitUtils
extends Object

Helper class to get input splits.

Method Summary

Methods

Modifier and Type	Method and Description
static String	expandInputFolder(org.apache.hadoop.conf.Configuration conf) Expand folder to all files to support all files in that folder
static int	getBlockIndex(org.apache.hadoop.fs.BlockLocation[] blkLocations, long offset)
static List<List<org.apache.hadoop.mapreduce.InputSplit>>	getCombineGuaguaSplits(List<org.apache.hadoop.mapreduce.InputSplit> oneInputSplits, long maxCombinedSplitSize)
static List<org.apache.hadoop.mapreduce.InputSplit>	getFileSplits(org.apache.hadoop.conf.Configuration conf, long splitSize) Generate the list of files and make them into FileSplits.
static List<org.apache.hadoop.mapreduce.InputSplit>	getFinalCombineGuaguaSplits(List<org.apache.hadoop.mapreduce.InputSplit> newSplits, long combineSize) Copy from pig implementation, need to check this code logic.
static List<org.apache.hadoop.mapreduce.InputSplit>	getGuaguaSplits(org.apache.hadoop.conf.Configuration conf, long splitSize) Generate the list of files and make them into FileSplits.
static org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.conf.Configuration conf) List all the inputs files.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getFinalCombineGuaguaSplits

public static List<org.apache.hadoop.mapreduce.InputSplit> getFinalCombineGuaguaSplits(List<org.apache.hadoop.mapreduce.InputSplit> newSplits,
                                                                       long combineSize)
                                                                                throws IOException

Copy from pig implementation, need to check this code logic.

Throws:

IOException

listStatus

public static org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.conf.Configuration conf)
                                                    throws IOException

List all the inputs files. Better to follow FileInputFormat#listStatus

Throws:

IOException

expandInputFolder

public static String expandInputFolder(org.apache.hadoop.conf.Configuration conf)
                                throws IOException

Expand folder to all files to support all files in that folder

Throws:

IOException

getFileSplits

public static List<org.apache.hadoop.mapreduce.InputSplit> getFileSplits(org.apache.hadoop.conf.Configuration conf,
                                                         long splitSize)
                                                                  throws IOException

Generate the list of files and make them into FileSplits.

Throws:

IOException

getGuaguaSplits

public static List<org.apache.hadoop.mapreduce.InputSplit> getGuaguaSplits(org.apache.hadoop.conf.Configuration conf,
                                                           long splitSize)
                                                                    throws IOException

Generate the list of files and make them into FileSplits.

Throws:

IOException

getBlockIndex

public static int getBlockIndex(org.apache.hadoop.fs.BlockLocation[] blkLocations,
                long offset)

getCombineGuaguaSplits

public static List<List<org.apache.hadoop.mapreduce.InputSplit>> getCombineGuaguaSplits(List<org.apache.hadoop.mapreduce.InputSplit> oneInputSplits,
                                                                        long maxCombinedSplitSize)
                                                                                 throws IOException,
                                                                                        InterruptedException

Throws:

IOException

InterruptedException