InputSplitUtils (guagua-yarn 0.7.8 API)

java.lang.Object
- ml.shifu.guagua.yarn.util.InputSplitUtils

public final class InputSplitUtils
extends Object

Helper class to get input splits.

Method Summary

All Methods Static Methods Concrete Methods
Modifier and Type	Method and Description
`static String`	`expandInputFolder(org.apache.hadoop.conf.Configuration conf)` Expand folder to all files to support all files in that folder
`static int`	`getBlockIndex(org.apache.hadoop.fs.BlockLocation[] blkLocations, long offset)`
`static List<List<org.apache.hadoop.mapreduce.InputSplit>>`	`getCombineGuaguaSplits(List<org.apache.hadoop.mapreduce.InputSplit> oneInputSplits, long maxCombinedSplitSize)`
`static List<org.apache.hadoop.mapreduce.InputSplit>`	`getFileSplits(org.apache.hadoop.conf.Configuration conf, long splitSize)` Generate the list of files and make them into FileSplits.
`static List<org.apache.hadoop.mapreduce.InputSplit>`	`getFinalCombineGuaguaSplits(List<org.apache.hadoop.mapreduce.InputSplit> newSplits, long combineSize)` Copy from pig implementation, need to check this code logic.
`static List<org.apache.hadoop.mapreduce.InputSplit>`	`getGuaguaSplits(org.apache.hadoop.conf.Configuration conf, long splitSize)` Generate the list of files and make them into FileSplits.
`static org.apache.hadoop.fs.FileStatus[]`	`listStatus(org.apache.hadoop.conf.Configuration conf)` List all the inputs files.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getFinalCombineGuaguaSplits

public static List<org.apache.hadoop.mapreduce.InputSplit> getFinalCombineGuaguaSplits(List<org.apache.hadoop.mapreduce.InputSplit> newSplits,
                                                                                       long combineSize)
                                                                                throws IOException

Copy from pig implementation, need to check this code logic.

Throws:: IOException

listStatus

public static org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.conf.Configuration conf)
                                                    throws IOException

List all the inputs files. Better to follow FileInputFormat#listStatus

Throws:: IOException

expandInputFolder

public static String expandInputFolder(org.apache.hadoop.conf.Configuration conf)
                                throws IOException

Expand folder to all files to support all files in that folder

Throws:: IOException

getFileSplits

public static List<org.apache.hadoop.mapreduce.InputSplit> getFileSplits(org.apache.hadoop.conf.Configuration conf,
                                                                         long splitSize)
                                                                  throws IOException

Generate the list of files and make them into FileSplits.

Throws:: IOException

getGuaguaSplits

public static List<org.apache.hadoop.mapreduce.InputSplit> getGuaguaSplits(org.apache.hadoop.conf.Configuration conf,
                                                                           long splitSize)
                                                                    throws IOException