ml.shifu.guagua.hadoop.io

Class GuaguaOptionsParser

java.lang.Object

ml.shifu.guagua.hadoop.io.GuaguaOptionsParser

public class GuaguaOptionsParser
extends Object

GuaguaOptionsParser is a utility to parse command line arguments generic to the Hadoop framework. GenericOptionsParser recognizes several standard command line arguments, enabling applications to easily specify a namenode, a jobtracker, additional configuration resources etc.

Generic Options

The supported generic options are:

     -conf <configuration file>     specify a configuration file
     -D <property=value>            use value for given property
     -fs <local|namenode:port>      specify a namenode
     -jt <local|jobtracker:port>    specify a job tracker
     -files <comma separated list of files>    specify comma separated
                            files to be copied to the map reduce cluster
     -libjars <comma separated list of jars>   specify comma separated
                            jar files to include in the classpath.
     -archives <comma separated list of archives>    specify comma
             separated archives to be unarchived on the compute machines.
 
 

The general command line syntax is:

 bin/hadoop command [genericOptions] [commandOptions]
 

Generic command line arguments might modify Configuration objects, given to constructors.

The functionality is implemented using Commons CLI.

Examples:

 $ bin/hadoop dfs -fs darwin:8020 -ls /data
 list /data directory in dfs with namenode darwin:8020
 
 $ bin/hadoop dfs -D fs.default.name=darwin:8020 -ls /data
 list /data directory in dfs with namenode darwin:8020
     
 $ bin/hadoop dfs -conf hadoop-site.xml -ls /data
 list /data directory in dfs with conf specified in hadoop-site.xml
     
 $ bin/hadoop job -D mapred.job.tracker=darwin:50020 -submit job.xml
 submit a job to job tracker darwin:50020
     
 $ bin/hadoop job -jt darwin:50020 -submit job.xml
 submit a job to job tracker darwin:50020
     
 $ bin/hadoop job -jt local -submit job.xml
 submit a job to local runner
 
 $ bin/hadoop jar -libjars testlib.jar -archives test.tgz -files file.txt inputjar args 
 job submission with libjars, files and archives
 

Copy from hadoop GenericOptionsParser and extend it.

See Also:

Tool, ToolRunner

Constructor Summary

Constructors

Constructor and Description
GuaguaOptionsParser(org.apache.hadoop.conf.Configuration conf, org.apache.commons.cli.Options options, String[] args) Create a GuaguaOptionsParser to parse given options as well as generic Hadoop options.
GuaguaOptionsParser(org.apache.hadoop.conf.Configuration conf, String[] args) Create a GuaguaOptionsParser to parse only the generic Hadoop arguments.
GuaguaOptionsParser(org.apache.commons.cli.Options opts, String[] args) Create an options parser with the given options to parse the args.
GuaguaOptionsParser(String[] args) Create an options parser to parse the args.

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.commons.cli.CommandLine	getCommandLine() Returns the commons-cli CommandLine object to process the parsed arguments.
org.apache.hadoop.conf.Configuration	getConfiguration() Get the modified configuration
static URL[]	getLibJars(org.apache.hadoop.conf.Configuration conf) If libjars are set in the conf, parse the libjars.
String[]	getRemainingArgs() Returns an array of Strings containing only application-specific arguments.
static void	printGenericCommandUsage(PrintStream out) Print the usage message for generic command-line options supported.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GuaguaOptionsParser

public GuaguaOptionsParser(org.apache.commons.cli.Options opts,
                   String[] args)
                    throws IOException

Create an options parser with the given options to parse the args.

Parameters:

opts - the options

args - the command line arguments

Throws:

IOException

GuaguaOptionsParser

public GuaguaOptionsParser(String[] args)
                    throws IOException

Create an options parser to parse the args.

Parameters:

args - the command line arguments

Throws:

IOException

GuaguaOptionsParser

public GuaguaOptionsParser(org.apache.hadoop.conf.Configuration conf,
                   String[] args)
                    throws IOException

Create a GuaguaOptionsParser to parse only the generic Hadoop arguments. The array of string arguments other than the generic arguments can be obtained by getRemainingArgs().

Parameters:

conf - the Configuration to modify.

args - command-line arguments.

Throws:

IOException

GuaguaOptionsParser

public GuaguaOptionsParser(org.apache.hadoop.conf.Configuration conf,
                   org.apache.commons.cli.Options options,
                   String[] args)
                    throws IOException

Create a GuaguaOptionsParser to parse given options as well as generic Hadoop options. The resulting CommandLine object can be obtained by getCommandLine().

Parameters:

conf - the configuration to modify

options - options built by the caller

args - User-specified arguments

Throws:

IOException

Method Detail

getRemainingArgs

public String[] getRemainingArgs()

Returns an array of Strings containing only application-specific arguments.

Returns:

array of Strings containing the un-parsed arguments or empty array if commandLine was not defined.

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

Get the modified configuration

Returns:

the configuration that has the modified parameters.

getCommandLine

public org.apache.commons.cli.CommandLine getCommandLine()

Returns the commons-cli CommandLine object to process the parsed arguments. Note: If the object is created with GuaguaOptionsParser(Configuration, String[]), then returned object will only contain parsed generic options.

Returns:

CommandLine representing list of arguments parsed against Options descriptor.

getLibJars

public static URL[] getLibJars(org.apache.hadoop.conf.Configuration conf)
                        throws IOException

If libjars are set in the conf, parse the libjars.

Throws:

IOException

printGenericCommandUsage

public static void printGenericCommandUsage(PrintStream out)

Print the usage message for generic command-line options supported.

Parameters:

out - stream to print the usage message to.