ml.shifu.guagua.yarn.util

Class YarnUtils

java.lang.Object

ml.shifu.guagua.yarn.util.YarnUtils

public final class YarnUtils
extends Object

YarnUtils is yarn-related helper class.

Field Summary

Fields

Modifier and Type	Field and Description
static String	GUAGUA_YARN_TMP

Method Summary

Methods

Modifier and Type	Method and Description
static void	addFileToResourceMap(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path target) Boilerplate to add a file to the local resources..
static void	addLocalClasspathToEnv(Map<String,String> env, org.apache.hadoop.conf.Configuration conf) Populate the environment string map to be added to the environment vars in a remote execution container.
static void	copyLocalResourcesToFs(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId) Copy batch of resources to hdfs app directory for master and containers.
static void	copyLocalResourceToFs(String src, String dst, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId) Copy local file to a named-file in remote FS
static void	exportGuaguaConfiguration(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId) Export our populated Configuration as an XML file to be used by the ApplicationMaster's exec container, and register it with LocalResources.
static org.apache.hadoop.fs.Path	getAppDirectory(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.yarn.api.records.ApplicationId appId) Working folder to store jars, files and other resources
static List<String>	getCommand(String mainClass, String vmArgs, String programArgs, String memory) Return java command according to main class, vm args, program args and memory settings.
static Map<String,org.apache.hadoop.yarn.api.records.LocalResource>	getLocalResourceMap(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.ApplicationId appId) Build local resources including main app jar, lib jars, log4j.properties and guagua-conf.xml.
static org.apache.hadoop.fs.Path	getPathForResource(org.apache.hadoop.fs.FileSystem fs, String loc, org.apache.hadoop.yarn.api.records.ApplicationId appId) Get path to store local resources on hdfs.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

GUAGUA_YARN_TMP

public static final String GUAGUA_YARN_TMP

See Also:

Constant Field Values

Method Detail

getLocalResourceMap

public static Map<String,org.apache.hadoop.yarn.api.records.LocalResource> getLocalResourceMap(org.apache.hadoop.conf.Configuration conf,
                                                                               org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                                                                        throws IOException

Build local resources including main app jar, lib jars, log4j.properties and guagua-conf.xml.

Throws:

IOException

getCommand

public static List<String> getCommand(String mainClass,
                      String vmArgs,
                      String programArgs,
                      String memory)

Return java command according to main class, vm args, program args and memory settings.

addLocalClasspathToEnv

public static void addLocalClasspathToEnv(Map<String,String> env,
                          org.apache.hadoop.conf.Configuration conf)

Populate the environment string map to be added to the environment vars in a remote execution container.

Parameters:

env - the map of env var values.

conf - the Configuration to pull values from.

getPathForResource

public static org.apache.hadoop.fs.Path getPathForResource(org.apache.hadoop.fs.FileSystem fs,
                                           String loc,
                                           org.apache.hadoop.yarn.api.records.ApplicationId appId)

Get path to store local resources on hdfs.

getAppDirectory

public static org.apache.hadoop.fs.Path getAppDirectory(org.apache.hadoop.fs.FileSystem fs,
                                        org.apache.hadoop.yarn.api.records.ApplicationId appId)

Working folder to store jars, files and other resources

copyLocalResourcesToFs

public static void copyLocalResourcesToFs(org.apache.hadoop.conf.Configuration conf,
                          org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                   throws IOException

Copy batch of resources to hdfs app directory for master and containers.

Throws:

IOException

addFileToResourceMap

public static void addFileToResourceMap(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources,
                        org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.fs.Path target)
                                 throws IOException

Boilerplate to add a file to the local resources..

Parameters:

localResources - the LocalResources map to populate.

fs - handle to the HDFS file system.

target - the file to send to the remote container.

Throws:

IOException

copyLocalResourceToFs

public static void copyLocalResourceToFs(String src,
                         String dst,
                         org.apache.hadoop.conf.Configuration conf,
                         org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                  throws IOException

Copy local file to a named-file in remote FS

Throws:

IOException

exportGuaguaConfiguration

public static void exportGuaguaConfiguration(org.apache.hadoop.conf.Configuration conf,
                             org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                      throws IOException

Export our populated Configuration as an XML file to be used by the ApplicationMaster's exec container, and register it with LocalResources.

Parameters:

conf - the current Configuration object to be published.

appId - the ApplicationId to stamp this app's base HDFS resources dir.

Throws:

IOException