ml.shifu.guagua.hadoop.util

Class HDPUtils

java.lang.Object

ml.shifu.guagua.hadoop.util.HDPUtils

public class HDPUtils
extends Object

For HortonWorks HDP 2.2.4 two issues not compatible with other hadoop 2.x versions.

Two major issues include:

• 1. Cannot find hdp.version on the fly, we need find the version firstly and we need find such number from dependency jar and set it on Hadoop Configuration object.
• 2. HADOOP_CONF_DIR is not set as classpath in classpath of every container, which make Shifu UDF and mapper/reducer cannot read configuration from HDFS. To solve it we have to find such configuration files like 'yarn-site.xml', 'core-site.xml', 'hdfs-site.xml' and 'mapred-site.xml' on client class path and ship them to container classpath as distributed cache files or jars.

Author:

Zhang David (pengzhang@paypal.com)

Constructor Summary

Constructors

Constructor and Description
HDPUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static void	addFileToClassPath(String file, org.apache.hadoop.conf.Configuration conf) Add a hdfs file to classpath of one container.
static String	findContainingFile(String fileName) Find a real file that contains file name in class path.
static String	findContainingJar(Class my_class)
static String	getHdpVersionForHDP224() A ugly method to retrieve hdp version like 2.2.4.6633 from hadoop-hdfs jar name.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HDPUtils

public HDPUtils()

Method Detail

getHdpVersionForHDP224

public static String getHdpVersionForHDP224()

A ugly method to retrieve hdp version like 2.2.4.6633 from hadoop-hdfs jar name. The jar name is like 'hadoop-hdfs-2.6.0.2.2.4.6633.jar'.

addFileToClassPath

public static void addFileToClassPath(String file,
                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Add a hdfs file to classpath of one container.

Throws:

IOException

findContainingFile

public static String findContainingFile(String fileName)

Find a real file that contains file name in class path.

Parameters:

file - name

Returns:

real file name

findContainingJar

public static String findContainingJar(Class my_class)