Package ml.shifu.guagua.hadoop.util
Class HDPUtils
- java.lang.Object
-
- ml.shifu.guagua.hadoop.util.HDPUtils
-
public class HDPUtils extends Object
For HortonWorks HDP 2.2.4 two issues not caomptible with other hadoop 2.x versions.Two major issues include:
- 1. Cannot find hdp.version on the fly, we need find the version firstly and we need find such number from dependency jar and set it on Hadoop Configuration object.
- 2. HADOOP_CONF_DIR is not set as classpath in classpath of every container, which make Shifu UDF and mapper/reducer cannot read configuration from HDFS. To solve it we have to find such configuration files like 'yarn-site.xml', 'core-site.xml', 'hdfs-site.xml' and 'mapred-site.xml' on client class path and ship them to container classpath as distributed cache files or jars.
- Author:
- Zhang David (pengzhang@paypal.com)
-
-
Constructor Summary
Constructors Constructor Description HDPUtils()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static voidaddFileToClassPath(String file, org.apache.hadoop.conf.Configuration conf)Add a hdfs file to classpath of one container.static StringfindContainingFile(String fileName)Find a real file that contains file name in class path.static StringfindContainingJar(Class my_class)static StringgetHdpVersionForHDP224()A ugly method to retrieve hdp version like 2.2.4.6633 from hadoop-hdfs jar name.
-
-
-
Method Detail
-
getHdpVersionForHDP224
public static String getHdpVersionForHDP224()
A ugly method to retrieve hdp version like 2.2.4.6633 from hadoop-hdfs jar name. The jar name is like 'hadoop-hdfs-2.6.0.2.2.4.6633.jar'.
-
addFileToClassPath
public static void addFileToClassPath(String file, org.apache.hadoop.conf.Configuration conf) throws IOException
Add a hdfs file to classpath of one container.- Throws:
IOException
-
findContainingFile
public static String findContainingFile(String fileName)
Find a real file that contains file name in class path.- Parameters:
fileName- the file name- Returns:
- real file name
-
-