Class HDPUtils


  • public class HDPUtils
    extends Object
    For HortonWorks HDP 2.2.4 two issues not caomptible with other hadoop 2.x versions.

    Two major issues include:

    • 1. Cannot find hdp.version on the fly, we need find the version firstly and we need find such number from dependency jar and set it on Hadoop Configuration object.
    • 2. HADOOP_CONF_DIR is not set as classpath in classpath of every container, which make Shifu UDF and mapper/reducer cannot read configuration from HDFS. To solve it we have to find such configuration files like 'yarn-site.xml', 'core-site.xml', 'hdfs-site.xml' and 'mapred-site.xml' on client class path and ship them to container classpath as distributed cache files or jars.
    Author:
    Zhang David (pengzhang@paypal.com)
    • Constructor Detail

      • HDPUtils

        public HDPUtils()
    • Method Detail

      • getHdpVersionForHDP224

        public static String getHdpVersionForHDP224()
        A ugly method to retrieve hdp version like 2.2.4.6633 from hadoop-hdfs jar name. The jar name is like 'hadoop-hdfs-2.6.0.2.2.4.6633.jar'.
      • addFileToClassPath

        public static void addFileToClassPath​(String file,
                                              org.apache.hadoop.conf.Configuration conf)
                                       throws IOException
        Add a hdfs file to classpath of one container.
        Throws:
        IOException
      • findContainingFile

        public static String findContainingFile​(String fileName)
        Find a real file that contains file name in class path.
        Parameters:
        fileName - the file name
        Returns:
        real file name
      • findContainingJar

        public static String findContainingJar​(Class my_class)