Package org.gorpipe.spark.redis

Class RedisBatchConsumer

  • All Implemented Interfaces:
    java.io.Serializable, java.lang.AutoCloseable, org.apache.spark.api.java.function.VoidFunction2<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>,​java.lang.Long>
    public class RedisBatchConsumer
extends java.lang.Object
implements org.apache.spark.api.java.function.VoidFunction2<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>,​java.lang.Long>, java.lang.AutoCloseable
    See Also:
    Serialized Form

    • Constructor Summary

      Constructors 
      Constructor Description
      RedisBatchConsumer​(org.apache.spark.sql.SparkSession sparkSession, java.lang.String redisUri)  

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void call​(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> v1, java.lang.Long v2)  
      void close()  
      static void main​(java.lang.String[] args)
      For standalone gor query sparkapplication
      java.util.concurrent.Future<java.util.List<java.lang.String>> runGorJobs​(java.lang.String projectDirStr, java.util.Set<java.lang.Integer> gorJobs, java.lang.String[] queries, java.lang.String[] fingerprints, java.lang.String[] jobIds, java.lang.String[] cachefiles, java.lang.String[] secCtxs)  
      java.util.Map<java.lang.String,​java.util.concurrent.Future<java.util.List<java.lang.String>>> runJobBatch​(java.util.List<java.lang.String[]> lstr)  
      java.util.concurrent.Future<java.util.List<java.lang.String>> runSparkJob​(java.lang.String projectDirStr, java.lang.String[] creates, java.lang.String cmd, java.lang.String jobId, java.lang.String cacheFile, java.lang.String securityContext)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • RedisBatchConsumer

        public RedisBatchConsumer​(org.apache.spark.sql.SparkSession sparkSession,
                          java.lang.String redisUri)

    • Method Detail

      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable

      • runGorJobs

        public java.util.concurrent.Future<java.util.List<java.lang.String>> runGorJobs​(java.lang.String projectDirStr,
                                                                                java.util.Set<java.lang.Integer> gorJobs,
                                                                                java.lang.String[] queries,
                                                                                java.lang.String[] fingerprints,
                                                                                java.lang.String[] jobIds,
                                                                                java.lang.String[] cachefiles,
                                                                                java.lang.String[] secCtxs)

      • runSparkJob

        public java.util.concurrent.Future<java.util.List<java.lang.String>> runSparkJob​(java.lang.String projectDirStr,
                                                                                 java.lang.String[] creates,
                                                                                 java.lang.String cmd,
                                                                                 java.lang.String jobId,
                                                                                 java.lang.String cacheFile,
                                                                                 java.lang.String securityContext)

      • runJobBatch

        public java.util.Map<java.lang.String,​java.util.concurrent.Future<java.util.List<java.lang.String>>> runJobBatch​(java.util.List<java.lang.String[]> lstr)

      • call

        public void call​(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> v1,
                 java.lang.Long v2)
        Specified by:
        call in interface org.apache.spark.api.java.function.VoidFunction2<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>,​java.lang.Long>

      • main

        public static void main​(java.lang.String[] args)
        For standalone gor query sparkapplication
        Parameters:
        args -