Package org.gorpipe.spark.platform

Class GorClusterBase

  • All Implemented Interfaces:
    java.lang.AutoCloseable, java.lang.Runnable
    Direct Known Subclasses:
    GeneralSparkCluster
    public abstract class GorClusterBase
extends java.lang.Object
implements java.lang.Runnable, java.lang.AutoCloseable

    • Constructor Detail

      • GorClusterBase

        public GorClusterBase()

    • Method Detail

      • getJobRetention

        public abstract java.time.Duration getJobRetention()

      • getJobExpiration

        public abstract java.time.Duration getJobExpiration()

      • logInfo

        public abstract void logInfo​(java.lang.String message,
                             java.lang.Throwable th)

      • logDebug

        public abstract void logDebug​(java.lang.String message)

      • logWarn

        public abstract void logWarn​(java.lang.String message,
                             java.lang.Throwable th)

      • logError

        public abstract void logError​(java.lang.String message,
                              java.lang.Throwable th)

      • logProgress

        public abstract void logProgress​(java.lang.String jobId,
                                 java.lang.String progress)

      • getValue

        public abstract java.lang.String getValue​(java.lang.String jobId,
                                          JobField field)

      • setValue

        public abstract void setValue​(java.lang.String jobId,
                              JobField field,
                              java.lang.String value)

      • yieldFor

        public abstract void yieldFor​(java.util.Collection<GorJob> jobs,
                              java.lang.String queue,
                              JobMonitor monitor)
                       throws java.lang.InterruptedException
        Throws:
        java.lang.InterruptedException

      • locks

        public abstract GorLocks locks()

      • pool

        public abstract redis.clients.jedis.JedisPool pool()

      • findJob

        public abstract GorJob findJob​(java.lang.String jobId)

      • getJobStatusesByRequestId

        public abstract java.util.Map<JobStatus,​java.lang.Integer> getJobStatusesByRequestId​(java.lang.String requestId)

      • getJobMessage

        public abstract java.lang.String getJobMessage​(java.lang.String requestId)

      • subscribeGorMonitor

        public abstract void subscribeGorMonitor​(java.lang.String requestId,
                                         org.gorpipe.gor.monitor.GorMonitor gorMonitor)

      • jobExists

        public abstract boolean jobExists​(java.lang.String jobId)

      • findJobByFingerprint

        public abstract GorJob findJobByFingerprint​(java.lang.String fingerprint,
                                            java.util.function.Function<java.net.URI,​GorClusterBase> clusterLookup)

      • registerJobByFingerprint

        public abstract void registerJobByFingerprint​(java.lang.String fingerprint,
                                              GorJob job)

      • forget

        protected abstract void forget​(GorTaskBase task)

      • submit

        public abstract GorJob submit​(java.lang.String queue,
                              java.lang.String task,
                              java.lang.Object... args)

      • submit

        public abstract java.util.List<GorJob> submit​(java.lang.String queue,
                                              BatchSubmission submission)

      • prioritySubmit

        public abstract GorJob prioritySubmit​(java.lang.String queue,
                                      java.lang.String task,
                                      java.lang.Object... args)

      • addScore

        public abstract void addScore​(java.lang.String partition,
                              int seconds)

      • storeSecurityContext

        public abstract java.lang.String storeSecurityContext​(java.lang.String securityContext)

      • getSecurityContext

        public abstract java.lang.String getSecurityContext​(java.lang.String key)

      • createQueryHandler

        public abstract org.gorpipe.gor.model.GorParallelQueryHandler createQueryHandler​(boolean workOff,
                                                                                 GorQuery query,
                                                                                 java.lang.String securityContext)

      • createWorker

        public abstract java.lang.Object createWorker()

      • createWorker

        public abstract java.lang.Object createWorker​(java.lang.String queue)

      • waitFor

        public void waitFor​(java.lang.Iterable<GorJob> jobs,
                    JobMonitor monitor)
             throws java.lang.InterruptedException
        Wait for jobs to finish
        Parameters:
        jobs - Jobs to wait for
        monitor - To receive callbacks
        Throws:
        java.lang.InterruptedException - if interrupted

      • cancel

        public static void cancel​(java.lang.Iterable<GorJob> jobs)
        Do best effort of canceling a list of jobs. Exceptions will be ignored but logged.

      • getLocalHostName

        protected java.lang.String getLocalHostName()

      • workerFatalException

        public void workerFatalException​(java.lang.Object source,
                                 java.lang.Throwable t)
        Worker should notify cluster that it is exiting because of exception.
        Parameters:
        source - Object
        t - Exception

      • onWorkerException

        public void onWorkerException​(java.lang.Runnable runnable)
        Can register (single) runnable to run on worker exception
        Parameters:
        runnable - Runnable action