Package org.gorpipe.spark.platform

Class GorClusterBase

java.lang.Object
org.gorpipe.spark.platform.GorClusterBase
All Implemented Interfaces:
java.lang.AutoCloseable, java.lang.Runnable
Direct Known Subclasses:
GeneralSparkCluster
public abstract class GorClusterBase
extends java.lang.Object
implements java.lang.Runnable, java.lang.AutoCloseable

  • Nested Class Summary

    Nested Classes 
    Modifier and Type Class Description
    class  GorClusterBase.Statistics  

  • Constructor Summary

    Constructors 
    Constructor Description
    GorClusterBase()  

  • Method Summary

    Modifier and Type Method Description
    abstract GorLogSubscription addLogSubscription​(GorLogReceiver receiver, java.lang.String... jobIds)  
    abstract void addScore​(java.lang.String partition, int seconds)  
    static void cancel​(java.lang.Iterable<GorJob> jobs)
    Do best effort of canceling a list of jobs.
    abstract org.gorpipe.gor.model.GorParallelQueryHandler createQueryHandler​(boolean workOff, GorQuery query, java.lang.String securityContext)  
    abstract java.lang.Object createWorker()  
    abstract java.lang.Object createWorker​(java.lang.String queue)  
    abstract GorJob findJob​(java.lang.String jobId)  
    abstract GorJob findJobByFingerprint​(java.lang.String fingerprint, java.util.function.Function<java.net.URI,​GorClusterBase> clusterLookup)  
    protected abstract void forget​(GorTaskBase task)  
    abstract GorClusterConfig getConfig()  
    abstract GorClusterBase.Statistics getCurrentStatistics()  
    abstract java.time.Duration getJobExpiration()  
    abstract java.lang.String getJobMessage​(java.lang.String requestId)  
    abstract java.time.Duration getJobRetention()  
    abstract java.util.Map<JobStatus,​java.lang.Integer> getJobStatusesByRequestId​(java.lang.String requestId)  
    protected java.lang.String getLocalHostName()  
    abstract java.lang.String getSecurityContext​(java.lang.String key)  
    abstract java.lang.String getValue​(java.lang.String jobId, JobField field)  
    abstract boolean jobExists​(java.lang.String jobId)  
    abstract GorLocks locks()  
    abstract void logDebug​(java.lang.String message)  
    abstract void logError​(java.lang.String message, java.lang.Throwable th)  
    abstract void logInfo​(java.lang.String message, java.lang.Throwable th)  
    abstract void logProgress​(java.lang.String jobId, java.lang.String progress)  
    abstract void logWarn​(java.lang.String message, java.lang.Throwable th)  
    void onWorkerException​(java.lang.Runnable runnable)
    Can register (single) runnable to run on worker exception
    abstract redis.clients.jedis.JedisPool pool()  
    abstract GorJob prioritySubmit​(java.lang.String queue, java.lang.String task, java.lang.Object... args)  
    abstract void registerJobByFingerprint​(java.lang.String fingerprint, GorJob job)  
    abstract void setValue​(java.lang.String jobId, JobField field, java.lang.String value)  
    abstract java.lang.String storeSecurityContext​(java.lang.String securityContext)  
    abstract GorJob submit​(java.lang.String queue, java.lang.String task, java.lang.Object... args)  
    abstract java.util.List<GorJob> submit​(java.lang.String queue, BatchSubmission submission)  
    abstract void subscribeGorMonitor​(java.lang.String requestId, org.gorpipe.gor.monitor.GorMonitor gorMonitor)  
    void waitFor​(java.lang.Iterable<GorJob> jobs, JobMonitor monitor)
    Wait for jobs to finish
    void workerFatalException​(java.lang.Object source, java.lang.Throwable t)
    Worker should notify cluster that it is exiting because of exception.
    abstract void yieldFor​(java.util.Collection<GorJob> jobs, java.lang.String queue, JobMonitor monitor)  

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface java.lang.AutoCloseable

    close

    Methods inherited from interface java.lang.Runnable

    run

  • Constructor Details

  • Method Details

    • getJobRetention

      public abstract java.time.Duration getJobRetention()

    • getJobExpiration

      public abstract java.time.Duration getJobExpiration()

    • getConfig

      public abstract GorClusterConfig getConfig()

    • logInfo

      public abstract void logInfo​(java.lang.String message, java.lang.Throwable th)

    • logDebug

      public abstract void logDebug​(java.lang.String message)

    • logWarn

      public abstract void logWarn​(java.lang.String message, java.lang.Throwable th)

    • logError

      public abstract void logError​(java.lang.String message, java.lang.Throwable th)

    • logProgress

      public abstract void logProgress​(java.lang.String jobId, java.lang.String progress)

    • getValue

      public abstract java.lang.String getValue​(java.lang.String jobId, JobField field)

    • setValue

      public abstract void setValue​(java.lang.String jobId, JobField field, java.lang.String value)

    • yieldFor

      public abstract void yieldFor​(java.util.Collection<GorJob> jobs, java.lang.String queue, JobMonitor monitor) throws java.lang.InterruptedException
      Throws:
      java.lang.InterruptedException

    • locks

      public abstract GorLocks locks()

    • pool

      public abstract redis.clients.jedis.JedisPool pool()

    • addLogSubscription

      public abstract GorLogSubscription addLogSubscription​(GorLogReceiver receiver, java.lang.String... jobIds)

    • findJob

      public abstract GorJob findJob​(java.lang.String jobId)

    • getJobStatusesByRequestId

      public abstract java.util.Map<JobStatus,​java.lang.Integer> getJobStatusesByRequestId​(java.lang.String requestId)

    • getJobMessage

      public abstract java.lang.String getJobMessage​(java.lang.String requestId)

    • subscribeGorMonitor

      public abstract void subscribeGorMonitor​(java.lang.String requestId, org.gorpipe.gor.monitor.GorMonitor gorMonitor)

    • jobExists

      public abstract boolean jobExists​(java.lang.String jobId)

    • findJobByFingerprint

      public abstract GorJob findJobByFingerprint​(java.lang.String fingerprint, java.util.function.Function<java.net.URI,​GorClusterBase> clusterLookup)

    • registerJobByFingerprint

      public abstract void registerJobByFingerprint​(java.lang.String fingerprint, GorJob job)

    • forget

      protected abstract void forget​(GorTaskBase task)

    • getCurrentStatistics

      public abstract GorClusterBase.Statistics getCurrentStatistics()

    • submit

      public abstract GorJob submit​(java.lang.String queue, java.lang.String task, java.lang.Object... args)

    • submit

      public abstract java.util.List<GorJob> submit​(java.lang.String queue, BatchSubmission submission)

    • prioritySubmit

      public abstract GorJob prioritySubmit​(java.lang.String queue, java.lang.String task, java.lang.Object... args)

    • addScore

      public abstract void addScore​(java.lang.String partition, int seconds)

    • storeSecurityContext

      public abstract java.lang.String storeSecurityContext​(java.lang.String securityContext)

    • getSecurityContext

      public abstract java.lang.String getSecurityContext​(java.lang.String key)

    • createQueryHandler

      public abstract org.gorpipe.gor.model.GorParallelQueryHandler createQueryHandler​(boolean workOff, GorQuery query, java.lang.String securityContext)

    • createWorker

      public abstract java.lang.Object createWorker()

    • createWorker

      public abstract java.lang.Object createWorker​(java.lang.String queue)

    • waitFor

      public void waitFor​(java.lang.Iterable<GorJob> jobs, JobMonitor monitor) throws java.lang.InterruptedException
      Wait for jobs to finish
      Parameters:
      jobs - Jobs to wait for
      monitor - To receive callbacks
      Throws:
      java.lang.InterruptedException - if interrupted

    • cancel

      public static void cancel​(java.lang.Iterable<GorJob> jobs)
      Do best effort of canceling a list of jobs. Exceptions will be ignored but logged.

    • getLocalHostName

      protected java.lang.String getLocalHostName()

    • workerFatalException

      public void workerFatalException​(java.lang.Object source, java.lang.Throwable t)
      Worker should notify cluster that it is exiting because of exception.
      Parameters:
      source - Object
      t - Exception

    • onWorkerException

      public void onWorkerException​(java.lang.Runnable runnable)
      Can register (single) runnable to run on worker exception
      Parameters:
      runnable - Runnable action