c

org.apache.spark.network.shuffle

ExternalBlockHandler

class ExternalBlockHandler extends RpcHandler

RPC Handler for a server which can serve both RDD blocks and shuffle blocks from outside of an Executor process.

Handles registering executors and opening shuffle or disk persisted RDD blocks from them. Blocks are registered with the "one-for-one" strategy, meaning each Transport-layer Chunk is equivalent to one block.

Linear Supertypes
RpcHandler, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ExternalBlockHandler
  2. RpcHandler
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ExternalBlockHandler(streamManager: OneForOneStreamManager, blockManager: ExternalShuffleBlockResolver)

    Enables mocking out the StreamManager and BlockManager.

  2. new ExternalBlockHandler(conf: TransportConf, registeredExecutorFile: File)

Type Members

  1. class ShuffleMetrics extends MetricSet

    A simple class to wrap all shuffle service wrapper metrics

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def applicationRemoved(appId: String, cleanupLocalDirs: Boolean): Unit

    Removes an application (once it has been terminated), and optionally will clean up any local directories associated with the executors of that application in a separate thread.

  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def channelActive(client: TransportClient): Unit
    Definition Classes
    ExternalBlockHandler → RpcHandler
  7. def channelInactive(client: TransportClient): Unit
    Definition Classes
    ExternalBlockHandler → RpcHandler
  8. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  9. def close(): Unit
  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  12. def exceptionCaught(cause: Throwable, client: TransportClient): Unit
    Definition Classes
    ExternalBlockHandler → RpcHandler
  13. def executorRemoved(executorId: String, appId: String): Unit

    Clean up any non-shuffle files in any local directories associated with an finished executor.

  14. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  15. def getAllMetrics(): MetricSet
  16. def getBlockResolver(): ExternalShuffleBlockResolver
  17. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  18. def getStreamManager(): StreamManager
    Definition Classes
    ExternalBlockHandler → RpcHandler
  19. def handleMessage(msgObj: BlockTransferMessage, client: TransportClient, callback: RpcResponseCallback): Unit
    Attributes
    protected[shuffle]
  20. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  21. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  22. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  23. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  24. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  25. def receive(client: TransportClient, message: ByteBuffer, callback: RpcResponseCallback): Unit
    Definition Classes
    ExternalBlockHandler → RpcHandler
  26. def receive(arg0: TransportClient, arg1: ByteBuffer): Unit
    Definition Classes
    RpcHandler
  27. def receiveStream(arg0: TransportClient, arg1: ByteBuffer, arg2: RpcResponseCallback): StreamCallbackWithID
    Definition Classes
    RpcHandler
  28. def reregisterExecutor(appExecId: AppExecId, executorInfo: ExecutorShuffleInfo): Unit

    Register an (application, executor) with the given shuffle info.

    Register an (application, executor) with the given shuffle info.

    The "re-" is meant to highlight the intended use of this method -- when this service is restarted, this is used to restore the state of executors from before the restart. Normal registration will happen via a message handled in receive()

  29. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  30. def toString(): String
    Definition Classes
    AnyRef → Any
  31. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  32. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  33. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from RpcHandler

Inherited from AnyRef

Inherited from Any

Ungrouped