Type Members

1.  abstract class BlockTransferMessage extends Encodable

   Messages handled by the ExternalBlockHandler, or by Spark's NettyBlockTransferService.

   Messages handled by the ExternalBlockHandler, or by Spark's NettyBlockTransferService.

   At a high level:

   • OpenBlock is logically only handled by the NettyBlockTransferService, but for the capability for old version Spark, we still keep it in external shuffle service. It returns a StreamHandle.
   • UploadBlock is only handled by the NettyBlockTransferService.
   • RegisterExecutor is only handled by the external shuffle service.
   • RemoveBlocks is only handled by the external shuffle service.
   • FetchShuffleBlocks is handled by both services for shuffle files. It returns a StreamHandle.
2.  class BlocksRemoved extends BlockTransferMessage

   The reply to remove blocks giving back the number of removed blocks.

3.  class ExecutorShuffleInfo extends Encodable

   Contains all configuration necessary for locating the shuffle files of an executor.

4.  class FetchShuffleBlocks extends BlockTransferMessage

   Request to read a set of blocks.

   Request to read a set of blocks. Returns StreamHandle.

5.  class GetLocalDirsForExecutors extends BlockTransferMessage

   Request to get the local dirs for the given executors.

6.  class LocalDirsForExecutors extends BlockTransferMessage

   The reply to get local dirs giving back the dirs for each of the requested executors.

7.  class OpenBlocks extends BlockTransferMessage

   Request to read a set of blocks.

   Request to read a set of blocks. Returns StreamHandle.

8.  class RegisterExecutor extends BlockTransferMessage

   Initial registration message between an executor and its local shuffle server.

   Initial registration message between an executor and its local shuffle server. Returns nothing (empty byte array).

9.  class RemoveBlocks extends BlockTransferMessage

   Request to remove a set of blocks.

10.  class StreamHandle extends BlockTransferMessage

    Identifier for a fixed number of chunks to read from a stream created by an "open blocks" message.

    Identifier for a fixed number of chunks to read from a stream created by an "open blocks" message. This is used by org.apache.spark.network.shuffle.OneForOneBlockFetcher.

11.  class UploadBlock extends BlockTransferMessage

    Request to upload a block with a certain StorageLevel.

    Request to upload a block with a certain StorageLevel. Returns nothing (empty byte array).

12.  class UploadBlockStream extends BlockTransferMessage

    A request to Upload a block, which the destination should receive as a stream.

    A request to Upload a block, which the destination should receive as a stream.

    The actual block data is not contained here. It will be passed to the StreamCallbackWithID that is returned from RpcHandler.receiveStream()