Type Members

1.  trait BlockFetchingListener extends EventListener
2.  abstract class BlockStoreClient extends Closeable

   Provides an interface for reading both shuffle files and RDD blocks, either from an Executor or external service.

3.  class Constants extends AnyRef
4.  trait DownloadFile extends AnyRef

   A handle on the file used when fetching remote data to disk.

   A handle on the file used when fetching remote data to disk. Used to ensure the lifecycle of writing the data, reading it back, and then cleaning it up is followed. Specific implementations may also handle encryption. The data can be read only via DownloadFileWritableChannel, which ensures data is not read until after the writer is closed.

5.  trait DownloadFileManager extends AnyRef

   A manager to create temp block files used when fetching remote data to reduce the memory usage.

   A manager to create temp block files used when fetching remote data to reduce the memory usage. It will clean files when they won't be used any more.

6.  trait DownloadFileWritableChannel extends WritableByteChannel

   A channel for writing data which is fetched to disk, which allows access to the written data only after the writer has been closed.

   A channel for writing data which is fetched to disk, which allows access to the written data only after the writer has been closed. Used with DownloadFile and DownloadFileManager.

7.  class ExecutorDiskUtils extends AnyRef
8.  class ExternalBlockHandler extends RpcHandler

   RPC Handler for a server which can serve both RDD blocks and shuffle blocks from outside of an Executor process.

   RPC Handler for a server which can serve both RDD blocks and shuffle blocks from outside of an Executor process.

   Handles registering executors and opening shuffle or disk persisted RDD blocks from them. Blocks are registered with the "one-for-one" strategy, meaning each Transport-layer Chunk is equivalent to one block.

9.  class ExternalBlockStoreClient extends BlockStoreClient

   Client for reading both RDD blocks and shuffle blocks which points to an external (outside of executor) server.

   Client for reading both RDD blocks and shuffle blocks which points to an external (outside of executor) server. This is instead of reading blocks directly from other executors (via BlockTransferService), which has the downside of losing the data if we lose the executors.

10.  class ExternalShuffleBlockResolver extends AnyRef

    Manages converting shuffle BlockIds into physical segments of local files, from a process outside of Executors.

    Manages converting shuffle BlockIds into physical segments of local files, from a process outside of Executors. Each Executor must register its own configuration about where it stores its files (local dirs) and how (shuffle manager). The logic for retrieval of individual files is replicated from Spark's IndexShuffleBlockResolver.

11.  class OneForOneBlockFetcher extends AnyRef

    Simple wrapper on top of a TransportClient which interprets each chunk as a whole block, and invokes the BlockFetchingListener appropriately.

    Simple wrapper on top of a TransportClient which interprets each chunk as a whole block, and invokes the BlockFetchingListener appropriately. This class is agnostic to the actual RPC handler, as long as there is a single "open blocks" message which returns a ShuffleStreamHandle, and Java serialization is used.

    Note that this typically corresponds to a org.apache.spark.network.server.OneForOneStreamManager on the server side.

12.  class RetryingBlockFetcher extends AnyRef

    Wraps another BlockFetcher with the ability to automatically retry fetches which fail due to IOExceptions, which we hope are due to transient network conditions.

    Wraps another BlockFetcher with the ability to automatically retry fetches which fail due to IOExceptions, which we hope are due to transient network conditions.

    This fetcher provides stronger guarantees regarding the parent BlockFetchingListener. In particular, the listener will be invoked exactly once per blockId, with a success or failure.

13.  class ShuffleIndexInformation extends AnyRef

    Keeps the index information for a particular map output as an in-memory LongBuffer.

14.  class ShuffleIndexRecord extends AnyRef

    Contains offset and length of the shuffle block data.

15.  class SimpleDownloadFile extends DownloadFile

    A DownloadFile that does not take any encryption settings into account for reading and writing data.

    A DownloadFile that does not take any encryption settings into account for reading and writing data.

    This does *not* mean the data in the file is un-encrypted -- it could be that the data is already encrypted when its written, and subsequent layer is responsible for decrypting.