Type Members

1.  trait ContinuousPartitionReader[T] extends PartitionReader[T]

   A variation on PartitionReader for use with continuous streaming processing.

   A variation on PartitionReader for use with continuous streaming processing.

   Since

   3.0.0

2.  trait ContinuousPartitionReaderFactory extends PartitionReaderFactory

   A variation on PartitionReaderFactory that returns ContinuousPartitionReader instead of PartitionReader.

   A variation on PartitionReaderFactory that returns ContinuousPartitionReader instead of PartitionReader. It's used for continuous streaming processing.

   Since

   3.0.0

3.  trait ContinuousStream extends SparkDataStream

   A SparkDataStream for streaming queries with continuous mode.

   A SparkDataStream for streaming queries with continuous mode.

   Since

   3.0.0

4.  trait MicroBatchStream extends SparkDataStream

   A SparkDataStream for streaming queries with micro-batch mode.

   A SparkDataStream for streaming queries with micro-batch mode.

   Since

   3.0.0

5.  abstract class Offset extends AnyRef

   An abstract representation of progress through a MicroBatchStream or ContinuousStream.

   An abstract representation of progress through a MicroBatchStream or ContinuousStream. During execution, offsets provided by the data source implementation will be logged and used as restart checkpoints. Each source should provide an offset implementation which the source can use to reconstruct a position in the stream up to which data has been seen/processed.

   Since

   3.0.0

6.  trait PartitionOffset extends Serializable

   Used for per-partition offsets in continuous processing.

   Used for per-partition offsets in continuous processing. ContinuousReader implementations will provide a method to merge these into a global Offset.

   These offsets must be serializable.

   Since

   3.0.0

7.  final class ReadAllAvailable extends ReadLimit

   Represents a ReadLimit where the MicroBatchStream must scan all the data available at the streaming source.

   Represents a ReadLimit where the MicroBatchStream must scan all the data available at the streaming source. This is meant to be a hard specification as being able to return all available data is necessary for Trigger.Once() to work correctly. If a source is unable to scan all available data, then it must throw an error.

   Since

   3.0.0

   See also

   SupportsAdmissionControl#latestOffset(Offset, ReadLimit)

8.  trait ReadLimit extends AnyRef

   Interface representing limits on how much to read from a MicroBatchStream when it implements SupportsAdmissionControl.

   Interface representing limits on how much to read from a MicroBatchStream when it implements SupportsAdmissionControl. There are several child interfaces representing various kinds of limits.

   Since

   3.0.0

   See also

   SupportsAdmissionControl#latestOffset(Offset, ReadLimit)

   ReadAllAvailable

   ReadMaxRows

9.  class ReadMaxFiles extends ReadLimit

   Represents a ReadLimit where the MicroBatchStream should scan approximately the given maximum number of files.

   Represents a ReadLimit where the MicroBatchStream should scan approximately the given maximum number of files.

   Since

   3.0.0

   See also

   SupportsAdmissionControl#latestOffset(Offset, ReadLimit)

10.  final class ReadMaxRows extends ReadLimit

    Represents a ReadLimit where the MicroBatchStream should scan approximately the given maximum number of rows.

    Represents a ReadLimit where the MicroBatchStream should scan approximately the given maximum number of rows.

    Since

    3.0.0

    See also

    SupportsAdmissionControl#latestOffset(Offset, ReadLimit)

11.  trait SparkDataStream extends AnyRef

    The base interface representing a readable data stream in a Spark streaming query.

    The base interface representing a readable data stream in a Spark streaming query. It's responsible to manage the offsets of the streaming source in the streaming query.

    Data sources should implement concrete data stream interfaces: MicroBatchStream and ContinuousStream.

    Since

    3.0.0

12.  trait SupportsAdmissionControl extends SparkDataStream

    A mix-in interface for SparkDataStream streaming sources to signal that they can control the rate of data ingested into the system.

    A mix-in interface for SparkDataStream streaming sources to signal that they can control the rate of data ingested into the system. These rate limits can come implicitly from the contract of triggers, e.g. Trigger.Once() requires that a micro-batch process all data available to the system at the start of the micro-batch. Alternatively, sources can decide to limit ingest through data source options.

    Through this interface, a MicroBatchStream should be able to return the next offset that it will process until given a ReadLimit.

    Since

    3.0.0