Packages

t

org.apache.spark.sql.connector.read

SupportsReportPartitioning

trait SupportsReportPartitioning extends Scan

A mix in interface for Scan. Data sources can implement this interface to report data partitioning and try to avoid shuffle at Spark side.

Note that, when a Scan implementation creates exactly one InputPartition, Spark may avoid adding a shuffle even if the reader does not implement this interface.

Since

3.0.0

Linear Supertypes
Scan, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SupportsReportPartitioning
  2. Scan
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def outputPartitioning(): Partitioning

    Returns the output data partitioning that this reader guarantees.

  2. abstract def readSchema(): StructType

    Returns the actual schema of this data source scan, which may be different from the physical schema of the underlying storage, as column pruning or other optimizations may happen.

    Returns the actual schema of this data source scan, which may be different from the physical schema of the underlying storage, as column pruning or other optimizations may happen.

    Definition Classes
    Scan

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def description(): String

    A description string of this scan, which may includes information like: what filters are configured for this scan, what's the value of some important options like path, etc.

    A description string of this scan, which may includes information like: what filters are configured for this scan, what's the value of some important options like path, etc. The description doesn't need to include #readSchema(), as Spark already knows it.

    By default this returns the class name of the implementation. Please override it to provide a meaningful description.

    Definition Classes
    Scan
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  17. def toBatch(): Batch

    Returns the physical representation of this scan for batch query.

    Returns the physical representation of this scan for batch query. By default this method throws exception, data sources must overwrite this method to provide an implementation, if the Table that creates this scan returns TableCapability#BATCH_READ support in its Table#capabilities().

    Definition Classes
    Scan
    Exceptions thrown
  18. def toContinuousStream(checkpointLocation: String): ContinuousStream

    Returns the physical representation of this scan for streaming query with continuous mode.

    Returns the physical representation of this scan for streaming query with continuous mode. By default this method throws exception, data sources must overwrite this method to provide an implementation, if the Table that creates this scan returns TableCapability#CONTINUOUS_READ support in its Table#capabilities().

    checkpointLocation

    a path to Hadoop FS scratch space that can be used for failure recovery. Data streams for the same logical source in the same query will be given the same checkpointLocation.

    Definition Classes
    Scan
    Exceptions thrown
  19. def toMicroBatchStream(checkpointLocation: String): MicroBatchStream

    Returns the physical representation of this scan for streaming query with micro-batch mode.

    Returns the physical representation of this scan for streaming query with micro-batch mode. By default this method throws exception, data sources must overwrite this method to provide an implementation, if the Table that creates this scan returns TableCapability#MICRO_BATCH_READ support in its Table#capabilities().

    checkpointLocation

    a path to Hadoop FS scratch space that can be used for failure recovery. Data streams for the same logical source in the same query will be given the same checkpointLocation.

    Definition Classes
    Scan
    Exceptions thrown
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Scan

Inherited from AnyRef

Inherited from Any

Ungrouped