t

org.apache.spark.sql.connector.catalog

SupportsCatalogOptions

trait SupportsCatalogOptions extends TableProvider

An interface, which TableProviders can implement, to support table existence checks and creation through a catalog, without having to use table identifiers. For example, when file based data sources use the DataFrameWriter.save(path) method, the option path can translate to a PathIdentifier. A catalog can then use this PathIdentifier to check the existence of a table, or whether a table can be created at a given directory.

Since

3.0.0

Linear Supertypes
TableProvider, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SupportsCatalogOptions
  2. TableProvider
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def extractIdentifier(options: CaseInsensitiveStringMap): Identifier

    Return a Identifier instance that can identify a table for a DataSource given DataFrame[Reader|Writer] options.

    Return a Identifier instance that can identify a table for a DataSource given DataFrame[Reader|Writer] options.

    options

    the user-specified options that can identify a table, e.g. file path, Kafka topic name, etc. It's an immutable case-insensitive string-to-string map.

  2. abstract def getTable(schema: StructType, partitioning: Array[Transform], properties: Map[String, String]): Table

    Return a Table instance with the specified table schema, partitioning and properties to do read/write.

    Return a Table instance with the specified table schema, partitioning and properties to do read/write. The returned table should report the same schema and partitioning with the specified ones, or Spark may fail the operation.

    schema

    The specified table schema.

    partitioning

    The specified table partitioning.

    properties

    The specified table properties. It's case preserving (contains exactly what users specified) and implementations are free to use it case sensitively or insensitively. It should be able to identify a table, e.g. file path, Kafka topic name, etc.

    Definition Classes
    TableProvider
  3. abstract def inferSchema(options: CaseInsensitiveStringMap): StructType

    Infer the schema of the table identified by the given options.

    Infer the schema of the table identified by the given options.

    options

    an immutable case-insensitive string-to-string map that can identify a table, e.g. file path, Kafka topic name, etc.

    Definition Classes
    TableProvider

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def extractCatalog(options: CaseInsensitiveStringMap): String

    Return the name of a catalog that can be used to check the existence of, load, and create a table for this DataSource given the identifier that will be extracted by extractIdentifier.

    Return the name of a catalog that can be used to check the existence of, load, and create a table for this DataSource given the identifier that will be extracted by extractIdentifier. A null value can be used to defer to the V2SessionCatalog.

    options

    the user-specified options that can identify a table, e.g. file path, Kafka topic name, etc. It's an immutable case-insensitive string-to-string map.

  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def inferPartitioning(options: CaseInsensitiveStringMap): Array[Transform]

    Infer the partitioning of the table identified by the given options.

    Infer the partitioning of the table identified by the given options.

    By default this method returns empty partitioning, please override it if this source support partitioning.

    options

    an immutable case-insensitive string-to-string map that can identify a table, e.g. file path, Kafka topic name, etc.

    Definition Classes
    TableProvider
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. def supportsExternalMetadata(): Boolean

    Returns true if the source has the ability of accepting external table metadata when getting tables.

    Returns true if the source has the ability of accepting external table metadata when getting tables. The external table metadata includes user-specified schema from DataFrameReader/DataStreamReader and schema/partitioning stored in Spark catalog.

    By default this method returns false, which means the schema and partitioning passed to getTable are from the infer methods. Please override it if this source has expensive schema/partitioning inference and wants external table metadata to avoid inference.

    Definition Classes
    TableProvider
  18. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  19. def toString(): String
    Definition Classes
    AnyRef → Any
  20. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from TableProvider

Inherited from AnyRef

Inherited from Any

Ungrouped