Packages

object TypeCoercion

A collection of Rule that can be used to coerce differing types that participate in operations into compatible ones.

Notes about type widening / tightest common types: Broadly, there are two cases when we need to widen data types (e.g. union, binary comparison). In case 1, we are looking for a common data type for two or more data types, and in this case no loss of precision is allowed. Examples include type inference in JSON (e.g. what's the column's data type if one row is an integer while the other row is a long?). In case 2, we are looking for a widened data type with some acceptable loss of precision (e.g. there is no common type for double and decimal because double's range is larger than decimal, and yet decimal is more precise than double, but in union we would cast the decimal into double).

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. TypeCoercion
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. case class ConcatCoercion(conf: SQLConf) extends Rule[LogicalPlan] with TypeCoercionRule with Product with Serializable

    Coerces the types of Concat children to expected ones.

    Coerces the types of Concat children to expected ones.

    If spark.sql.function.concatBinaryAsString is false and all children types are binary, the expected types are binary. Otherwise, the expected ones are strings.

  2. case class EltCoercion(conf: SQLConf) extends Rule[LogicalPlan] with TypeCoercionRule with Product with Serializable

    Coerces the types of Elt children to expected ones.

    Coerces the types of Elt children to expected ones.

    If spark.sql.function.eltOutputAsString is false and all children types are binary, the expected types are binary. Otherwise, the expected ones are strings.

  3. case class InConversion(conf: SQLConf) extends Rule[LogicalPlan] with TypeCoercionRule with Product with Serializable

    Handles type coercion for both IN expression with subquery and IN expressions without subquery.

    Handles type coercion for both IN expression with subquery and IN expressions without subquery. 1. In the first case, find the common type by comparing the left hand side (LHS) expression types against corresponding right hand side (RHS) expression derived from the subquery expression's plan output. Inject appropriate casts in the LHS and RHS side of IN expression.

    2. In the second case, convert the value and in list expressions to the common operator type by looking at all the argument types and finding the closest one that all the arguments can be cast to. When no common operator type is found the original expression will be returned and an Analysis Exception will be raised at the type checking phase.

  4. case class PromoteStrings(conf: SQLConf) extends Rule[LogicalPlan] with TypeCoercionRule with Product with Serializable

    Promotes strings that appear in arithmetic expressions.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. def findCommonTypeDifferentOnlyInNullFlags(types: Seq[DataType]): Option[DataType]
  10. def findCommonTypeDifferentOnlyInNullFlags(t1: DataType, t2: DataType): Option[DataType]

    The method finds a common type for data types that differ only in nullable flags, including nullable, containsNull of ArrayType and valueContainsNull of MapType.

    The method finds a common type for data types that differ only in nullable flags, including nullable, containsNull of ArrayType and valueContainsNull of MapType. If the input types are different besides nullable flags, None is returned.

  11. val findTightestCommonType: (DataType, DataType) ⇒ Option[DataType]

    Case 1 type widening (see the classdoc comment above for TypeCoercion).

    Case 1 type widening (see the classdoc comment above for TypeCoercion).

    Find the tightest common type of two types that might be used in a binary expression. This handles all numeric types except fixed-precision decimals interacting with each other or with primitive types, because in that case the precision and scale of the result depends on the operation. Those rules are implemented in DecimalPrecision.

  12. def findWiderTypeForTwo(t1: DataType, t2: DataType): Option[DataType]

    Case 2 type widening (see the classdoc comment above for TypeCoercion).

    Case 2 type widening (see the classdoc comment above for TypeCoercion).

    i.e. the main difference with findTightestCommonType is that here we allow some loss of precision when widening decimal and double, and promotion to string.

  13. def findWiderTypeWithoutStringPromotion(types: Seq[DataType]): Option[DataType]
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def hasStringType(dt: DataType): Boolean

    Whether the data type contains StringType.

  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def haveSameType(types: Seq[DataType]): Boolean

    Check whether the given types are equal ignoring nullable, containsNull and valueContainsNull.

  18. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. val numericPrecedence: IndexedSeq[NumericType with Product with Serializable]
  23. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  24. def toString(): String
    Definition Classes
    AnyRef → Any
  25. def typeCoercionRules(conf: SQLConf): List[Rule[LogicalPlan]]
  26. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  29. object BooleanEquality extends Rule[LogicalPlan]

    Changes numeric values to booleans so that expressions like true = 1 can be evaluated.

  30. object CaseWhenCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces the type of different branches of a CASE WHEN statement to a common type.

  31. object DateTimeOperations extends Rule[LogicalPlan]
  32. object Division extends Rule[LogicalPlan] with TypeCoercionRule

    Hive only performs integral division with the DIV operator.

    Hive only performs integral division with the DIV operator. The arguments to / are always converted to fractional types.

  33. object FunctionArgumentConversion extends Rule[LogicalPlan] with TypeCoercionRule

    This ensure that the types for various functions are as expected.

  34. object IfCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces the type of different branches of If statement to a common type.

  35. object ImplicitTypeCasts extends Rule[LogicalPlan] with TypeCoercionRule

    Casts types according to the expected input types for Expressions.

  36. object IntegralDivision extends Rule[LogicalPlan] with TypeCoercionRule

    The DIV operator always returns long-type value.

    The DIV operator always returns long-type value. This rule cast the integral inputs to long type, to avoid overflow during calculation.

  37. object MapZipWithCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces key types of two different MapType arguments of the MapZipWith expression to a common type.

  38. object StackCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces NullTypes in the Stack expression to the column types of the corresponding positions.

  39. object StringLiteralCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    A special rule to support string literal as the second argument of date_add/date_sub functions, to keep backward compatibility as a temporary workaround.

    A special rule to support string literal as the second argument of date_add/date_sub functions, to keep backward compatibility as a temporary workaround. TODO(SPARK-28589): implement ANSI type type coercion and handle string literals.

  40. object WidenSetOperationTypes extends Rule[LogicalPlan]

    Widens numeric types and converts strings to numbers when appropriate.

    Widens numeric types and converts strings to numbers when appropriate.

    Loosely based on rules from "Hadoop: The Definitive Guide" 2nd edition, by Tom White

    The implicit conversion rules can be summarized as follows:

    • Any integral numeric type can be implicitly converted to a wider type.
    • All the integral numeric types, FLOAT, and (perhaps surprisingly) STRING can be implicitly converted to DOUBLE.
    • TINYINT, SMALLINT, and INT can all be converted to FLOAT.
    • BOOLEAN types cannot be converted to any other type.
    • Any integral numeric type can be implicitly converted to decimal type.
    • two different decimal types will be converted into a wider decimal type for both of them.
    • decimal type will be converted into double if there float or double together with it.

    Additionally, all types when UNION-ed with strings will be promoted to strings. Other string conversions are handled by PromoteStrings.

    Widening types might result in loss of precision in the following cases: - IntegerType to FloatType - LongType to FloatType - LongType to DoubleType - DecimalType to Double

    This rule is only applied to Union/Except/Intersect

  41. object WindowFrameCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Cast WindowFrame boundaries to the type they operate upon.

Inherited from AnyRef

Inherited from Any

Ungrouped