Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark
    Definition Classes
    apache
  • package sql
    Definition Classes
    spark
  • package catalyst

    Catalyst is a library for manipulating relational query plans.

    Catalyst is a library for manipulating relational query plans. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

    Definition Classes
    sql
  • package expressions

    A set of classes that can be used to represent trees of relational expressions.

    A set of classes that can be used to represent trees of relational expressions. A key goal of the expression library is to hide the details of naming and scoping from developers who want to manipulate trees of relational operators. As such, the library defines a special type of expression, a NamedExpression in addition to the standard collection of expressions.

    Standard Expressions

    A library of standard expressions (e.g., Add, EqualTo), aggregates (e.g., SUM, COUNT), and other computations (e.g. UDFs). Each expression type is capable of determining its output schema as a function of its children's output schema.

    Named Expressions

    Some expression are named and thus can be referenced by later operators in the dataflow graph. The two types of named expressions are AttributeReferences and Aliases. AttributeReferences refer to attributes of the input tuple for a given operator and form the leaves of some expression trees. Aliases assign a name to intermediate computations. For example, in the SQL statement SELECT a+b AS c FROM ..., the expressions a and b would be represented by AttributeReferences and c would be represented by an Alias.

    During analysis, all named expressions are assigned a globally unique expression id, which can be used for equality comparisons. While the original names are kept around for debugging purposes, they should never be used to check if two attributes refer to the same value, as plan transformations can result in the introduction of naming ambiguity. For example, consider a plan that contains subqueries, both of which are reading from the same table. If an optimization removes the subqueries, scoping information would be destroyed, eliminating the ability to reason about which subquery produced a given attribute.

    Evaluation

    The result of expressions can be evaluated using the Expression.apply(Row) method.

    Definition Classes
    catalyst
  • package objects
    Definition Classes
    expressions
  • AssertNotNull
  • CatalystToExternalMap
  • CreateExternalRow
  • DecodeUsingSerializer
  • EncodeUsingSerializer
  • ExternalMapToCatalyst
  • GetExternalRowField
  • InitializeJavaBean
  • Invoke
  • InvokeLike
  • LambdaVariable
  • MapObjects
  • NewInstance
  • SerializerSupport
  • StaticInvoke
  • UnresolvedCatalystToExternalMap
  • UnresolvedMapObjects
  • UnwrapOption
  • ValidateExternalType
  • WrapOption

case class DecodeUsingSerializer[T](child: Expression, tag: ClassTag[T], kryo: Boolean) extends UnaryExpression with NonSQLExpression with SerializerSupport with Product with Serializable

Serializes an input object using a generic serializer (Kryo or Java). Note that the ClassTag is not an implicit parameter because TreeNode cannot copy implicit parameters.

kryo

if true, use Kryo. Otherwise, use Java.

Linear Supertypes
Serializable, Serializable, SerializerSupport, NonSQLExpression, UnaryExpression, Expression, TreeNode[Expression], Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DecodeUsingSerializer
  2. Serializable
  3. Serializable
  4. SerializerSupport
  5. NonSQLExpression
  6. UnaryExpression
  7. Expression
  8. TreeNode
  9. Product
  10. Equals
  11. AnyRef
  12. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DecodeUsingSerializer(child: Expression, tag: ClassTag[T], kryo: Boolean)

    kryo

    if true, use Kryo. Otherwise, use Java.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addImmutableSerializerIfNeeded(ctx: CodegenContext): String

    Adds a immutable state to the generated class containing a reference to the serializer.

    Adds a immutable state to the generated class containing a reference to the serializer.

    returns

    a string containing the name of the variable referencing the serializer

    Definition Classes
    SerializerSupport
  5. def apply(number: Int): TreeNode[_]

    Returns the tree node at the specified number, used primarily for interactive debugging.

    Returns the tree node at the specified number, used primarily for interactive debugging. Numbers for each node can be found in the numberedTreeString.

    Note that this cannot return BaseType because logical plan's plan node might return physical plan for innerChildren, e.g. in-memory relation logical plan node has a reference to the physical plan node it is referencing.

    Definition Classes
    TreeNode
  6. def argString(maxFields: Int): String

    Returns a string representing the arguments to this node, minus any children

    Returns a string representing the arguments to this node, minus any children

    Definition Classes
    TreeNode
  7. def asCode: String

    Returns a 'scala code' representation of this TreeNode and its children.

    Returns a 'scala code' representation of this TreeNode and its children. Intended for use when debugging where the prettier toString function is obfuscating the actual structure. In the case of 'pure' TreeNodes that only contain primitives and other TreeNodes, the result can be pasted in the REPL to build an equivalent Tree.

    Definition Classes
    TreeNode
  8. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  9. lazy val canonicalized: Expression

    Returns an expression where a best effort attempt has been made to transform this in a way that preserves the result but removes cosmetic variations (case sensitivity, ordering for commutative operations, etc.) See Canonicalize for more details.

    Returns an expression where a best effort attempt has been made to transform this in a way that preserves the result but removes cosmetic variations (case sensitivity, ordering for commutative operations, etc.) See Canonicalize for more details.

    deterministic expressions where this.canonicalized == other.canonicalized will always evaluate to the same result.

    Definition Classes
    Expression
  10. def checkInputDataTypes(): TypeCheckResult

    Checks the input data types, returns TypeCheckResult.success if it's valid, or returns a TypeCheckResult with an error message if invalid.

    Checks the input data types, returns TypeCheckResult.success if it's valid, or returns a TypeCheckResult with an error message if invalid. Note: it's not valid to call this method until childrenResolved == true.

    Definition Classes
    Expression
  11. val child: Expression
    Definition Classes
    DecodeUsingSerializer → UnaryExpression
  12. final def children: Seq[Expression]

    Returns a Seq of the children of this node.

    Returns a Seq of the children of this node. Children should not change. Immutability required for containsChild optimization

    Definition Classes
    UnaryExpression → TreeNode
  13. def childrenResolved: Boolean

    Returns true if all the children of this expression have been resolved to a specific schema and false if any still contains any unresolved placeholders.

    Returns true if all the children of this expression have been resolved to a specific schema and false if any still contains any unresolved placeholders.

    Definition Classes
    Expression
  14. def clone(): Expression
    Definition Classes
    TreeNode → AnyRef
  15. def collect[B](pf: PartialFunction[Expression, B]): Seq[B]

    Returns a Seq containing the result of applying a partial function to all elements in this tree on which the function is defined.

    Returns a Seq containing the result of applying a partial function to all elements in this tree on which the function is defined.

    Definition Classes
    TreeNode
  16. def collectFirst[B](pf: PartialFunction[Expression, B]): Option[B]

    Finds and returns the first TreeNode of the tree for which the given partial function is defined (pre-order), and applies the partial function to it.

    Finds and returns the first TreeNode of the tree for which the given partial function is defined (pre-order), and applies the partial function to it.

    Definition Classes
    TreeNode
  17. def collectLeaves(): Seq[Expression]

    Returns a Seq containing the leaves in this tree.

    Returns a Seq containing the leaves in this tree.

    Definition Classes
    TreeNode
  18. lazy val containsChild: Set[TreeNode[_]]
    Definition Classes
    TreeNode
  19. def copyTagsFrom(other: Expression): Unit
    Attributes
    protected
    Definition Classes
    TreeNode
  20. def dataType: DataType

    Returns the DataType of the result of evaluating this expression.

    Returns the DataType of the result of evaluating this expression. It is invalid to query the dataType of an unresolved expression (i.e., when resolved == false).

    Definition Classes
    DecodeUsingSerializer → Expression
  21. def defineCodeGen(ctx: CodegenContext, ev: ExprCode, f: (String) ⇒ String): ExprCode

    Called by unary expressions to generate a code block that returns null if its parent returns null, and if not null, use f to generate the expression.

    Called by unary expressions to generate a code block that returns null if its parent returns null, and if not null, use f to generate the expression.

    As an example, the following does a boolean inversion (i.e. NOT).

    defineCodeGen(ctx, ev, c => s"!($c)")
    f

    function that accepts a variable name and returns Java code to compute the output.

    Attributes
    protected
    Definition Classes
    UnaryExpression
  22. lazy val deterministic: Boolean

    Returns true when the current expression always return the same result for fixed inputs from children.

    Returns true when the current expression always return the same result for fixed inputs from children. The non-deterministic expressions should not change in number and order. They should not be evaluated during the query planning.

    Note that this means that an expression should be considered as non-deterministic if: - it relies on some mutable internal state, or - it relies on some implicit input that is not part of the children expression list. - it has non-deterministic child or children. - it assumes the input satisfies some certain condition via the child operator.

    An example would be SparkPartitionID that relies on the partition id returned by TaskContext. By default leaf expressions are deterministic as Nil.forall(_.deterministic) returns true.

    Definition Classes
    Expression
  23. def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode

    Returns Java source code that can be compiled to evaluate this expression.

    Returns Java source code that can be compiled to evaluate this expression. The default behavior is to call the eval method of the expression. Concrete expression implementations should override this to do actual code generation.

    ctx

    a CodegenContext

    ev

    an ExprCode with unique terms.

    returns

    an ExprCode containing the Java source code to generate the given expression

    Attributes
    protected
    Definition Classes
    DecodeUsingSerializer → Expression
  24. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  25. def eval(input: InternalRow): Any

    Default behavior of evaluation according to the default nullability of UnaryExpression.

    Default behavior of evaluation according to the default nullability of UnaryExpression. If subclass of UnaryExpression override nullable, probably should also override this.

    Definition Classes
    UnaryExpression → Expression
  26. def fastEquals(other: TreeNode[_]): Boolean

    Faster version of equality which short-circuits when two treeNodes are the same instance.

    Faster version of equality which short-circuits when two treeNodes are the same instance. We don't just override Object.equals, as doing so prevents the scala compiler from generating case class equals methods

    Definition Classes
    TreeNode
  27. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  28. def find(f: (Expression) ⇒ Boolean): Option[Expression]

    Find the first TreeNode that satisfies the condition specified by f.

    Find the first TreeNode that satisfies the condition specified by f. The condition is recursively applied to this node and all of its children (pre-order).

    Definition Classes
    TreeNode
  29. def flatArguments: Iterator[Any]
    Attributes
    protected
    Definition Classes
    Expression
  30. def flatMap[A](f: (Expression) ⇒ TraversableOnce[A]): Seq[A]

    Returns a Seq by applying a function to all nodes in this tree and using the elements of the resulting collections.

    Returns a Seq by applying a function to all nodes in this tree and using the elements of the resulting collections.

    Definition Classes
    TreeNode
  31. def foldable: Boolean

    Returns true when an expression is a candidate for static evaluation before the query is executed.

    Returns true when an expression is a candidate for static evaluation before the query is executed. A typical use case: org.apache.spark.sql.catalyst.optimizer.ConstantFolding

    The following conditions are used to determine suitability for constant folding:

    Definition Classes
    UnaryExpression → Expression
  32. def foreach(f: (Expression) ⇒ Unit): Unit

    Runs the given function on this node and then recursively on children.

    Runs the given function on this node and then recursively on children.

    f

    the function to be applied to each node in the tree.

    Definition Classes
    TreeNode
  33. def foreachUp(f: (Expression) ⇒ Unit): Unit

    Runs the given function recursively on children then on this node.

    Runs the given function recursively on children then on this node.

    f

    the function to be applied to each node in the tree.

    Definition Classes
    TreeNode
  34. def genCode(ctx: CodegenContext): ExprCode

    Returns an ExprCode, that contains the Java source code to generate the result of evaluating the expression on an input row.

    Returns an ExprCode, that contains the Java source code to generate the result of evaluating the expression on an input row.

    ctx

    a CodegenContext

    returns

    ExprCode

    Definition Classes
    Expression
  35. def generateTreeString(depth: Int, lastChildren: Seq[Boolean], append: (String) ⇒ Unit, verbose: Boolean, prefix: String = "", addSuffix: Boolean = false, maxFields: Int, printNodeId: Boolean): Unit

    Appends the string representation of this node and its children to the given Writer.

    Appends the string representation of this node and its children to the given Writer.

    The i-th element in lastChildren indicates whether the ancestor of the current node at depth i + 1 is the last child of its own parent node. The depth of the root node is 0, and lastChildren for the root node should be empty.

    Note that this traversal (numbering) order must be the same as getNodeNumbered.

    Definition Classes
    TreeNode
  36. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  37. def getTagValue[T](tag: TreeNodeTag[T]): Option[T]
    Definition Classes
    TreeNode
  38. def hashCode(): Int
    Definition Classes
    TreeNode → AnyRef → Any
  39. def innerChildren: Seq[TreeNode[_]]

    All the nodes that should be shown as a inner nested tree of this node.

    All the nodes that should be shown as a inner nested tree of this node. For example, this can be used to show sub-queries.

    Definition Classes
    TreeNode
  40. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  41. def jsonFields: List[JField]
    Attributes
    protected
    Definition Classes
    TreeNode
  42. val kryo: Boolean

    If true, Kryo serialization is used, otherwise the Java one is used

    If true, Kryo serialization is used, otherwise the Java one is used

    Definition Classes
    DecodeUsingSerializer → SerializerSupport
  43. def makeCopy(newArgs: Array[AnyRef]): Expression

    Creates a copy of this type of tree node after a transformation.

    Creates a copy of this type of tree node after a transformation. Must be overridden by child classes that have constructor arguments that are not present in the productIterator.

    newArgs

    the new product arguments.

    Definition Classes
    TreeNode
  44. def map[A](f: (Expression) ⇒ A): Seq[A]

    Returns a Seq containing the result of applying the given function to each node in this tree in a preorder traversal.

    Returns a Seq containing the result of applying the given function to each node in this tree in a preorder traversal.

    f

    the function to be applied.

    Definition Classes
    TreeNode
  45. def mapChildren(f: (Expression) ⇒ Expression): Expression

    Returns a copy of this node where f has been applied to all the nodes in children.

    Returns a copy of this node where f has been applied to all the nodes in children.

    Definition Classes
    TreeNode
  46. def mapProductIterator[B](f: (Any) ⇒ B)(implicit arg0: ClassTag[B]): Array[B]

    Efficient alternative to productIterator.map(f).toArray.

    Efficient alternative to productIterator.map(f).toArray.

    Attributes
    protected
    Definition Classes
    TreeNode
  47. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  48. def nodeName: String

    Returns the name of this type of TreeNode.

    Returns the name of this type of TreeNode. Defaults to the class name. Note that we remove the "Exec" suffix for physical operators here.

    Definition Classes
    TreeNode
  49. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  50. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  51. def nullSafeCodeGen(ctx: CodegenContext, ev: ExprCode, f: (String) ⇒ String): ExprCode

    Called by unary expressions to generate a code block that returns null if its parent returns null, and if not null, use f to generate the expression.

    Called by unary expressions to generate a code block that returns null if its parent returns null, and if not null, use f to generate the expression.

    f

    function that accepts the non-null evaluation result name of child and returns Java code to compute the output.

    Attributes
    protected
    Definition Classes
    UnaryExpression
  52. def nullSafeEval(input: Any): Any

    Called by default eval implementation.

    Called by default eval implementation. If subclass of UnaryExpression keep the default nullability, they can override this method to save null-check code. If we need full control of evaluation process, we should override eval.

    Definition Classes
    DecodeUsingSerializer → UnaryExpression
  53. def nullable: Boolean
    Definition Classes
    UnaryExpression → Expression
  54. def numberedTreeString: String

    Returns a string representation of the nodes in this tree, where each operator is numbered.

    Returns a string representation of the nodes in this tree, where each operator is numbered. The numbers can be used with TreeNode.apply to easily access specific subtrees.

    The numbers are based on depth-first traversal of the tree (with innerChildren traversed first before children).

    Definition Classes
    TreeNode
  55. val origin: Origin
    Definition Classes
    TreeNode
  56. def otherCopyArgs: Seq[AnyRef]

    Args to the constructor that should be copied, but not transformed.

    Args to the constructor that should be copied, but not transformed. These are appended to the transformed args automatically by makeCopy

    Attributes
    protected
    Definition Classes
    TreeNode
  57. def p(number: Int): Expression

    Returns the tree node at the specified number, used primarily for interactive debugging.

    Returns the tree node at the specified number, used primarily for interactive debugging. Numbers for each node can be found in the numberedTreeString.

    This is a variant of apply that returns the node as BaseType (if the type matches).

    Definition Classes
    TreeNode
  58. def prettyJson: String
    Definition Classes
    TreeNode
  59. def prettyName: String

    Returns a user-facing string representation of this expression's name.

    Returns a user-facing string representation of this expression's name. This should usually match the name of the function in SQL.

    Definition Classes
    Expression
  60. def references: AttributeSet
    Definition Classes
    Expression
  61. lazy val resolved: Boolean

    Returns true if this expression and all its children have been resolved to a specific schema and input data types checking passed, and false if it still contains any unresolved placeholders or has data types mismatch.

    Returns true if this expression and all its children have been resolved to a specific schema and input data types checking passed, and false if it still contains any unresolved placeholders or has data types mismatch. Implementations of expressions should override this if the resolution of this type of expression involves more than just the resolution of its children and type checking.

    Definition Classes
    Expression
  62. def semanticEquals(other: Expression): Boolean

    Returns true when two expressions will always compute the same result, even if they differ cosmetically (i.e.

    Returns true when two expressions will always compute the same result, even if they differ cosmetically (i.e. capitalization of names in attributes may be different).

    See Canonicalize for more details.

    Definition Classes
    Expression
  63. def semanticHash(): Int

    Returns a hashCode for the calculation performed by this expression.

    Returns a hashCode for the calculation performed by this expression. Unlike the standard hashCode, an attempt has been made to eliminate cosmetic differences.

    See Canonicalize for more details.

    Definition Classes
    Expression
  64. lazy val serializerInstance: SerializerInstance

    The serializer instance to be used for serialization/deserialization in interpreted execution

    The serializer instance to be used for serialization/deserialization in interpreted execution

    Definition Classes
    SerializerSupport
  65. def setTagValue[T](tag: TreeNodeTag[T], value: T): Unit
    Definition Classes
    TreeNode
  66. def simpleString(maxFields: Int): String

    ONE line description of this node.

    ONE line description of this node.

    maxFields

    Maximum number of fields that will be converted to strings. Any elements beyond the limit will be dropped.

    Definition Classes
    Expression → TreeNode
  67. def simpleStringWithNodeId(): String

    ONE line description of this node containing the node identifier.

    ONE line description of this node containing the node identifier.

    Definition Classes
    Expression → TreeNode
  68. final def sql: String

    Returns SQL representation of this expression.

    Returns SQL representation of this expression. For expressions extending NonSQLExpression, this method may return an arbitrary user facing string.

    Definition Classes
    NonSQLExpression → Expression
  69. def stringArgs: Iterator[Any]

    The arguments that should be included in the arg string.

    The arguments that should be included in the arg string. Defaults to the productIterator.

    Attributes
    protected
    Definition Classes
    TreeNode
  70. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  71. val tag: ClassTag[T]
  72. def toJSON: String
    Definition Classes
    TreeNode
  73. def toString(): String
    Definition Classes
    Expression → TreeNode → AnyRef → Any
  74. def transform(rule: PartialFunction[Expression, Expression]): Expression

    Returns a copy of this node where rule has been recursively applied to the tree.

    Returns a copy of this node where rule has been recursively applied to the tree. When rule does not apply to a given node it is left unchanged. Users should not expect a specific directionality. If a specific directionality is needed, transformDown or transformUp should be used.

    rule

    the function use to transform this nodes children

    Definition Classes
    TreeNode
  75. def transformDown(rule: PartialFunction[Expression, Expression]): Expression

    Returns a copy of this node where rule has been recursively applied to it and all of its children (pre-order).

    Returns a copy of this node where rule has been recursively applied to it and all of its children (pre-order). When rule does not apply to a given node it is left unchanged.

    rule

    the function used to transform this nodes children

    Definition Classes
    TreeNode
  76. def transformUp(rule: PartialFunction[Expression, Expression]): Expression

    Returns a copy of this node where rule has been recursively applied first to all of its children and then itself (post-order).

    Returns a copy of this node where rule has been recursively applied first to all of its children and then itself (post-order). When rule does not apply to a given node, it is left unchanged.

    rule

    the function use to transform this nodes children

    Definition Classes
    TreeNode
  77. def treeString(append: (String) ⇒ Unit, verbose: Boolean, addSuffix: Boolean, maxFields: Int, printOperatorId: Boolean): Unit
    Definition Classes
    TreeNode
  78. final def treeString(verbose: Boolean, addSuffix: Boolean = false, maxFields: Int = SQLConf.get.maxToStringFields, printOperatorId: Boolean = false): String
    Definition Classes
    TreeNode
  79. final def treeString: String

    Returns a string representation of the nodes in this tree

    Returns a string representation of the nodes in this tree

    Definition Classes
    TreeNode
  80. def unsetTagValue[T](tag: TreeNodeTag[T]): Unit
    Definition Classes
    TreeNode
  81. final def verboseString(maxFields: Int): String

    ONE line description of this node with more information

    ONE line description of this node with more information

    Definition Classes
    Expression → TreeNode
  82. def verboseStringWithSuffix(maxFields: Int): String

    ONE line description of this node with some suffix information

    ONE line description of this node with some suffix information

    Definition Classes
    TreeNode
  83. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  84. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  85. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  86. def withNewChildren(newChildren: Seq[Expression]): Expression

    Returns a copy of this node with the children replaced.

    Returns a copy of this node with the children replaced. TODO: Validate somewhere (in debug mode?) that children are ordered correctly.

    Definition Classes
    TreeNode

Inherited from Serializable

Inherited from Serializable

Inherited from SerializerSupport

Inherited from NonSQLExpression

Inherited from UnaryExpression

Inherited from Expression

Inherited from TreeNode[Expression]

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped