expressions

package expressions

A set of classes that can be used to represent trees of relational expressions. A key goal of the expression library is to hide the details of naming and scoping from developers who want to manipulate trees of relational operators. As such, the library defines a special type of expression, a NamedExpression in addition to the standard collection of expressions.

Standard Expressions

A library of standard expressions (e.g., Add, EqualTo), aggregates (e.g., SUM, COUNT), and other computations (e.g. UDFs). Each expression type is capable of determining its output schema as a function of its children's output schema.

Named Expressions

Some expression are named and thus can be referenced by later operators in the dataflow graph. The two types of named expressions are AttributeReferences and Aliases. AttributeReferences refer to attributes of the input tuple for a given operator and form the leaves of some expression trees. Aliases assign a name to intermediate computations. For example, in the SQL statement SELECT a+b AS c FROM ..., the expressions a and b would be represented by AttributeReferences and c would be represented by an Alias.

During analysis, all named expressions are assigned a globally unique expression id, which can be used for equality comparisons. While the original names are kept around for debugging purposes, they should never be used to check if two attributes refer to the same value, as plan transformations can result in the introduction of naming ambiguity. For example, consider a plan that contains subqueries, both of which are reading from the same table. If an optimization removes the subqueries, scoping information would be destroyed, eliminating the ability to reason about which subquery produced a given attribute.

Evaluation

The result of expressions can be evaluated using the Expression.apply(Row) method.

Linear Supertypes

AnyRef, Any

Ordering

Alphabetic
By Inheritance

Inherited

expressions
AnyRef
Any

Hide All
Show All

Visibility

Public
All

Type Members

case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable
A function that get the absolute value of the numeric value.
A function that get the absolute value of the numeric value.

Annotations
@ExpressionDescription()
case class Acos(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Acosh(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Add(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

Annotations
@ExpressionDescription()
case class AddMonths(startDate: Expression, numMonths: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the date that is num_months after start_date.
Returns the date that is num_months after start_date.

Annotations
@ExpressionDescription()
abstract class AggregateWindowFunction extends DeclarativeAggregate with WindowFunction
case class Alias(child: Expression, name: String)(exprId: ExprId = NamedExpression.newExprId, qualifier: Seq[String] = Seq.empty, explicitMetadata: Option[Metadata] = None) extends UnaryExpression with NamedExpression with Product with Serializable
Used to assign a new name to a computation.
Used to assign a new name to a computation. For example the SQL expression "1 + 1 AS a" could be represented as follows: Alias(Add(Literal(1), Literal(1)), "a")()
Note that exprId and qualifiers are in a separate parameter list because we only pattern match on child and name.
Note that when creating a new Alias, all the AttributeReference that refer to the original alias should be updated to the new one.
child
The computation being performed
name
The name to be associated with the result of computing child.
exprId
A globally unique id used to check if an AttributeReference refers to this alias. Auto-assigned if left blank.
qualifier
An optional Seq of string that can be used to refer to this attribute in a fully qualified way. Consider the examples tableName.name, subQueryAlias.name. tableName and subQueryAlias are possible qualifiers.
explicitMetadata
Explicit metadata associated with this alias that overwrites child's.
case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate with Product with Serializable

Annotations
@ExpressionDescription()
case class AnsiCast(child: Expression, dataType: DataType, timeZoneId: Option[String] = None) extends CastBase with Product with Serializable
Cast the child expression to the target data type as per ANSI SQL standard.
Cast the child expression to the target data type as per ANSI SQL standard. A runtime exception will be thrown on casting failure such as converting an out-of-range value to an integral type.
When cast from/to timezone related types, we need timeZoneId, which will be resolved with session local timezone by an analyzer ResolveTimeZone.
case class ArrayAggregate(argument: Expression, zero: Expression, merge: Expression, finish: Expression) extends Expression with HigherOrderFunction with CodegenFallback with Product with Serializable
Applies a binary operator to a start value and all elements in the array.
Applies a binary operator to a start value and all elements in the array.

Annotations
@ExpressionDescription()
trait ArrayBasedSimpleHigherOrderFunction extends Expression with SimpleHigherOrderFunction
trait ArrayBinaryLike extends BinaryExpression with BinaryArrayExpressionWithImplicitCast with ArraySetLike
Will become common base class for ArrayUnion, ArrayIntersect, and ArrayExcept.
case class ArrayContains(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Checks if the array (left) has the element (right)
Checks if the array (left) has the element (right)

Annotations
@ExpressionDescription()
case class ArrayDistinct(child: Expression) extends UnaryExpression with ArraySetLike with ExpectsInputTypes with Product with Serializable
Removes duplicate values from the array.
Removes duplicate values from the array.

Annotations
@ExpressionDescription()
case class ArrayExcept(left: Expression, right: Expression) extends BinaryExpression with ArrayBinaryLike with ComplexTypeMergingExpression with Product with Serializable
Returns an array of the elements in the intersect of x and y, without duplicates
Returns an array of the elements in the intersect of x and y, without duplicates

Annotations
@ExpressionDescription()
case class ArrayExists(argument: Expression, function: Expression, followThreeValuedLogic: Boolean) extends Expression with ArrayBasedSimpleHigherOrderFunction with CodegenFallback with Product with Serializable
Tests whether a predicate holds for one or more elements in the array.
Tests whether a predicate holds for one or more elements in the array.

Annotations
@ExpressionDescription()
case class ArrayFilter(argument: Expression, function: Expression) extends Expression with ArrayBasedSimpleHigherOrderFunction with CodegenFallback with Product with Serializable
Filters the input array using the given lambda function.
Filters the input array using the given lambda function.

Annotations
@ExpressionDescription()
case class ArrayForAll(argument: Expression, function: Expression) extends Expression with ArrayBasedSimpleHigherOrderFunction with CodegenFallback with Product with Serializable
Tests whether a predicate holds for all elements in the array.
Tests whether a predicate holds for all elements in the array.

Annotations
@ExpressionDescription()
case class ArrayIntersect(left: Expression, right: Expression) extends BinaryExpression with ArrayBinaryLike with ComplexTypeMergingExpression with Product with Serializable
Returns an array of the elements in the intersect of x and y, without duplicates
Returns an array of the elements in the intersect of x and y, without duplicates

Annotations
@ExpressionDescription()
case class ArrayJoin(array: Expression, delimiter: Expression, nullReplacement: Option[Expression]) extends Expression with ExpectsInputTypes with Product with Serializable
Creates a String containing all the elements of the input array separated by the delimiter.
Creates a String containing all the elements of the input array separated by the delimiter.

Annotations
@ExpressionDescription()
case class ArrayMax(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the maximum value in the array.
Returns the maximum value in the array.

Annotations
@ExpressionDescription()
case class ArrayMin(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the minimum value in the array.
Returns the minimum value in the array.

Annotations
@ExpressionDescription()
case class ArrayPosition(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the position of the first occurrence of element in the given array as long.
Returns the position of the first occurrence of element in the given array as long. Returns 0 if the given value could not be found in the array. Returns null if either of the arguments are null
NOTE: that this is not zero based, but 1-based index. The first element in the array has index 1.

Annotations
@ExpressionDescription()
case class ArrayRemove(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Remove all elements that equal to element from the given array
Remove all elements that equal to element from the given array

Annotations
@ExpressionDescription()
case class ArrayRepeat(left: Expression, right: Expression) extends BinaryExpression with ExpectsInputTypes with Product with Serializable
Returns the array containing the given input value (left) count (right) times.
Returns the array containing the given input value (left) count (right) times.

Annotations
@ExpressionDescription()
trait ArraySetLike extends AnyRef
Will become common base class for ArrayDistinct, ArrayUnion, ArrayIntersect, and ArrayExcept.
case class ArraySort(argument: Expression, function: Expression) extends Expression with ArrayBasedSimpleHigherOrderFunction with CodegenFallback with Product with Serializable
Sorts elements in an array using a comparator function.
Sorts elements in an array using a comparator function.

Annotations
@ExpressionDescription()
trait ArraySortLike extends Expression with ExpectsInputTypes
Common base class for SortArray and ArraySort.
case class ArrayTransform(argument: Expression, function: Expression) extends Expression with ArrayBasedSimpleHigherOrderFunction with CodegenFallback with Product with Serializable
Transform elements in an array using the transform function.
Transform elements in an array using the transform function. This is similar to a map in functional programming.

Annotations
@ExpressionDescription()
case class ArrayUnion(left: Expression, right: Expression) extends BinaryExpression with ArrayBinaryLike with ComplexTypeMergingExpression with Product with Serializable
Returns an array of the elements in the union of x and y, without duplicates
Returns an array of the elements in the union of x and y, without duplicates

Annotations
@ExpressionDescription()
case class ArraysOverlap(left: Expression, right: Expression) extends BinaryExpression with BinaryArrayExpressionWithImplicitCast with Product with Serializable
Checks if the two arrays contain at least one common element.
Checks if the two arrays contain at least one common element.

Annotations
@ExpressionDescription()
case class ArraysZip(children: Seq[Expression]) extends Expression with ExpectsInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the numeric value of the first character of str.
Returns the numeric value of the first character of str.

Annotations
@ExpressionDescription()
case class Asin(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Asinh(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class AssertTrue(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
A function throws an exception if 'condition' is not true.
A function throws an exception if 'condition' is not true.

Annotations
@ExpressionDescription()
case class AtLeastNNonNulls(n: Int, children: Seq[Expression]) extends Expression with Predicate with Product with Serializable
A predicate that is evaluated to be true if there are at least n non-null and non-NaN values.
case class Atan(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Atan2(left: Expression, right: Expression) extends BinaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Atanh(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
abstract class Attribute extends LeafExpression with NamedExpression with NullIntolerant
class AttributeEquals extends AnyRef

Attributes
protected
class AttributeMap[A] extends Map[Attribute, A] with Serializable
case class AttributeReference(name: String, dataType: DataType, nullable: Boolean = true, metadata: Metadata = Metadata.empty)(exprId: ExprId = NamedExpression.newExprId, qualifier: Seq[String] = Seq.empty[String]) extends Attribute with Unevaluable with Product with Serializable
A reference to an attribute produced by another operator in the tree.
A reference to an attribute produced by another operator in the tree.
name
The name of this attribute, should only be used during analysis or for debugging.
dataType
The DataType of this attribute.
nullable
True if null is a valid value for this attribute.
metadata
The metadata of this attribute.
exprId
A globally unique id used to check if different AttributeReferences refer to the same attribute.
qualifier
An optional string that can be used to referred to this attribute in a fully qualified way. Consider the examples tableName.name, subQueryAlias.name. tableName and subQueryAlias are possible qualifiers.
implicit class AttributeSeq extends Serializable
Helper functions for working with Seq[Attribute].
class AttributeSet extends Iterable[Attribute] with Serializable
A Set designed to hold AttributeReference objects, that performs equality checking using expression id instead of standard java equality.
A Set designed to hold AttributeReference objects, that performs equality checking using expression id instead of standard java equality. Using expression id means that these sets will correctly test for membership, even when the AttributeReferences in question differ cosmetically (e.g., the names have different capitalizations).
Note that we do not override equality for Attribute references as it is really weird when AttributeReference("a"...) == AttributeReference("b", ...). This tactic leads to broken tests, and also makes doing transformations hard (we always try keep older trees instead of new ones when the transformation was a no-op).
case class BRound(child: Expression, scale: Expression) extends RoundBase with Serializable with ImplicitCastInputTypes with Product
Round an expression to d decimal places using HALF_EVEN rounding mode, also known as Gaussian rounding or bankers' rounding.
Round an expression to d decimal places using HALF_EVEN rounding mode, also known as Gaussian rounding or bankers' rounding. round(2.5) = 2.0, round(3.5) = 4.0.

Annotations
@ExpressionDescription()
case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Converts the argument from binary to a base 64 string.
Converts the argument from binary to a base 64 string.

Annotations
@ExpressionDescription()
trait BaseGenericInternalRow extends InternalRow
An extended version of InternalRow that implements all special getters, toString and equals/hashCode by genericGet.
class BaseOrdering extends Ordering[InternalRow]
A base class for generated/interpreted row ordering.
abstract class BasePredicate extends AnyRef
A base class for generated/interpreted predicate
case class Bin(child: Expression) extends UnaryExpression with Serializable with ImplicitCastInputTypes with Product

Annotations
@ExpressionDescription()
abstract class BinaryArithmetic extends BinaryOperator with NullIntolerant
trait BinaryArrayExpressionWithImplicitCast extends BinaryExpression with ImplicitCastInputTypes
Base trait for BinaryExpressions with two arrays of the same element type and implicit casting.
abstract class BinaryComparison extends BinaryOperator with Predicate
abstract class BinaryExpression extends Expression
An expression with two inputs and one output.
An expression with two inputs and one output. The output is by default evaluated to null if any input is evaluated to null.
abstract class BinaryMathExpression extends BinaryExpression with Serializable with ImplicitCastInputTypes
A binary expression specifically for math functions that take two Doubles as input and returns a Double.
abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
A BinaryExpression that is an operator, with two properties:
A BinaryExpression that is an operator, with two properties:
1. The string representation is "x symbol y", rather than "funcName(x, y)". 2. Two inputs are expected to be of the same type. If the two inputs have different types, the analyzer will find the tightest common type and do the proper type casting.
case class BitLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
A function that returns the bit length of the given string or binary expression.
A function that returns the bit length of the given string or binary expression.

Annotations
@ExpressionDescription()
case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable
A function that calculates bitwise and(&) of two numbers.
A function that calculates bitwise and(&) of two numbers.
Code generation inherited from BinaryArithmetic.

Annotations
@ExpressionDescription()
case class BitwiseCount(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable
A function that calculates bitwise not(~) of a number.
A function that calculates bitwise not(~) of a number.

Annotations
@ExpressionDescription()
case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable
A function that calculates bitwise or(|) of two numbers.
A function that calculates bitwise or(|) of two numbers.
Code generation inherited from BinaryArithmetic.

Annotations
@ExpressionDescription()
case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable
A function that calculates bitwise xor(^) of two numbers.
A function that calculates bitwise xor(^) of two numbers.
Code generation inherited from BinaryArithmetic.

Annotations
@ExpressionDescription()
case class BoundReference(ordinal: Int, dataType: DataType, nullable: Boolean) extends LeafExpression with Product with Serializable
A bound reference points to a specific slot in the input tuple, allowing the actual value to be retrieved more efficiently.
A bound reference points to a specific slot in the input tuple, allowing the actual value to be retrieved more efficiently. However, since operations like column pruning can change the layout of intermediate tuples, BindReferences should be run after all such transformations.
case class Bucket(numBuckets: Literal, child: Expression) extends PartitionTransformExpression with Product with Serializable
Expression for the v2 partition transform bucket.
case class CallMethodViaReflection(children: Seq[Expression]) extends Expression with CodegenFallback with Product with Serializable
An expression that invokes a method on a class via reflection.
An expression that invokes a method on a class via reflection.
For now, only types defined in Reflect.typeMapping are supported (basically primitives and string) as input types, and the output is turned automatically to a string.
Note that unlike Hive's reflect function, this expression calls only static methods (i.e. does not support calling non-static methods).
We should also look into how to consolidate this expression with org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke in the future.
children
the first element should be a literal string for the class name, and the second element should be a literal string for the method name, and the remaining are input arguments to the Java method.

Annotations
@ExpressionDescription()
case class CaseWhen(branches: Seq[(Expression, Expression)], elseValue: Option[Expression] = None) extends Expression with ComplexTypeMergingExpression with Serializable with Product
Case statements of the form "CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END".
Case statements of the form "CASE WHEN a THEN b [WHEN c THEN d]* [ELSE e] END". When a = true, returns b; when c = true, returns d; else returns e.
branches
seq of (branch condition, branch value)
elseValue
optional value for the else branch

Annotations
@ExpressionDescription()
case class Cast(child: Expression, dataType: DataType, timeZoneId: Option[String] = None) extends CastBase with Product with Serializable
Cast the child expression to the target data type.
Cast the child expression to the target data type.
When cast from/to timezone related types, we need timeZoneId, which will be resolved with session local timezone by an analyzer ResolveTimeZone.

Annotations
@ExpressionDescription()
abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression with NullIntolerant
case class Cbrt(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Ceil(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class CheckOverflow(child: Expression, dataType: DecimalType, nullOnOverflow: Boolean) extends UnaryExpression with Product with Serializable
Rounds the decimal to given scale and check whether the decimal can fit in provided precision or not.
Rounds the decimal to given scale and check whether the decimal can fit in provided precision or not. If not, if nullOnOverflow is true, it returns null; otherwise an ArithmeticException is thrown.
case class Chr(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the ASCII character having the binary equivalent to n.
Returns the ASCII character having the binary equivalent to n. If n is larger than 256 the result is equivalent to chr(n % 256)

Annotations
@ExpressionDescription()
case class Coalesce(children: Seq[Expression]) extends Expression with ComplexTypeMergingExpression with Product with Serializable
An expression that is evaluated to the first non-null input.
An expression that is evaluated to the first non-null input.
```
coalesce(1, 2) => 1
coalesce(null, 1, 2) => 1
coalesce(null, null, 2) => 2
coalesce(null, null, null) => null
```
Annotations
@ExpressionDescription()
abstract class CodeGeneratorWithInterpretedFallback[IN, OUT] extends Logging
A codegen object generator which creates objects with codegen path first.
A codegen object generator which creates objects with codegen path first. Once any compile error happens, it can fallback to interpreted implementation. In tests, we can use a SQL config SQLConf.CODEGEN_FACTORY_MODE to control fallback behavior.
trait CollectionGenerator extends Expression with Generator
A collection producing Generator.
A collection producing Generator. This trait provides a different path for code generation, by allowing code generation to return either an ArrayData or a MapData object.
trait ComplexTypeMergingExpression extends Expression
A trait used for resolving nullable flags, including nullable, containsNull of ArrayType and valueContainsNull of MapType, containsNull, valueContainsNull flags of the output date type.
A trait used for resolving nullable flags, including nullable, containsNull of ArrayType and valueContainsNull of MapType, containsNull, valueContainsNull flags of the output date type. This is usually utilized by the expressions (e.g. CaseWhen) that combine data from multiple child expressions of non-primitive types.
case class Concat(children: Seq[Expression]) extends Expression with ComplexTypeMergingExpression with Product with Serializable
Concatenates multiple input columns together into a single column.
Concatenates multiple input columns together into a single column. The function works with strings, binary and compatible array columns.

Annotations
@ExpressionDescription()
case class ConcatWs(children: Seq[Expression]) extends Expression with ImplicitCastInputTypes with Product with Serializable
An expression that concatenates multiple input strings or array of strings into a single string, using a given separator (the first child).
An expression that concatenates multiple input strings or array of strings into a single string, using a given separator (the first child).
Returns null if the separator is null. Otherwise, concat_ws skips all null values.

Annotations
@ExpressionDescription()
case class Contains(left: Expression, right: Expression) extends StringPredicate with Product with Serializable
A function that returns true if the string left contains the string right.
case class Conv(numExpr: Expression, fromBaseExpr: Expression, toBaseExpr: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
Convert a num from one base to another
Convert a num from one base to another
numExpr
the number to be converted
fromBaseExpr
from which base
toBaseExpr
to which base

Annotations
@ExpressionDescription()
case class Cos(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Cosh(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Cot(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
A function that computes a cyclic redundancy check value and returns it as a bigint For input of type BinaryType
A function that computes a cyclic redundancy check value and returns it as a bigint For input of type BinaryType

Annotations
@ExpressionDescription()
case class CreateArray(children: Seq[Expression], useStringTypeWhenEmpty: Boolean) extends Expression with Product with Serializable
Returns an Array containing the evaluation of all children expressions.
Returns an Array containing the evaluation of all children expressions.

Annotations
@ExpressionDescription()
case class CreateMap(children: Seq[Expression], useStringTypeWhenEmpty: Boolean) extends Expression with Product with Serializable
Returns a catalyst Map containing the evaluation of all children expressions as keys and values.
Returns a catalyst Map containing the evaluation of all children expressions as keys and values. The children are a flatted sequence of kv pairs, e.g. (key1, value1, key2, value2, ...)

Annotations
@ExpressionDescription()
case class CreateNamedStruct(children: Seq[Expression]) extends Expression with Product with Serializable
Creates a struct with the given field names and values
Creates a struct with the given field names and values
children
Seq(name1, val1, name2, val2, ...)

Annotations
@ExpressionDescription()
case class CsvToStructs(schema: StructType, options: Map[String, String], child: Expression, timeZoneId: Option[String] = None) extends UnaryExpression with TimeZoneAwareExpression with CodegenFallback with ExpectsInputTypes with NullIntolerant with Product with Serializable
Converts a CSV input string to a StructType with the specified schema.
Converts a CSV input string to a StructType with the specified schema.

Annotations
@ExpressionDescription()
case class Cube(groupByExprs: Seq[Expression]) extends Expression with GroupingSet with Product with Serializable

Annotations
@ExpressionDescription()
case class CumeDist() extends RowNumberLike with SizeBasedWindowFunction with Product with Serializable
The CumeDist function computes the position of a value relative to all values in the partition.
The CumeDist function computes the position of a value relative to all values in the partition. The result is the number of rows preceding or equal to the current row in the ordering of the partition divided by the total number of rows in the window partition. Any tie values in the ordering will evaluate to the same position.
This documentation has been based upon similar documentation for the Hive and Presto projects.

Annotations
@ExpressionDescription()
case class CurrentBatchTimestamp(timestampMs: Long, dataType: DataType, timeZoneId: Option[String] = None) extends LeafExpression with TimeZoneAwareExpression with Nondeterministic with CodegenFallback with Product with Serializable
Expression representing the current batch time, which is used by StreamExecution to 1.
Expression representing the current batch time, which is used by StreamExecution to 1. prevent optimizer from pushing this expression below a stateful operator 2. allow IncrementalExecution to substitute this expression with a Literal(timestamp)
There is no code generation since this expression should be replaced with a literal.
case class CurrentDatabase() extends LeafExpression with Unevaluable with Product with Serializable
Returns the current database of the SessionCatalog.
Returns the current database of the SessionCatalog.

Annotations
@ExpressionDescription()
case class CurrentDate(timeZoneId: Option[String] = None) extends LeafExpression with TimeZoneAwareExpression with CodegenFallback with Product with Serializable
Returns the current date at the start of query evaluation.
Returns the current date at the start of query evaluation. All calls of current_date within the same query return the same value.
There is no code generation since this expression should get constant folded by the optimizer.

Annotations
@ExpressionDescription()
case class CurrentTimestamp() extends CurrentTimestampLike with Product with Serializable
Returns the current timestamp at the start of query evaluation.
Returns the current timestamp at the start of query evaluation. All calls of current_timestamp within the same query return the same value.
There is no code generation since this expression should get constant folded by the optimizer.

Annotations
@ExpressionDescription()
abstract class CurrentTimestampLike extends LeafExpression with CodegenFallback
case class DateAdd(startDate: Expression, days: Expression) extends BinaryExpression with ExpectsInputTypes with Product with Serializable
Adds a number of days to startdate.
Adds a number of days to startdate.

Annotations
@ExpressionDescription()
case class DateAddInterval(start: Expression, interval: Expression, timeZoneId: Option[String] = None, ansiEnabled: Boolean = SQLConf.get.ansiEnabled) extends BinaryExpression with ExpectsInputTypes with TimeZoneAwareExpression with Product with Serializable
Adds date and an interval.
Adds date and an interval.
When ansi mode is on, the microseconds part of interval needs to be 0, otherwise a runtime IllegalArgumentException will be raised. When ansi mode is off, if the microseconds part of interval is 0, we perform date + interval for better performance. if the microseconds part is not 0, then the date will be converted to a timestamp to add with the whole interval parts.
case class DateDiff(endDate: Expression, startDate: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the number of days from startDate to endDate.
Returns the number of days from startDate to endDate.

Annotations
@ExpressionDescription()
case class DateFormatClass(left: Expression, right: Expression, timeZoneId: Option[String] = None) extends BinaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class DatePart(field: Expression, source: Expression, child: Expression) extends UnaryExpression with RuntimeReplaceable with Product with Serializable

Annotations
@ExpressionDescription()
case class DateSub(startDate: Expression, days: Expression) extends BinaryExpression with ExpectsInputTypes with Product with Serializable
Subtracts a number of days to startdate.
Subtracts a number of days to startdate.

Annotations
@ExpressionDescription()
case class DayOfMonth(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class DayOfWeek(child: Expression) extends DayWeek with Product with Serializable

Annotations
@ExpressionDescription()
case class DayOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
abstract class DayWeek extends UnaryExpression with ImplicitCastInputTypes
case class Days(child: Expression) extends PartitionTransformExpression with Product with Serializable
Expression for the v2 partition transform days.
case class Decode(bin: Expression, charset: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Decodes the first argument into a String using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
Decodes the first argument into a String using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.

Annotations
@ExpressionDescription()
case class DenseRank(children: Seq[Expression]) extends RankLike with Product with Serializable
The DenseRank function computes the rank of a value in a group of values.
The DenseRank function computes the rank of a value in a group of values. The result is one plus the previously assigned rank value. Unlike Rank, DenseRank will not produce gaps in the ranking sequence.
This documentation has been based upon similar documentation for the Hive and Presto projects.

Annotations
@ExpressionDescription()
trait DivModLike extends BinaryArithmetic
case class Divide(left: Expression, right: Expression) extends BinaryArithmetic with DivModLike with Product with Serializable

Annotations
@ExpressionDescription()
case class DivideInterval(interval: Expression, num: Expression, checkOverflow: Boolean = SQLConf.get.ansiEnabled) extends IntervalNumOperation with Product with Serializable
trait DynamicPruning extends Expression with Predicate
case class DynamicPruningExpression(child: Expression) extends UnaryExpression with DynamicPruning with Product with Serializable
Marker for a planned DynamicPruning expression.
Marker for a planned DynamicPruning expression. The expression is created during planning, and it defers to its child for evaluation.
child
underlying predicate.
case class DynamicPruningSubquery(pruningKey: Expression, buildQuery: LogicalPlan, buildKeys: Seq[Expression], broadcastKeyIndex: Int, onlyInBroadcast: Boolean, exprId: ExprId = NamedExpression.newExprId) extends SubqueryExpression with DynamicPruning with Unevaluable with Product with Serializable
The DynamicPruningSubquery expression is only used in join operations to prune one side of the join with a filter from the other side of the join.
The DynamicPruningSubquery expression is only used in join operations to prune one side of the join with a filter from the other side of the join. It is inserted in cases where partition pruning can be applied.
pruningKey
the filtering key of the plan to be pruned.
buildQuery
the build side of the join.
buildKeys
the join keys corresponding to the build side of the join
broadcastKeyIndex
the index of the filtering key collected from the broadcast
onlyInBroadcast
when set to false it indicates that the pruning filter is likely to be beneficial and so it should be executed even if it cannot reuse the results of the broadcast through ReuseExchange; otherwise, it will use the filter only if it can reuse the results of the broadcast through ReuseExchange
case class ElementAt(left: Expression, right: Expression) extends BinaryExpression with GetMapValueUtil with GetArrayItemUtil with Product with Serializable
Returns the value of index right in Array left or the value for key right in Map left.
Returns the value of index right in Array left or the value for key right in Map left.

Annotations
@ExpressionDescription()
case class Elt(children: Seq[Expression]) extends Expression with Product with Serializable
An expression that returns the n-th input in given inputs.
An expression that returns the n-th input in given inputs. If all inputs are binary, elt returns an output as binary. Otherwise, it returns as string. If any input is null, elt returns null.

Annotations
@ExpressionDescription()
case class Encode(value: Expression, charset: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Encodes the first argument into a BINARY using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
Encodes the first argument into a BINARY using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.

Annotations
@ExpressionDescription()
case class EndsWith(left: Expression, right: Expression) extends StringPredicate with Product with Serializable
A function that returns true if the string left ends with the string right.
case class EqualNullSafe(left: Expression, right: Expression) extends BinaryComparison with Product with Serializable

Annotations
@ExpressionDescription()
case class EqualTo(left: Expression, right: Expression) extends BinaryComparison with NullIntolerant with Product with Serializable

Annotations
@ExpressionDescription()
class EquivalentExpressions extends AnyRef
This class is used to compute equality of (sub)expression trees.
This class is used to compute equality of (sub)expression trees. Expressions can be added to this class and they subsequently query for expression equality. Expression trees are considered equal if for the same input(s), the same result is produced.
case class EulerNumber() extends LeafMathExpression with Product with Serializable
Euler's number.
Euler's number. Note that there is no code generation because this is only evaluated by the optimizer during constant folding.

Annotations
@ExpressionDescription()
case class Exists(plan: LogicalPlan, children: Seq[Expression] = Seq.empty, exprId: ExprId = NamedExpression.newExprId) extends SubqueryExpression with Predicate with Unevaluable with Product with Serializable
The Exists expression checks if a row exists in a subquery given some correlated condition or some uncorrelated condition.
The Exists expression checks if a row exists in a subquery given some correlated condition or some uncorrelated condition.
1. correlated condition:
For example (SQL):
```
SELECT  *
FROM    a
WHERE   EXISTS (SELECT  *
                FROM    b
                WHERE   b.id = a.id)
```
2. uncorrelated condition example:
For example (SQL):
```
SELECT  *
FROM    a
WHERE   EXISTS (SELECT  *
                FROM    b
                WHERE   b.id > 10)
```
case class Exp(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
trait ExpectsInputTypes extends Expression
A trait that gets mixin to define the expected input types of an expression.
A trait that gets mixin to define the expected input types of an expression.
This trait is typically used by operator expressions (e.g. Add, Subtract) to define expected input types without any implicit casting.
Most function expressions (e.g. Substring should extend ImplicitCastInputTypes) instead.
case class Explode(child: Expression) extends ExplodeBase with Product with Serializable
Given an input array produces a sequence of rows for each value in the array.
Given an input array produces a sequence of rows for each value in the array.
```
SELECT explode(array(10,20)) ->
10
20
```
Annotations
@ExpressionDescription()
abstract class ExplodeBase extends UnaryExpression with CollectionGenerator with Serializable
A base class for Explode and PosExplode.
case class Expm1(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class ExprId(id: Long, jvmId: UUID) extends Product with Serializable
A globally unique id for a given named expression.
A globally unique id for a given named expression. Used to identify which attribute output by a relation is being referenced in a subsequent computation.
The id field is unique within a given JVM, while the uuid is used to uniquely identify JVMs.
abstract class Expression extends TreeNode[Expression]
An expression in Catalyst.
An expression in Catalyst.
If an expression wants to be exposed in the function registry (so users can call it with "name(arguments...)", the concrete implementation must be a case class whose constructor arguments are all Expressions types. See Substring for an example.
There are a few important traits or abstract classes:
- Nondeterministic: an expression that is not deterministic. - Stateful: an expression that contains mutable state. For example, MonotonicallyIncreasingID and Rand. A stateful expression is always non-deterministic. - Unevaluable: an expression that is not supposed to be evaluated. - CodegenFallback: an expression that does not have code gen implemented and falls back to interpreted mode. - NullIntolerant: an expression that is null intolerant (i.e. any null input will result in null output). - NonSQLExpression: a common base trait for the expressions that do not have SQL expressions like representation. For example, ScalaUDF, ScalaUDAF, and object MapObjects and Invoke. - UserDefinedExpression: a common base trait for user-defined functions, including UDF/UDAF/UDTF. - HigherOrderFunction: a common base trait for higher order functions that take one or more (lambda) functions and applies these to some objects. The function produces a number of variables which can be consumed by some lambda functions. - NamedExpression: An Expression that is named. - TimeZoneAwareExpression: A common base trait for time zone aware expressions. - SubqueryExpression: A base interface for expressions that contain a LogicalPlan.
- LeafExpression: an expression that has no child. - UnaryExpression: an expression that has one child. - BinaryExpression: an expression that has two children. - TernaryExpression: an expression that has three children. - QuaternaryExpression: an expression that has four children. - BinaryOperator: a special case of BinaryExpression that requires two children to have the same output data type.
A few important traits used for type coercion rules: - ExpectsInputTypes: an expression that has the expected input types. This trait is typically used by operator expressions (e.g. Add, Subtract) to define expected input types without any implicit casting. - ImplicitCastInputTypes: an expression that has the expected input types, which can be implicitly castable using TypeCoercion.ImplicitTypeCasts. - ComplexTypeMergingExpression: to resolve output types of the complex expressions (e.g., CaseWhen).
class ExpressionDescription extends Annotation with Annotation with ClassfileAnnotation
class ExpressionInfo extends AnyRef
Expression information, will be used to describe a expression.
class ExpressionSet extends Set[Expression]
A Set where membership is determined based on determinacy and a canonical representation of an Expression (i.e.
A Set where membership is determined based on determinacy and a canonical representation of an Expression (i.e. one that attempts to ignore cosmetic differences). See Canonicalize for more details.
Internally this set uses the canonical representation, but keeps also track of the original expressions to ease debugging. Since different expressions can share the same canonical representation, this means that operations that extract expressions from this set are only guaranteed to see at least one such expression. For example:
```
val set = ExpressionSet(a + 1, 1 + a)

set.iterator => Iterator(a + 1)
set.contains(a + 1) => true
set.contains(1 + a) => true
set.contains(a + 2) => false
```
For non-deterministic expressions, they are always considered as not contained in the Set. On adding a non-deterministic expression, simply append it to the original expressions. This is consistent with how we define semanticEquals between two expressions.
trait ExpressionWithRandomSeed extends AnyRef
Represents the behavior of expressions which have a random seed and can renew the seed.
Represents the behavior of expressions which have a random seed and can renew the seed. Usually the random seed needs to be renewed at each execution under streaming queries.
case class Extract(field: Expression, source: Expression, child: Expression) extends UnaryExpression with RuntimeReplaceable with Product with Serializable

Annotations
@ExpressionDescription()
case class ExtractIntervalDays(child: Expression) extends ExtractIntervalPart with Product with Serializable
case class ExtractIntervalHours(child: Expression) extends ExtractIntervalPart with Product with Serializable
case class ExtractIntervalMinutes(child: Expression) extends ExtractIntervalPart with Product with Serializable
case class ExtractIntervalMonths(child: Expression) extends ExtractIntervalPart with Product with Serializable
abstract class ExtractIntervalPart extends UnaryExpression with ExpectsInputTypes with Serializable
case class ExtractIntervalSeconds(child: Expression) extends ExtractIntervalPart with Product with Serializable
case class ExtractIntervalYears(child: Expression) extends ExtractIntervalPart with Product with Serializable
trait ExtractValue extends Expression
case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class FindInSet(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
A function that returns the index (1-based) of the given string (left) in the comma- delimited list (right).
A function that returns the index (1-based) of the given string (left) in the comma- delimited list (right). Returns 0, if the string wasn't found or if the given string (left) contains a comma.

Annotations
@ExpressionDescription()
final class FixedLengthRowBasedKeyValueBatch extends RowBasedKeyValueBatch
An implementation of RowBasedKeyValueBatch in which all key-value records have same length.
An implementation of RowBasedKeyValueBatch in which all key-value records have same length.
The format for each record looks like this: [UnsafeRow for key of length klen] [UnsafeRow for Value of length vlen] [8 bytes pointer to next] Thus, record length = klen + vlen + 8
case class Flatten(child: Expression) extends UnaryExpression with Product with Serializable
Transforms an array of arrays into a single array.
Transforms an array of arrays into a single array.

Annotations
@ExpressionDescription()
case class Floor(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class FormatNumber(x: Expression, d: Expression) extends BinaryExpression with ExpectsInputTypes with Product with Serializable
Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string.
Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string. If D is 0, the result has no decimal point or fractional part.

Annotations
@ExpressionDescription()
case class FormatString(children: Expression*) extends Expression with ImplicitCastInputTypes with Product with Serializable
Returns the input formatted according do printf-style format strings
Returns the input formatted according do printf-style format strings

Annotations
@ExpressionDescription()
sealed trait FrameType extends AnyRef
The trait used to represent the type of a Window Frame.
case class FromUTCTimestamp(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
This is a common function for databases supporting TIMESTAMP WITHOUT TIMEZONE.
This is a common function for databases supporting TIMESTAMP WITHOUT TIMEZONE. This function takes a timestamp which is timezone-agnostic, and interprets it as a timestamp in UTC, and renders that timestamp as a timestamp in the given time zone.
However, timestamp in Spark represents number of microseconds from the Unix epoch, which is not timezone-agnostic. So in Spark this function just shift the timestamp value from UTC timezone to the given timezone.
This function may return confusing result if the input is a string with timezone, e.g. '2018-03-13T06:18:23+00:00'. The reason is that, Spark firstly cast the string to timestamp according to the timezone in the string, and finally display the result by converting the timestamp to string according to the session local timezone.

Annotations
@ExpressionDescription()
case class FromUnixTime(sec: Expression, format: Expression, timeZoneId: Option[String] = None) extends BinaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format. If the format is missing, using format like "1970-01-01 00:00:00". Note that hive Language Manual says it returns 0 if fail, but in fact it returns null.

Annotations
@ExpressionDescription()
trait Generator extends Expression
An expression that produces zero or more rows given a single input row.
An expression that produces zero or more rows given a single input row.
Generators produce multiple output rows instead of a single value like other expressions, and thus they must have a schema to associate with the rows that are output.
However, unlike row producing relational operators, which are either leaves or determine their output schema functionally from their input, generators can contain other expressions that might result in their modification by rules. This structure means that they might be copied multiple times after first determining their output schema. If a new output schema is created for each copy references up the tree might be rendered invalid. As a result generators must instead define a function makeOutput which is called only once when the schema is first requested. The attributes produced by this function will be automatically copied anytime rules result in changes to the Generator or its children.
case class GeneratorOuter(child: Generator) extends UnaryExpression with Generator with Product with Serializable
Wrapper around another generator to specify outer behavior.
Wrapper around another generator to specify outer behavior. This is used to implement functions such as explode_outer. This expression gets replaced during analysis.
class GenericInternalRow extends InternalRow with BaseGenericInternalRow
An internal row implementation that uses an array of objects as the underlying storage.
An internal row implementation that uses an array of objects as the underlying storage. Note that, while the array is not copied, and thus could technically be mutated after creation, this is not allowed.
class GenericRow extends Row
A row implementation that uses an array of objects as the underlying storage.
A row implementation that uses an array of objects as the underlying storage. Note that, while the array is not copied, and thus could technically be mutated after creation, this is not allowed.
class GenericRowWithSchema extends GenericRow
case class GetArrayItem(child: Expression, ordinal: Expression) extends BinaryExpression with GetArrayItemUtil with ExpectsInputTypes with ExtractValue with NullIntolerant with Product with Serializable
Returns the field at ordinal in the Array child.
Returns the field at ordinal in the Array child.
We need to do type checking here as ordinal expression maybe unresolved.
trait GetArrayItemUtil extends AnyRef
Common trait for GetArrayItem and ElementAt.
case class GetArrayStructFields(child: Expression, field: StructField, ordinal: Int, numFields: Int, containsNull: Boolean) extends UnaryExpression with ExtractValue with NullIntolerant with Product with Serializable
For a child whose data type is an array of structs, extracts the ordinal-th fields of all array elements, and returns them as a new array.
For a child whose data type is an array of structs, extracts the ordinal-th fields of all array elements, and returns them as a new array.
No need to do type checking since it is handled by ExtractValue.
case class GetJsonObject(json: Expression, path: Expression) extends BinaryExpression with ExpectsInputTypes with CodegenFallback with Product with Serializable
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object.
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object. It will return null if the input json string is invalid.

Annotations
@ExpressionDescription()
case class GetMapValue(child: Expression, key: Expression) extends BinaryExpression with GetMapValueUtil with ExtractValue with NullIntolerant with Product with Serializable
Returns the value of key key in Map child.
Returns the value of key key in Map child.
We need to do type checking here as key expression maybe unresolved.
trait GetMapValueUtil extends BinaryExpression with ImplicitCastInputTypes
Common trait for GetMapValue and ElementAt.
case class GetStructField(child: Expression, ordinal: Int, name: Option[String] = None) extends UnaryExpression with ExtractValue with NullIntolerant with Product with Serializable
Returns the value of fields in the Struct child.
Returns the value of fields in the Struct child.
No need to do type checking since it is handled by ExtractValue.
Note that we can pass in the field name directly to keep case preserving in toString. For example, when get field yEAr from <year: int, month: int>, we should pass in yEAr.
case class GreaterThan(left: Expression, right: Expression) extends BinaryComparison with NullIntolerant with Product with Serializable

Annotations
@ExpressionDescription()
case class GreaterThanOrEqual(left: Expression, right: Expression) extends BinaryComparison with NullIntolerant with Product with Serializable

Annotations
@ExpressionDescription()
case class Greatest(children: Seq[Expression]) extends Expression with ComplexTypeMergingExpression with Product with Serializable
A function that returns the greatest value of all parameters, skipping null values.
A function that returns the greatest value of all parameters, skipping null values. It takes at least 2 parameters, and returns null iff all parameters are null.

Annotations
@ExpressionDescription()
case class Grouping(child: Expression) extends Expression with Unevaluable with Product with Serializable
Indicates whether a specified column expression in a GROUP BY list is aggregated or not.
Indicates whether a specified column expression in a GROUP BY list is aggregated or not. GROUPING returns 1 for aggregated or 0 for not aggregated in the result set.

Annotations
@ExpressionDescription()
case class GroupingID(groupByExprs: Seq[Expression]) extends Expression with Unevaluable with Product with Serializable
GroupingID is a function that computes the level of grouping.
GroupingID is a function that computes the level of grouping.
If groupByExprs is empty, it means all grouping expressions in GroupingSets.

Annotations
@ExpressionDescription()
trait GroupingSet extends Expression with CodegenFallback
A placeholder expression for cube/rollup, which will be replaced by analyzer
abstract class HashExpression[E] extends Expression
A function that calculates hash value for a group of expressions.
A function that calculates hash value for a group of expressions. Note that the seed argument is not exposed to users and should only be set inside spark SQL.
The hash value for an expression depends on its type and seed:
- null: seed
- boolean: turn boolean into int, 1 for true, 0 for false, and then use murmur3 to hash this int with seed.
- byte, short, int: use murmur3 to hash the input as int with seed.
- long: use murmur3 to hash the long input with seed.
- float: turn it into int: java.lang.Float.floatToIntBits(input), and hash it.
- double: turn it into long: java.lang.Double.doubleToLongBits(input), and hash it.
- decimal: if it's a small decimal, i.e. precision <= 18, turn it into long and hash it. Else, turn it into bytes and hash it.
- calendar interval: hash microseconds first, and use the result as seed to hash months.
- binary: use murmur3 to hash the bytes with seed.
- string: get the bytes of string and hash it.
- array: The result starts with seed, then use result as seed, recursively calculate hash value for each element, and assign the element hash value to result.
- struct: The result starts with seed, then use result as seed, recursively calculate hash value for each field, and assign the field hash value to result.
Finally we aggregate the hash values for each expression by the same way of struct.
case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
If the argument is an INT or binary, hex returns the number as a STRING in hexadecimal format.
If the argument is an INT or binary, hex returns the number as a STRING in hexadecimal format. Otherwise if the number is a STRING, it converts each character into its hex representation and returns the resulting STRING. Negative numbers would be treated as two's complement.

Annotations
@ExpressionDescription()
trait HigherOrderFunction extends Expression with ExpectsInputTypes
A higher order function takes one or more (lambda) functions and applies these to some objects.
A higher order function takes one or more (lambda) functions and applies these to some objects. The function produces a number of variables which can be consumed by some lambda function.
case class HiveHash(children: Seq[Expression]) extends HashExpression[Int] with Product with Serializable
Simulates Hive's hashing function from Hive v1.2.1 at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils#hashcode()
Simulates Hive's hashing function from Hive v1.2.1 at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils#hashcode()
We should use this hash function for both shuffle and bucket of Hive tables, so that we can guarantee shuffle and bucketing have same data distribution

Annotations
@ExpressionDescription()
case class Hour(child: Expression, timeZoneId: Option[String] = None) extends UnaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class Hours(child: Expression) extends PartitionTransformExpression with Product with Serializable
Expression for the v2 partition transform hours.
case class Hypot(left: Expression, right: Expression) extends BinaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class If(predicate: Expression, trueValue: Expression, falseValue: Expression) extends Expression with ComplexTypeMergingExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class IfNull(left: Expression, right: Expression, child: Expression) extends UnaryExpression with RuntimeReplaceable with Product with Serializable

Annotations
@ExpressionDescription()
trait ImplicitCastInputTypes extends Expression with ExpectsInputTypes
A mixin for the analyzer to perform implicit type casting using org.apache.spark.sql.catalyst.analysis.TypeCoercion.ImplicitTypeCasts.
case class In(value: Expression, list: Seq[Expression]) extends Expression with Predicate with Product with Serializable
Evaluates to true if list contains value.
Evaluates to true if list contains value.

Annotations
@ExpressionDescription()
case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression with Predicate with Product with Serializable
Optimized version of In clause, when all filter values of In clause are static.
case class InSubquery(values: Seq[Expression], query: ListQuery) extends Expression with Predicate with Unevaluable with Product with Serializable
Evaluates to true if values are returned in query's result set.
case class InitCap(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns string, with the first letter of each word in uppercase, all other letters in lowercase.
Returns string, with the first letter of each word in uppercase, all other letters in lowercase. Words are delimited by whitespace.

Annotations
@ExpressionDescription()
case class Inline(child: Expression) extends UnaryExpression with CollectionGenerator with Product with Serializable
Explodes an array of structs into a table.
Explodes an array of structs into a table.

Annotations
@ExpressionDescription()
case class InputFileBlockLength() extends LeafExpression with Nondeterministic with Product with Serializable

Annotations
@ExpressionDescription()
case class InputFileBlockStart() extends LeafExpression with Nondeterministic with Product with Serializable

Annotations
@ExpressionDescription()
case class InputFileName() extends LeafExpression with Nondeterministic with Product with Serializable

Annotations
@ExpressionDescription()
case class IntegralDivide(left: Expression, right: Expression) extends BinaryArithmetic with DivModLike with Product with Serializable

Annotations
@ExpressionDescription()
abstract class InterpretedHashFunction extends AnyRef
Base class for interpreted hash functions.
class InterpretedMutableProjection extends MutableProjection
A MutableProjection that is calculated by calling eval on each of the specified expressions.
class InterpretedOrdering extends BaseOrdering
An interpreted row ordering comparator.
case class InterpretedPredicate(expression: Expression) extends BasePredicate with Product with Serializable
class InterpretedProjection extends Projection
A Projection that is calculated by calling the eval of each of the specified expressions.
class InterpretedSafeProjection extends Projection
An interpreted version of a safe projection.
class InterpretedUnsafeProjection extends UnsafeProjection
An interpreted unsafe projection.
An interpreted unsafe projection. This class reuses the UnsafeRow it produces, a consumer should copy the row if it is being buffered. This class is not thread safe.
abstract class IntervalNumOperation extends BinaryExpression with ImplicitCastInputTypes with Serializable
case class IsNaN(child: Expression) extends UnaryExpression with Predicate with ImplicitCastInputTypes with Product with Serializable
Evaluates to true iff it's NaN.
Evaluates to true iff it's NaN.

Annotations
@ExpressionDescription()
case class IsNotNull(child: Expression) extends UnaryExpression with Predicate with Product with Serializable
An expression that is evaluated to true if the input is not null.
An expression that is evaluated to true if the input is not null.

Annotations
@ExpressionDescription()
case class IsNull(child: Expression) extends UnaryExpression with Predicate with Product with Serializable
An expression that is evaluated to true if the input is null.
An expression that is evaluated to true if the input is null.

Annotations
@ExpressionDescription()
class JoinedRow extends InternalRow
A mutable wrapper that makes two rows appear as a single concatenated row.
A mutable wrapper that makes two rows appear as a single concatenated row. Designed to be instantiated once per thread and reused.
case class JsonToStructs(schema: DataType, options: Map[String, String], child: Expression, timeZoneId: Option[String] = None) extends UnaryExpression with TimeZoneAwareExpression with CodegenFallback with ExpectsInputTypes with Product with Serializable
Converts an json input string to a StructType, ArrayType or MapType with the specified schema.
Converts an json input string to a StructType, ArrayType or MapType with the specified schema.

Annotations
@ExpressionDescription()
case class JsonTuple(children: Seq[Expression]) extends Expression with Generator with CodegenFallback with Product with Serializable

Annotations
@ExpressionDescription()
case class KnownFloatingPointNormalized(child: Expression) extends UnaryExpression with TaggingExpression with Product with Serializable
case class KnownNotNull(child: Expression) extends UnaryExpression with TaggingExpression with Product with Serializable
case class Lag(input: Expression, offset: Expression, default: Expression) extends OffsetWindowFunction with Product with Serializable
The Lag function returns the value of input at the offsetth row before the current row in the window.
The Lag function returns the value of input at the offsetth row before the current row in the window. Offsets start at 0, which is the current row. The offset must be constant integer value. The default offset is 1. When the value of input is null at the offsetth row, null is returned. If there is no such offset row, the default expression is evaluated.

Annotations
@ExpressionDescription()
case class LambdaFunction(function: Expression, arguments: Seq[NamedExpression], hidden: Boolean = false) extends Expression with CodegenFallback with Product with Serializable
A lambda function and its arguments.
A lambda function and its arguments. A lambda function can be hidden when a user wants to process an completely independent expression in a HigherOrderFunction, the lambda function and its variables are then only used for internal bookkeeping within the higher order function.
case class LastDay(startDate: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the last day of the month which the date belongs to.
Returns the last day of the month which the date belongs to.

Annotations
@ExpressionDescription()
case class Lead(input: Expression, offset: Expression, default: Expression) extends OffsetWindowFunction with Product with Serializable
The Lead function returns the value of input at the offsetth row after the current row in the window.
The Lead function returns the value of input at the offsetth row after the current row in the window. Offsets start at 0, which is the current row. The offset must be constant integer value. The default offset is 1. When the value of input is null at the offsetth row, null is returned. If there is no such offset row, the default expression is evaluated.

Annotations
@ExpressionDescription()
abstract class LeafExpression extends Expression
A leaf expression, i.e.
A leaf expression, i.e. one without any child expressions.
abstract class LeafMathExpression extends LeafExpression with CodegenFallback with Serializable
A leaf expression specifically for math constants.
A leaf expression specifically for math constants. Math constants expect no input.
There is no code generation because they should get constant folded by the optimizer.
case class Least(children: Seq[Expression]) extends Expression with ComplexTypeMergingExpression with Product with Serializable
A function that returns the least value of all parameters, skipping null values.
A function that returns the least value of all parameters, skipping null values. It takes at least 2 parameters, and returns null iff all parameters are null.

Annotations
@ExpressionDescription()
case class Left(str: Expression, len: Expression, child: Expression) extends UnaryExpression with RuntimeReplaceable with Product with Serializable
Returns the leftmost n characters from the string.
Returns the leftmost n characters from the string.

Annotations
@ExpressionDescription()
case class Length(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
A function that returns the char length of the given string expression or number of bytes of the given binary expression.
A function that returns the char length of the given string expression or number of bytes of the given binary expression.

Annotations
@ExpressionDescription()
case class LessThan(left: Expression, right: Expression) extends BinaryComparison with NullIntolerant with Product with Serializable

Annotations
@ExpressionDescription()
case class LessThanOrEqual(left: Expression, right: Expression) extends BinaryComparison with NullIntolerant with Product with Serializable

Annotations
@ExpressionDescription()
case class Levenshtein(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
A function that return the Levenshtein distance between the two given strings.
A function that return the Levenshtein distance between the two given strings.

Annotations
@ExpressionDescription()
case class Like(left: Expression, right: Expression, escapeChar: Char) extends StringRegexExpression with Product with Serializable
Simple RegEx pattern matching function
Simple RegEx pattern matching function

Annotations
@ExpressionDescription()
case class ListQuery(plan: LogicalPlan, children: Seq[Expression] = Seq.empty, exprId: ExprId = NamedExpression.newExprId, childOutputs: Seq[Attribute] = Seq.empty) extends SubqueryExpression with Unevaluable with Product with Serializable
A ListQuery expression defines the query which we want to search in an IN subquery expression.
A ListQuery expression defines the query which we want to search in an IN subquery expression. It should and can only be used in conjunction with an IN expression.
For example (SQL):
```
SELECT  *
FROM    a
WHERE   a.id IN (SELECT  id
                 FROM    b)
```
case class Literal(value: Any, dataType: DataType) extends LeafExpression with Product with Serializable
In order to do type checking, use Literal.create() instead of constructor
case class Log(child: Expression) extends UnaryLogExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Log10(child: Expression) extends UnaryLogExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Log1p(child: Expression) extends UnaryLogExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Log2(child: Expression) extends UnaryLogExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Logarithm(left: Expression, right: Expression) extends BinaryMathExpression with Product with Serializable
Computes the logarithm of a number.
Computes the logarithm of a number.
left
the logarithm base, default to e.
right
the number to compute the logarithm of.

Annotations
@ExpressionDescription()
case class Lower(child: Expression) extends UnaryExpression with String2StringExpression with Product with Serializable
A function that converts the characters of a string to lowercase.
A function that converts the characters of a string to lowercase.

Annotations
@ExpressionDescription()
case class MakeDate(year: Expression, month: Expression, day: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class MakeDecimal(child: Expression, precision: Int, scale: Int, nullOnOverflow: Boolean) extends UnaryExpression with Product with Serializable
Create a Decimal from an unscaled Long value.
Create a Decimal from an unscaled Long value. Note: this expression is internal and created only by the optimizer, we don't need to do type check for it.
case class MakeInterval(years: Expression, months: Expression, weeks: Expression, days: Expression, hours: Expression, mins: Expression, secs: Expression) extends SeptenaryExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class MakeTimestamp(year: Expression, month: Expression, day: Expression, hour: Expression, min: Expression, sec: Expression, timezone: Option[Expression] = None, timeZoneId: Option[String] = None) extends SeptenaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
trait MapBasedSimpleHigherOrderFunction extends Expression with SimpleHigherOrderFunction
case class MapConcat(children: Seq[Expression]) extends Expression with ComplexTypeMergingExpression with Product with Serializable
Returns the union of all the given maps.
Returns the union of all the given maps.

Annotations
@ExpressionDescription()
case class MapEntries(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable
Returns an unordered array of all entries in the given map.
Returns an unordered array of all entries in the given map.

Annotations
@ExpressionDescription()
case class MapFilter(argument: Expression, function: Expression) extends Expression with MapBasedSimpleHigherOrderFunction with CodegenFallback with Product with Serializable
Filters entries in a map using the provided function.
Filters entries in a map using the provided function.

Annotations
@ExpressionDescription()
case class MapFromArrays(left: Expression, right: Expression) extends BinaryExpression with ExpectsInputTypes with Product with Serializable
Returns a catalyst Map containing the two arrays in children expressions as keys and values.
Returns a catalyst Map containing the two arrays in children expressions as keys and values.

Annotations
@ExpressionDescription()
case class MapFromEntries(child: Expression) extends UnaryExpression with Product with Serializable
Returns a map created from the given array of entries.
Returns a map created from the given array of entries.

Annotations
@ExpressionDescription()
case class MapKeys(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable
Returns an unordered array containing the keys of the map.
Returns an unordered array containing the keys of the map.

Annotations
@ExpressionDescription()
case class MapValues(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable
Returns an unordered array containing the values of the map.
Returns an unordered array containing the values of the map.

Annotations
@ExpressionDescription()
case class MapZipWith(left: Expression, right: Expression, function: Expression) extends Expression with HigherOrderFunction with CodegenFallback with Product with Serializable
Merges two given maps into a single map by applying function to the pair of values with the same key.
Merges two given maps into a single map by applying function to the pair of values with the same key.

Annotations
@ExpressionDescription()
case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
A function that calculates an MD5 128-bit checksum and returns it as a hex string For input of type BinaryType
A function that calculates an MD5 128-bit checksum and returns it as a hex string For input of type BinaryType

Annotations
@ExpressionDescription()
case class Minute(child: Expression, timeZoneId: Option[String] = None) extends UnaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class MonotonicallyIncreasingID() extends LeafExpression with Stateful with Product with Serializable
Returns monotonically increasing 64-bit integers.
Returns monotonically increasing 64-bit integers.
The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the lower 33 bits represent the record number within each partition. The assumption is that the data frame has less than 1 billion partitions, and each partition has less than 8 billion records.
Since this expression is stateful, it cannot be a case object.

Annotations
@ExpressionDescription()
case class Month(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class Months(child: Expression) extends PartitionTransformExpression with Product with Serializable
Expression for the v2 partition transform months.
case class MonthsBetween(date1: Expression, date2: Expression, roundOff: Expression, timeZoneId: Option[String] = None) extends TernaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable
Returns number of months between times timestamp1 and timestamp2.
Returns number of months between times timestamp1 and timestamp2. If timestamp1 is later than timestamp2, then the result is positive. If timestamp1 and timestamp2 are on the same day of month, or both are the last day of month, time of day will be ignored. Otherwise, the difference is calculated based on 31 days per month, and rounded to 8 digits unless roundOff=false.

Annotations
@ExpressionDescription()
case class Multiply(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

Annotations
@ExpressionDescription()
case class MultiplyInterval(interval: Expression, num: Expression, checkOverflow: Boolean = SQLConf.get.ansiEnabled) extends IntervalNumOperation with Product with Serializable
case class Murmur3Hash(children: Seq[Expression], seed: Int) extends HashExpression[Int] with Product with Serializable
A MurMur3 Hash expression.
A MurMur3 Hash expression.
We should use this hash function for both shuffle and bucket, so that we can guarantee shuffle and bucketing have same data distribution.

Annotations
@ExpressionDescription()
final class MutableAny extends MutableValue
final class MutableBoolean extends MutableValue
final class MutableByte extends MutableValue
final class MutableDouble extends MutableValue
final class MutableFloat extends MutableValue
final class MutableInt extends MutableValue
final class MutableLong extends MutableValue
abstract class MutableProjection extends Projection
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row.
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row. If the schema of the input row is specified, then the given expression will be bound to that schema.
In contrast to a normal projection, a MutableProjection reuses the same underlying row object each time an input row is added. This significantly reduces the cost of calculating the projection, but means that it is not safe to hold on to a reference to a InternalRow after next() has been called on the Iterator that produced it. Instead, the user must call InternalRow.copy() and hold on to the returned InternalRow before calling next().
final class MutableShort extends MutableValue

abstract class MutableValue extends Serializable

A parent class for mutable container objects that are reused when the values are changed, resulting in less garbage.

A parent class for mutable container objects that are reused when the values are changed, resulting in less garbage. These values are held by a SpecificInternalRow.

The following code was roughly used to generate these objects:

val types = "Int,Float,Boolean,Double,Short,Long,Byte,Any".split(",")
types.map {tpe =>
s"""
final class Mutable$tpe extends MutableValue {
  var value: $tpe = 0
  def boxed = if (isNull) null else value
  def update(v: Any) = value = {
    isNull = false
    v.asInstanceOf[$tpe]
  }
  def copy() = {
    val newCopy = new Mutable$tpe
    newCopy.isNull = isNull
    newCopy.value = value
    newCopy
  }
}"""
}.foreach(println)

types.map { tpe =>
s"""
  override def set$tpe(ordinal: Int, value: $tpe): Unit = {
    val currentValue = values(ordinal).asInstanceOf[Mutable$tpe]
    currentValue.isNull = false
    currentValue.value = value
  }

  override def get$tpe(i: Int): $tpe = {
    values(i).asInstanceOf[Mutable$tpe].value
  }"""
}.foreach(println)

case class NTile(buckets: Expression) extends RowNumberLike with SizeBasedWindowFunction with Product with Serializable
The NTile function divides the rows for each window partition into n buckets ranging from 1 to at most n.
The NTile function divides the rows for each window partition into n buckets ranging from 1 to at most n. Bucket values will differ by at most 1. If the number of rows in the partition does not divide evenly into the number of buckets, then the remainder values are distributed one per bucket, starting with the first bucket.
The NTile function is particularly useful for the calculation of tertiles, quartiles, deciles and other common summary statistics
The function calculates two variables during initialization: The size of a regular bucket, and the number of buckets that will have one extra row added to it (when the rows do not evenly fit into the number of buckets); both variables are based on the size of the current partition. During the calculation process the function keeps track of the current row number, the current bucket number, and the row number at which the bucket will change (bucketThreshold). When the current row number reaches bucket threshold, the bucket value is increased by one and the threshold is increased by the bucket size (plus one extra if the current bucket is padded).
This documentation has been based upon similar documentation for the Hive and Presto projects.

Annotations
@ExpressionDescription()
case class NaNvl(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
An Expression evaluates to left iff it's not NaN, or evaluates to right otherwise.
An Expression evaluates to left iff it's not NaN, or evaluates to right otherwise. This Expression is useful for mapping NaN values to null.

Annotations
@ExpressionDescription()
trait NamedExpression extends Expression
An Expression that is named.
case class NamedLambdaVariable(name: String, dataType: DataType, nullable: Boolean, exprId: ExprId = NamedExpression.newExprId, value: AtomicReference[Any] = new AtomicReference()) extends LeafExpression with NamedExpression with CodegenFallback with Product with Serializable
A named lambda variable.
case class NextDay(startDate: Expression, dayOfWeek: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the first date which is later than startDate and named as dayOfWeek.
Returns the first date which is later than startDate and named as dayOfWeek. For example, NextDay(2015-07-27, Sunday) would return 2015-08-02, which is the first Sunday later than 2015-07-27.
Allowed "dayOfWeek" is defined in DateTimeUtils.getDayOfWeekFromString.

Annotations
@ExpressionDescription()
trait NonSQLExpression extends Expression
Expressions that don't have SQL representation should extend this trait.
Expressions that don't have SQL representation should extend this trait. Examples are ScalaUDF, ScalaUDAF, and object expressions like MapObjects and Invoke.
trait Nondeterministic extends Expression
An expression that is nondeterministic.
case class Not(child: Expression) extends UnaryExpression with Predicate with ImplicitCastInputTypes with NullIntolerant with Product with Serializable

Annotations
@ExpressionDescription()
case class Now() extends CurrentTimestampLike with Product with Serializable

Annotations
@ExpressionDescription()
case class NullIf(left: Expression, right: Expression, child: Expression) extends UnaryExpression with RuntimeReplaceable with Product with Serializable

Annotations
@ExpressionDescription()
trait NullIntolerant extends Expression
When an expression inherits this, meaning the expression is null intolerant (i.e.
When an expression inherits this, meaning the expression is null intolerant (i.e. any null input will result in null output). We will use this information during constructing IsNotNull constraints.
sealed abstract class NullOrdering extends AnyRef
case class Nvl(left: Expression, right: Expression, child: Expression) extends UnaryExpression with RuntimeReplaceable with Product with Serializable

Annotations
@ExpressionDescription()
case class Nvl2(expr1: Expression, expr2: Expression, expr3: Expression, child: Expression) extends UnaryExpression with RuntimeReplaceable with Product with Serializable

Annotations
@ExpressionDescription()
case class OctetLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
A function that returns the byte length of the given string or binary expression.
A function that returns the byte length of the given string or binary expression.

Annotations
@ExpressionDescription()
abstract class OffsetWindowFunction extends Expression with WindowFunction with Unevaluable with ImplicitCastInputTypes
An offset window function is a window function that returns the value of the input column offset by a number of rows within the partition.
An offset window function is a window function that returns the value of the input column offset by a number of rows within the partition. For instance: an OffsetWindowfunction for value x with offset -2, will get the value of x 2 rows back in the partition.
case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate with Product with Serializable

Annotations
@ExpressionDescription()
case class OuterReference(e: NamedExpression) extends LeafExpression with NamedExpression with Unevaluable with Product with Serializable
A place holder used to hold a reference that has been resolved to a field outside of the current plan.
A place holder used to hold a reference that has been resolved to a field outside of the current plan. This is used for correlated subqueries.
case class Overlay(input: Expression, replace: Expression, pos: Expression, len: Expression) extends QuaternaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable

Annotations
@ExpressionDescription()
case class ParseToDate(left: Expression, format: Option[Expression], child: Expression) extends UnaryExpression with RuntimeReplaceable with Product with Serializable
Parses a column to a date based on the given format.
Parses a column to a date based on the given format.

Annotations
@ExpressionDescription()
case class ParseToTimestamp(left: Expression, format: Option[Expression], child: Expression) extends UnaryExpression with RuntimeReplaceable with Product with Serializable
Parses a column to a timestamp based on the supplied format.
Parses a column to a timestamp based on the supplied format.

Annotations
@ExpressionDescription()
case class ParseUrl(children: Seq[Expression]) extends Expression with ExpectsInputTypes with CodegenFallback with Product with Serializable
Extracts a part from a URL
Extracts a part from a URL

Annotations
@ExpressionDescription()
abstract class PartitionTransformExpression extends Expression with Unevaluable
Base class for expressions that are converted to v2 partition transforms.
Base class for expressions that are converted to v2 partition transforms.
Subclasses represent abstract transform functions with concrete implementations that are determined by data source implementations. Because the concrete implementation is not known, these expressions are Unevaluable.
These expressions are used to pass transformations from the DataFrame API:
```
df.writeTo("catalog.db.table").partitionedBy($"category", days($"timestamp")).create()
```
case class PercentRank(children: Seq[Expression]) extends RankLike with SizeBasedWindowFunction with Product with Serializable
The PercentRank function computes the percentage ranking of a value in a group of values.
The PercentRank function computes the percentage ranking of a value in a group of values. The result the rank of the minus one divided by the total number of rows in the partition minus one: (r - 1) / (n - 1). If a partition only contains one row, the function will return 0.
The PercentRank function is similar to the CumeDist function, but it uses rank values instead of row counts in the its numerator.
This documentation has been based upon similar documentation for the Hive and Presto projects.

Annotations
@ExpressionDescription()
case class Pi() extends LeafMathExpression with Product with Serializable
Pi.
Pi. Note that there is no code generation because this is only evaluated by the optimizer during constant folding.

Annotations
@ExpressionDescription()
abstract class PlanExpression[T <: QueryPlan[_]] extends Expression
An interface for expressions that contain a QueryPlan.
case class Pmod(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

Annotations
@ExpressionDescription()
case class PosExplode(child: Expression) extends ExplodeBase with Product with Serializable
Given an input array produces a sequence of rows for each position and value in the array.
Given an input array produces a sequence of rows for each position and value in the array.
```
SELECT posexplode(array(10,20)) ->
0  10
1  20
```
Annotations
@ExpressionDescription()
case class Pow(left: Expression, right: Expression) extends BinaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class PreciseTimestampConversion(child: Expression, fromType: DataType, toType: DataType) extends UnaryExpression with ExpectsInputTypes with Product with Serializable
Expression used internally to convert the TimestampType to Long and back without losing precision, i.e.
Expression used internally to convert the TimestampType to Long and back without losing precision, i.e. in microseconds. Used in time windowing.
trait Predicate extends Expression
An Expression that returns a boolean value.
trait PredicateHelper extends AnyRef
case class PrettyAttribute(name: String, dataType: DataType = NullType) extends Attribute with Unevaluable with Product with Serializable
A place holder used when printing expressions without debugging information such as the expression id or the unresolved indicator.
case class PrintToStderr(child: Expression) extends UnaryExpression with Product with Serializable
Print the result of an expression to stderr (used for debugging codegen).
abstract class Projection extends (InternalRow) ⇒ InternalRow
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row.
Converts a InternalRow to another Row given a sequence of expression that define each column of the new row. If the schema of the input row is specified, then the given expression will be bound to that schema.
case class ProjectionOverSchema(schema: StructType) extends Product with Serializable
A Scala extractor that projects an expression over a given schema.
A Scala extractor that projects an expression over a given schema. Data types, field indexes and field counts of complex type extractors and attributes are adjusted to fit the schema. All other expressions are left as-is. This class is motivated by columnar nested schema pruning.
case class PromotePrecision(child: Expression) extends UnaryExpression with Product with Serializable
An expression used to wrap the children when promote the precision of DecimalType to avoid promote multiple times.
case class PythonUDF(name: String, func: PythonFunction, dataType: DataType, children: Seq[Expression], evalType: Int, udfDeterministic: Boolean, resultId: ExprId = NamedExpression.newExprId) extends Expression with Unevaluable with NonSQLExpression with UserDefinedExpression with Product with Serializable
A serialized version of a Python lambda function.
A serialized version of a Python lambda function. This is a special expression, which needs a dedicated physical operator to execute it, and thus can't be pushed down to data sources.
case class Quarter(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
abstract class QuaternaryExpression extends Expression
An expression with four inputs and one output.
An expression with four inputs and one output. The output is by default evaluated to null if any input is evaluated to null.
abstract class RDG extends UnaryExpression with ExpectsInputTypes with Stateful
A Random distribution generating expression.
A Random distribution generating expression. TODO: This can be made generic to generate any type of random distribution, or any type of StructType.
Since this expression is stateful, it cannot be a case object.
case class RLike(left: Expression, right: Expression) extends StringRegexExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Rand(child: Expression) extends RDG with ExpressionWithRandomSeed with Product with Serializable
Generate a random column with i.i.d.
Generate a random column with i.i.d. uniformly distributed values in [0, 1).

Annotations
@ExpressionDescription()
case class Randn(child: Expression) extends RDG with ExpressionWithRandomSeed with Product with Serializable
Generate a random column with i.i.d.
Generate a random column with i.i.d. values drawn from the standard normal distribution.

Annotations
@ExpressionDescription()
case class Rank(children: Seq[Expression]) extends RankLike with Product with Serializable
The Rank function computes the rank of a value in a group of values.
The Rank function computes the rank of a value in a group of values. The result is one plus the number of rows preceding or equal to the current row in the ordering of the partition. The values will produce gaps in the sequence.
This documentation has been based upon similar documentation for the Hive and Presto projects.

Annotations
@ExpressionDescription()
abstract class RankLike extends AggregateWindowFunction
A RankLike function is a WindowFunction that changes its value based on a change in the value of the order of the window in which is processed.
A RankLike function is a WindowFunction that changes its value based on a change in the value of the order of the window in which is processed. For instance, when the value of input changes in a window ordered by input the rank function also changes. The size of the change of the rank function is (typically) not dependent on the size of the change in input.
This documentation has been based upon similar documentation for the Hive and Presto projects.
case class RegExpExtract(subject: Expression, regexp: Expression, idx: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
Extract a specific(idx) group identified by a Java regex.
Extract a specific(idx) group identified by a Java regex.
NOTE: this expression is not THREAD-SAFE, as it has some internal mutable status.

Annotations
@ExpressionDescription()
case class RegExpReplace(subject: Expression, regexp: Expression, rep: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
Replace all substrings of str that match regexp with rep.
Replace all substrings of str that match regexp with rep.
NOTE: this expression is not THREAD-SAFE, as it has some internal mutable status.

Annotations
@ExpressionDescription()
case class Remainder(left: Expression, right: Expression) extends BinaryArithmetic with DivModLike with Product with Serializable

Annotations
@ExpressionDescription()
case class ReplicateRows(children: Seq[Expression]) extends Expression with Generator with CodegenFallback with Product with Serializable
Replicate the row N times.
Replicate the row N times. N is specified as the first argument to the function. This is an internal function solely used by optimizer to rewrite EXCEPT ALL AND INTERSECT ALL queries.
case class Reverse(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns a reversed string or an array with reverse order of elements.
Returns a reversed string or an array with reverse order of elements.

Annotations
@ExpressionDescription()
case class Right(str: Expression, len: Expression, child: Expression) extends UnaryExpression with RuntimeReplaceable with Product with Serializable
Returns the rightmost n characters from the string.
Returns the rightmost n characters from the string.

Annotations
@ExpressionDescription()
case class Rint(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Rollup(groupByExprs: Seq[Expression]) extends Expression with GroupingSet with Product with Serializable

Annotations
@ExpressionDescription()
case class Round(child: Expression, scale: Expression) extends RoundBase with Serializable with ImplicitCastInputTypes with Product
Round an expression to d decimal places using HALF_UP rounding mode.
Round an expression to d decimal places using HALF_UP rounding mode. round(2.5) == 3.0, round(3.5) == 4.0.

Annotations
@ExpressionDescription()
abstract class RoundBase extends BinaryExpression with Serializable with ImplicitCastInputTypes
Round the child's result to scale decimal place when scale >= 0 or round at integral part when scale < 0.
Round the child's result to scale decimal place when scale >= 0 or round at integral part when scale < 0.
Child of IntegralType would round to itself when scale >= 0. Child of FractionalType whose value is NaN or Infinite would always round to itself.
Round's dataType would always equal to child's dataType except for DecimalType, which would lead scale decrease from the origin DecimalType.
abstract class RowBasedKeyValueBatch extends MemoryConsumer with Closeable
RowBasedKeyValueBatch stores key value pairs in contiguous memory region.
RowBasedKeyValueBatch stores key value pairs in contiguous memory region.
Each key or value is stored as a single UnsafeRow. Each record contains one key and one value and some auxiliary data, which differs based on implementation: i.e., FixedLengthRowBasedKeyValueBatch and VariableLengthRowBasedKeyValueBatch.
We use FixedLengthRowBasedKeyValueBatch if all fields in the key and the value are fixed-length data types. Otherwise we use VariableLengthRowBasedKeyValueBatch.
RowBasedKeyValueBatch is backed by a single page / MemoryBlock (ranges from 1 to 64MB depending on the system configuration). If the page is full, the aggregate logic should fallback to a second level, larger hash map. We intentionally use the single-page design because it simplifies memory address encoding & decoding for each key-value pair. Because the maximum capacity for RowBasedKeyValueBatch is only 2^16, it is unlikely we need a second page anyway. Filling the page requires an average size for key value pairs to be larger than 1024 bytes.
case class RowNumber() extends RowNumberLike with Product with Serializable
The RowNumber function computes a unique, sequential number to each row, starting with one, according to the ordering of rows within the window partition.
The RowNumber function computes a unique, sequential number to each row, starting with one, according to the ordering of rows within the window partition.
This documentation has been based upon similar documentation for the Hive and Presto projects.

Annotations
@ExpressionDescription()
abstract class RowNumberLike extends AggregateWindowFunction
trait RuntimeReplaceable extends UnaryExpression with Unevaluable
An expression that gets replaced at runtime (currently by the optimizer) into a different expression for evaluation.
An expression that gets replaced at runtime (currently by the optimizer) into a different expression for evaluation. This is mainly used to provide compatibility with other databases. For example, we use this to support "nvl" by replacing it with "coalesce".
A RuntimeReplaceable should have the original parameters along with a "child" expression in the case class constructor, and define a normal constructor that accepts only the original parameters. For an example, see Nvl. To make sure the explain plan and expression SQL works correctly, the implementation should also override flatArguments method and sql method.
case class ScalaUDF(function: AnyRef, dataType: DataType, children: Seq[Expression], inputEncoders: Seq[Option[ExpressionEncoder[_]]] = Nil, udfName: Option[String] = None, nullable: Boolean = true, udfDeterministic: Boolean = true) extends Expression with NonSQLExpression with UserDefinedExpression with Product with Serializable
User-defined function.
User-defined function.
function
The user defined scala function to run. Note that if you use primitive parameters, you are not able to check if it is null or not, and the UDF will return null for you if the primitive input is null. Use boxed type or Option if you wanna do the null-handling yourself.
dataType
Return type of function.
children
The input expressions of this UDF.
inputEncoders
ExpressionEncoder for each input parameters. For a input parameter which serialized as struct will use encoder instead of CatalystTypeConverters to convert internal value to Scala value.
udfName
The user-specified name of this UDF.
nullable
True if the UDF can return null value.
udfDeterministic
True if the UDF is deterministic. Deterministic UDF returns same result each time it is invoked with a particular input.
case class ScalarSubquery(plan: LogicalPlan, children: Seq[Expression] = Seq.empty, exprId: ExprId = NamedExpression.newExprId) extends SubqueryExpression with Unevaluable with Product with Serializable
A subquery that will return only one row and one column.
A subquery that will return only one row and one column. This will be converted into a physical scalar subquery during planning.
Note: exprId is used to have a unique name in explain string output.
case class SchemaOfCsv(child: Expression, options: Map[String, String]) extends UnaryExpression with CodegenFallback with Product with Serializable
A function infers schema of CSV string.
A function infers schema of CSV string.

Annotations
@ExpressionDescription()
case class SchemaOfJson(child: Expression, options: Map[String, String]) extends UnaryExpression with CodegenFallback with Product with Serializable
A function infers schema of JSON string.
A function infers schema of JSON string.

Annotations
@ExpressionDescription()
case class Second(child: Expression, timeZoneId: Option[String] = None) extends UnaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class SecondWithFraction(child: Expression, timeZoneId: Option[String] = None) extends UnaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable
case class Sentences(str: Expression, language: Expression = Literal(""), country: Expression = Literal("")) extends Expression with ImplicitCastInputTypes with CodegenFallback with Product with Serializable
Splits a string into arrays of sentences, where each sentence is an array of words.
Splits a string into arrays of sentences, where each sentence is an array of words. The 'lang' and 'country' arguments are optional, and if omitted, the default locale is used.

Annotations
@ExpressionDescription()
abstract class SeptenaryExpression extends Expression
An expression with six inputs + 7th optional input and one output.
An expression with six inputs + 7th optional input and one output. The output is by default evaluated to null if any input is evaluated to null.
case class Sequence(start: Expression, stop: Expression, stepOpt: Option[Expression], timeZoneId: Option[String] = None) extends Expression with TimeZoneAwareExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
A function that calculates a sha1 hash value and returns it as a hex string For input of type BinaryType or StringType
A function that calculates a sha1 hash value and returns it as a hex string For input of type BinaryType or StringType

Annotations
@ExpressionDescription()
case class Sha2(left: Expression, right: Expression) extends BinaryExpression with Serializable with ImplicitCastInputTypes with Product
A function that calculates the SHA-2 family of functions (SHA-224, SHA-256, SHA-384, and SHA-512) and returns it as a hex string.
A function that calculates the SHA-2 family of functions (SHA-224, SHA-256, SHA-384, and SHA-512) and returns it as a hex string. The first argument is the string or binary to be hashed. The second argument indicates the desired bit length of the result, which must have a value of 224, 256, 384, 512, or 0 (which is equivalent to 256). SHA-224 is supported starting from Java 8. If asking for an unsupported SHA function, the return value is NULL. If either argument is NULL or the hash length is not one of the permitted values, the return value is NULL.

Annotations
@ExpressionDescription()
case class ShiftLeft(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Bitwise left shift.
Bitwise left shift.
left
the base number to shift.
right
number of bits to left shift.

Annotations
@ExpressionDescription()
case class ShiftRight(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Bitwise (signed) right shift.
Bitwise (signed) right shift.
left
the base number to shift.
right
number of bits to right shift.

Annotations
@ExpressionDescription()
case class ShiftRightUnsigned(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Bitwise unsigned right shift, for integer and long data type.
Bitwise unsigned right shift, for integer and long data type.
left
the base number.
right
the number of bits to right shift.

Annotations
@ExpressionDescription()
case class Shuffle(child: Expression, randomSeed: Option[Long] = None) extends UnaryExpression with ExpectsInputTypes with Stateful with ExpressionWithRandomSeed with Product with Serializable
Returns a random permutation of the given array.
Returns a random permutation of the given array.

Annotations
@ExpressionDescription()
case class Signum(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
trait SimpleHigherOrderFunction extends Expression with HigherOrderFunction
Trait for functions having as input one argument and one function.
case class Sin(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Sinh(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Size(child: Expression, legacySizeOfNull: Boolean) extends UnaryExpression with ExpectsInputTypes with Product with Serializable
Given an array or map, returns total number of elements in it.
Given an array or map, returns total number of elements in it.

Annotations
@ExpressionDescription()
trait SizeBasedWindowFunction extends AggregateWindowFunction
A SizeBasedWindowFunction needs the size of the current window for its calculation.
case class Slice(x: Expression, start: Expression, length: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
Slices an array according to the requested start index and length
Slices an array according to the requested start index and length

Annotations
@ExpressionDescription()
case class SortArray(base: Expression, ascendingOrder: Expression) extends BinaryExpression with ArraySortLike with Product with Serializable
Sorts the input array in ascending / descending order according to the natural ordering of the array elements and returns it.
Sorts the input array in ascending / descending order according to the natural ordering of the array elements and returns it.

Annotations
@ExpressionDescription()
sealed abstract class SortDirection extends AnyRef
case class SortOrder(child: Expression, direction: SortDirection, nullOrdering: NullOrdering, sameOrderExpressions: Set[Expression]) extends UnaryExpression with Unevaluable with Product with Serializable
An expression that can be used to sort a tuple.
An expression that can be used to sort a tuple. This class extends expression primarily so that transformations over expression will descend into its child. sameOrderExpressions is a set of expressions with the same sort order as the child. It is derived from equivalence relation in an operator, e.g. left/right keys of an inner sort merge join.
case class SortPrefix(child: SortOrder) extends UnaryExpression with Product with Serializable
An expression to generate a 64-bit long prefix used in sorting.
An expression to generate a 64-bit long prefix used in sorting. If the sort must operate over null keys as well, this.nullValue can be used in place of emitted null prefixes in the sort.
case class SoundEx(child: Expression) extends UnaryExpression with ExpectsInputTypes with Product with Serializable
A function that return Soundex code of the given string expression.
A function that return Soundex code of the given string expression.

Annotations
@ExpressionDescription()
case class SparkPartitionID() extends LeafExpression with Nondeterministic with Product with Serializable
Expression that returns the current partition id.
Expression that returns the current partition id.

Annotations
@ExpressionDescription()
case class SparkVersion() extends LeafExpression with CodegenFallback with Product with Serializable

Annotations
@ExpressionDescription()
sealed trait SpecialFrameBoundary extends Expression with Unevaluable
The trait used to represent special boundaries used in a window frame.
trait SpecializedGetters extends AnyRef
final class SpecializedGettersReader extends AnyRef
final class SpecificInternalRow extends InternalRow with BaseGenericInternalRow
A row type that holds an array specialized container objects, of type MutableValue, chosen based on the dataTypes of each column.
A row type that holds an array specialized container objects, of type MutableValue, chosen based on the dataTypes of each column. The intent is to decrease garbage when modifying the values of primitive columns.
case class SpecifiedWindowFrame(frameType: FrameType, lower: Expression, upper: Expression) extends Expression with WindowFrame with Product with Serializable
A specified Window Frame.
A specified Window Frame. The val lower/uppper can be either a foldable Expression or a SpecialFrameBoundary.
case class Sqrt(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Stack(children: Seq[Expression]) extends Expression with Generator with Product with Serializable
Separate v1, ..., vk into n rows.
Separate v1, ..., vk into n rows. Each row will have k/n columns. n must be constant.
```
SELECT stack(2, 1, 2, 3) ->
1      2
3      NULL
```
Annotations
@ExpressionDescription()
case class StartsWith(left: Expression, right: Expression) extends StringPredicate with Product with Serializable
A function that returns true if the string left starts with the string right.
trait Stateful extends Expression with Nondeterministic
An expression that contains mutable state.
An expression that contains mutable state. A stateful expression is always non-deterministic because the results it produces during evaluation are not only dependent on the given input but also on its internal state.
The state of the expressions is generally not exposed in the parameter list and this makes comparing stateful expressions problematic because similar stateful expressions (with the same parameter list) but with different internal state will be considered equal. This is especially problematic during tree transformations. In order to counter this the fastEquals method for stateful expressions only returns true for the same reference.
A stateful expression should never be evaluated multiple times for a single row. This should only be a problem for interpreted execution. This can be prevented by creating fresh copies of the stateful expression before execution, these can be made using the freshCopy function.
trait String2StringExpression extends Expression with ImplicitCastInputTypes
trait String2TrimExpression extends Expression with ImplicitCastInputTypes
case class StringInstr(str: Expression, substr: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
A function that returns the position of the first occurrence of substr in the given string.
A function that returns the position of the first occurrence of substr in the given string. Returns null if either of the arguments are null and returns 0 if substr could not be found in str.
NOTE: that this is not zero based, but 1-based index. The first character in str has index 1.

Annotations
@ExpressionDescription()
case class StringLPad(str: Expression, len: Expression, pad: Expression = Literal(" ")) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns str, left-padded with pad to a length of len.
Returns str, left-padded with pad to a length of len.

Annotations
@ExpressionDescription()
case class StringLocate(substr: Expression, str: Expression, start: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
A function that returns the position of the first occurrence of substr in given string after position pos.
A function that returns the position of the first occurrence of substr in given string after position pos.

Annotations
@ExpressionDescription()
abstract class StringPredicate extends BinaryExpression with Predicate with ImplicitCastInputTypes with NullIntolerant
A base trait for functions that compare two strings, returning a boolean.
case class StringRPad(str: Expression, len: Expression, pad: Expression = Literal(" ")) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns str, right-padded with pad to a length of len.
Returns str, right-padded with pad to a length of len.

Annotations
@ExpressionDescription()
abstract class StringRegexExpression extends BinaryExpression with ImplicitCastInputTypes with NullIntolerant
case class StringRepeat(str: Expression, times: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the string which repeat the given string value n times.
Returns the string which repeat the given string value n times.

Annotations
@ExpressionDescription()
case class StringReplace(srcExpr: Expression, searchExpr: Expression, replaceExpr: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
Replace all occurrences with string.
Replace all occurrences with string.

Annotations
@ExpressionDescription()
case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns a string consisting of n spaces.
Returns a string consisting of n spaces.

Annotations
@ExpressionDescription()
case class StringSplit(str: Expression, regex: Expression, limit: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
Splits str around matches of the given regex.
Splits str around matches of the given regex.

Annotations
@ExpressionDescription()
case class StringToMap(text: Expression, pairDelim: Expression, keyValueDelim: Expression) extends TernaryExpression with ExpectsInputTypes with Product with Serializable
Creates a map after splitting the input text into key/value pairs using delimiters
Creates a map after splitting the input text into key/value pairs using delimiters

Annotations
@ExpressionDescription()
case class StringTranslate(srcExpr: Expression, matchingExpr: Expression, replaceExpr: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
A function translate any character in the srcExpr by a character in replaceExpr.
A function translate any character in the srcExpr by a character in replaceExpr. The characters in replaceExpr is corresponding to the characters in matchingExpr. The translate will happen when any character in the string matching with the character in the matchingExpr.

Annotations
@ExpressionDescription()
case class StringTrim(srcStr: Expression, trimStr: Option[Expression] = None) extends Expression with String2TrimExpression with Product with Serializable
A function that takes a character string, removes the leading and trailing characters matching with any character in the trim string, returns the new string.
A function that takes a character string, removes the leading and trailing characters matching with any character in the trim string, returns the new string. If BOTH and trimStr keywords are not specified, it defaults to remove space character from both ends. The trim function will have one argument, which contains the source string. If BOTH and trimStr keywords are specified, it trims the characters from both ends, and the trim function will have two arguments, the first argument contains trimStr, the second argument contains the source string. trimStr: A character string to be trimmed from the source string, if it has multiple characters, the function searches for each character in the source string, removes the characters from the source string until it encounters the first non-match character. BOTH: removes any character from both ends of the source string that matches characters in the trim string.

Annotations
@ExpressionDescription()
case class StringTrimLeft(srcStr: Expression, trimStr: Option[Expression] = None) extends Expression with String2TrimExpression with Product with Serializable
A function that trims the characters from left end for a given string.
A function that trims the characters from left end for a given string. If LEADING and trimStr keywords are not specified, it defaults to remove space character from the left end. The ltrim function will have one argument, which contains the source string. If LEADING and trimStr keywords are not specified, it trims the characters from left end. The ltrim function will have two arguments, the first argument contains trimStr, the second argument contains the source string. trimStr: the function removes any character from the left end of the source string which matches with the characters from trimStr, it stops at the first non-match character. LEADING: removes any character from the left end of the source string that matches characters in the trim string.

Annotations
@ExpressionDescription()
case class StringTrimRight(srcStr: Expression, trimStr: Option[Expression] = None) extends Expression with String2TrimExpression with Product with Serializable
A function that trims the characters from right end for a given string.
A function that trims the characters from right end for a given string. If TRAILING and trimStr keywords are not specified, it defaults to remove space character from the right end. The rtrim function will have one argument, which contains the source string. If TRAILING and trimStr keywords are specified, it trims the characters from right end. The rtrim function will have two arguments, the first argument contains trimStr, the second argument contains the source string. trimStr: the function removes any character from the right end of source string which matches with the characters from trimStr, it stops at the first non-match character. TRAILING: removes any character from the right end of the source string that matches characters in the trim string.

Annotations
@ExpressionDescription()
case class StructsToCsv(options: Map[String, String], child: Expression, timeZoneId: Option[String] = None) extends UnaryExpression with TimeZoneAwareExpression with CodegenFallback with ExpectsInputTypes with Product with Serializable
Converts a StructType to a CSV output string.
Converts a StructType to a CSV output string.

Annotations
@ExpressionDescription()
case class StructsToJson(options: Map[String, String], child: Expression, timeZoneId: Option[String] = None) extends UnaryExpression with TimeZoneAwareExpression with CodegenFallback with ExpectsInputTypes with Product with Serializable
Converts a StructType, ArrayType or MapType to a JSON output string.
Converts a StructType, ArrayType or MapType to a JSON output string.

Annotations
@ExpressionDescription()
abstract class SubqueryExpression extends PlanExpression[LogicalPlan]
A base interface for expressions that contain a LogicalPlan.
case class Substring(str: Expression, pos: Expression, len: Expression) extends TernaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
A function that takes a substring of its first argument starting at a given position.
A function that takes a substring of its first argument starting at a given position. Defined for String and Binary types.
NOTE: that this is not zero based, but 1-based index. The first character in str has index 1.

Annotations
@ExpressionDescription()
case class SubstringIndex(strExpr: Expression, delimExpr: Expression, countExpr: Expression) extends TernaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the substring from string str before count occurrences of the delimiter delim.
Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything the left of the final delimiter (counting from left) is returned. If count is negative, every to the right of the final delimiter (counting from the right) is returned. substring_index performs a case-sensitive match when searching for delim.

Annotations
@ExpressionDescription()
case class Subtract(left: Expression, right: Expression) extends BinaryArithmetic with Product with Serializable

Annotations
@ExpressionDescription()
case class SubtractDates(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
Returns the interval from the left date (inclusive) to the right date (exclusive).
case class SubtractTimestamps(endTimestamp: Expression, startTimestamp: Expression) extends BinaryExpression with ExpectsInputTypes with Product with Serializable
Returns the interval from startTimestamp to endTimestamp in which the months and day field is set to 0 and the microseconds field is initialized to the microsecond difference between the given timestamps.
trait TaggingExpression extends UnaryExpression
case class Tan(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class Tanh(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
abstract class TernaryExpression extends Expression
An expression with three inputs and one output.
An expression with three inputs and one output. The output is by default evaluated to null if any input is evaluated to null.
case class TimeAdd(start: Expression, interval: Expression, timeZoneId: Option[String] = None) extends BinaryExpression with TimeZoneAwareExpression with ExpectsInputTypes with Product with Serializable
Adds an interval to timestamp.
case class TimeSub(start: Expression, interval: Expression, timeZoneId: Option[String] = None) extends BinaryExpression with TimeZoneAwareExpression with ExpectsInputTypes with Product with Serializable
Subtracts an interval from timestamp.
case class TimeWindow(timeColumn: Expression, windowDuration: Long, slideDuration: Long, startTime: Long) extends UnaryExpression with ImplicitCastInputTypes with Unevaluable with NonSQLExpression with Product with Serializable
trait TimeZoneAwareExpression extends Expression
Common base class for time zone aware expressions.
case class ToDegrees(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class ToRadians(child: Expression) extends UnaryMathExpression with Product with Serializable

Annotations
@ExpressionDescription()
abstract class ToTimestamp extends BinaryExpression with TimeZoneAwareExpression with ExpectsInputTypes
case class ToUTCTimestamp(left: Expression, right: Expression) extends BinaryExpression with ImplicitCastInputTypes with Product with Serializable
This is a common function for databases supporting TIMESTAMP WITHOUT TIMEZONE.
This is a common function for databases supporting TIMESTAMP WITHOUT TIMEZONE. This function takes a timestamp which is timezone-agnostic, and interprets it as a timestamp in the given timezone, and renders that timestamp as a timestamp in UTC.
However, timestamp in Spark represents number of microseconds from the Unix epoch, which is not timezone-agnostic. So in Spark this function just shift the timestamp value from the given timezone to UTC timezone.
This function may return confusing result if the input is a string with timezone, e.g. '2018-03-13T06:18:23+00:00'. The reason is that, Spark firstly cast the string to timestamp according to the timezone in the string, and finally display the result by converting the timestamp to string according to the session local timezone.

Annotations
@ExpressionDescription()
case class ToUnixTimestamp(timeExp: Expression, format: Expression, timeZoneId: Option[String] = None) extends UnixTime with Product with Serializable
Converts time string with given pattern.
Converts time string with given pattern. Deterministic version of UnixTimestamp, must have at least one parameter.

Annotations
@ExpressionDescription()
case class TransformKeys(argument: Expression, function: Expression) extends Expression with MapBasedSimpleHigherOrderFunction with CodegenFallback with Product with Serializable
Transform Keys for every entry of the map by applying the transform_keys function.
Transform Keys for every entry of the map by applying the transform_keys function. Returns map with transformed key entries

Annotations
@ExpressionDescription()
case class TransformValues(argument: Expression, function: Expression) extends Expression with MapBasedSimpleHigherOrderFunction with CodegenFallback with Product with Serializable
Returns a map that applies the function to each value of the map.
Returns a map that applies the function to each value of the map.

Annotations
@ExpressionDescription()
case class TruncDate(date: Expression, format: Expression) extends BinaryExpression with TruncInstant with Product with Serializable
Returns date truncated to the unit specified by the format.
Returns date truncated to the unit specified by the format.

Annotations
@ExpressionDescription()
trait TruncInstant extends BinaryExpression with ImplicitCastInputTypes
case class TruncTimestamp(format: Expression, timestamp: Expression, timeZoneId: Option[String] = None) extends BinaryExpression with TruncInstant with TimeZoneAwareExpression with Product with Serializable
Returns timestamp truncated to the unit specified by the format.
Returns timestamp truncated to the unit specified by the format.

Annotations
@ExpressionDescription()
case class TypeOf(child: Expression) extends UnaryExpression with Product with Serializable

Annotations
@ExpressionDescription()
case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Converts the argument from a base 64 string to BINARY.
Converts the argument from a base 64 string to BINARY.

Annotations
@ExpressionDescription()
abstract class UnaryExpression extends Expression
An expression with one input and one output.
An expression with one input and one output. The output is by default evaluated to null if the input is evaluated to null.
abstract class UnaryLogExpression extends UnaryMathExpression
abstract class UnaryMathExpression extends UnaryExpression with Serializable with ImplicitCastInputTypes
A unary expression specifically for math functions.
A unary expression specifically for math functions. Math Functions expect a specific type of input format, therefore these functions extend ExpectsInputTypes.
case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable

Annotations
@ExpressionDescription()
case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable

Annotations
@ExpressionDescription()
trait Unevaluable extends Expression
An expression that cannot be evaluated.
An expression that cannot be evaluated. These expressions don't live past analysis or optimization time (e.g. Star) and should not be evaluated during query planning and execution.
trait UnevaluableAggregate extends DeclarativeAggregate
An aggregate expression that gets rewritten (currently by the optimizer) into a different aggregate expression for evaluation.
An aggregate expression that gets rewritten (currently by the optimizer) into a different aggregate expression for evaluation. This is mainly used to provide compatibility with other databases. For example, we use this to support every, any/some aggregates by rewriting them with Min and Max respectively.
case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
Performs the inverse operation of HEX.
Performs the inverse operation of HEX. Resulting characters are returned as a byte array.

Annotations
@ExpressionDescription()
abstract class UnixTime extends ToTimestamp
case class UnixTimestamp(timeExp: Expression, format: Expression, timeZoneId: Option[String] = None) extends UnixTime with Product with Serializable
Converts time string with given pattern to Unix time stamp (in seconds), returns null if fail.
Converts time string with given pattern to Unix time stamp (in seconds), returns null if fail. See Datetime Patterns. Note that hive Language Manual says it returns 0 if fail, but in fact it returns null. If the second parameter is missing, use "yyyy-MM-dd HH:mm:ss". If no parameters provided, the first parameter will be current_timestamp. If the first parameter is a Date or Timestamp instead of String, we will ignore the second parameter.

Annotations
@ExpressionDescription()
case class UnresolvedNamedLambdaVariable(nameParts: Seq[String]) extends LeafExpression with NamedExpression with Unevaluable with Product with Serializable
A placeholder of lambda variables to prevent unexpected resolution of LambdaFunction.
case class UnresolvedWindowExpression(child: Expression, windowSpec: WindowSpecReference) extends UnaryExpression with Unevaluable with Product with Serializable
final class UnsafeArrayData extends ArrayData with Externalizable with KryoSerializable
An Unsafe implementation of Array which is backed by raw memory instead of Java objects.
An Unsafe implementation of Array which is backed by raw memory instead of Java objects.
Each array has four parts: [numElements][null bits][values or offset&length][variable length portion]
The numElements is 8 bytes storing the number of elements of this array.
In the null bits region, we store 1 bit per element, represents whether an element is null Its total size is ceil(numElements / 8) bytes, and it is aligned to 8-byte boundaries.
In the values or offset&length region, we store the content of elements. For fields that hold fixed-length primitive types, such as long, double, or int, we store the value directly in the field. The whole fixed-length portion (even for byte) is aligned to 8-byte boundaries. For fields with non-primitive or variable-length values, we store a relative offset (w.r.t. the base address of the array) that points to the beginning of the variable-length field and length (they are combined into a long). For variable length portion, each is aligned to 8-byte boundaries.
Instances of UnsafeArrayData act as pointers to row data stored in this format.
final class UnsafeMapData extends MapData with Externalizable with KryoSerializable
An Unsafe implementation of Map which is backed by raw memory instead of Java objects.
An Unsafe implementation of Map which is backed by raw memory instead of Java objects.
Currently we just use 2 UnsafeArrayData to represent UnsafeMapData, with extra 8 bytes at head to indicate the number of bytes of the unsafe key array. [unsafe key array numBytes] [unsafe key array] [unsafe value array]
Note that, user is responsible to guarantee that the key array does not have duplicated elements, otherwise the behavior is undefined.
abstract class UnsafeProjection extends Projection
A projection that returns UnsafeRow.
A projection that returns UnsafeRow.
CAUTION: the returned projection object should *not* be assumed to be thread-safe.
final class UnsafeRow extends InternalRow with Externalizable with KryoSerializable
An Unsafe implementation of Row which is backed by raw memory instead of Java objects.
An Unsafe implementation of Row which is backed by raw memory instead of Java objects.
Each tuple has three parts: [null bit set] [values] [variable length portion]
The bit set is used for null tracking and is aligned to 8-byte word boundaries. It stores one bit per field.
In the values region, we store one 8-byte word per field. For fields that hold fixed-length primitive types, such as long, double, or int, we store the value directly in the word. For fields with non-primitive or variable-length values, we store a relative offset (w.r.t. the base address of the row) that points to the beginning of the variable-length field, and length (they are combined into a long).
Instances of UnsafeRow act as pointers to row data stored in this format.
case class UnscaledValue(child: Expression) extends UnaryExpression with Product with Serializable
Return the unscaled Long value of a Decimal, assuming it fits in a Long.
Return the unscaled Long value of a Decimal, assuming it fits in a Long. Note: this expression is internal and created only by the optimizer, we don't need to do type check for it.
case class UpCast(child: Expression, target: AbstractDataType, walkedTypePath: Seq[String] = Nil) extends UnaryExpression with Unevaluable with Product with Serializable
Cast the child expression to the target data type, but will throw error if the cast might truncate, e.g.
Cast the child expression to the target data type, but will throw error if the cast might truncate, e.g. long -> int, timestamp -> data.
Note: target is AbstractDataType, so that we can put object DecimalType, which means we accept DecimalType with any valid precision/scale.
case class Upper(child: Expression) extends UnaryExpression with String2StringExpression with Product with Serializable
A function that converts the characters of a string to uppercase.
A function that converts the characters of a string to uppercase.

Annotations
@ExpressionDescription()
trait UserDefinedExpression extends AnyRef
Common base trait for user-defined functions, including UDF/UDAF/UDTF of different languages and Hive function wrappers.
case class UserDefinedGenerator(elementSchema: StructType, function: (Row) ⇒ TraversableOnce[InternalRow], children: Seq[Expression]) extends Expression with Generator with CodegenFallback with Product with Serializable
A generator that produces its output using the provided lambda function.
case class Uuid(randomSeed: Option[Long] = None) extends LeafExpression with Stateful with ExpressionWithRandomSeed with Product with Serializable

Annotations
@ExpressionDescription()
final class VariableLengthRowBasedKeyValueBatch extends RowBasedKeyValueBatch
An implementation of RowBasedKeyValueBatch in which key-value records have variable lengths.
An implementation of RowBasedKeyValueBatch in which key-value records have variable lengths.
The format for each record looks like this (in case of uaoSize = 4): [4 bytes total size = (klen + vlen + 4)] [4 bytes key size = klen] [UnsafeRow for key of length klen] [UnsafeRow for Value of length vlen] [8 bytes pointer to next] Thus, record length = 4 + 4 + klen + vlen + 8
case class WeekDay(child: Expression) extends DayWeek with Product with Serializable

Annotations
@ExpressionDescription()
case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class WindowExpression(windowFunction: Expression, windowSpec: WindowSpecDefinition) extends Expression with Unevaluable with Product with Serializable
sealed trait WindowFrame extends Expression with Unevaluable
Represents a window frame.
trait WindowFunction extends Expression
A window function is a function that can only be evaluated in the context of a window operator.
sealed trait WindowFunctionType extends AnyRef
Case objects that describe whether a window function is a SQL window function or a Python user-defined window function.
sealed trait WindowSpec extends AnyRef
The trait of the Window Specification (specified in the OVER clause or WINDOW clause) for Window Functions.
case class WindowSpecDefinition(partitionSpec: Seq[Expression], orderSpec: Seq[SortOrder], frameSpecification: WindowFrame) extends Expression with WindowSpec with Unevaluable with Product with Serializable
The specification for a window function.
The specification for a window function.
partitionSpec
It defines the way that input rows are partitioned.
orderSpec
It defines the ordering of rows in a partition.
frameSpecification
It defines the window frame in a partition.
case class WindowSpecReference(name: String) extends WindowSpec with Product with Serializable
A Window specification reference that refers to the WindowSpecDefinition defined under the name name.
final class XXH64 extends AnyRef
xxHash64.
xxHash64. A high quality and fast 64 bit hash code by Yann Colet and Mathias Westerdahl. The class below is modelled like its Murmur3_x86_32 cousin.
This was largely based on the following (original) C and Java implementations: https://github.com/Cyan4973/xxHash/blob/master/xxhash.c https://github.com/OpenHFT/Zero-Allocation-Hashing/blob/master/src/main/java/net/openhft/hashing/XxHash_r39.java https://github.com/airlift/slice/blob/master/src/main/java/io/airlift/slice/XxHash64.java
case class XxHash64(children: Seq[Expression], seed: Long) extends HashExpression[Long] with Product with Serializable
A xxHash64 64-bit hash expression.
A xxHash64 64-bit hash expression.

Annotations
@ExpressionDescription()
case class Year(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable

Annotations
@ExpressionDescription()
case class YearOfWeek(child: Expression) extends UnaryExpression with ImplicitCastInputTypes with Product with Serializable
case class Years(child: Expression) extends PartitionTransformExpression with Product with Serializable
Expression for the v2 partition transform years.
case class ZipWith(left: Expression, right: Expression, function: Expression) extends Expression with HigherOrderFunction with CodegenFallback with Product with Serializable

Annotations
@ExpressionDescription()

Value Members

val EmptyRow: InternalRow
Used as input into expressions whose output does not depend on any input value.
object ArrayBinaryLike
object ArrayExists extends Serializable
object ArraySort extends Serializable
object ArraySortLike
object ArrayUnion extends Serializable
object Ascending extends SortDirection with Product with Serializable
object AttributeMap extends Serializable
Builds a map that is keyed by an Attribute's expression id.
Builds a map that is keyed by an Attribute's expression id. Using the expression id allows values to be looked up even when the attributes used differ cosmetically (i.e., the capitalization of the name, or the expected nullability).
object AttributeSet extends Serializable
object BinaryArithmetic
object BinaryComparison
object BinaryOperator
object BindReferences extends Logging
object CallMethodViaReflection extends Serializable
object Canonicalize
Rewrites an expression using rules that are guaranteed preserve the result while attempting to remove cosmetic variations.
Rewrites an expression using rules that are guaranteed preserve the result while attempting to remove cosmetic variations. Deterministic expressions that are equal after canonicalization will always return the same answer given the same input (i.e. false positives should not be possible). However, it is possible that two canonical expressions that are not equal will in fact return the same answer given any input (i.e. false negatives are possible).
The following rules are applied:
- Names and nullability hints for org.apache.spark.sql.types.DataTypes are stripped.
- Names for GetStructField are stripped.
- TimeZoneId for Cast and AnsiCast are stripped if needsTimeZone is false.
- Commutative and associative operations (Add and Multiply) have their children ordered by hashCode.
- EqualTo and EqualNullSafe are reordered by hashCode.
- Other comparisons (GreaterThan, LessThan) are reversed by hashCode.
- Elements in In are reordered by hashCode.
object CaseKeyWhen
Case statements of the form "CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END".
Case statements of the form "CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END". When a = b, returns c; when a = d, returns e; else returns f.
object CaseWhen extends Serializable
Factory methods for CaseWhen.
object Cast extends Serializable
object CodegenObjectFactoryMode extends Enumeration
Defines values for SQLConf config of fallback mode.
Defines values for SQLConf config of fallback mode. Use for test only.
object CreateArray extends Serializable
object CreateMap extends Serializable
object CreateStruct
Returns a Row containing the evaluation of all children expressions.
object CurrentRow extends Expression with SpecialFrameBoundary with Product with Serializable
CURRENT ROW boundary.
object DatePart extends Serializable
object DecimalLiteral
Extractor for and other utility methods for decimal literals.
object Descending extends SortDirection with Product with Serializable
object DoubleLiteral
Extractor for retrieving Double literals.
object Equality
An extractor that matches both standard 3VL equality and null-safe equality.
object ExpectsInputTypes
object ExprId extends Serializable
object ExprUtils
object ExpressionSet
object ExtractIntervalPart extends Serializable
object ExtractValue
object Factorial extends Serializable
object FloatLiteral
Extractor for retrieving Float literals.
object GetStructFieldObject
A Scala extractor that extracts the child expression and struct field from a GetStructField.
A Scala extractor that extracts the child expression and struct field from a GetStructField. This is in contrast to the GetStructField case class extractor which returns the field ordinal instead of the field itself.
object Hex extends Serializable
object HiveHashFunction extends InterpretedHashFunction
object IdentityProjection extends Projection
An identity projection.
An identity projection. This returns the input row.
object IntegerLiteral
Extractor for retrieving Int literals.
object IntegralDivide extends Serializable
object InterpretedMutableProjection
Helper functions for creating an InterpretedMutableProjection.
object InterpretedOrdering extends Serializable
object InterpretedSafeProjection
Helper functions for creating an InterpretedSafeProjection.
object InterpretedUnsafeProjection
Helper functions for creating an InterpretedUnsafeProjection.
object IsNotUnknown
object IsUnknown
IS UNKNOWN and IS NOT UNKNOWN are the same as IS NULL and IS NOT NULL, respectively, except that the input expression must be of a boolean type.
object LambdaFunction extends Serializable
object Literal extends Serializable
object MakeDecimal extends Serializable
object Murmur3HashFunction extends InterpretedHashFunction
object MutableProjection extends CodeGeneratorWithInterpretedFallback[Seq[Expression], MutableProjection]
The factory object for MutableProjection.
object NamePlaceholder extends LeafExpression with Unevaluable with Product with Serializable
An expression representing a not yet available attribute name.
An expression representing a not yet available attribute name. This expression is unevaluable and as its name suggests it is a temporary place holder until we're able to determine the actual attribute name.
object NamedExpression
object NonNullLiteral
An extractor that matches non-null literal values
object NullsFirst extends NullOrdering with Product with Serializable
object NullsLast extends NullOrdering with Product with Serializable
object Overlay extends Serializable
object ParseUrl extends Serializable
object Predicate extends CodeGeneratorWithInterpretedFallback[Expression, BasePredicate]
The factory object for BasePredicate.
object PythonUDF extends Serializable
Helper functions for PythonUDF
object Rand extends Serializable
object Randn extends Serializable
object RangeFrame extends FrameType with Product with Serializable
RangeFrame treats rows in a partition as groups of peers.
RangeFrame treats rows in a partition as groups of peers. All rows having the same ORDER BY ordering are considered as peers. Values used in a range frame are considered to be logical offsets. For example, assuming the value of the current row's ORDER BY expression expr is v, RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING represents a frame containing rows whose values expr are in the range of [v-1, v+1].
If ORDER BY clause is not defined, all rows in the partition are considered as peers of the current row.
object RegExpExtract extends Serializable
object RowFrame extends FrameType with Product with Serializable
RowFrame treats rows in a partition individually.
RowFrame treats rows in a partition individually. Values used in a row frame are considered to be physical offsets. For example, ROW BETWEEN 1 PRECEDING AND 1 FOLLOWING represents a 3-row frame, from the row that precedes the current row to the row that follows the current row.
object RowOrdering extends CodeGeneratorWithInterpretedFallback[Seq[SortOrder], BaseOrdering]
object SafeProjection extends CodeGeneratorWithInterpretedFallback[Seq[Expression], Projection]
A projection that could turn UnsafeRow into GenericInternalRow
object ScalarSubquery extends Serializable
object SchemaPruning
object SelectedField
A Scala extractor that builds a org.apache.spark.sql.types.StructField from a Catalyst complex type extractor.
A Scala extractor that builds a org.apache.spark.sql.types.StructField from a Catalyst complex type extractor. For example, consider a relation with the following schema:
```
root
|-- name: struct (nullable = true)
|    |-- first: string (nullable = true)
|    |-- last: string (nullable = true)
```
Further, suppose we take the select expression name.first. This will parse into an Alias(child, "first"). Ignoring the alias, child matches the following pattern:
```
GetStructFieldObject(
  AttributeReference("name", StructType(_), _, _),
  StructField("first", StringType, _, _))
```
SelectedField converts that expression into
```
StructField("name", StructType(Array(StructField("first", StringType))))
```
by mapping each complex type extractor to a org.apache.spark.sql.types.StructField with the same name as its child (or "parent" going right to left in the select expression) and a data type appropriate to the complex type extractor. In our example, the name of the child expression is "name" and its data type is a org.apache.spark.sql.types.StructType with a single string field named "first".
object Sequence extends Serializable
object Size extends Serializable
object SizeBasedWindowFunction extends Serializable
object SortOrder extends Serializable
object StringLiteral
Extractor for retrieving String literals.
object StringTranslate extends Serializable
object StringTrim extends Serializable
object StringTrimLeft extends Serializable
object StringTrimRight extends Serializable
object SubExprUtils extends PredicateHelper
object SubqueryExpression
object TimeWindow extends Serializable
object UnboundedFollowing extends Expression with SpecialFrameBoundary with Product with Serializable
object UnboundedPreceding extends Expression with SpecialFrameBoundary with Product with Serializable
UNBOUNDED boundary.
object UnsafeProjection extends CodeGeneratorWithInterpretedFallback[Seq[Expression], UnsafeProjection]
The factory object for UnsafeProjection.
object UnspecifiedFrame extends Expression with WindowFrame with Product with Serializable
Used as a placeholder when a frame specification is not defined.
object VirtualColumn
object WindowFunctionType
object XxHash64Function extends InterpretedHashFunction

Packages

Standard Expressions

Named Expressions

Evaluation

expressions

package expressions

Standard Expressions

Named Expressions

Evaluation

Type Members

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

Packages

Standard Expressions

Named Expressions

Evaluation

expressions 

package expressions

Standard Expressions

Named Expressions

Evaluation

Type Members

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

expressions