Type Members

1.  case class AggregateExpression(aggregateFunction: AggregateFunction, mode: AggregateMode, isDistinct: Boolean, filter: Option[Expression], resultId: ExprId) extends Expression with Unevaluable with Product with Serializable

   A container for an AggregateFunction with its AggregateMode and a field (isDistinct) indicating if DISTINCT keyword is specified for this function and a field (filter) indicating if filter clause is specified for this function.

2.  abstract class AggregateFunction extends Expression

   AggregateFunction is the superclass of two aggregation function interfaces:

   AggregateFunction is the superclass of two aggregation function interfaces:

   • ImperativeAggregate is for aggregation functions that are specified in terms of initialize(), update(), and merge() functions that operate on Row-based aggregation buffers.
   • DeclarativeAggregate is for aggregation functions that are specified using Catalyst expressions.

   In both interfaces, aggregates must define the schema (aggBufferSchema) and attributes (aggBufferAttributes) of an aggregation buffer which is used to hold partial aggregate results. At runtime, multiple aggregate functions are evaluated by the same operator using a combined aggregation buffer which concatenates the aggregation buffers of the individual aggregate functions. Please note that aggregate functions should be stateless.

   Code which accepts AggregateFunction instances should be prepared to handle both types of aggregate functions.

3.  sealed trait AggregateMode extends AnyRef

   The mode of an AggregateFunction.

4.  case class ApproxCountDistinctForIntervals(child: Expression, endpointsExpression: Expression, relativeSD: Double = 0.05, mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0) extends TypedImperativeAggregate[Array[Long]] with ExpectsInputTypes with Product with Serializable

   This function counts the approximate number of distinct values (ndv) in intervals constructed from endpoints specified in endpointsExpression.

   This function counts the approximate number of distinct values (ndv) in intervals constructed from endpoints specified in endpointsExpression. The endpoints should be sorted into ascending order. E.g., given an array of endpoints (endpoint_1, endpoint_2, ... endpoint_N), returns the approximate ndv's for intervals [endpoint_1, endpoint_2], (endpoint_2, endpoint_3], ... (endpoint_N-1, endpoint_N]. To count ndv's in these intervals, apply the HyperLogLogPlusPlus algorithm in each of them.

   child

   to estimate the ndv's of.

   endpointsExpression

   An array expression to construct the intervals. It must be foldable, and its elements should be sorted into ascending order. Duplicate endpoints are allowed, e.g. (1, 5, 5, 10), and ndv for interval (5, 5] would be 1.

   relativeSD

   The maximum relative standard deviation allowed in the HyperLogLogPlusPlus algorithm.

5.  case class ApproximatePercentile(child: Expression, percentageExpression: Expression, accuracyExpression: Expression, mutableAggBufferOffset: Int, inputAggBufferOffset: Int) extends TypedImperativeAggregate[PercentileDigest] with ImplicitCastInputTypes with Product with Serializable

   The ApproximatePercentile function returns the approximate percentile(s) of a column at the given percentage(s).

   The ApproximatePercentile function returns the approximate percentile(s) of a column at the given percentage(s). A percentile is a watermark value below which a given percentage of the column values fall. For example, the percentile of column col at percentage 50% is the median of column col.

   This function supports partial aggregation.

   child

   child expression that can produce column value with child.eval(inputRow)

   percentageExpression

   Expression that represents a single percentage value or an array of percentage values. Each percentage value must be between 0.0 and 1.0.

   accuracyExpression

   Integer literal expression of approximation accuracy. Higher value yields better accuracy, the default value is DEFAULT_PERCENTILE_ACCURACY.

   Annotations

   @ExpressionDescription()

6.  case class Average(child: Expression) extends DeclarativeAggregate with ImplicitCastInputTypes with Product with Serializable

   Annotations

   @ExpressionDescription()

7.  abstract class BitAggregate extends DeclarativeAggregate with ExpectsInputTypes
8.  case class BitAndAgg(child: Expression) extends BitAggregate with Product with Serializable

   Annotations

   @ExpressionDescription()

9.  case class BitOrAgg(child: Expression) extends BitAggregate with Product with Serializable

   Annotations

   @ExpressionDescription()

10.  case class BitXorAgg(child: Expression) extends BitAggregate with Product with Serializable

    Annotations

    @ExpressionDescription()

11.  case class BoolAnd(arg: Expression) extends UnevaluableBooleanAggBase with Product with Serializable

    Annotations

    @ExpressionDescription()

12.  case class BoolOr(arg: Expression) extends UnevaluableBooleanAggBase with Product with Serializable

    Annotations

    @ExpressionDescription()

13.  abstract class CentralMomentAgg extends DeclarativeAggregate with ImplicitCastInputTypes

    A central moment is the expected value of a specified power of the deviation of a random variable from the mean.

    A central moment is the expected value of a specified power of the deviation of a random variable from the mean. Central moments are often used to characterize the properties of about the shape of a distribution.

    This class implements online, one-pass algorithms for computing the central moments of a set of points.

    Behavior:

    • null values are ignored
    • returns Double.NaN when the column contains Double.NaN values

    References:

    • Xiangrui Meng. "Simpler Online Updates for Arbitrary-Order Central Moments." 2015. http://arxiv.org/abs/1510.04923

    See also

    Algorithms for calculating variance (Wikipedia)

14.  abstract class Collect[T <: Growable[Any] with Iterable[Any]] extends TypedImperativeAggregate[T]

    A base class for collect_list and collect_set aggregate functions.

    A base class for collect_list and collect_set aggregate functions.

    We have to store all the collected elements in memory, and so notice that too many elements can cause GC paused and eventually OutOfMemory Errors.

15.  case class CollectList(child: Expression, mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0) extends Collect[ArrayBuffer[Any]] with Product with Serializable

    Collect a list of elements.

    Collect a list of elements.

    Annotations

    @ExpressionDescription()

16.  case class CollectSet(child: Expression, mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0) extends Collect[HashSet[Any]] with Product with Serializable

    Collect a set of unique elements.

    Collect a set of unique elements.

    Annotations

    @ExpressionDescription()

17.  case class Corr(x: Expression, y: Expression) extends PearsonCorrelation with Product with Serializable

    Annotations

    @ExpressionDescription()

18.  case class Count(children: Seq[Expression]) extends DeclarativeAggregate with Product with Serializable

    Annotations

    @ExpressionDescription()

19.  case class CountIf(predicate: Expression) extends DeclarativeAggregate with UnevaluableAggregate with ImplicitCastInputTypes with Product with Serializable

    Annotations

    @ExpressionDescription()

20.  case class CountMinSketchAgg(child: Expression, epsExpression: Expression, confidenceExpression: Expression, seedExpression: Expression, mutableAggBufferOffset: Int, inputAggBufferOffset: Int) extends TypedImperativeAggregate[CountMinSketch] with ExpectsInputTypes with Product with Serializable

    This function returns a count-min sketch of a column with the given esp, confidence and seed.

    This function returns a count-min sketch of a column with the given esp, confidence and seed. A count-min sketch is a probabilistic data structure used for summarizing streams of data in sub-linear space, which is useful for equality predicates and join size estimation. The result returned by the function is an array of bytes, which should be deserialized to a CountMinSketch before usage.

    child

    child expression that can produce column value with child.eval(inputRow)

    epsExpression

    relative error, must be positive

    confidenceExpression

    confidence, must be positive and less than 1.0

    seedExpression

    random seed

    Annotations

    @ExpressionDescription()

21.  case class CovPopulation(left: Expression, right: Expression) extends Covariance with Product with Serializable

    Annotations

    @ExpressionDescription()

22.  case class CovSample(left: Expression, right: Expression) extends Covariance with Product with Serializable

    Annotations

    @ExpressionDescription()

23.  abstract class Covariance extends DeclarativeAggregate with ImplicitCastInputTypes

    Compute the covariance between two expressions.

    Compute the covariance between two expressions. When applied on empty data (i.e., count is zero), it returns NULL.

24.  abstract class DeclarativeAggregate extends AggregateFunction with Serializable with Unevaluable

    API for aggregation functions that are expressed in terms of Catalyst expressions.

    API for aggregation functions that are expressed in terms of Catalyst expressions.

    When implementing a new expression-based aggregate function, start by implementing bufferAttributes, defining attributes for the fields of the mutable aggregation buffer. You can then use these attributes when defining updateExpressions, mergeExpressions, and evaluateExpressions.

    Please note that children of an aggregate function can be unresolved (it will happen when we create this function in DataFrame API). So, if there is any fields in the implemented class that need to access fields of its children, please make those fields lazy vals.

25.  case class First(child: Expression, ignoreNulls: Boolean) extends DeclarativeAggregate with ExpectsInputTypes with Product with Serializable

    Returns the first value of child for a group of rows.

    Returns the first value of child for a group of rows. If the first value of child is null, it returns null (respecting nulls). Even if First is used on an already sorted column, if we do partial aggregation and final aggregation (when mergeExpression is used) its result will not be deterministic (unless the input table is sorted and has a single partition, and we use a single reducer to do the aggregation.).

    Annotations

    @ExpressionDescription()

26.  case class HyperLogLogPlusPlus(child: Expression, relativeSD: Double = 0.05, mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0) extends ImperativeAggregate with Product with Serializable

    HyperLogLog++ (HLL++) is a state of the art cardinality estimation algorithm.

    HyperLogLog++ (HLL++) is a state of the art cardinality estimation algorithm. This class implements the dense version of the HLL++ algorithm as an Aggregate Function.

    This implementation has been based on the following papers: HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf

    HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/40671.pdf

    Appendix to HyperLogLog in Practice: Algorithmic Engineering of a State of the Art Cardinality Estimation Algorithm https://docs.google.com/document/d/1gyjfMHy43U9OWBXxfaeG-3MjGzejW1dlpyMwEYAAWEI/view?fullscreen#

    child

    to estimate the cardinality of.

    relativeSD

    the maximum relative standard deviation allowed.

    Annotations

    @ExpressionDescription()

27.  abstract class ImperativeAggregate extends AggregateFunction with CodegenFallback

    API for aggregation functions that are expressed in terms of imperative initialize(), update(), and merge() functions which operate on Row-based aggregation buffers.

    API for aggregation functions that are expressed in terms of imperative initialize(), update(), and merge() functions which operate on Row-based aggregation buffers.

    Within these functions, code should access fields of the mutable aggregation buffer by adding the bufferSchema-relative field number to mutableAggBufferOffset then using this new field number to access the buffer Row. This is necessary because this aggregation function's buffer is embedded inside of a larger shared aggregation buffer when an aggregation operator evaluates multiple aggregate functions at the same time.

    We need to perform similar field number arithmetic when merging multiple intermediate aggregate buffers together in merge() (in this case, use inputAggBufferOffset when accessing the input buffer).

    Correct ImperativeAggregate evaluation depends on the correctness of mutableAggBufferOffset and inputAggBufferOffset, but not on the correctness of the attribute ids in aggBufferAttributes and inputAggBufferAttributes.

28.  case class Kurtosis(child: Expression) extends CentralMomentAgg with Product with Serializable

    Annotations

    @ExpressionDescription()

29.  case class Last(child: Expression, ignoreNulls: Boolean) extends DeclarativeAggregate with ExpectsInputTypes with Product with Serializable

    Returns the last value of child for a group of rows.

    Returns the last value of child for a group of rows. If the last value of child is null, it returns null (respecting nulls). Even if Last is used on an already sorted column, if we do partial aggregation and final aggregation (when mergeExpression is used) its result will not be deterministic (unless the input table is sorted and has a single partition, and we use a single reducer to do the aggregation.).

    Annotations

    @ExpressionDescription()

30.  case class Max(child: Expression) extends DeclarativeAggregate with Product with Serializable

    Annotations

    @ExpressionDescription()

31.  case class MaxBy(valueExpr: Expression, orderingExpr: Expression) extends MaxMinBy with Product with Serializable

    Annotations

    @ExpressionDescription()

32.  abstract class MaxMinBy extends DeclarativeAggregate

    The shared abstract superclass for MaxBy and MinBy SQL aggregate functions.

33.  case class Min(child: Expression) extends DeclarativeAggregate with Product with Serializable

    Annotations

    @ExpressionDescription()

34.  case class MinBy(valueExpr: Expression, orderingExpr: Expression) extends MaxMinBy with Product with Serializable

    Annotations

    @ExpressionDescription()

35.  abstract class PearsonCorrelation extends DeclarativeAggregate with ImplicitCastInputTypes

    Base class for computing Pearson correlation between two expressions.

    Base class for computing Pearson correlation between two expressions. When applied on empty data (i.e., count is zero), it returns NULL.

    Definition of Pearson correlation can be found at http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient

36.  case class Percentile(child: Expression, percentageExpression: Expression, frequencyExpression: Expression, mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0) extends TypedImperativeAggregate[OpenHashMap[AnyRef, Long]] with ImplicitCastInputTypes with Product with Serializable

    The Percentile aggregate function returns the exact percentile(s) of numeric column expr at the given percentage(s) with value range in [0.0, 1.0].

    The Percentile aggregate function returns the exact percentile(s) of numeric column expr at the given percentage(s) with value range in [0.0, 1.0].

    Because the number of elements and their partial order cannot be determined in advance. Therefore we have to store all the elements in memory, and so notice that too many elements can cause GC paused and eventually OutOfMemory Errors.

    child

    child expression that produce numeric column value with child.eval(inputRow)

    percentageExpression

    Expression that represents a single percentage value or an array of percentage values. Each percentage value must be in the range [0.0, 1.0].

    Annotations

    @ExpressionDescription()

37.  case class PivotFirst(pivotColumn: Expression, valueColumn: Expression, pivotColumnValues: Seq[Any], mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0) extends ImperativeAggregate with Product with Serializable

    PivotFirst is an aggregate function used in the second phase of a two phase pivot to do the required rearrangement of values into pivoted form.

    PivotFirst is an aggregate function used in the second phase of a two phase pivot to do the required rearrangement of values into pivoted form.

    For example on an input of A | B --+-- x | 1 y | 2 z | 3

    with pivotColumn=A, valueColumn=B, and pivotColumnValues=[z,y] the output is [3,2].

    pivotColumn

    column that determines which output position to put valueColumn in.

    valueColumn

    the column that is being rearranged.

    pivotColumnValues

    the list of pivotColumn values in the order of desired output. Values not listed here will be ignored.

38.  case class Skewness(child: Expression) extends CentralMomentAgg with Product with Serializable

    Annotations

    @ExpressionDescription()

39.  case class StddevPop(child: Expression) extends CentralMomentAgg with Product with Serializable

    Annotations

    @ExpressionDescription()

40.  case class StddevSamp(child: Expression) extends CentralMomentAgg with Product with Serializable

    Annotations

    @ExpressionDescription()

41.  case class Sum(child: Expression) extends DeclarativeAggregate with ImplicitCastInputTypes with Product with Serializable

    Annotations

    @ExpressionDescription()

42.  abstract class TypedImperativeAggregate[T] extends ImperativeAggregate

    Aggregation function which allows **arbitrary** user-defined java object to be used as internal aggregation buffer.

    Aggregation function which allows **arbitrary** user-defined java object to be used as internal aggregation buffer.

    aggregation buffer for normal aggregation function `avg`            aggregate buffer for `sum`
              |                                                                  |
              v                                                                  v
            +--------------+---------------+-----------------------------------+-------------+
            |  sum1 (Long) | count1 (Long) | generic user-defined java objects | sum2 (Long) |
            +--------------+---------------+-----------------------------------+-------------+
                                             ^
                                             |
              aggregation buffer object for `TypedImperativeAggregate` aggregation function

    General work flow:

    Stage 1: initialize aggregate buffer object.

    1. The framework calls initialize(buffer: MutableRow) to set up the empty aggregate buffer. 2. In initialize, we call createAggregationBuffer(): T to get the initial buffer object, and set it to the global buffer row.

    Stage 2: process input rows.

    If the aggregate mode is Partial or Complete:

    1. The framework calls update(buffer: MutableRow, input: InternalRow) to process the input row. 2. In update, we get the buffer object from the global buffer row and call update(buffer: T, input: InternalRow): Unit.

    If the aggregate mode is PartialMerge or Final:

    1. The framework call merge(buffer: MutableRow, inputBuffer: InternalRow) to process the input row, which are serialized buffer objects shuffled from other nodes. 2. In merge, we get the buffer object from the global buffer row, and get the binary data from input row and deserialize it to buffer object, then we call merge(buffer: T, input: T): Unit to merge these 2 buffer objects.

    Stage 3: output results.

    If the aggregate mode is Partial or PartialMerge:

    1. The framework calls serializeAggregateBufferInPlace to replace the buffer object in the global buffer row with binary data. 2. In serializeAggregateBufferInPlace, we get the buffer object from the global buffer row and call serialize(buffer: T): Array[Byte] to serialize the buffer object to binary. 3. The framework outputs buffer attributes and shuffle them to other nodes.

    If the aggregate mode is Final or Complete:

    1. The framework calls eval(buffer: InternalRow) to calculate the final result. 2. In eval, we get the buffer object from the global buffer row and call eval(buffer: T): Any to get the final result. 3. The framework outputs these final results.

    Window function work flow: The framework calls update(buffer: MutableRow, input: InternalRow) several times and then call eval(buffer: InternalRow), so there is no need for window operator to call serializeAggregateBufferInPlace.

    NOTE: SQL with TypedImperativeAggregate functions is planned in sort based aggregation, instead of hash based aggregation, as TypedImperativeAggregate use BinaryType as aggregation buffer's storage format, which is not supported by hash based aggregation. Hash based aggregation only support aggregation buffer of mutable types (like LongType, IntType that have fixed length and can be mutated in place in UnsafeRow). NOTE: The newly added ObjectHashAggregateExec supports TypedImperativeAggregate functions in hash based aggregation under some constraints.

43.  abstract class UnevaluableBooleanAggBase extends DeclarativeAggregate with UnevaluableAggregate with ImplicitCastInputTypes
44.  case class VariancePop(child: Expression) extends CentralMomentAgg with Product with Serializable

    Annotations

    @ExpressionDescription()

45.  case class VarianceSamp(child: Expression) extends CentralMomentAgg with Product with Serializable

    Annotations

    @ExpressionDescription()