Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark
    Definition Classes
    apache
  • package sql
    Definition Classes
    spark
  • package catalyst

    Catalyst is a library for manipulating relational query plans.

    Catalyst is a library for manipulating relational query plans. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

    Definition Classes
    sql
  • package analysis

    Provides a logical query plan Analyzer and supporting classes for performing analysis.

    Provides a logical query plan Analyzer and supporting classes for performing analysis. Analysis consists of translating UnresolvedAttributes and UnresolvedRelations into fully typed objects using information in a schema Catalog.

    Definition Classes
    catalyst
  • package catalog
    Definition Classes
    catalyst
  • package csv
    Definition Classes
    catalyst
  • package dsl

    A collection of implicit conversions that create a DSL for constructing catalyst data structures.

    A collection of implicit conversions that create a DSL for constructing catalyst data structures.

    scala> import org.apache.spark.sql.catalyst.dsl.expressions._

// Standard operators are added to expressions.
scala> import org.apache.spark.sql.catalyst.expressions.Literal
scala> Literal(1) + Literal(1)
res0: org.apache.spark.sql.catalyst.expressions.Add = (1 + 1)

// There is a conversion from 'symbols to unresolved attributes.
scala> 'a.attr
res1: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 'a

// These unresolved attributes can be used to create more complicated expressions.
scala> 'a === 'b
res2: org.apache.spark.sql.catalyst.expressions.EqualTo = ('a = 'b)

// SQL verbs can be used to construct logical query plans.
scala> import org.apache.spark.sql.catalyst.plans.logical._
scala> import org.apache.spark.sql.catalyst.dsl.plans._
scala> LocalRelation('key.int, 'value.string).where('key === 1).select('value).analyze
res3: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan =
Project [value#3]
 Filter (key#2 = 1)
  LocalRelation [key#2,value#3], []
    Definition Classes
    catalyst
  • package encoders
    Definition Classes
    catalyst
  • package errors

    Functions for attaching and retrieving trees that are associated with errors.

    Functions for attaching and retrieving trees that are associated with errors.

    Definition Classes
    catalyst
  • package expressions

    A set of classes that can be used to represent trees of relational expressions.

    A set of classes that can be used to represent trees of relational expressions. A key goal of the expression library is to hide the details of naming and scoping from developers who want to manipulate trees of relational operators. As such, the library defines a special type of expression, a NamedExpression in addition to the standard collection of expressions.

    Standard Expressions

    A library of standard expressions (e.g., Add, EqualTo), aggregates (e.g., SUM, COUNT), and other computations (e.g. UDFs). Each expression type is capable of determining its output schema as a function of its children's output schema.

    Named Expressions

    Some expression are named and thus can be referenced by later operators in the dataflow graph. The two types of named expressions are AttributeReferences and Aliases. AttributeReferences refer to attributes of the input tuple for a given operator and form the leaves of some expression trees. Aliases assign a name to intermediate computations. For example, in the SQL statement SELECT a+b AS c FROM ..., the expressions a and b would be represented by AttributeReferences and c would be represented by an Alias.

    During analysis, all named expressions are assigned a globally unique expression id, which can be used for equality comparisons. While the original names are kept around for debugging purposes, they should never be used to check if two attributes refer to the same value, as plan transformations can result in the introduction of naming ambiguity. For example, consider a plan that contains subqueries, both of which are reading from the same table. If an optimization removes the subqueries, scoping information would be destroyed, eliminating the ability to reason about which subquery produced a given attribute.

    Evaluation

    The result of expressions can be evaluated using the Expression.apply(Row) method.

    Definition Classes
    catalyst
  • package json
    Definition Classes
    catalyst
  • package optimizer
    Definition Classes
    catalyst
  • package parser
    Definition Classes
    catalyst
  • package planning

    Contains classes for enumerating possible physical plans for a given logical query plan.

    Contains classes for enumerating possible physical plans for a given logical query plan.

    Definition Classes
    catalyst
  • package plans

    A collection of common abstractions for query plans as well as a base logical plan representation.

    A collection of common abstractions for query plans as well as a base logical plan representation.

    Definition Classes
    catalyst
  • package rules

    A framework for applying batches rewrite rules to trees, possibly to fixed point.

    A framework for applying batches rewrite rules to trees, possibly to fixed point.

    Definition Classes
    catalyst
  • package trees

    A library for easily manipulating trees of operators.

    A library for easily manipulating trees of operators. Operators that extend TreeNode are granted the following interface:

    • Scala collection like methods (foreach, map, flatMap, collect, etc)

    - transform - accepts a partial function that is used to generate a new tree. When the partial function can be applied to a given tree segment, that segment is replaced with the result. After attempting to apply the partial function to a given node, the transform function recursively attempts to apply the function to that node's children.

    • debugging support - pretty printing, easy splicing of trees, etc.
    Definition Classes
    catalyst
  • package util
    Definition Classes
    catalyst
  • ArrayBasedMapBuilder
  • ArrayBasedMapData
  • ArrayData
  • ArrayDataIndexedSeq
  • BadRecordException
  • CaseInsensitiveMap
  • CompressionCodecs
  • DataTypeJsonUtils
  • DateFormatter
  • DateTimeFormatterHelper
  • DateTimeUtils
  • DropMalformedMode
  • FailFastMode
  • FailureSafeParser
  • FractionTimestampFormatter
  • GenericArrayData
  • HyperLogLogPlusPlusHelper
  • IntervalUtils
  • Iso8601DateFormatter
  • Iso8601TimestampFormatter
  • LegacyDateFormats
  • LegacyDateFormatter
  • LegacyFastDateFormatter
  • LegacyFastTimestampFormatter
  • LegacySimpleDateFormatter
  • LegacySimpleTimestampFormatter
  • MapData
  • MicrosCalendar
  • NumberConverter
  • ParseMode
  • PartialResultException
  • PermissiveMode
  • QuantileSummaries
  • RandomIndicesGenerator
  • RandomUUIDGenerator
  • RebaseDateTime
  • StringKeyHashMap
  • StringUtils
  • TimestampFormatter
  • TypeUtils

package util

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. util
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. class ArrayBasedMapBuilder extends Serializable

    A builder of ArrayBasedMapData, which fails if a null map key is detected, and removes duplicated map keys w.r.t.

    A builder of ArrayBasedMapData, which fails if a null map key is detected, and removes duplicated map keys w.r.t. the last wins policy.

  2. class ArrayBasedMapData extends MapData

    A simple MapData implementation which is backed by 2 arrays.

    A simple MapData implementation which is backed by 2 arrays.

    Note that, user is responsible to guarantee that the key array does not have duplicated elements, otherwise the behavior is undefined.

  3. abstract class ArrayData extends SpecializedGetters with Serializable
  4. class ArrayDataIndexedSeq[T] extends IndexedSeq[T]

    Implements an IndexedSeq interface for ArrayData.

    Implements an IndexedSeq interface for ArrayData. Notice that if the original ArrayData is a primitive array and contains null elements, it is better to ask for IndexedSeq[Any], instead of IndexedSeq[Int], in order to keep the null elements.

  5. case class BadRecordException(record: () ⇒ UTF8String, partialResult: () ⇒ Option[InternalRow], cause: Throwable) extends Exception with Product with Serializable

    Exception thrown when the underlying parser meet a bad record and can't parse it.

    Exception thrown when the underlying parser meet a bad record and can't parse it.

    record

    a function to return the record that cause the parser to fail

    partialResult

    a function that returns an optional row, which is the partial result of parsing this bad record.

    cause

    the actual exception about why the record is bad and can't be parsed.

  6. class CaseInsensitiveMap[T] extends Map[String, T] with Serializable

    Builds a map in which keys are case insensitive.

    Builds a map in which keys are case insensitive. Input map can be accessed for cases where case-sensitive information is required. The primary constructor is marked private to avoid nested case-insensitive map creation, otherwise the keys in the original map will become case-insensitive in this scenario. Note: CaseInsensitiveMap is serializable. However, after transformation, e.g. filterKeys(), it may become not serializable.

  7. sealed trait DateFormatter extends Serializable
  8. trait DateTimeFormatterHelper extends AnyRef
  9. class FailureSafeParser[IN] extends AnyRef
  10. class FractionTimestampFormatter extends Iso8601TimestampFormatter

    The formatter parses/formats timestamps according to the pattern yyyy-MM-dd HH:mm:ss.[..fff..] where [..fff..] is a fraction of second up to microsecond resolution.

    The formatter parses/formats timestamps according to the pattern yyyy-MM-dd HH:mm:ss.[..fff..] where [..fff..] is a fraction of second up to microsecond resolution. The formatter does not output trailing zeros in the fraction. For example, the timestamp 2019-03-05 15:00:01.123400 is formatted as the string 2019-03-05 15:00:01.1234.

  11. class GenericArrayData extends ArrayData
  12. class HyperLogLogPlusPlusHelper extends Serializable
  13. class Iso8601DateFormatter extends DateFormatter with DateTimeFormatterHelper
  14. class Iso8601TimestampFormatter extends TimestampFormatter with DateTimeFormatterHelper
  15. trait LegacyDateFormatter extends DateFormatter
  16. class LegacyFastDateFormatter extends LegacyDateFormatter

    The legacy formatter is based on Apache Commons FastDateFormat.

    The legacy formatter is based on Apache Commons FastDateFormat. The formatter uses the default JVM time zone intentionally for compatibility with Spark 2.4 and earlier versions.

    Note: Using of the default JVM time zone makes the formatter compatible with the legacy DateTimeUtils methods toJavaDate and fromJavaDate that are based on the default JVM time zone too.

  17. class LegacyFastTimestampFormatter extends TimestampFormatter
  18. class LegacySimpleDateFormatter extends LegacyDateFormatter

    The legacy formatter is based on java.text.SimpleDateFormat.

    The legacy formatter is based on java.text.SimpleDateFormat. The formatter uses the default JVM time zone intentionally for compatibility with Spark 2.4 and earlier versions.

    Note: Using of the default JVM time zone makes the formatter compatible with the legacy DateTimeUtils methods toJavaDate and fromJavaDate that are based on the default JVM time zone too.

  19. class LegacySimpleTimestampFormatter extends TimestampFormatter
  20. abstract class MapData extends Serializable

    This is an internal data representation for map type in Spark SQL.

    This is an internal data representation for map type in Spark SQL. This should not implement equals and hashCode because the type cannot be used as join keys, grouping keys, or in equality tests. See SPARK-9415 and PR#13847 for the discussions.

  21. class MicrosCalendar extends GregorianCalendar

    The custom sub-class of GregorianCalendar is needed to get access to protected fields immediately after parsing.

    The custom sub-class of GregorianCalendar is needed to get access to protected fields immediately after parsing. We cannot use the get() method because it performs normalization of the fraction part. Accordingly, the MILLISECOND field doesn't contain original value.

    Also this class allows to set raw value to the MILLISECOND field directly before formatting.

  22. sealed trait ParseMode extends AnyRef
  23. case class PartialResultException(partialResult: InternalRow, cause: Throwable) extends Exception with Product with Serializable

    Exception thrown when the underlying parser returns a partial result of parsing.

    Exception thrown when the underlying parser returns a partial result of parsing.

    partialResult

    the partial result of parsing a bad record.

    cause

    the actual exception about why the parser cannot return full result.

  24. class QuantileSummaries extends Serializable

    Helper class to compute approximate quantile summary.

    Helper class to compute approximate quantile summary. This implementation is based on the algorithm proposed in the paper: "Space-efficient Online Computation of Quantile Summaries" by Greenwald, Michael and Khanna, Sanjeev. (https://doi.org/10.1145/375663.375670)

    In order to optimize for speed, it maintains an internal buffer of the last seen samples, and only inserts them after crossing a certain size threshold. This guarantees a near-constant runtime complexity compared to the original algorithm.

  25. case class RandomIndicesGenerator(randomSeed: Long) extends Product with Serializable

    This class is used to generate a random indices of given length.

    This class is used to generate a random indices of given length.

    This implementation uses the "inside-out" version of Fisher-Yates algorithm. Reference: https://en.wikipedia.org/wiki/Fisher%E2%80%93Yates_shuffle#The_%22inside-out%22_algorithm

  26. case class RandomUUIDGenerator(randomSeed: Long) extends Product with Serializable

    This class is used to generate a UUID from Pseudo-Random Numbers.

    This class is used to generate a UUID from Pseudo-Random Numbers.

    For the algorithm, see RFC 4122: A Universally Unique IDentifier (UUID) URN Namespace, section 4.4 "Algorithms for Creating a UUID from Truly Random or Pseudo-Random Numbers".

  27. class StringKeyHashMap[T] extends AnyRef
  28. sealed trait TimestampFormatter extends Serializable

Value Members

  1. def escapeSingleQuotedString(str: String): String
  2. def fileToString(file: File, encoding: Charset = UTF_8): String
  3. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  4. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  5. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  6. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  7. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  8. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  9. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  10. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  11. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  12. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  13. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  14. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  15. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  16. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  17. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  18. def quietly[A](f: ⇒ A): A

    Silences output to stderr or stdout for the duration of f

  19. def quoteIdentifier(name: String): String
  20. def resourceToBytes(resource: String, classLoader: ClassLoader = Utils.getSparkClassLoader): Array[Byte]
  21. def resourceToString(resource: String, encoding: String = UTF_8.name(), classLoader: ClassLoader = Utils.getSparkClassLoader): String
  22. def sideBySide(left: Seq[String], right: Seq[String]): Seq[String]
  23. def sideBySide(left: String, right: String): Seq[String]
  24. def stackTraceToString(t: Throwable): String
  25. def stringToFile(file: File, str: String): File
  26. def toPrettySQL(e: Expression): String
  27. def truncatedString[T](seq: Seq[T], sep: String, maxFields: Int): String

    Shorthand for calling truncatedString() without start or end strings.

  28. def truncatedString[T](seq: Seq[T], start: String, sep: String, end: String, maxFields: Int): String

    Format a sequence with semantics similar to calling .mkString().

    Format a sequence with semantics similar to calling .mkString(). Any elements beyond maxNumToStringFields will be dropped and replaced by a "... N more fields" placeholder.

    returns

    the trimmed and formatted string.

  29. def usePrettyExpression(e: Expression): Expression
  30. object ArrayBasedMapData extends Serializable
  31. object ArrayData extends Serializable
  32. object CaseInsensitiveMap extends Serializable
  33. object CompressionCodecs
  34. object DataTypeJsonUtils
  35. object DateFormatter extends Serializable
  36. object DateTimeUtils

    Helper functions for converting between internal and external date and time representations.

    Helper functions for converting between internal and external date and time representations. Dates are exposed externally as java.sql.Date and are represented internally as the number of dates since the Unix epoch (1970-01-01). Timestamps are exposed externally as java.sql.Timestamp and are stored internally as longs, which are capable of storing timestamps with microsecond precision.

  37. object DropMalformedMode extends ParseMode with Product with Serializable

    This mode ignores the whole corrupted records.

  38. object FailFastMode extends ParseMode with Product with Serializable

    This mode throws an exception when it meets corrupted records.

  39. object HyperLogLogPlusPlusHelper extends Serializable

    Constants used in the implementation of the HyperLogLogPlusPlus aggregate function.

    Constants used in the implementation of the HyperLogLogPlusPlus aggregate function.

    See the Appendix to HyperLogLog in Practice: Algorithmic Engineering of a State of the Art Cardinality (https://docs.google.com/document/d/1gyjfMHy43U9OWBXxfaeG-3MjGzejW1dlpyMwEYAAWEI/view?fullscreen) for more information.

  40. object IntervalUtils
  41. object LegacyDateFormats extends Enumeration
  42. object NumberConverter
  43. object ParseMode extends Logging
  44. object PermissiveMode extends ParseMode with Product with Serializable

    This mode permissively parses the records.

  45. object QuantileSummaries extends Serializable
  46. object RebaseDateTime

    The collection of functions for rebasing days and microseconds from/to the hybrid calendar (Julian + Gregorian since 1582-10-15) which is used by Spark 2.4 and earlier versions to/from Proleptic Gregorian calendar which is used by Spark since version 3.0.

    The collection of functions for rebasing days and microseconds from/to the hybrid calendar (Julian + Gregorian since 1582-10-15) which is used by Spark 2.4 and earlier versions to/from Proleptic Gregorian calendar which is used by Spark since version 3.0. See SPARK-26651.

  47. object StringKeyHashMap

    Build a map with String type of key, and it also supports either key case sensitive or insensitive.

  48. object StringUtils extends Logging
  49. object TimestampFormatter extends Serializable
  50. object TypeUtils

    Functions to help with checking for valid data types and value comparison of various types.

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped