Packages

c

org.apache.spark.sql.vectorized

ArrowColumnVector

final class ArrowColumnVector extends ColumnVector

A column vector backed by Apache Arrow. Currently calendar interval type and map type are not supported.

Linear Supertypes
ColumnVector, AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ArrowColumnVector
  2. ColumnVector
  3. AutoCloseable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ArrowColumnVector(vector: ValueVector)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def close(): Unit

    Cleans up memory for this column vector.

    Cleans up memory for this column vector. The column vector is not usable after this.

    This overwrites AutoCloseable.close to remove the throws clause, as column vector is in-memory and we don't expect any exception to happen during closing.

    Definition Classes
    ArrowColumnVector → ColumnVector → AutoCloseable
  7. final def dataType(): DataType

    Returns the data type of this column vector.

    Returns the data type of this column vector.

    Definition Classes
    ColumnVector
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. def getArray(rowId: Int): ColumnarArray

    Returns the array type value for rowId.

    Returns the array type value for rowId. If the slot for rowId is null, it should return null.

    To support array type, implementations must construct an ColumnarArray and return it in this method. ColumnarArray requires a ColumnVector that stores the data of all the elements of all the arrays in this vector, and an offset and length which points to a range in that ColumnVector, and the range represents the array for rowId. Implementations are free to decide where to put the data vector and offsets and lengths. For example, we can use the first child vector as the data vector, and store offsets and lengths in 2 int arrays in this vector.

    Definition Classes
    ArrowColumnVector → ColumnVector
  12. def getBinary(rowId: Int): Array[Byte]

    Returns the binary type value for rowId.

    Returns the binary type value for rowId. If the slot for rowId is null, it should return null.

    Definition Classes
    ArrowColumnVector → ColumnVector
  13. def getBoolean(rowId: Int): Boolean

    Returns the boolean type value for rowId.

    Returns the boolean type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
  14. def getBooleans(rowId: Int, count: Int): Array[Boolean]

    Gets boolean type values from [rowId, rowId + count).

    Gets boolean type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  15. def getByte(rowId: Int): Byte

    Returns the byte type value for rowId.

    Returns the byte type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
  16. def getBytes(rowId: Int, count: Int): Array[Byte]

    Gets byte type values from [rowId, rowId + count).

    Gets byte type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  17. def getChild(ordinal: Int): ArrowColumnVector

    returns

    child ColumnVector at the given ordinal.

    Definition Classes
    ArrowColumnVector → ColumnVector
  18. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  19. def getDecimal(rowId: Int, precision: Int, scale: Int): Decimal

    Returns the decimal type value for rowId.

    Returns the decimal type value for rowId. If the slot for rowId is null, it should return null.

    Definition Classes
    ArrowColumnVector → ColumnVector
  20. def getDouble(rowId: Int): Double

    Returns the double type value for rowId.

    Returns the double type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
  21. def getDoubles(rowId: Int, count: Int): Array[Double]

    Gets double type values from [rowId, rowId + count).

    Gets double type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  22. def getFloat(rowId: Int): Float

    Returns the float type value for rowId.

    Returns the float type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
  23. def getFloats(rowId: Int, count: Int): Array[Float]

    Gets float type values from [rowId, rowId + count).

    Gets float type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  24. def getInt(rowId: Int): Int

    Returns the int type value for rowId.

    Returns the int type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
  25. final def getInterval(rowId: Int): CalendarInterval

    Returns the calendar interval type value for rowId.

    Returns the calendar interval type value for rowId. If the slot for rowId is null, it should return null.

    In Spark, calendar interval type value is basically two integer values representing the number of months and days in this interval, and a long value representing the number of microseconds in this interval. An interval type vector is the same as a struct type vector with 3 fields: months, days and microseconds.

    To support interval type, implementations must implement #getChild(int) and define 3 child vectors: the first child vector is an int type vector, containing all the month values of all the interval values in this vector. The second child vector is an int type vector, containing all the day values of all the interval values in this vector. The third child vector is a long type vector, containing all the microsecond values of all the interval values in this vector.

    Definition Classes
    ColumnVector
  26. def getInts(rowId: Int, count: Int): Array[Int]

    Gets int type values from [rowId, rowId + count).

    Gets int type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  27. def getLong(rowId: Int): Long

    Returns the long type value for rowId.

    Returns the long type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
  28. def getLongs(rowId: Int, count: Int): Array[Long]

    Gets long type values from [rowId, rowId + count).

    Gets long type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  29. def getMap(rowId: Int): ColumnarMap

    Returns the map type value for rowId.

    Returns the map type value for rowId. If the slot for rowId is null, it should return null.

    In Spark, map type value is basically a key data array and a value data array. A key from the key array with a index and a value from the value array with the same index contribute to an entry of this map type value.

    To support map type, implementations must construct a ColumnarMap and return it in this method. ColumnarMap requires a ColumnVector that stores the data of all the keys of all the maps in this vector, and another ColumnVector that stores the data of all the values of all the maps in this vector, and a pair of offset and length which specify the range of the key/value array that belongs to the map type value at rowId.

    Definition Classes
    ArrowColumnVector → ColumnVector
  30. def getShort(rowId: Int): Short

    Returns the short type value for rowId.

    Returns the short type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
  31. def getShorts(rowId: Int, count: Int): Array[Short]

    Gets short type values from [rowId, rowId + count).

    Gets short type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  32. final def getStruct(rowId: Int): ColumnarRow

    Returns the struct type value for rowId.

    Returns the struct type value for rowId. If the slot for rowId is null, it should return null.

    To support struct type, implementations must implement #getChild(int) and make this vector a tree structure. The number of child vectors must be same as the number of fields of the struct type, and each child vector is responsible to store the data for its corresponding struct field.

    Definition Classes
    ColumnVector
  33. def getUTF8String(rowId: Int): UTF8String

    Returns the string type value for rowId.

    Returns the string type value for rowId. If the slot for rowId is null, it should return null. Note that the returned UTF8String may point to the data of this column vector, please copy it if you want to keep it after this column vector is freed.

    Definition Classes
    ArrowColumnVector → ColumnVector
  34. def hasNull(): Boolean

    Returns true if this column vector contains any null values.

    Returns true if this column vector contains any null values.

    Definition Classes
    ArrowColumnVector → ColumnVector
  35. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  36. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  37. def isNullAt(rowId: Int): Boolean

    Returns whether the value at rowId is NULL.

    Returns whether the value at rowId is NULL.

    Definition Classes
    ArrowColumnVector → ColumnVector
  38. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  39. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  40. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  41. def numNulls(): Int

    Returns the number of nulls in this column vector.

    Returns the number of nulls in this column vector.

    Definition Classes
    ArrowColumnVector → ColumnVector
  42. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  43. def toString(): String
    Definition Classes
    AnyRef → Any
  44. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  45. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  46. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from ColumnVector

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped