Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final val JULIAN_DAY_OF_EPOCH: Int(2440588)
5.  val TIMEZONE_OPTION: String
6.  final val TimeZoneGMT: TimeZone
7.  final val TimeZoneUTC: TimeZone
8.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

9.  def cleanLegacyTimestampStr(s: String): String
10.  def clone(): AnyRef

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()

11.  def convertSpecialDate(input: String, zoneId: ZoneId): Option[SQLDate]

    Converts notational shorthands that are converted to ordinary dates.

    Converts notational shorthands that are converted to ordinary dates.

    input

    A trimmed string

    zoneId

    Zone identifier used to get the current date.

    returns

    Some of days since the epoch if the conversion completed successfully otherwise None.

12.  def convertSpecialTimestamp(input: String, zoneId: ZoneId): Option[SQLTimestamp]

    Converts notational shorthands that are converted to ordinary timestamps.

    Converts notational shorthands that are converted to ordinary timestamps.

    input

    A trimmed string

    zoneId

    Zone identifier used to get the current date.

    returns

    Some of microseconds since the epoch if the conversion completed successfully otherwise None.

13.  def convertTz(ts: SQLTimestamp, fromZone: ZoneId, toZone: ZoneId): SQLTimestamp

    Convert the timestamp ts from one timezone to another.

    Convert the timestamp ts from one timezone to another.

    TODO: Because of DST, the conversion between UTC and human time is not exactly one-to-one mapping, the conversion here may return wrong result, we should make the timestamp timezone-aware.

14.  def currentDate(zoneId: ZoneId): SQLDate
15.  def currentTimestamp(): SQLTimestamp
16.  def dateAddInterval(start: SQLDate, interval: CalendarInterval): SQLDate

    Add the date and the interval's months and days.

    Add the date and the interval's months and days. Returns a date value, expressed in days since 1.1.1970.

    Exceptions thrown

    DateTimeException if the result exceeds the supported date range

    IllegalArgumentException if the interval has microseconds part

17.  def dateAddMonths(days: SQLDate, months: Int): SQLDate

    Add date and year-month interval.

    Add date and year-month interval. Returns a date value, expressed in days since 1.1.1970.

18.  def daysToLocalDate(days: Int): LocalDate
19.  def daysToMillis(days: SQLDate, zoneId: ZoneId): Long
20.  def daysToMillis(days: SQLDate): Long
21.  def defaultTimeZone(): TimeZone
22.  def epochDaysToMicros(epochDays: SQLDate, zoneId: ZoneId): SQLTimestamp
23.  final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

24.  def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

25.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

26.  def fromJavaDate(date: Date): SQLDate

    Converts a local date at the default JVM time zone to the number of days since 1970-01-01 in the hybrid calendar (Julian + Gregorian) by discarding the time part.

    Converts a local date at the default JVM time zone to the number of days since 1970-01-01 in the hybrid calendar (Julian + Gregorian) by discarding the time part. The resulted days are rebased from the hybrid to Proleptic Gregorian calendar. The days rebasing is performed via UTC time zone for simplicity because the difference between two calendars is the same in any given time zone and UTC time zone.

    Note: The date is shifted by the offset of the default JVM time zone for backward compatibility with Spark 2.4 and earlier versions. The goal of the shift is to get a local date derived from the number of days that has the same date fields (year, month, day) as the original date at the default JVM time zone.

    date

    It represents a specific instant in time based on the hybrid calendar which combines Julian and Gregorian calendars.

    returns

    The number of days since the epoch in Proleptic Gregorian calendar.

27.  def fromJavaTimestamp(t: Timestamp): SQLTimestamp

    Converts an instance of java.sql.Timestamp to the number of microseconds since 1970-01-01T00:00:00.000000Z.

    Converts an instance of java.sql.Timestamp to the number of microseconds since 1970-01-01T00:00:00.000000Z. It extracts date-time fields from the input, builds a local timestamp in Proleptic Gregorian calendar from the fields, and binds the timestamp to the system time zone. The resulted instant is converted to microseconds since the epoch.

    The conversion is performed via the system time zone because it is used internally in java.sql.Timestamp while extracting date-time fields.

    The goal of the function is to have the same local date-time in the original calendar - the hybrid calendar (Julian + Gregorian) and in the target calendar which is Proleptic Gregorian calendar, see SPARK-26651.

    t

    It represents a specific instant in time based on the hybrid calendar which combines Julian and Gregorian calendars.

    returns

    The number of micros since epoch from java.sql.Timestamp.

28.  def fromJulianDay(day: Int, nanoseconds: Long): SQLTimestamp

    Returns the number of microseconds since epoch from Julian day and nanoseconds in a day

29.  def fromMillis(millis: Long): SQLTimestamp
30.  def fromUTCTime(time: SQLTimestamp, timeZone: String): SQLTimestamp

    Returns a timestamp of given timezone from utc timestamp, with the same string representation in their timezone.

31.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

32.  def getDayInYear(date: SQLDate): Int

    Returns the 'day in year' value for the given date.

    Returns the 'day in year' value for the given date. The date is expressed in days since 1.1.1970.

33.  def getDayOfMonth(date: SQLDate): Int

    Returns the 'day of month' value for the given date.

    Returns the 'day of month' value for the given date. The date is expressed in days since 1.1.1970.

34.  def getDayOfWeekFromString(string: UTF8String): Int
35.  def getHours(microsec: SQLTimestamp, zoneId: ZoneId): Int

    Returns the hour value of a given timestamp value.

    Returns the hour value of a given timestamp value. The timestamp is expressed in microseconds.

36.  def getLastDayOfMonth(date: SQLDate): SQLDate

    Returns last day of the month for the given date.

    Returns last day of the month for the given date. The date is expressed in days since 1.1.1970.

37.  def getMicroseconds(timestamp: SQLTimestamp, zoneId: ZoneId): Int

    Returns seconds, including fractional parts, multiplied by 1000000.

    Returns seconds, including fractional parts, multiplied by 1000000. The timestamp is expressed in microseconds since the epoch.

38.  def getMinutes(microsec: SQLTimestamp, zoneId: ZoneId): Int

    Returns the minute value of a given timestamp value.

    Returns the minute value of a given timestamp value. The timestamp is expressed in microseconds.

39.  def getMonth(date: SQLDate): Int

    Returns the month value for the given date.

    Returns the month value for the given date. The date is expressed in days since 1.1.1970. January is month 1.

40.  def getNextDateForDayOfWeek(startDate: SQLDate, dayOfWeek: Int): SQLDate

    Returns the first date which is later than startDate and is of the given dayOfWeek.

    Returns the first date which is later than startDate and is of the given dayOfWeek. dayOfWeek is an integer ranges in [0, 6], and 0 is Thu, 1 is Fri, etc,.

41.  def getQuarter(date: SQLDate): Int

    Returns the quarter for the given date.

    Returns the quarter for the given date. The date is expressed in days since 1.1.1970.

42.  def getSeconds(microsec: SQLTimestamp, zoneId: ZoneId): Int

    Returns the second value of a given timestamp value.

    Returns the second value of a given timestamp value. The timestamp is expressed in microseconds.

43.  def getSecondsWithFraction(microsec: SQLTimestamp, zoneId: ZoneId): Decimal

    Returns the seconds part and its fractional part with microseconds.

44.  def getTimeZone(timeZoneId: String): TimeZone
45.  def getWeekBasedYear(date: SQLDate): Int

    Returns the year which conforms to ISO 8601.

    Returns the year which conforms to ISO 8601. Each ISO 8601 week-numbering year begins with the Monday of the week containing the 4th of January.

46.  def getYear(date: SQLDate): Int

    Returns the year value for the given date.

    Returns the year value for the given date. The date is expressed in days since 1.1.1970.

47.  def getZoneId(timeZoneId: String): ZoneId
48.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

49.  def instantToDays(instant: Instant): Int
50.  def instantToMicros(instant: Instant): Long
51.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

52.  final val julianCommonEraStart: Timestamp
53.  def localDateToDays(localDate: LocalDate): Int
54.  def microsToEpochDays(epochMicros: SQLTimestamp, zoneId: ZoneId): SQLDate
55.  def microsToInstant(us: Long): Instant
56.  def millisToDays(millisUtc: Long, zoneId: ZoneId): SQLDate
57.  def millisToDays(millisUtc: Long): SQLDate
58.  def monthsBetween(time1: SQLTimestamp, time2: SQLTimestamp, roundOff: Boolean, zoneId: ZoneId): Double

    Returns number of months between time1 and time2.

    Returns number of months between time1 and time2. time1 and time2 are expressed in microseconds since 1.1.1970. If time1 is later than time2, the result is positive.

    If time1 and time2 are on the same day of month, or both are the last day of month, returns, time of day will be ignored.

    Otherwise, the difference is calculated based on 31 days per month. The result is rounded to 8 decimal places if roundOff is set to true.

59.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

60.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

61.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

62.  def parseTruncLevel(format: UTF8String): Int

    Returns the truncate level, could be from TRUNC_TO_MICROSECOND to TRUNC_TO_YEAR, or TRUNC_INVALID, TRUNC_INVALID means unsupported truncate level.

63.  def splitDate(date: SQLDate): (Int, Int, Int, Int)

    Split date (expressed in days since 1.1.1970) into four fields: year, month (Jan is Month 1), dayInMonth, daysToMonthEnd (0 if it's last day of month).

64.  def stringToDate(s: UTF8String, zoneId: ZoneId): Option[SQLDate]

    Trim and parse a given UTF8 date string to a corresponding Int value.

    Trim and parse a given UTF8 date string to a corresponding Int value. The return type is Option in order to distinguish between 0 and null. The following formats are allowed:

    yyyy yyyy-[m]m yyyy-[m]m-[d]d yyyy-[m]m-[d]d yyyy-[m]m-[d]d * yyyy-[m]m-[d]dT*

65.  def stringToTimestamp(s: UTF8String, timeZoneId: ZoneId): Option[SQLTimestamp]

    Trim and parse a given UTF8 date string to the corresponding a corresponding Long value.

    Trim and parse a given UTF8 date string to the corresponding a corresponding Long value. The return type is Option in order to distinguish between 0L and null. The following formats are allowed:

    yyyy yyyy-[m]m yyyy-[m]m-[d]d yyyy-[m]m-[d]d yyyy-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] yyyy-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] T[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id]

    where zone_id should have one of the forms:

    • Z - Zulu time zone UTC+0
    • +|-[h]h:[m]m
    • A short id, see https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html#SHORT_IDS
    • An id with one of the prefixes UTC+, UTC-, GMT+, GMT-, UT+ or UT-, and a suffix in the formats:
      • +|-h[h]
      • +|-hh[:]mm
      • +|-hh:mm:ss
      • +|-hhmmss

    • Region-based zone IDs in the form area/city, such as Europe/Paris
66.  def subtractDates(endDate: SQLDate, startDate: SQLDate): CalendarInterval

    Subtracts two dates.

    Subtracts two dates.

    endDate

    The end date, exclusive

    startDate

    The start date, inclusive

    returns

    An interval between two dates. The interval can be negative if the end date is before the start date.

67.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

68.  def timestampAddInterval(start: SQLTimestamp, months: Int, days: Int, microseconds: Long, zoneId: ZoneId): SQLTimestamp

    Add timestamp and full interval.

    Add timestamp and full interval. Returns a timestamp value, expressed in microseconds since 1.1.1970 00:00:00.

69.  def timestampToString(tf: TimestampFormatter, us: SQLTimestamp): String
70.  def toJavaDate(daysSinceEpoch: SQLDate): Date

    Converts days since the epoch 1970-01-01 in Proleptic Gregorian calendar to a local date at the default JVM time zone in the hybrid calendar (Julian + Gregorian).

    Converts days since the epoch 1970-01-01 in Proleptic Gregorian calendar to a local date at the default JVM time zone in the hybrid calendar (Julian + Gregorian). It rebases the given days from Proleptic Gregorian to the hybrid calendar at UTC time zone for simplicity because the difference between two calendars doesn't depend on any time zone. The result is shifted by the time zone offset in wall clock to have the same date fields (year, month, day) at the default JVM time zone as the input daysSinceEpoch in Proleptic Gregorian calendar.

    Note: The date is shifted by the offset of the default JVM time zone for backward compatibility with Spark 2.4 and earlier versions.

    daysSinceEpoch

    The number of days since 1970-01-01 in Proleptic Gregorian calendar.

    returns

    A local date in the hybrid calendar as java.sql.Date from number of days since epoch.

71.  def toJavaTimestamp(us: SQLTimestamp): Timestamp

    Converts microseconds since the epoch to an instance of java.sql.Timestamp via creating a local timestamp at the system time zone in Proleptic Gregorian calendar, extracting date and time fields like year and hours, and forming new timestamp in the hybrid calendar from the extracted fields.

    Converts microseconds since the epoch to an instance of java.sql.Timestamp via creating a local timestamp at the system time zone in Proleptic Gregorian calendar, extracting date and time fields like year and hours, and forming new timestamp in the hybrid calendar from the extracted fields.

    The conversion is based on the JVM system time zone because the java.sql.Timestamp uses the time zone internally.

    The method performs the conversion via local timestamp fields to have the same date-time representation as year, month, day, ..., seconds in the original calendar and in the target calendar.

    us

    The number of microseconds since 1970-01-01T00:00:00.000000Z.

    returns

    A java.sql.Timestamp from number of micros since epoch.

72.  def toJulianDay(us: SQLTimestamp): (Int, Long)

    Returns Julian day and nanoseconds in a day from the number of microseconds

    Returns Julian day and nanoseconds in a day from the number of microseconds

    Note: support timestamp since 4717 BC (without negative nanoseconds, compatible with Hive).

73.  def toMillis(us: SQLTimestamp): Long
74.  def toString(): String

    Definition Classes

    AnyRef → Any

75.  def toUTCTime(time: SQLTimestamp, timeZone: String): SQLTimestamp

    Returns a utc timestamp from a given timestamp from a given timezone, with the same string representation in their timezone.

76.  def truncDate(d: SQLDate, level: Int): SQLDate

    Returns the trunc date from original date and trunc level.

    Returns the trunc date from original date and trunc level. Trunc level should be generated using parseTruncLevel(), should be between 6 and 9.

77.  def truncTimestamp(t: SQLTimestamp, level: Int, zoneId: ZoneId): SQLTimestamp

    Returns the trunc date time from original date time and trunc level.

    Returns the trunc date time from original date time and trunc level. Trunc level should be generated using parseTruncLevel(), should be between 0 and 9.

78.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

79.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

80.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()