ai.databand.parameters

Class DatasetPreview

java.lang.Object

ai.databand.parameters.DatasetPreview

All Implemented Interfaces:

TaskParameterPreview<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Direct Known Subclasses:

DatasetOperationPreview

public class DatasetPreview
extends java.lang.Object
implements TaskParameterPreview<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Constructor Summary

Constructors

Constructor and Description
DatasetPreview()

Method Summary

Modifier and Type	Method and Description
java.lang.String	compact(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input)
java.util.List<java.lang.Long>	dimensions(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input) This method calculates the exact size of the dataframe.
java.lang.String	full(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input)
java.lang.Object	schema(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input)
java.lang.String	typeName(java.lang.Class<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>> input)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DatasetPreview

public DatasetPreview()

Method Detail

compact

public java.lang.String compact(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input)

Specified by:

compact in interface TaskParameterPreview<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

full

public java.lang.String full(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input)

Specified by:

full in interface TaskParameterPreview<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

typeName

public java.lang.String typeName(java.lang.Class<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>> input)

Specified by:

typeName in interface TaskParameterPreview<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

schema

public java.lang.Object schema(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input)

Specified by:

schema in interface TaskParameterPreview<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

dimensions

public java.util.List<java.lang.Long> dimensions(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input)

This method calculates the exact size of the dataframe. TODO: opt-out of dimensions calculation because count() kicks out new job and it may take a lot of time. There is no easy way to calculate dataframe size without converting to RDD and using rdd.estimateCount()

Specified by:

dimensions in interface TaskParameterPreview<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Parameters:

input -

Returns: