Type Members

1.  class RikaiSparkSessionExtensions extends (SparkSessionExtensions) => Unit

   Rikai SparkSession extensions to enable Spark SQL ML.

2.  trait SparkRunnable extends AnyRef

   Make ai.eto.rikai.sql.model.Model runnable on Spark.

   Make ai.eto.rikai.sql.model.Model runnable on Spark.

   For a ML_PREDICT expression in Spark SQL,

   SELECT ML_PREDICT(model_zoo, col1, col2, col3) FROM t1

   It generates a LogicalPlan equivalent to

   SELECT <Model{model_zoo}.asSpark(col1, col2, col3)> FROM t1

   Example:

   1. To implement a ai.eto.rikai.sql.model.Model for RegistryFoo:

      class FooModel(name, uri) extends Model with SparkRunnable {
      
          /** Use a Spark UDF with the same name to run RegistryFoo's model */
         def asSpark(args: Seq[Expression]) : Expression = {
             UnresolvedFunction(
                new FunctionIdentifier(s"${name}"),
                arguments,
                isDistinct = false,
                Option.empty
             )
         }

   Note

   It is the ai.eto.rikai.sql.model.Registry's responsibility to implement a Model that runs with Spark SQL.