org.apache.spark.ml

odkl

package odkl

Visibility
  1. Public
  2. All

Type Members

  1. class AutoAssembler extends Estimator[PipelineModel] with AutoAssemblerParams with DefaultParamsWritable

    Utility for automatically assembling columns into a vector of features.

  2. trait AutoAssemblerParams extends HasColumnsSets with HasOutputCol with HasColumnAttributeMap

    Params for automatic feature-vector assembler.

  3. class BinaryClassificationEvaluator extends Evaluator[BinaryClassificationEvaluator]

    Simple evaluator based on the mllib.

  4. class CRRSamplerEstimator extends Estimator[CRRSamplerModel] with DefaultParamsWritable with CRRSamplerParams

    Estimator is used to select the proper item sample rate to achive desired size of the resulting sample.

  5. class CRRSamplerModel extends Model[CRRSamplerModel] with DefaultParamsWritable with CRRSamplerParams with HasNetlibBlas

    Model applied as a transformer, but the resulting data set is not determenistic (each pass produces different results).

  6. trait CRRSamplerParams extends HasInputCol with HasGroupByColumns with HasLabelCol

    Follows ideas from Combined Regression and Ranking paper (http://www.

  7. class ColumnsExtractor extends Transformer with DefaultParamsWritable

    Used to extract a set of columns from the underlying data frame based on names and/or SQL expresions.

  8. class CombinedLinearModelUnwrappedFeatureSelector[M <: LinearModel[M], C <: CombinedModel[M, C]] extends GenericFeatureSelector[CombinedLinearModelUnwrappedFeatureSelector[M, C]] with ModelTransformer[C, CombinedLinearModelUnwrappedFeatureSelector[M, C]]

  9. abstract class CombinedModel[M <: ModelWithSummary[M], C <: CombinedModel[M, C]] extends Model[C] with ModelWithSummary[C] with HasDescriminantColumn with HasDirectTransformOption with HasPredictionCol with ForkedModelParams

    Base class for combined model holding a named map of nested models.

  10. class CrossValidator[M <: ModelWithSummary[M]] extends ForkedEstimatorSameType[M, Int] with HasIsTestCol with HasFolds

    Used to train and evaluate model in folds.

  11. abstract class DSVRGD[M <: ModelWithSummary[M]] extends Estimator[M] with SummarizableEstimator[M] with HasPredictionCol with HasFeaturesCol with HasLabelCol with HasRegParam with HasElasticNetParam with HasNetlibBlas with HasMaxIter with HasTol with HasCacheTrainData

    Created by dmitriybugaichenko on 10.

  12. abstract class DeVectorizedDSVRGD[M <: ModelWithSummary[M]] extends DSVRGD[M]

    Helper class for training single-label models.

  13. abstract class Evaluator[S <: Evaluator[S]] extends Transformer with HasLabelCol with HasPredictionCol

    Base class for evaluators.

  14. class ExtendedMultivariateOnlineSummarizer extends MultivariateOnlineSummarizer with Serializable with Logging

    Created by dmitriybugaichenko on 30.

  15. abstract class FoldedFeatureSelector[SelectingModel <: ModelWithSummary[SelectingModel] with HasWeights, Filter <: GenericFeatureSelector[Filter]] extends Estimator[Filter] with SummarizableEstimator[Filter] with HasWeights with HasFeaturesCol with HasFeaturesSignificance

    Created by dmitriybugaichenko on 29.

  16. abstract class ForkedEstimator[ModelIn <: ModelWithSummary[ModelIn], ForeKeyType, ModelOut <: ModelWithSummary[ModelOut]] extends Estimator[ModelOut] with SummarizableEstimator[ModelOut] with ForkedModelParams

    Utility used to split training into forks (per type, per class, per fold).

  17. abstract class ForkedEstimatorSameType[ModelIn <: ModelWithSummary[ModelIn], ForeKeyType] extends ForkedEstimator[ModelIn, ForeKeyType, ModelIn]

    Specific case of forked estimator which does not change the type of the underlying model.

  18. trait ForkedModelParams extends AnyRef

  19. abstract class GenericFeatureSelector[M <: ModelWithSummary[M]] extends Model[M] with ModelWithSummary[M] with HasFeaturesCol

  20. trait HasBatchSize extends Params

    Used for evaluators with batch support

  21. trait HasCacheTrainData extends AnyRef

    For estimators capable of caching training data.

  22. trait HasClassesCol extends Params

    Adds parameter with column for instance classes.

  23. trait HasClassesWeights extends Params

    Adds parameter wot classes weights (defaults to 1.

  24. trait HasColumnAttributeMap extends AnyRef

    For vector assemblers used to provide better naming for metadata attrbiutes.

  25. trait HasColumnsSets extends Params

    Parameters for specifying which columns to include or exclude.

  26. trait HasDescriminantColumn extends Params

    Created by dmitriybugaichenko on 30.

  27. trait HasDirectTransformOption extends Transformer

    Supplementary train used for optimization (moving transformation out of the execution plan into UDF)

  28. trait HasFeaturesSignificance extends AnyRef

    Block with information regarding features significance stat, produced during the features selection stage.

  29. trait HasFolds extends Params

    Adds parameters for folding - number of folds and name of column with fold number.

  30. trait HasGroupByColumns extends AnyRef

    For transformers performing grouping by a certain columns.

  31. trait HasIsTestCol extends Params

    Adds parameter with the name of test/train split column

  32. trait HasMetricsBlock extends AnyRef

    Metrics block is added by the evaluators.

  33. trait HasNetlibBlas extends AnyRef

    Created by dmitriybugaichenko on 19.

  34. trait HasRegularizeLast extends Params

    Used to indicate that last weight should not be considered as a part of regularization (typically if it is the intercept)

  35. trait HasSortByColumns extends AnyRef

    For transformers performing sorting by a certain columns.

  36. trait HasTypeCol extends Params

    Adds parameter with column for instance type.

  37. trait HasWeights extends AnyRef

    Block produced by a models with concept of feature weights (eg.

  38. class Interceptor extends Transformer with HasFeaturesCol with DefaultParamsWritable

    Adds extra column to features vector with a fixed value of 1.

  39. class IsotonicRegression extends regression.IsotonicRegression

    :: Experimental :: Isotonic regression.

  40. class JacksonParam[T] extends Param[T] with Logging

    ml.

  41. class LinearCombinationModel[N <: ModelWithSummary[N]] extends MultiClassCombinationModelBase[N, LinearCombinationModel[N]]

    Combination model which evaluates ALL nested model and combines results based on linear weights.

  42. class LinearDSVRGD extends DeVectorizedDSVRGD[LinearRegressionModel]

    Single-label linear regresion with DSVRGD

  43. abstract class LinearEstimator[M <: LinearModel[M], T <: LinearEstimator[M, T]] extends Predictor[Vector, T, M] with SummarizableEstimator[M] with LinearModelParams with HasWeights

  44. class LinearMatrixDSVRGD extends DSVRGD[LinearCombinationModel[LinearRegressionModel]]

    Multi-label linear regresion with DSVRGD

  45. abstract class LinearModel[M <: LinearModel[M]] extends PredictionModel[Vector, M] with DirectPredictionModel[Vector, M] with ModelWithSummary[M] with LinearModelParams with HasWeights

  46. trait LinearModelParams extends PredictorParams

  47. class LinearModelUnwrappedFeatureSelector[M <: LinearModel[M]] extends GenericFeatureSelector[LinearModelUnwrappedFeatureSelector[M]] with ModelTransformer[M, LinearModelUnwrappedFeatureSelector[M]]

  48. class LinearRegressionModel extends LinearModel[LinearRegressionModel]

  49. class LinearRegressionSGD extends LinearRegressor[LinearRegressionModel, GradientDescent, LinearRegressionSGD] with HasRegParam with HasTol with HasMaxIter with HasStepSize

  50. abstract class LinearRegressor[M <: LinearModel[M], O <: Optimizer, T <: LinearRegressor[M, O, T]] extends LinearEstimator[M, T] with DefaultParamsWritable with Logging with HasCacheTrainData

  51. class LogisticDSVRGD extends DeVectorizedDSVRGD[LogisticRegressionModel]

    Multi-label logistic regresion with DSVRGD

  52. class LogisticMatrixDSVRGD extends DSVRGD[LinearCombinationModel[LogisticRegressionModel]]

    Multi-label logistic regresion with DSVRGD

  53. class LogisticRegressionLBFSG extends LinearRegressor[LogisticRegressionModel, LogisticRegressionLBFSG, LogisticRegressionLBFSG] with HasRegParam with HasTol with HasMaxIter with Optimizer with HasElasticNetParam with HasRegularizeLast with HasBatchSize

  54. class LogisticRegressionModel extends LinearModel[LogisticRegressionModel]

  55. class MLWrapper[M <: Model[M]] extends Estimator[MLWrapperModel[M]] with SummarizableEstimator[MLWrapperModel[M]]

    Utility used to bridge default spark ML models into our advanced pipelines.

  56. class MLWrapperModel[M <: Model[M]] extends Model[MLWrapperModel[M]] with ModelWithSummary[MLWrapperModel[M]]

  57. class MatrixLBFGS extends Estimator[LinearCombinationModel[LogisticRegressionModel]] with SummarizableEstimator[LinearCombinationModel[LogisticRegressionModel]] with PredictorParams with HasTol with HasMaxIter with HasRegParam with HasRegularizeLast with HasBatchSize with HasNetlibBlas

    Created by dmitriybugaichenko on 24.

  58. class MetadataParam extends Param[Metadata]

    Created by alexander.

  59. class ModelSummary extends Serializable

    One of main extensions to the base concept of model - each model might return a summary represented by a named collection of dataframes.

  60. trait ModelTransformer[M <: ModelWithSummary[M], T <: ModelTransformer[M, T]] extends Model[T] with Logging

    In case if we can avoid certain stages used during training while predicting we need to propagate some changes to the model (eg.

  61. trait ModelWithSummary[M <: ModelWithSummary[M]] extends Model[M] with MLWritable

    Model which has a summary.

  62. class MultiClassCombinationModel[N <: ModelWithSummary[N]] extends MultiClassCombinationModelBase[N, MultiClassCombinationModel[N]]

    Combination model which evaluates ALL nested model and returns vector.

  63. abstract class MultiClassCombinationModelBase[N <: ModelWithSummary[N], M <: MultiClassCombinationModelBase[N, M]] extends CombinedModel[N, M] with HasClassesCol with HasPredictionCol with HasFeaturesCol

    Base class for models, evaluated per each class.

  64. class MultinominalExtractor extends Estimator[MultinominalExtractorModel] with MultinominalExtractorParams with DefaultParamsWritable

    Utility for converting columns with string or a set of stings into a vector of 0/1 with the cardinality equal to the number of unique string values used.

  65. class MultinominalExtractorModel extends Model[MultinominalExtractorModel] with MultinominalExtractorParams with DefaultParamsWritable

    Model produced by the multinominal extractor.

  66. trait MultinominalExtractorParams extends HasInputCol with HasOutputCol

    Parameters for multinominal feature extractor.

  67. class NaNToMeanReplacerEstimator extends Estimator[NaNToMeanReplacerModel] with NaNToMeanReplacerParams

    Estimates mean values ignoring NaN's

  68. class NaNToMeanReplacerModel extends Model[NaNToMeanReplacerModel] with NaNToMeanReplacerParams with DefaultParamsWritable

    Model used to replace values with pre-computed defaults before training/predicting.

  69. trait NaNToMeanReplacerParams extends HasInputCol with HasOutputCol

    Set of parameters for the replacer

  70. class NameAssigner extends Transformer with HasInputCols

    Assuming there is a metadata attached to a integer field can be used to replace ints with corresponding attribute names.

  71. class NullToDefaultReplacer extends Transformer with HasColumnsSets with DefaultParamsWritable

    Utility used to replace null values with defaults (zero or false).

  72. class NullToNaNVectorAssembler extends Transformer with HasInputCols with HasOutputCol with HasColumnAttributeMap with DefaultParamsWritable

    :: Experimental :: A feature transformer that merges multiple columns into a vector column.

  73. class PartitionedRankingEvaluator extends Evaluator[PartitionedRankingEvaluator] with HasOutputCol with HasGroupByColumns

    Evaluator used to compute metrics for predictions grouped by a certain criteria (typically by a user id).

  74. trait PartitioningParams extends Params with HasSortByColumns

    Settings for partitioning, except the number of partitions.

  75. class PipelinedFeatureSelector extends GenericFeatureSelector[PipelinedFeatureSelector]

  76. class PipelinedFoldedFeatureSelector[SelectingModel <: ModelWithSummary[SelectingModel] with HasWeights] extends FoldedFeatureSelector[SelectingModel, PipelinedFeatureSelector]

  77. class ScalerEstimator[M <: ModelWithSummary[M]] extends Estimator[Unscaler[M]] with ScalerParams with DefaultParamsWritable

    This is a specific implementation of the scaler for linear models.

  78. trait ScalerParams extends Params with HasFeaturesCol

    Scaler parameters.

  79. class SelectingModel[N <: ModelWithSummary[N]] extends CombinedModel[N, SelectingModel[N]] with HasTypeCol with HasFeaturesCol

    Selecting model applies exactly one model based on instance type and return its result.

  80. class SeriallizableAvlTreeDigest extends Serializer[SeriallizableAvlTreeDigest] with Serializable

    Serializable wrapper over the TDigest

  81. class SqlFilter extends Transformer with DefaultParamsWritable

    Simple utility used to apply SQL WHERE filter

  82. trait SummarizableEstimator[M <: ModelWithSummary[M]] extends Estimator[M]

    Estimator with produces model with summary.

  83. class TopKTransformer[B] extends Transformer with DefaultParamsWritable with HasGroupByColumns

    Created by eugeny.

  84. class TopKUDAF[B] extends UserDefinedAggregateFunction with Logging

    Created by eugeny.

  85. class UnwrappedStage[M <: ModelWithSummary[M], T <: ModelTransformer[M, T]] extends Estimator[M] with SummarizableEstimator[M]

    In case if we can avoid certain stages used during training while predicting we need to propagate some changes to the model (eg.

  86. class VectorExplode extends Transformer with DefaultParamsWritable

    Utility used to extract nested values from vectors into dedicated columns.

  87. class VectorStatCollector extends Transformer with HasInputCol with HasGroupByColumns with DefaultParamsWritable

    Utility used to collect detailed stat for vectors grouped by a certain keys.

  88. case class WeightedFeature(index: Int, name: String, weight: Double) extends Product with Serializable

    Utility used for reporting single indexed feature weight.

  89. case class WeightsStat(stats: Array[WeightsStatRecord]) extends Product with Serializable

  90. case class WeightsStatRecord(index: Int, name: String, descriminant: String, average: Double, significance: Double, isRelevant: Boolean) extends Product with Serializable

Value Members

  1. object AutoAssembler extends DefaultParamsReadable[AutoAssembler] with Serializable

    Adds read logic

  2. object ColumnsExtractor extends DefaultParamsReadable[ColumnsExtractor] with Serializable

    Adds read ability.

  3. object CombinedModel extends MLReadable[PipelineStage] with Serializable

  4. object CrossValidator extends DefaultParamsReadable[CrossValidator[_]] with Serializable

  5. object DSVRGD extends Serializable with HasNetlibBlas

  6. object Evaluator extends Serializable

  7. object FoldedFeatureSelector extends Serializable with Logging

  8. object ForkedEstimator extends Serializable

    Helper used to inject common task support with thread count limit into all forked estimators.

  9. object HasNetlibBlas extends Serializable

  10. object Interceptor extends DefaultParamsReadable[Interceptor] with Serializable

  11. object IsotonicRegression extends DefaultParamsReadable[IsotonicRegression] with Serializable

    Annotations
    @Since( "1.6.0" )
  12. object JacksonParam extends Serializable

  13. object LinearDSVRGD extends DefaultParamsReadable[LinearDSVRGD] with Serializable

  14. object LinearMatrixDSVRGD extends DefaultParamsReadable[LinearMatrixDSVRGD] with Serializable

  15. object LinearRegressionModel extends MLReadable[LinearRegressionModel] with Serializable

  16. object LogisticDSVRGD extends DefaultParamsReadable[LogisticDSVRGD] with Serializable

  17. object LogisticMatrixDSVRGD extends DefaultParamsReadable[LogisticMatrixDSVRGD] with Serializable

  18. object LogisticRegressionModel extends MLReadable[LogisticRegressionModel] with Serializable

  19. object MLWrapperModel extends MLReadable[PipelineStage] with Serializable

  20. object MatrixLBFGS extends Logging with HasNetlibBlas with Serializable

  21. object ModelWithSummary extends MLReadable[PipelineStage] with Serializable

    Helper for reading and writing models in a typed way.

  22. object MultinominalExtractor extends DefaultParamsReadable[MultinominalExtractor] with Serializable

    Adds read logic

  23. object MultinominalExtractorModel extends DefaultParamsReadable[MultinominalExtractorModel] with Serializable

    Adds read ability

  24. object NaNToMeanReplacerModel extends DefaultParamsReadable[NaNToMeanReplacerModel] with Serializable

    Adds support for reading.

  25. object NullToDefaultReplacer extends DefaultParamsReadable[NullToDefaultReplacer] with Serializable

    Adds read ability.

  26. object NullToNaNVectorAssembler extends DefaultParamsReadable[NullToNaNVectorAssembler] with Serializable

    Annotations
    @Since( "1.6.0" )
  27. object PartitionedRankingEvaluator extends Serializable

  28. object Scaler extends Serializable

  29. object SqlFilter extends DefaultParamsReadable[SqlFilter] with Serializable

  30. object UnwrappedStage extends Serializable

  31. package texts

Ungrouped