feature-training/org.pointyware.disco.training.entities.optimizers

Package-level declarations

Types

class AdaGrad : MultiPassOptimizer, EpochStatistics, BatchStatistics, SampleStatistics

data class Adam(val alpha: Double = 0.001, val beta1: Double = 0.9, val beta2: Double = 0.999, val epsilon: Double = 1.0E-8, val statistics: Statistics) : MultiPassOptimizer, Statistics

Adam optimizer. Based on research paper: https://arxiv.org/pdf/1412.6980

ConstantLearningRate

data class ConstantLearningRate(val learningRate: Float) : LearningRateSchedule

GradientDescent

open class GradientDescent(val learningRate: LearningRateSchedule, val entropy: Random = Random.Default) : SinglePassOptimizer

An optimizer that works by caching calculations during the forward pass and calculating gradients during the backward pass.

LearningRateSchedule

interface LearningRateSchedule

Allows an SinglePassOptimizer to delegate the learning rate to increase composability of different optimizers.

MomentumOptimizer

class MomentumOptimizer : SinglePassOptimizer, EpochStatistics, BatchStatistics, SampleStatistics

MultiPassOptimizer

interface MultiPassOptimizer : Optimizer

An optimizer that performs multiple passes over training data, updating the model parameters multiple times per epoch.

Optimizer

interface Optimizer

A Trainer will track the activations and derivatives of the model during the forward pass and provide them to the SinglePassOptimizer to update the model parameters.

RmsProp

class RmsProp : SinglePassOptimizer

Implemented per lecture slides: https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf

SinglePassOptimizer

interface SinglePassOptimizer : Optimizer

An optimizer that only performs a single pass over training data before updating model parameters.

StatisticalOptimizer

interface StatisticalOptimizer : Statistics, Optimizer

StepLearningRateSchedule

interface StepLearningRateSchedule : LearningRateSchedule

StochasticGradientDescent

class StochasticGradientDescent(val batchSize: Int, val learningRate: LearningRateSchedule, val entropy: Random = Random.Default, val discardExtras: Boolean = false) : GradientDescent

Stochastic Gradient Descent (SGD) optimizer with adjustable learning rate.

WarmRestartExponentialLearningRate

data class WarmRestartExponentialLearningRate(val initialLearningRate: Float, val decayMax: Float = 100.0f, val decayPeriod: Int = 10000) : LearningRateSchedule

This learning rate schedule begins at the initialLearningRate and decays exponentially until it reaches initialLearningRate / decayMax over decayPeriod epochs.