ai.databand

Class DefaultDbndRun

java.lang.Object

ai.databand.DefaultDbndRun

All Implemented Interfaces:

DbndRun

public class DefaultDbndRun
extends java.lang.Object
implements DbndRun

Constructor Summary

Constructors

Constructor and Description
DefaultDbndRun(DbndClient dbndClient, DbndConfig config)

Method Summary

Modifier and Type	Method and Description
protected TaskRunsInfo	buildRootRun(java.lang.reflect.Method method, java.lang.Object[] args) Builds root run.
protected RunAndDefinition	buildRunAndDefinition(java.lang.reflect.Method method, java.lang.Object[] args, boolean hasUpstreams)
protected java.util.List<TaskParamDefinition>	buildTaskParamDefinitions(java.lang.reflect.Method method)
protected Pair<java.util.List<TaskRunParam>,java.util.List<LogTarget>>	buildTaskRunParamsAndTargets(java.lang.reflect.Method method, java.lang.Object[] args, java.lang.String taskRunUid, java.lang.String methodName, java.lang.String taskRunAttemptUid, java.lang.String taskDefinitionUid)
void	completeTask(java.lang.reflect.Method method, java.lang.Object result) Set task state to 'completed'.
void	error(java.lang.Throwable error) Stop run.
void	errorTask(java.lang.reflect.Method method, java.lang.Throwable error) Set task state to 'error'.
protected java.lang.String	extractSourceCode(java.lang.reflect.Method method)
protected java.lang.String	extractStackTrace(java.lang.Throwable error)
java.lang.String	getTaskName(java.lang.reflect.Method method) Extract task name either from method name or annotation value.
void	init(java.lang.reflect.Method method, java.lang.Object[] args) Init run in DBND using pipeline root execution method pointcut.
void	logDataframe(java.lang.String key, org.apache.spark.sql.Dataset<?> value, HistogramRequest histogramRequest) Log Spark dataframe
void	logDatasetOperation(java.lang.String path, DatasetOperationType type, DatasetOperationStatus status, org.apache.spark.sql.Dataset<?> data, java.lang.Throwable error, LogDatasetRequest params) Log dataset operations with options like preview and schema generation.
void	logDatasetOperation(java.lang.String path, DatasetOperationType type, DatasetOperationStatus status, java.lang.String error, java.lang.String valuePreview, java.util.List<java.lang.Long> dataDimensions, java.lang.Object dataSchema, java.lang.Boolean withPartition, java.util.List<ColumnStats> columnStats) Log dataset operations.
void	logHistogram(java.util.Map<java.lang.String,java.lang.Object> histogram) Log histogram object.
void	logMetric(java.lang.String key, java.lang.Object value) Log metric and attach it to the current task.
void	logMetric(TaskRun taskRun, java.lang.String key, java.lang.Object value, java.lang.String source)
void	logMetric(TaskRun taskRun, java.lang.String key, java.lang.Object value, java.lang.String source, boolean compact)
void	logMetrics(java.util.Map<java.lang.String,java.lang.Object> metrics) Log metrics batch and attach it to the current task.
void	logMetrics(java.util.Map<java.lang.String,java.lang.Object> metrics, java.lang.String source) Log metrics batch with source
void	logMetrics(TaskRun taskRun, java.util.Map<java.lang.String,java.lang.Object> metrics, java.lang.String source)
void	saveLog(org.apache.log4j.spi.LoggingEvent event, java.lang.String formattedEvent) Save log and attach it to the current task and all parent tasks.
void	saveSparkMetrics(org.apache.spark.scheduler.SparkListenerStageCompleted event) Save spark metrics.
void	setDriverTask(TaskRun driverTask) Override task run to avoid creating duplicate runs.
void	startTask(java.lang.reflect.Method method, java.lang.Object[] args) Start task in the run context.
void	stop() Stop run.
void	stopExternal() Submit driver task metrics when run context was external.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DefaultDbndRun

public DefaultDbndRun(DbndClient dbndClient,
                      DbndConfig config)

Method Detail

init

public void init(java.lang.reflect.Method method,
                 java.lang.Object[] args)

Description copied from interface: DbndRun

Init run in DBND using pipeline root execution method pointcut.

Specified by:

init in interface DbndRun

buildRootRun

protected TaskRunsInfo buildRootRun(java.lang.reflect.Method method,
                                    java.lang.Object[] args)

Builds root run.

Returns:

extractSourceCode

protected java.lang.String extractSourceCode(java.lang.reflect.Method method)

startTask

public void startTask(java.lang.reflect.Method method,
                      java.lang.Object[] args)

Description copied from interface: DbndRun

Start task in the run context.

Specified by:

startTask in interface DbndRun

buildTaskParamDefinitions

protected java.util.List<TaskParamDefinition> buildTaskParamDefinitions(java.lang.reflect.Method method)

buildTaskRunParamsAndTargets

protected Pair<java.util.List<TaskRunParam>,java.util.List<LogTarget>> buildTaskRunParamsAndTargets(java.lang.reflect.Method method,
                                                                                                    java.lang.Object[] args,
                                                                                                    java.lang.String taskRunUid,
                                                                                                    java.lang.String methodName,
                                                                                                    java.lang.String taskRunAttemptUid,
                                                                                                    java.lang.String taskDefinitionUid)

getTaskName

public java.lang.String getTaskName(java.lang.reflect.Method method)

Description copied from interface: DbndRun

Extract task name either from method name or annotation value.

Specified by:

getTaskName in interface DbndRun

Returns:

task name extracted from method.

buildRunAndDefinition

protected RunAndDefinition buildRunAndDefinition(java.lang.reflect.Method method,
                                                 java.lang.Object[] args,
                                                 boolean hasUpstreams)

errorTask

public void errorTask(java.lang.reflect.Method method,
                      java.lang.Throwable error)

Description copied from interface: DbndRun

Set task state to 'error'.

Specified by:

errorTask in interface DbndRun

extractStackTrace

protected java.lang.String extractStackTrace(java.lang.Throwable error)

completeTask

public void completeTask(java.lang.reflect.Method method,
                         java.lang.Object result)

Description copied from interface: DbndRun

Set task state to 'completed'.

Specified by:

completeTask in interface DbndRun

stop

public void stop()

Description copied from interface: DbndRun

Stop run. Set run state to 'completed'.

Specified by:

stop in interface DbndRun

stopExternal

public void stopExternal()

Description copied from interface: DbndRun

Submit driver task metrics when run context was external.

Specified by:

stopExternal in interface DbndRun

error

public void error(java.lang.Throwable error)

Description copied from interface: DbndRun

Stop run. Set run state to 'failed'.

Specified by:

error in interface DbndRun

logMetric

public void logMetric(java.lang.String key,
                      java.lang.Object value)

Description copied from interface: DbndRun

Log metric and attach it to the current task.

Specified by:

logMetric in interface DbndRun

logMetrics

public void logMetrics(java.util.Map<java.lang.String,java.lang.Object> metrics)

Description copied from interface: DbndRun

Log metrics batch and attach it to the current task.

Specified by:

logMetrics in interface DbndRun

logMetrics

public void logMetrics(java.util.Map<java.lang.String,java.lang.Object> metrics,
                       java.lang.String source)

Description copied from interface: DbndRun

Log metrics batch with source

Specified by:

logMetrics in interface DbndRun

logDataframe

public void logDataframe(java.lang.String key,
                         org.apache.spark.sql.Dataset<?> value,
                         HistogramRequest histogramRequest)

Description copied from interface: DbndRun

Log Spark dataframe

Specified by:

logDataframe in interface DbndRun

logHistogram

public void logHistogram(java.util.Map<java.lang.String,java.lang.Object> histogram)

Description copied from interface: DbndRun

Log histogram object.

Specified by:

logHistogram in interface DbndRun

logDatasetOperation

public void logDatasetOperation(java.lang.String path,
                                DatasetOperationType type,
                                DatasetOperationStatus status,
                                java.lang.String error,
                                java.lang.String valuePreview,
                                java.util.List<java.lang.Long> dataDimensions,
                                java.lang.Object dataSchema,
                                java.lang.Boolean withPartition,
                                java.util.List<ColumnStats> columnStats)

Description copied from interface: DbndRun

Log dataset operations.

Specified by:

logDatasetOperation in interface DbndRun

logDatasetOperation

public void logDatasetOperation(java.lang.String path,
                                DatasetOperationType type,
                                DatasetOperationStatus status,
                                org.apache.spark.sql.Dataset<?> data,
                                java.lang.Throwable error,
                                LogDatasetRequest params)

Description copied from interface: DbndRun

Log dataset operations with options like preview and schema generation.

Specified by:

logDatasetOperation in interface DbndRun

logMetric

public void logMetric(TaskRun taskRun,
                      java.lang.String key,
                      java.lang.Object value,
                      java.lang.String source)

logMetric

public void logMetric(TaskRun taskRun,
                      java.lang.String key,
                      java.lang.Object value,
                      java.lang.String source,
                      boolean compact)

logMetrics

public void logMetrics(TaskRun taskRun,
                       java.util.Map<java.lang.String,java.lang.Object> metrics,
                       java.lang.String source)

saveLog

public void saveLog(org.apache.log4j.spi.LoggingEvent event,
                    java.lang.String formattedEvent)

Description copied from interface: DbndRun

Save log and attach it to the current task and all parent tasks.

Specified by:

saveLog in interface DbndRun

saveSparkMetrics

public void saveSparkMetrics(org.apache.spark.scheduler.SparkListenerStageCompleted event)

Description copied from interface: DbndRun

Save spark metrics.

Specified by:

saveSparkMetrics in interface DbndRun

setDriverTask

public void setDriverTask(TaskRun driverTask)

Description copied from interface: DbndRun

Override task run to avoid creating duplicate runs.

Specified by:

setDriverTask in interface DbndRun

Parameters:

driverTask - task run