ai.databand

Interface DbndRun

All Known Implementing Classes:

DefaultDbndRun, NoopDbndRun

public interface DbndRun

DBND run.

Method Summary

Modifier and Type	Method and Description
void	completeTask(java.lang.reflect.Method method, java.lang.Object result) Set task state to 'completed'.
void	error(java.lang.Throwable error) Stop run.
void	errorTask(java.lang.reflect.Method method, java.lang.Throwable error) Set task state to 'error'.
java.lang.String	getTaskName(java.lang.reflect.Method method) Extract task name either from method name or annotation value.
void	init(java.lang.reflect.Method method, java.lang.Object[] args) Init run in DBND using pipeline root execution method pointcut.
void	logDataframe(java.lang.String key, org.apache.spark.sql.Dataset<?> value, HistogramRequest histogramRequest) Log Spark dataframe
void	logDatasetOperation(java.lang.String path, DatasetOperationType type, DatasetOperationStatus status, org.apache.spark.sql.Dataset<?> data, java.lang.Throwable error, LogDatasetRequest params) Log dataset operations with options like preview and schema generation.
void	logDatasetOperation(java.lang.String path, DatasetOperationType type, DatasetOperationStatus status, java.lang.String valuePreview, java.lang.String error, java.util.List<java.lang.Long> dataDimensions, java.lang.Object dataSchema, java.lang.Boolean withPartition, java.util.List<ColumnStats> columnStats) Log dataset operations.
void	logHistogram(java.util.Map<java.lang.String,java.lang.Object> histogram) Log histogram object.
void	logMetric(java.lang.String key, java.lang.Object value) Log metric and attach it to the current task.
void	logMetrics(java.util.Map<java.lang.String,java.lang.Object> metrics) Log metrics batch and attach it to the current task.
void	logMetrics(java.util.Map<java.lang.String,java.lang.Object> metrics, java.lang.String source) Log metrics batch with source
void	saveLog(org.apache.log4j.spi.LoggingEvent event, java.lang.String formattedEvent) Save log and attach it to the current task and all parent tasks.
void	saveSparkMetrics(org.apache.spark.scheduler.SparkListenerStageCompleted event) Save spark metrics.
void	setDriverTask(TaskRun taskRun) Override task run to avoid creating duplicate runs.
void	startTask(java.lang.reflect.Method method, java.lang.Object[] args) Start task in the run context.
void	stop() Stop run.
void	stopExternal() Submit driver task metrics when run context was external.

Method Detail

init

void init(java.lang.reflect.Method method,
          java.lang.Object[] args)

Init run in DBND using pipeline root execution method pointcut.

Parameters:

method -

args -

startTask

void startTask(java.lang.reflect.Method method,
               java.lang.Object[] args)

Start task in the run context.

Parameters:

method -

args -

errorTask

void errorTask(java.lang.reflect.Method method,
               java.lang.Throwable error)

Set task state to 'error'.

Parameters:

method -

error -

completeTask

void completeTask(java.lang.reflect.Method method,
                  java.lang.Object result)

Set task state to 'completed'.

Parameters:

method -

result -

stop

void stop()

Stop run. Set run state to 'completed'.

stopExternal

void stopExternal()

Submit driver task metrics when run context was external.

error

void error(java.lang.Throwable error)

Stop run. Set run state to 'failed'.

Parameters:

error -

logMetric

void logMetric(java.lang.String key,
               java.lang.Object value)

Log metric and attach it to the current task.

Parameters:

key -

value -

logDataframe

void logDataframe(java.lang.String key,
                  org.apache.spark.sql.Dataset<?> value,
                  HistogramRequest histogramRequest)

Log Spark dataframe

Parameters:

key -

value -

histogramRequest -

logHistogram

void logHistogram(java.util.Map<java.lang.String,java.lang.Object> histogram)

Log histogram object.

Parameters:

histogram -

logDatasetOperation

void logDatasetOperation(java.lang.String path,
                         DatasetOperationType type,
                         DatasetOperationStatus status,
                         java.lang.String valuePreview,
                         java.lang.String error,
                         java.util.List<java.lang.Long> dataDimensions,
                         java.lang.Object dataSchema,
                         java.lang.Boolean withPartition,
                         java.util.List<ColumnStats> columnStats)

Log dataset operations.

Parameters:

path -

type -

status -

valuePreview -

dataDimensions -

dataSchema -

columnStats -

logDatasetOperation

void logDatasetOperation(java.lang.String path,
                         DatasetOperationType type,
                         DatasetOperationStatus status,
                         org.apache.spark.sql.Dataset<?> data,
                         java.lang.Throwable error,
                         LogDatasetRequest params)

Log dataset operations with options like preview and schema generation.

Parameters:

path -

type -

status -

data -

params -

logMetrics

void logMetrics(java.util.Map<java.lang.String,java.lang.Object> metrics)

Log metrics batch and attach it to the current task.

Parameters:

metrics -

logMetrics

void logMetrics(java.util.Map<java.lang.String,java.lang.Object> metrics,
                java.lang.String source)

Log metrics batch with source

Parameters:

metrics -

source -

saveLog

void saveLog(org.apache.log4j.spi.LoggingEvent event,
             java.lang.String formattedEvent)

Save log and attach it to the current task and all parent tasks.

Parameters:

event -

formattedEvent -

saveSparkMetrics

void saveSparkMetrics(org.apache.spark.scheduler.SparkListenerStageCompleted event)

Save spark metrics.

Parameters:

event -

getTaskName

java.lang.String getTaskName(java.lang.reflect.Method method)

Extract task name either from method name or annotation value.

Parameters:

method -

Returns:

task name extracted from method.

setDriverTask

void setDriverTask(TaskRun taskRun)

Override task run to avoid creating duplicate runs.

Parameters:

taskRun - task run