sparkmeasure

Type Members

1. class FlightRecorderStageMetrics extends StageInfoRecorderListener

   

   FlightRecorderStageMetrics - Use Spark Listeners defined in stagemetrics.scala to record task metrics data aggregated at the Stage level, without changing the application code.

   FlightRecorderStageMetrics - Use Spark Listeners defined in stagemetrics.scala to record task metrics data aggregated at the Stage level, without changing the application code. The resulting data can be saved to a file and/or printed to stdout.

   Use: by adding the following configuration to spark-submit (or Spark Session) configuration --conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderStageMetrics

   Additional configuration parameters: --conf spark.sparkmeasure.outputFormat=<format>, valid values: java,json,json_to_hadoop default "json" note: json and java serialization formats, write to the driver local filesystem json_to_hadoop, writes to JSON serialized metrics to HDFS or to an Hadoop compliant filesystem, such as s3a

   --conf spark.sparkmeasure.outputFilename=<output file>, default: "/tmp/stageMetrics_flightRecorder" --conf spark.sparkmeasure.printToStdout=<true|false>, default false. Set to true to print JSON serialized metrics to stdout.

2. class FlightRecorderTaskMetrics extends TaskInfoRecorderListener

   

   FlightRecorderTaskMetrics - Use a Spark Listener to record task metrics data and save them to a file

   FlightRecorderTaskMetrics - Use a Spark Listener to record task metrics data and save them to a file

   Use: by adding the following configuration to spark-submit (or Spark Session) configuration --conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderTaskMetrics

   Additional configuration parameters: --conf spark.sparkmeasure.outputFormat=<format>, valid values: java,json,json_to_hadoop default "json" note: json and java serialization formats, write to the driver local filesystem json_to_hadoop, writes to JSON serialized metrics to HDFS or to an Hadoop compliant filesystem, such as s3a

   --conf spark.sparkmeasure.outputFilename=<output file>, default: "/tmp/taskMetrics_flightRecorder" --conf spark.sparkmeasure.printToStdout=<true|false>, default false. Set to true to print JSON serialized metrics to stdout.

3. class InfluxDBSink extends SparkListener

   

   InfluxDBSink: write Spark metrics and application info in near real-time to InfluxDB use this mode to monitor Spark execution workload use for Grafana dashboard and analytics of job execution How to use: attach the InfluxDBSInk to a Spark Context using the extra listener infrastructure.

   InfluxDBSink: write Spark metrics and application info in near real-time to InfluxDB use this mode to monitor Spark execution workload use for Grafana dashboard and analytics of job execution How to use: attach the InfluxDBSInk to a Spark Context using the extra listener infrastructure. Example: --conf spark.extraListeners=ch.cern.sparkmeasure.InfluxDBSink

   Configuration for InfluxDBSink is handled with Spark conf parameters:

   spark.sparkmeasure.influxdbURL (default "http://localhost:8086") spark.sparkmeasure.influxdbUsername (default "", not this can be empty if InfluxDB is configured with no authentication) spark.sparkmeasure.influxdbPassword (default "") spark.sparkmeasure.influxdbName (default "sparkmeasure") spark.sparkmeasure.influxdbStagemetrics, (boolean, default is false) spark.sparkmeasure.influxdbEnableBatch, boolean, default true Note: this is to improve write performance, but it requires to explicitly stopping Spark Session for clean exit: spark.stop() consider setting it to false if this is an issue

   This code depends on "influxdb.java", you may need to add the dependency: --packages org.influxdb:influxdb-java:2.14 Note currently we need to use version 2.14 as newer versions generate jar conflicts (tested up to Spark 3.3.0)

   InfluxDBExtended: provides additional and verbose info on Task execution use: --conf spark.extraListeners=ch.cern.sparkmeasure.InfluxDBSinkExtended

   InfluxDBSink: the amount of data generated is relatively small in most applications: O(number_of_stages) InfluxDBSInkExtended can generate a large amount of data O(Number_of_tasks), use with care

4. class InfluxDBSinkExtended extends InfluxDBSink

   

   InfluxDBSinkExtended extends the basic Influx Sink functionality with a verbose dump of Task metrics and task info into InfluxDB Note: this can generate a large amount of data O(Number_of_tasks) Configuration parameters and how-to use: see InfluxDBSink

5. class KafkaSink extends SparkListener

   

   KafkaSink: write Spark metrics and application info in near real-time to Kafka stream use this mode to monitor Spark execution workload use for Grafana dashboard and analytics of job execution

   KafkaSink: write Spark metrics and application info in near real-time to Kafka stream use this mode to monitor Spark execution workload use for Grafana dashboard and analytics of job execution

   How to use: attach the KafkaSink to a Spark Context using the extra listener infrastructure. Example: --conf spark.extraListeners=ch.cern.sparkmeasure.KafkaSink

   Configuration for KafkaSink is handled with Spark conf parameters:

   spark.sparkmeasure.kafkaBroker = Kafka broker endpoint URL example: --conf spark.sparkmeasure.kafkaBroker=kafka.your-site.com:9092 spark.sparkmeasure.kafkaTopic = Kafka topic example: --conf spark.sparkmeasure.kafkaTopic=sparkmeasure-stageinfo

   This code depends on "kafka clients", you may need to add the dependency: --packages org.apache.kafka:kafka-clients:3.2.1

   Output: each message contains the name, it is acknowledged as metrics name as well. Note: the amount of data generated is relatively small in most applications: O(number_of_stages)

6. class KafkaSinkExtended extends KafkaSink

   

   KafkaSinkExtended extends the basic KafkaSink functionality with a verbose dump of tasks metrics Note: this can generate a large amount of data O(Number_of_tasks) Configuration parameters and how-to use: see KafkaSink

7. case class PushGateway(serverIPnPort: String, metricsJob: String) extends Product with Serializable

   

   serverIPnPort: String with prometheus pushgateway hostIP:Port, metricsJob: job name

8. class StageInfoRecorderListener extends SparkListener

   

   StageInfoRecorderListener: this listener gathers metrics with Stage execution granularity It is based on the Spark Listener interface Stage metrics are stored in memory and use to produce a report that aggregates resource consumption they can also be consumed "raw" (transformed into a DataFrame and/or saved to a file) See StageMetrics

9. case class StageMetrics(sparkSession: SparkSession) extends Product with Serializable

   

   Stage Metrics: collects stage-level metrics with Stage granularity and provides aggregation and reporting functions for the end-user

   Stage Metrics: collects stage-level metrics with Stage granularity and provides aggregation and reporting functions for the end-user

   Example usage for stage metrics: val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark) stageMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show)

   The tool is based on using Spark Listeners as data source and collecting metrics in a ListBuffer of a case class that encapsulates Spark task metrics. The List Buffer is then transformed into a DataFrame for ease of reporting and analysis.

   Stage metrics are stored in memory and use to produce a report that aggregates resource consumption they can also be consumed "raw" (transformed into a DataFrame and/or saved to a file)

10. case class StageVals(jobId: Int, jobGroup: String, stageId: Int, name: String, submissionTime: Long, completionTime: Long, stageDuration: Long, numTasks: Int, executorRunTime: Long, executorCpuTime: Long, executorDeserializeTime: Long, executorDeserializeCpuTime: Long, resultSerializationTime: Long, jvmGCTime: Long, resultSize: Long, diskBytesSpilled: Long, memoryBytesSpilled: Long, peakExecutionMemory: Long, recordsRead: Long, bytesRead: Long, recordsWritten: Long, bytesWritten: Long, shuffleFetchWaitTime: Long, shuffleTotalBytesRead: Long, shuffleTotalBlocksFetched: Long, shuffleLocalBlocksFetched: Long, shuffleRemoteBlocksFetched: Long, shuffleLocalBytesRead: Long, shuffleRemoteBytesRead: Long, shuffleRemoteBytesReadToDisk: Long, shuffleRecordsRead: Long, shuffleWriteTime: Long, shuffleBytesWritten: Long, shuffleRecordsWritten: Long) extends Product with Serializable

    

11. class TaskInfoRecorderListener extends SparkListener

    

    TaskInfoRecorderListener: this listener gathers metrics with Task execution granularity It is based on the Spark Listener interface Task metrics are stored in memory and use to produce a report that aggregates resource consumption they can also be consumed "raw" (transformed into a DataFrame and/or saved to a file)

12. case class TaskMetrics(sparkSession: SparkSession) extends Product with Serializable

    

    Task Metrics: collects metrics data at Task granularity and provides aggregation and reporting functions for the end-user

    Task Metrics: collects metrics data at Task granularity and provides aggregation and reporting functions for the end-user

    Example of how to use task metrics: val taskMetrics = ch.cern.sparkmeasure.TaskMetrics(spark) taskMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show)

    The tool is based on using Spark Listeners as data source and collecting metrics in a ListBuffer of a case class that encapsulates Spark task metrics.

13. case class TaskVals(jobId: Int, jobGroup: String, stageId: Int, index: Long, launchTime: Long, finishTime: Long, duration: Long, schedulerDelay: Long, executorId: String, host: String, taskLocality: Int, speculative: Boolean, gettingResultTime: Long, successful: Boolean, executorRunTime: Long, executorCpuTime: Long, executorDeserializeTime: Long, executorDeserializeCpuTime: Long, resultSerializationTime: Long, jvmGCTime: Long, resultSize: Long, diskBytesSpilled: Long, memoryBytesSpilled: Long, peakExecutionMemory: Long, recordsRead: Long, bytesRead: Long, recordsWritten: Long, bytesWritten: Long, shuffleFetchWaitTime: Long, shuffleTotalBytesRead: Long, shuffleTotalBlocksFetched: Long, shuffleLocalBlocksFetched: Long, shuffleRemoteBlocksFetched: Long, shuffleLocalBytesRead: Long, shuffleRemoteBytesRead: Long, shuffleRemoteBytesReadToDisk: Long, shuffleRecordsRead: Long, shuffleWriteTime: Long, shuffleBytesWritten: Long, shuffleRecordsWritten: Long) extends Product with Serializable