c

ch.cern.sparkmeasure

InfluxDBSink

class InfluxDBSink extends SparkListener

InfluxDBSink: write Spark metrics and application info in near real-time to InfluxDB v 1.x use this mode to monitor Spark execution workload use for Grafana dashboard and analytics of job execution How to use: attach the InfluxDBSInk to a Spark Context using the extra listener infrastructure. Note: this is for InfluxDB v1.x Example: --conf spark.extraListeners=ch.cern.sparkmeasure.InfluxDBSink

Configuration for InfluxDBSink is handled with Spark conf parameters:

spark.sparkmeasure.influxdbURL (default "http://localhost:8086") spark.sparkmeasure.influxdbUsername (default "", this can be empty if InfluxDB is configured with no authentication) spark.sparkmeasure.influxdbPassword (default "") spark.sparkmeasure.influxdbName (default "sparkmeasure") spark.sparkmeasure.influxdbStagemetrics, (boolean, default is false) spark.sparkmeasure.influxdbEnableBatch, boolean, default true Note: this is to improve write performance, but it requires to explicitly stopping Spark Session for clean exit: spark.stop() consider setting it to false if this is an issue

This code depends on "influxdb.java", you may need to add the dependency: --packages org.influxdb:influxdb-java:2.14 Note currently we need to use version 2.14 as newer versions generate jar conflicts (tested up to Spark 3.3.0)

InfluxDBExtended: provides additional and verbose info on Task execution use: --conf spark.extraListeners=ch.cern.sparkmeasure.InfluxDBSinkExtended

InfluxDBSink: the amount of data generated is relatively small in most applications: O(number_of_stages) InfluxDBSInkExtended can generate a large amount of data O(Number_of_tasks), use with care

Linear Supertypes
SparkListener, SparkListenerInterface, AnyRef, Any
Known Subclasses
InfluxDBSinkExtended
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. InfluxDBSink
  2. SparkListener
  3. SparkListenerInterface
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new InfluxDBSink(conf: SparkConf)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. var appId: String
  5. var appName: String
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @IntrinsicCandidate()
  8. val database: InfluxDB
  9. val dbName: String
  10. var dynamicAllocationEnabled: String
  11. val enableBatch: Boolean
  12. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  15. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  16. val influxDB: InfluxDB
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. val logStageMetrics: Boolean
  19. lazy val logger: Logger
  20. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  22. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  23. def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit
    Definition Classes
    InfluxDBSink → SparkListener → SparkListenerInterface
  24. def onApplicationStart(applicationStart: SparkListenerApplicationStart): Unit
    Definition Classes
    InfluxDBSink → SparkListener → SparkListenerInterface
  25. def onBlockManagerAdded(blockManagerAdded: SparkListenerBlockManagerAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  26. def onBlockManagerRemoved(blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  27. def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  28. def onEnvironmentUpdate(environmentUpdate: SparkListenerEnvironmentUpdate): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  29. def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit
    Definition Classes
    InfluxDBSink → SparkListener → SparkListenerInterface
  30. def onExecutorBlacklisted(executorBlacklisted: SparkListenerExecutorBlacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  31. def onExecutorBlacklistedForStage(executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  32. def onExecutorExcluded(executorExcluded: SparkListenerExecutorExcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  33. def onExecutorExcludedForStage(executorExcludedForStage: SparkListenerExecutorExcludedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  34. def onExecutorMetricsUpdate(executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  35. def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  36. def onExecutorUnblacklisted(executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  37. def onExecutorUnexcluded(executorUnexcluded: SparkListenerExecutorUnexcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  38. def onJobEnd(jobEnd: SparkListenerJobEnd): Unit
    Definition Classes
    InfluxDBSink → SparkListener → SparkListenerInterface
  39. def onJobStart(jobStart: SparkListenerJobStart): Unit
    Definition Classes
    InfluxDBSink → SparkListener → SparkListenerInterface
  40. def onNodeBlacklisted(nodeBlacklisted: SparkListenerNodeBlacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  41. def onNodeBlacklistedForStage(nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  42. def onNodeExcluded(nodeExcluded: SparkListenerNodeExcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  43. def onNodeExcludedForStage(nodeExcludedForStage: SparkListenerNodeExcludedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  44. def onNodeUnblacklisted(nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  45. def onNodeUnexcluded(nodeUnexcluded: SparkListenerNodeUnexcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  46. def onOtherEvent(event: SparkListenerEvent): Unit
    Definition Classes
    InfluxDBSink → SparkListener → SparkListenerInterface
  47. def onResourceProfileAdded(event: SparkListenerResourceProfileAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  48. def onSpeculativeTaskSubmitted(speculativeTask: SparkListenerSpeculativeTaskSubmitted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  49. def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit
    Definition Classes
    InfluxDBSink → SparkListener → SparkListenerInterface
  50. def onStageExecutorMetrics(executorMetrics: SparkListenerStageExecutorMetrics): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  51. def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit
    Definition Classes
    InfluxDBSink → SparkListener → SparkListenerInterface
  52. def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  53. def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  54. def onTaskStart(taskStart: SparkListenerTaskStart): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  55. def onUnpersistRDD(unpersistRDD: SparkListenerUnpersistRDD): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  56. def onUnschedulableTaskSetAdded(unschedulableTaskSetAdded: SparkListenerUnschedulableTaskSetAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  57. def onUnschedulableTaskSetRemoved(unschedulableTaskSetRemoved: SparkListenerUnschedulableTaskSetRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  58. val password: String
  59. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  60. def toString(): String
    Definition Classes
    AnyRef → Any
  61. val url: String
  62. val username: String
  63. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  64. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  65. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from SparkListener

Inherited from SparkListenerInterface

Inherited from AnyRef

Inherited from Any

Ungrouped