Class

ch.cern.sparkmeasure

TaskInfoRecorderListener

Related Doc: package sparkmeasure

Permalink

class TaskInfoRecorderListener extends SparkListener

TaskInfoRecorderListener: this listener gathers metrics with Task execution granularity It is based on the Spark Listener interface Task metrics are stored in memory and use to produce a report that aggregates resource consumption they can also be consumed "raw" (transformed into a DataFrame and/or saved to a file)

Linear Supertypes
SparkListener, SparkListenerInterface, AnyRef, Any
Known Subclasses
FlightRecorderTaskMetrics
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. TaskInfoRecorderListener
  2. SparkListener
  3. SparkListenerInterface
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new TaskInfoRecorderListener()

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. val StageIdtoJobGroup: HashMap[Int, String]

    Permalink

  5. val StageIdtoJobId: HashMap[Int, Int]

    Permalink

  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  8. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  9. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  10. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  11. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  12. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  13. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  14. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  15. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  16. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  17. def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  18. def onApplicationStart(applicationStart: SparkListenerApplicationStart): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  19. def onBlockManagerAdded(blockManagerAdded: SparkListenerBlockManagerAdded): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  20. def onBlockManagerRemoved(blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  21. def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  22. def onEnvironmentUpdate(environmentUpdate: SparkListenerEnvironmentUpdate): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  23. def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  24. def onExecutorBlacklisted(executorBlacklisted: SparkListenerExecutorBlacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  25. def onExecutorBlacklistedForStage(executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  26. def onExecutorMetricsUpdate(executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  27. def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  28. def onExecutorUnblacklisted(executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  29. def onJobEnd(jobEnd: SparkListenerJobEnd): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  30. def onJobStart(jobStart: SparkListenerJobStart): Unit

    Permalink
    Definition Classes
    TaskInfoRecorderListener → SparkListener → SparkListenerInterface

  31. def onNodeBlacklisted(nodeBlacklisted: SparkListenerNodeBlacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  32. def onNodeBlacklistedForStage(nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  33. def onNodeUnblacklisted(nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  34. def onOtherEvent(event: SparkListenerEvent): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  35. def onSpeculativeTaskSubmitted(speculativeTask: SparkListenerSpeculativeTaskSubmitted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  36. def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  37. def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  38. def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit

    Permalink

    This methods fires at the end of each Task and collects metrics flattened into the taskMetricsData ListBuffer Note all times are in ms, cpu time and shufflewrite are originally in nanosec, thus in the code are divided by 1e6

    This methods fires at the end of each Task and collects metrics flattened into the taskMetricsData ListBuffer Note all times are in ms, cpu time and shufflewrite are originally in nanosec, thus in the code are divided by 1e6

    Definition Classes
    TaskInfoRecorderListener → SparkListener → SparkListenerInterface

  39. def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  40. def onTaskStart(taskStart: SparkListenerTaskStart): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  41. def onUnpersistRDD(unpersistRDD: SparkListenerUnpersistRDD): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  42. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  43. val taskMetricsData: ListBuffer[TaskVals]

    Permalink

  44. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  45. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  46. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  47. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from SparkListener

Inherited from SparkListenerInterface

Inherited from AnyRef

Inherited from Any

Ungrouped