Class

org.apache.spark.ml.spark

ProgressListener

Related Doc: package spark

Permalink

class ProgressListener extends SparkListener

One progress listener for each instance of model creation. Each listener gets all events so this particular one will respond only to the ones corresponding to a given RDD used for training (which should not be shared) and to a particular job type (depending on the algorithm).

It might not be 100% accurate depending on how many "non primary" jobs will be run.

Linear Supertypes
SparkListener, SparkListenerInterface, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ProgressListener
  2. SparkListener
  3. SparkListenerInterface
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ProgressListener(sc: SparkContext, job: Job[_], rddInfo: RDDInfo, stageNames: Iterable[String])

    Permalink

    job

    Job which progress is to be updated

    rddInfo

    events triggered for this RDD (or its children) trigger progress change

    stageNames

    which stages are supposed to change the progress bar %

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. var currentJobs: Map[Int, Int]

    Permalink

  7. var currentStages: Map[Int, String]

    Permalink

  8. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  9. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  10. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  11. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  12. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  13. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  14. val job: Job[_]

    Permalink

    Job which progress is to be updated

  15. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  16. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  17. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  18. def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  19. def onApplicationStart(applicationStart: SparkListenerApplicationStart): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  20. def onBlockManagerAdded(blockManagerAdded: SparkListenerBlockManagerAdded): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  21. def onBlockManagerRemoved(blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  22. def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  23. def onEnvironmentUpdate(environmentUpdate: SparkListenerEnvironmentUpdate): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  24. def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  25. def onExecutorBlacklisted(executorBlacklisted: SparkListenerExecutorBlacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  26. def onExecutorMetricsUpdate(executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  27. def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  28. def onExecutorUnblacklisted(executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  29. def onJobEnd(jobEnd: SparkListenerJobEnd): Unit

    Permalink
    Definition Classes
    ProgressListener → SparkListener → SparkListenerInterface

  30. def onJobStart(jobStart: SparkListenerJobStart): Unit

    Permalink
    Definition Classes
    ProgressListener → SparkListener → SparkListenerInterface

  31. def onNodeBlacklisted(nodeBlacklisted: SparkListenerNodeBlacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  32. def onNodeUnblacklisted(nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  33. def onOtherEvent(event: SparkListenerEvent): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  34. def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit

    Permalink
    Definition Classes
    ProgressListener → SparkListener → SparkListenerInterface

  35. def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit

    Permalink
    Definition Classes
    ProgressListener → SparkListener → SparkListenerInterface

  36. def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  37. def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  38. def onTaskStart(taskStart: SparkListenerTaskStart): Unit

    Permalink
    Definition Classes
    ProgressListener → SparkListener → SparkListenerInterface

  39. def onUnpersistRDD(unpersistRDD: SparkListenerUnpersistRDD): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface

  40. val rddInfo: RDDInfo

    Permalink

    events triggered for this RDD (or its children) trigger progress change

  41. val sc: SparkContext

    Permalink

  42. val stageNames: Iterable[String]

    Permalink

    which stages are supposed to change the progress bar %

  43. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  44. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  45. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  46. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  47. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from SparkListener

Inherited from SparkListenerInterface

Inherited from AnyRef

Inherited from Any

Ungrouped