Class

org.apache.spark

SparkParallelismTracker

Related Doc: package spark

Permalink

class SparkParallelismTracker extends AnyRef

A tracker that ensures enough number of executor cores are alive. Throws an exception when the number of alive cores is less than nWorkers.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkParallelismTracker
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SparkParallelismTracker(sc: SparkContext, timeout: Long, numWorkers: Int)

    Permalink

    sc

    The SparkContext object

    timeout

    The maximum time to wait for enough number of workers.

    numWorkers

    nWorkers used in an XGBoost Job

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  8. def execute[T](body: ⇒ T): T

    Permalink

    Execute a blocking function call with two checks on enough nWorkers:

    Execute a blocking function call with two checks on enough nWorkers:

    • Before the function starts, wait until there are enough executor cores.
    • During the execution, throws an exception if there is any executor lost.
    T

    Return type

    body

    A blocking function call

    returns

    The return of body

  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  10. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  11. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  12. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  13. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  14. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  15. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  16. val sc: SparkContext

    Permalink

    The SparkContext object

  17. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  18. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  19. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  20. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  21. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped