Accumulable

A data type that can be accumulated, i.e. has a commutative and associative "add" operation, but where the result type, R, may be different from the element type being added, T.

You must define how to add data, and how to merge two of these together. For some data types, such as a counter, these might be the same operation. In that case, you can use the simpler org.apache.spark.Accumulator. They won't always be the same, though -- e.g., imagine you are accumulating a set. You will add items to the set, and you will union two sets together.

Operations are not thread-safe.

R: the full accumulated data (result type)
T: partial data that can be added in

Annotations: @deprecated
Deprecated: (Since version 2.0.0) use AccumulatorV2
Source: Accumulable.scala

Linear Supertypes

Serializable, AnyRef, Any

Known Subclasses

Accumulator

Instance Constructors

new Accumulable(initialValue: R, param: AccumulableParam[R, T])

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
def ++=(term: R): Unit

Merge two accumulable objects together
Merge two accumulable objects together
Normally, a user will not want to use this version, but will instead call +=.
term
the other R that will get merged with this
def +=(term: T): Unit

Add more data to this accumulator / accumulable
Add more data to this accumulator / accumulable
term
the data to add
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def add(term: T): Unit

Add more data to this accumulator / accumulable
Add more data to this accumulator / accumulable
term
the data to add
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
val id: Long

ID of this accumulator; for internal use only.
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def localValue: R

Get the current value of this accumulator from within a task.
Get the current value of this accumulator from within a task.
This is NOT the global value of the accumulator. To get the global value after a completed operation on the dataset, call value.
The typical use of this method is to directly mutate the local value, eg., to add an element to a Set.
def merge(term: R): Unit

Merge two accumulable objects together
Merge two accumulable objects together
Normally, a user will not want to use this version, but will instead call add.
term
the other R that will get merged with this
val name: Option[String]

human-readable name for use in Spark's web UI
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def setValue(newValue: R): Unit

Set the accumulator's value.
Set the accumulator's value. For internal use only.
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Accumulable → AnyRef → Any
def value: R

Access the accumulator's current value; only allowed on driver.
def value_=(newValue: R): Unit

Set the accumulator's value; only allowed on driver.
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
val zero: R

Related Doc: package spark

class Accumulable[R, T] extends Serializable

Instance Constructors

new Accumulable(initialValue: R, param: AccumulableParam[R, T])

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

def ++=(term: R): Unit

def +=(term: T): Unit

final def ==(arg0: Any): Boolean

def add(term: T): Unit

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

val id: Long

final def isInstanceOf[T0]: Boolean

def localValue: R

def merge(term: R): Unit

val name: Option[String]

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def setValue(newValue: R): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def value: R

def value_=(newValue: R): Unit

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

val zero: R

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped