org.apache.hadoop.hive.ql.exec.spark

Class RemoteHiveSparkClient

java.lang.Object

org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, HiveSparkClient

public class RemoteHiveSparkClient
extends Object
implements HiveSparkClient

RemoteSparkClient is a wrapper of SparkClient, which wrap a spark job request and send to an remote SparkContext.

See Also:

Serialized Form

Method Summary

Modifier and Type	Method and Description
void	close()
SparkJobRef	execute(DriverContext driverContext, SparkWork sparkWork) HiveSparkClient should generate Spark RDD graph by given sparkWork and driverContext, and submit RDD graph to Spark cluster.
int	getDefaultParallelism() For standalone mode, this can be used to get total number of cores.
int	getExecutorCount()
org.apache.spark.SparkConf	getSparkConf()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getSparkConf

public org.apache.spark.SparkConf getSparkConf()

Specified by:

getSparkConf in interface HiveSparkClient

Returns:

spark configuration

getExecutorCount

public int getExecutorCount()
                     throws Exception

Specified by:

getExecutorCount in interface HiveSparkClient

Returns:

the number of executors

Throws:

Exception

getDefaultParallelism

public int getDefaultParallelism()
                          throws Exception

Description copied from interface: HiveSparkClient

For standalone mode, this can be used to get total number of cores.

Specified by:

getDefaultParallelism in interface HiveSparkClient

Returns:

default parallelism.

Throws:

Exception

execute

public SparkJobRef execute(DriverContext driverContext,
                           SparkWork sparkWork)
                    throws Exception

Description copied from interface: HiveSparkClient

HiveSparkClient should generate Spark RDD graph by given sparkWork and driverContext, and submit RDD graph to Spark cluster.

Specified by:

execute in interface HiveSparkClient

Returns:

SparkJobRef could be used to track spark job progress and metrics.

Throws:

Exception

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable