org.apache.hadoop.hive.ql.exec.spark

Interface HiveSparkClient

All Superinterfaces:

AutoCloseable, Closeable, Serializable

All Known Implementing Classes:

LocalHiveSparkClient, RemoteHiveSparkClient

public interface HiveSparkClient
extends Serializable, Closeable

Method Summary

Modifier and Type	Method and Description
SparkJobRef	execute(DriverContext driverContext, SparkWork sparkWork) HiveSparkClient should generate Spark RDD graph by given sparkWork and driverContext, and submit RDD graph to Spark cluster.
int	getDefaultParallelism() For standalone mode, this can be used to get total number of cores.
int	getExecutorCount()
org.apache.spark.SparkConf	getSparkConf()

Methods inherited from interface java.io.Closeable

close

Method Detail

execute

SparkJobRef execute(DriverContext driverContext,
                    SparkWork sparkWork)
             throws Exception

HiveSparkClient should generate Spark RDD graph by given sparkWork and driverContext, and submit RDD graph to Spark cluster.

Parameters:

driverContext -

sparkWork -

Returns:

SparkJobRef could be used to track spark job progress and metrics.

Throws:

Exception

getSparkConf

org.apache.spark.SparkConf getSparkConf()

Returns:

spark configuration

getExecutorCount

int getExecutorCount()
              throws Exception

Returns:

the number of executors

Throws:

Exception

getDefaultParallelism

int getDefaultParallelism()
                   throws Exception

For standalone mode, this can be used to get total number of cores.

Returns:

default parallelism.

Throws:

Exception