org.apache.hadoop.hive.ql.exec.spark

Class LocalHiveSparkClient

java.lang.Object

org.apache.hadoop.hive.ql.exec.spark.LocalHiveSparkClient

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, HiveSparkClient

public class LocalHiveSparkClient
extends Object
implements HiveSparkClient

LocalSparkClient submit Spark job in local driver, it's responsible for build spark client environment and execute spark work.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	LOG

Method Summary

Modifier and Type	Method and Description
void	close()
SparkJobRef	execute(DriverContext driverContext, SparkWork sparkWork) HiveSparkClient should generate Spark RDD graph by given sparkWork and driverContext, and submit RDD graph to Spark cluster.
int	getDefaultParallelism() For standalone mode, this can be used to get total number of cores.
int	getExecutorCount()
static LocalHiveSparkClient	getInstance(org.apache.spark.SparkConf sparkConf, HiveConf hiveConf)
org.apache.spark.SparkConf	getSparkConf()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final transient org.slf4j.Logger LOG

Method Detail

getInstance

public static LocalHiveSparkClient getInstance(org.apache.spark.SparkConf sparkConf,
                                               HiveConf hiveConf)
                                        throws FileNotFoundException,
                                               MalformedURLException

Throws:

FileNotFoundException

MalformedURLException

getSparkConf

public org.apache.spark.SparkConf getSparkConf()

Specified by:

getSparkConf in interface HiveSparkClient

Returns:

spark configuration

getExecutorCount

public int getExecutorCount()

Specified by:

getExecutorCount in interface HiveSparkClient

Returns:

the number of executors

getDefaultParallelism

public int getDefaultParallelism()
                          throws Exception

Description copied from interface: HiveSparkClient

For standalone mode, this can be used to get total number of cores.

Specified by:

getDefaultParallelism in interface HiveSparkClient

Returns:

default parallelism.

Throws:

Exception

execute

public SparkJobRef execute(DriverContext driverContext,
                           SparkWork sparkWork)
                    throws Exception

Description copied from interface: HiveSparkClient

HiveSparkClient should generate Spark RDD graph by given sparkWork and driverContext, and submit RDD graph to Spark cluster.

Specified by:

execute in interface HiveSparkClient

Returns:

SparkJobRef could be used to track spark job progress and metrics.

Throws:

Exception

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable