org.apache.hadoop.hive.ql.exec.mr

Class ExecDriver

java.lang.Object

org.apache.hadoop.hive.ql.exec.Task<MapredWork>

org.apache.hadoop.hive.ql.exec.mr.ExecDriver

All Implemented Interfaces:

Serializable, HadoopJobExecHook, Node

Direct Known Subclasses:

MapRedTask

public class ExecDriver
extends Task<MapredWork>
implements Serializable, HadoopJobExecHook

ExecDriver is the central class in co-ordinating execution of any map-reduce task. It's main responsibilities are: - Converting the plan (MapredWork) into a MR Job (JobConf) - Submitting a MR job to the cluster via JobClient and ExecHelper - Executing MR job in local execution mode (where applicable)

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Task

Task.FeedType, Task.TaskState

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.mapred.JobConf	job
protected HadoopJobExecHelper	jobExecHelper
protected static org.slf4j.Logger	LOG
static MemoryMXBean	memoryMXBean

Fields inherited from class org.apache.hadoop.hive.ql.exec.Task

BACKUP_COMMON_JOIN, backupChildrenTasks, backupTask, childTasks, clonedConf, COMMON_JOIN, conf, console, CONVERTED_MAPJOIN, CONVERTED_MAPJOIN_LOCAL, CONVERTED_SORTMERGEJOIN, driverContext, feedSubscribers, HINTED_MAPJOIN, HINTED_MAPJOIN_LOCAL, id, jobID, MAPJOIN_ONLY_NOBACKUP, NO_TAG, parentTasks, queryDisplay, queryPlan, queryState, rootTask, taskCounters, taskHandle, taskTag, work

Constructor Summary

Constructors

Constructor and Description
ExecDriver() Constructor when invoked from QL.
ExecDriver(MapredWork plan, org.apache.hadoop.mapred.JobConf job, boolean isSilent) Constructor/Initialization for invocation as independent utility.

Method Summary

Modifier and Type	Method and Description
boolean	checkFatalErrors(org.apache.hadoop.mapred.Counters ctrs, StringBuilder errMsg) Fatal errors are those errors that cannot be recovered by retries.
int	execute(DriverContext driverContext) Execute a query plan using Hadoop.
static String	generateCmdLine(HiveConf hconf, Context ctx) Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver.
static String	getCurrentDB()
String	getExternalHandle()
Collection<MapWork>	getMapWork()
String	getName() Gets the name of the node.
protected static String	getResource(HiveConf conf, SessionState.ResourceType resType) Retrieve the resources from the current session and configuration for the given type.
Collection<Operator<? extends OperatorDesc>>	getTopOperators()
org.apache.hadoop.hive.ql.plan.api.StageType	getType() Should be overridden to return the type of the specific task among the types in StageType.
boolean	hasReduce()
void	initialize(QueryState queryState, QueryPlan queryPlan, DriverContext driverContext, CompilationOpContext opContext) Initialization when invoked from QL.
boolean	isMapRedTask()
boolean	isTaskShutdown()
void	logPlanProgress(SessionState ss)
static void	main(String[] args)
boolean	mapDone()
boolean	mapStarted()
static void	propagateSplitSettings(org.apache.hadoop.mapred.JobConf job, MapWork work)
boolean	reduceDone()
boolean	reduceStarted()
boolean	requireLock()
protected void	setInputAttributes(org.apache.hadoop.conf.Configuration conf) Set hive input format, and input format file if necessary.
void	shutdown()

Methods inherited from class org.apache.hadoop.hive.ql.exec.Task

addDependentTask, canExecuteInParallel, cloneConf, done, equals, executeTask, findLeafs, getAndInitBackupTask, getBackupChildrenTasks, getBackupTask, getChildren, getChildTasks, getCounters, getDependentTasks, getDiagnosticsMessage, getDriverContext, getFeedSubscribers, getHive, getId, getInitialized, getJobID, getNumChild, getNumParent, getParentTasks, getQueryPlan, getQueryState, getQueued, getReducer, getStatusMessage, getTaskHandle, getTaskState, getTaskTag, getTxnMgr, getWork, hashCode, isFetchSource, isLocalMode, isMapRedLocalTask, isNotInitialized, isRootTask, isRunnable, pushFeed, receiveFeed, removeDependentTask, removeFromChildrenTasks, setBackupChildrenTasks, setBackupTask, setChildTasks, setConf, setConsole, setDiagnosticMessage, setDone, setDriverContext, setException, setFeedSubscribers, setFetchSource, setId, setInitialized, setLocalMode, setParentTasks, setQueryDisplay, setQueryPlan, setQueued, setRootTask, setStarted, setState, setStatusMessage, setTaskTag, setWork, started, subscribeFeed, toString, updateStatusInQueryDisplay, updateTaskMetrics

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

job

protected transient org.apache.hadoop.mapred.JobConf job

memoryMXBean

public static MemoryMXBean memoryMXBean

jobExecHelper

protected HadoopJobExecHelper jobExecHelper

LOG

protected static final transient org.slf4j.Logger LOG

Constructor Detail

ExecDriver

public ExecDriver()

Constructor when invoked from QL.

ExecDriver

public ExecDriver(MapredWork plan,
                  org.apache.hadoop.mapred.JobConf job,
                  boolean isSilent)
           throws HiveException

Constructor/Initialization for invocation as independent utility.

Throws:

HiveException

Method Detail

requireLock

public boolean requireLock()

Overrides:

requireLock in class Task<MapredWork>

getResource

protected static String getResource(HiveConf conf,
                                    SessionState.ResourceType resType)

Retrieve the resources from the current session and configuration for the given type.

Returns:

Comma-separated list of resources

initialize

public void initialize(QueryState queryState,
                       QueryPlan queryPlan,
                       DriverContext driverContext,
                       CompilationOpContext opContext)

Initialization when invoked from QL.

Overrides:

initialize in class Task<MapredWork>

checkFatalErrors

public boolean checkFatalErrors(org.apache.hadoop.mapred.Counters ctrs,
                                StringBuilder errMsg)

Fatal errors are those errors that cannot be recovered by retries. These are application dependent. Examples of fatal errors include: - the small table in the map-side joins is too large to be feasible to be handled by one mapper. The job should fail and the user should be warned to use regular joins rather than map-side joins. Fatal errors are indicated by counters that are set at execution time. If the counter is non-zero, a fatal error occurred. The value of the counter indicates the error type.

Specified by:

checkFatalErrors in interface HadoopJobExecHook

Returns:

true if fatal errors happened during job execution, false otherwise.

execute

public int execute(DriverContext driverContext)

Execute a query plan using Hadoop.

Specified by:

execute in class Task<MapredWork>

Returns:

status of executing the task

propagateSplitSettings

public static void propagateSplitSettings(org.apache.hadoop.mapred.JobConf job,
                                          MapWork work)

setInputAttributes

protected void setInputAttributes(org.apache.hadoop.conf.Configuration conf)

Set hive input format, and input format file if necessary.

getCurrentDB

public static String getCurrentDB()

mapStarted

public boolean mapStarted()

reduceStarted

public boolean reduceStarted()

mapDone

public boolean mapDone()

reduceDone

public boolean reduceDone()

main

public static void main(String[] args)
                 throws IOException,
                        HiveException

Throws:

IOException

HiveException

generateCmdLine

public static String generateCmdLine(HiveConf hconf,
                                     Context ctx)
                              throws IOException

Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver.

Throws:

IOException

getMapWork

public Collection<MapWork> getMapWork()

Overrides:

getMapWork in class Task<MapredWork>

isMapRedTask

public boolean isMapRedTask()

Overrides:

isMapRedTask in class Task<MapredWork>

getTopOperators

public Collection<Operator<? extends OperatorDesc>> getTopOperators()

Overrides:

getTopOperators in class Task<MapredWork>

hasReduce

public boolean hasReduce()

Overrides:

hasReduce in class Task<MapredWork>

getType

public org.apache.hadoop.hive.ql.plan.api.StageType getType()

Description copied from class: Task

Should be overridden to return the type of the specific task among the types in StageType.

Specified by:

getType in class Task<MapredWork>

Returns:

StageType.* or null if not overridden

getName

public String getName()

Description copied from interface: Node

Gets the name of the node. This is used in the rule dispatchers.

Specified by:

getName in interface Node

Returns:

String

logPlanProgress

public void logPlanProgress(SessionState ss)
                     throws IOException

Specified by:

logPlanProgress in interface HadoopJobExecHook

Throws:

IOException

isTaskShutdown

public boolean isTaskShutdown()

shutdown

public void shutdown()

Overrides:

shutdown in class Task<MapredWork>

getExternalHandle

public String getExternalHandle()

Overrides:

getExternalHandle in class Task<MapredWork>