org.apache.hadoop.hive.llap.daemon.impl

Class TaskExecutorService

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService

All Implemented Interfaces:

Closeable, AutoCloseable, Scheduler<TaskRunnerCallable>, SchedulerFragmentCompletingListener, org.apache.hadoop.service.Service

public class TaskExecutorService
extends org.apache.hadoop.service.AbstractService
implements Scheduler<TaskRunnerCallable>, SchedulerFragmentCompletingListener

Task executor service provides method for scheduling tasks. Tasks submitted to executor service are submitted to wait queue for scheduling. Wait queue tasks are ordered based on the priority of the task. The internal wait queue scheduler moves tasks from wait queue when executor slots are available or when a higher priority task arrives and will schedule it for execution. When pre-emption is enabled, the tasks from wait queue can replace(pre-empt) a running task. The pre-empted task is reported back to the Application Master(AM) for it to be rescheduled.

Because of the concurrent nature of task submission, the position of the task in wait queue is held as long the scheduling of the task from wait queue (with or without pre-emption) is complete. The order of pre-emption is based on the ordering in the pre-emption queue. All tasks that cannot run to completion immediately (canFinish = false) are added to pre-emption queue.

When all the executor threads are occupied and wait queue is full, the task scheduler will return SubmissionState.REJECTED response

Task executor service can be shut down which will terminated all running tasks and reject all new tasks. Shutting down of the task executor service can be done gracefully or immediately.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TaskExecutorService.PreemptionQueueComparator
static class	TaskExecutorService.TaskWrapper

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.llap.daemon.impl.Scheduler

Scheduler.SubmissionState

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.llap.daemon.SchedulerFragmentCompletingListener

SchedulerFragmentCompletingListener.State

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Constructor Summary

Constructors

Constructor and Description
TaskExecutorService(int numExecutors, int waitQueueSize, String waitQueueComparatorClassName, boolean enablePreemption, ClassLoader classLoader, LlapDaemonExecutorMetrics metrics, org.apache.hadoop.yarn.util.Clock clock)

Method Summary

Modifier and Type	Method and Description
QueryIdentifier	findQueryByFragment(String fragmentId)
void	fragmentCompleting(String fragmentId, SchedulerFragmentCompletingListener.State state) Indicates that a fragment is about to complete.
Set<String>	getExecutorsStatusForReporting()
int	getNumActiveForReporting()
void	killFragment(String fragmentId) Attempt to kill the fragment with the specified fragmentId
Scheduler.SubmissionState	schedule(TaskRunnerCallable task) Schedule the task or throw RejectedExecutionException if queues are full
void	serviceStop()
void	shutDown(boolean awaitTermination)
boolean	updateFragment(String fragmentId, boolean isGuaranteed)

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, serviceStart, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

TaskExecutorService

public TaskExecutorService(int numExecutors,
                           int waitQueueSize,
                           String waitQueueComparatorClassName,
                           boolean enablePreemption,
                           ClassLoader classLoader,
                           LlapDaemonExecutorMetrics metrics,
                           org.apache.hadoop.yarn.util.Clock clock)

Method Detail

serviceStop

public void serviceStop()

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

getNumActiveForReporting

public int getNumActiveForReporting()

Specified by:

getNumActiveForReporting in interface Scheduler<TaskRunnerCallable>

getExecutorsStatusForReporting

public Set<String> getExecutorsStatusForReporting()

Specified by:

getExecutorsStatusForReporting in interface Scheduler<TaskRunnerCallable>

schedule

public Scheduler.SubmissionState schedule(TaskRunnerCallable task)

Description copied from interface: Scheduler

Schedule the task or throw RejectedExecutionException if queues are full

Specified by:

schedule in interface Scheduler<TaskRunnerCallable>

Parameters:

task - - task to schedule

Returns:

SubmissionState

updateFragment

public boolean updateFragment(String fragmentId,
                              boolean isGuaranteed)

Specified by:

updateFragment in interface Scheduler<TaskRunnerCallable>

findQueryByFragment

public QueryIdentifier findQueryByFragment(String fragmentId)

Specified by:

findQueryByFragment in interface Scheduler<TaskRunnerCallable>

killFragment

public void killFragment(String fragmentId)

Description copied from interface: Scheduler

Attempt to kill the fragment with the specified fragmentId

Specified by:

killFragment in interface Scheduler<TaskRunnerCallable>

fragmentCompleting

public void fragmentCompleting(String fragmentId,
                               SchedulerFragmentCompletingListener.State state)

Description copied from interface: SchedulerFragmentCompletingListener

Indicates that a fragment is about to complete.

Specified by:

fragmentCompleting in interface SchedulerFragmentCompletingListener

shutDown

public void shutDown(boolean awaitTermination)