org.apache.hadoop.hive.ql.exec.tez

Class WorkloadManager

java.lang.Object

org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.AbstractTriggerValidator

org.apache.hadoop.hive.ql.exec.tez.WorkloadManager

All Implemented Interfaces:

TezSessionPoolSession.Manager, WorkloadManagerMxBean

public class WorkloadManager
extends TezSessionPoolSession.AbstractTriggerValidator
implements TezSessionPoolSession.Manager, WorkloadManagerMxBean

Workload management entry point for HS2. Note on how this class operates. There are tons of things that could be happening in parallel that are a real pain to sync. Therefore, it uses an actor-ish model where the master thread, in processCurrentEvents method, processes a bunch of events that have accumulated since the previous iteration, repeatedly and quickly, doing physical work via async calls or via worker threads. That way the bulk of the state (pools, etc.) does not require any sync, and we mostly have a consistent view of the conflicting events when we process things. However, that also means none of that state can be accessed directly - most changes that touch pool state, or interact with background operations like init, need to go thru eventstate; see e.g. returnAfterUse.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WorkloadManager.NoPoolMappingException

Field Summary

Fields

Modifier and Type	Field and Description
protected Thread	wmThread The master thread the processes the events from EventState.

Method Summary

Modifier and Type	Method and Description
void	closeAndReopenExpiredSession(org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession session)
static WorkloadManager	create(String yarnQueue, HiveConf conf, org.apache.hadoop.hive.metastore.api.WMFullResourcePlan plan) Called once, when HS2 initializes.
protected WmTezSession	createSessionObject(String sessionId, HiveConf conf)
void	destroy(TezSessionState session)
protected HiveConf	getConf()
org.apache.hadoop.hive.ql.exec.tez.SessionExpirationTracker	getExpirationTracker()
static WorkloadManager	getInstance()
WmTezSession	getSession(TezSessionState session, UserPoolMapping.MappingInput input, HiveConf conf)
WmTezSession	getSession(TezSessionState session, UserPoolMapping.MappingInput input, HiveConf conf, WmContext wmContext)
List<String>	getWmStateDescription()
void	notifyInitializationCompleted(org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.SessionInitContext initCtx)
void	notifyOfClusterStateChange()
void	notifyOfInconsistentAllocation(WmTezSession session)
void	registerOpenSession(org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession session) Called by TezSessionPoolSession when opened.
TezSessionState	reopen(TezSessionState session)
void	returnAfterUse(org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession session)
void	start()
void	stop()
void	unregisterOpenSession(org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession session) Called by TezSessionPoolSession when closed.
com.google.common.util.concurrent.ListenableFuture<Boolean>	updateResourcePlanAsync(org.apache.hadoop.hive.metastore.api.WMFullResourcePlan plan)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

wmThread

protected final Thread wmThread

The master thread the processes the events from EventState.

Method Detail

getInstance

public static WorkloadManager getInstance()

create

public static WorkloadManager create(String yarnQueue,
                                     HiveConf conf,
                                     org.apache.hadoop.hive.metastore.api.WMFullResourcePlan plan)
                              throws ExecutionException,
                                     InterruptedException

Called once, when HS2 initializes.

Throws:

ExecutionException

InterruptedException

start

public void start()
           throws Exception

Throws:

Exception

stop

public void stop()
          throws Exception

Throws:

Exception

updateResourcePlanAsync

public com.google.common.util.concurrent.ListenableFuture<Boolean> updateResourcePlanAsync(org.apache.hadoop.hive.metastore.api.WMFullResourcePlan plan)

getSession

public WmTezSession getSession(TezSessionState session,
                               UserPoolMapping.MappingInput input,
                               HiveConf conf)
                        throws Exception

Throws:

Exception

getSession

public WmTezSession getSession(TezSessionState session,
                               UserPoolMapping.MappingInput input,
                               HiveConf conf,
                               WmContext wmContext)
                        throws Exception

Throws:

Exception

destroy

public void destroy(TezSessionState session)
             throws Exception

Specified by:

destroy in interface TezSessionPoolSession.Manager

Throws:

Exception

returnAfterUse

public void returnAfterUse(org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession session)
                    throws Exception

Specified by:

returnAfterUse in interface TezSessionPoolSession.Manager

Throws:

Exception

notifyOfInconsistentAllocation

public void notifyOfInconsistentAllocation(WmTezSession session)

notifyOfClusterStateChange

public void notifyOfClusterStateChange()

getWmStateDescription

public List<String> getWmStateDescription()

Specified by:

getWmStateDescription in interface WorkloadManagerMxBean

Returns:

The text-based description of current WM state.

notifyInitializationCompleted

public void notifyInitializationCompleted(org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.SessionInitContext initCtx)

reopen

public TezSessionState reopen(TezSessionState session)
                       throws Exception

Specified by:

reopen in interface TezSessionPoolSession.Manager

Throws:

Exception

closeAndReopenExpiredSession

public void closeAndReopenExpiredSession(org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession session)
                                  throws Exception

Throws:

Exception

createSessionObject

protected WmTezSession createSessionObject(String sessionId,
                                           HiveConf conf)

registerOpenSession

public void registerOpenSession(org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession session)

Called by TezSessionPoolSession when opened.

Specified by:

registerOpenSession in interface TezSessionPoolSession.Manager

unregisterOpenSession

public void unregisterOpenSession(org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession session)

Called by TezSessionPoolSession when closed.

Specified by:

unregisterOpenSession in interface TezSessionPoolSession.Manager

getExpirationTracker

public org.apache.hadoop.hive.ql.exec.tez.SessionExpirationTracker getExpirationTracker()

getConf

protected final HiveConf getConf()