org.apache.hadoop.hive.ql.exec.tez

Class DagUtils

java.lang.Object

org.apache.hadoop.hive.ql.exec.tez.DagUtils

public class DagUtils
extends Object

DagUtils. DagUtils is a collection of helper methods to convert map and reduce work to tez vertices and edges. It handles configuration objects, file localization and vertex/edge creation.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DagUtils.ValueHashPartitioner

Field Summary

Fields

Modifier and Type	Field and Description
static String	TEZ_MERGE_CURRENT_MERGE_FILE_PREFIX
static String	TEZ_MERGE_WORK_FILE_PREFIXES
static String	TEZ_TMP_DIR_KEY

Method Summary

Modifier and Type	Method and Description
void	addCredentials(BaseWork work, org.apache.tez.dag.api.DAG dag) Set up credentials for the base work on secure clusters
boolean	checkOrWaitForTheFile(org.apache.hadoop.fs.FileSystem srcFs, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dest, org.apache.hadoop.conf.Configuration conf, Object notifier, int waitAttempts, long sleepInterval, boolean doLog)
org.apache.hadoop.mapred.JobConf	createConfiguration(HiveConf hiveConf) Creates and initializes a JobConf object that can be used to execute the DAG.
String	createDagName(org.apache.hadoop.conf.Configuration conf, QueryPlan plan)
org.apache.tez.dag.api.Edge	createEdge(org.apache.hadoop.mapred.JobConf vConf, org.apache.tez.dag.api.Vertex v, org.apache.tez.dag.api.Vertex w, TezEdgeProperty edgeProp, BaseWork work, TezWork tezWork) Given two vertices and the configuration for the source vertex, createEdge will create an Edge object that connects the two.
org.apache.tez.dag.api.GroupInputEdge	createEdge(org.apache.tez.dag.api.VertexGroup group, org.apache.hadoop.mapred.JobConf vConf, org.apache.tez.dag.api.Vertex w, TezEdgeProperty edgeProp, BaseWork work, TezWork tezWork) Given a Vertex group and a vertex createEdge will create an Edge between them.
org.apache.tez.dag.api.PreWarmVertex	createPreWarmVertex(org.apache.tez.dag.api.TezConfiguration conf, int numContainers, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
org.apache.hadoop.fs.Path	createTezDir(org.apache.hadoop.fs.Path scratchDir, org.apache.hadoop.conf.Configuration conf) createTezDir creates a temporary directory in the scratchDir folder to be used with Tez.
static Map<String,org.apache.hadoop.yarn.api.records.LocalResource>	createTezLrMap(org.apache.hadoop.yarn.api.records.LocalResource appJarLr, Collection<org.apache.hadoop.yarn.api.records.LocalResource> additionalLr)
org.apache.tez.dag.api.Vertex	createVertex(org.apache.hadoop.mapred.JobConf conf, BaseWork work, org.apache.hadoop.fs.Path scratchDir, org.apache.hadoop.fs.FileSystem fileSystem, Context ctx, boolean hasChildren, TezWork tezWork, TezWork.VertexType vertexType, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources) Create a vertex from a given work object.
static String	getBaseName(org.apache.hadoop.yarn.api.records.LocalResource lr)
static org.apache.hadoop.yarn.api.records.Resource	getContainerResource(org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.fs.Path	getDefaultDestDir(org.apache.hadoop.conf.Configuration conf)
String	getExecJarPathLocal(org.apache.hadoop.conf.Configuration configuration)
org.apache.hadoop.fs.FileStatus	getHiveJarDirectory(org.apache.hadoop.conf.Configuration conf)
static DagUtils	getInstance() Singleton
String	getResourceBaseName(org.apache.hadoop.fs.Path path)
static Map<String,org.apache.hadoop.yarn.api.records.LocalResource>	getResourcesUpdatableForAm(Collection<org.apache.hadoop.yarn.api.records.LocalResource> allNonAppResources)
static String[]	getTempFilesFromConf(org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.fs.Path	getTezDir(org.apache.hadoop.fs.Path scratchDir) Gets the tez dir that belongs to the hive scratch dir
static String	getUserSpecifiedDagName(org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.mapred.JobConf	initializeVertexConf(org.apache.hadoop.mapred.JobConf conf, Context context, BaseWork work) Creates and initializes the JobConf object for a given BaseWork object.
org.apache.hadoop.yarn.api.records.LocalResource	localizeResource(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dest, org.apache.hadoop.yarn.api.records.LocalResourceType type, org.apache.hadoop.conf.Configuration conf) Localizes a resources.
List<org.apache.hadoop.yarn.api.records.LocalResource>	localizeTempFiles(String hdfsDirPathStr, org.apache.hadoop.conf.Configuration conf, String[] inputOutputJars, String[] skipJars) Localizes files, archives and jars from a provided array of names.
List<org.apache.hadoop.yarn.api.records.LocalResource>	localizeTempFilesFromConf(String hdfsDirPathStr, org.apache.hadoop.conf.Configuration conf) Localizes files, archives and jars the user has instructed us to provide on the cluster as resources for execution.
static org.apache.hadoop.fs.FileStatus	validateTargetDir(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_TMP_DIR_KEY

public static final String TEZ_TMP_DIR_KEY

See Also:

Constant Field Values

TEZ_MERGE_CURRENT_MERGE_FILE_PREFIX

public static final String TEZ_MERGE_CURRENT_MERGE_FILE_PREFIX

See Also:

Constant Field Values

TEZ_MERGE_WORK_FILE_PREFIXES

public static final String TEZ_MERGE_WORK_FILE_PREFIXES

See Also:

Constant Field Values

Method Detail

createEdge

public org.apache.tez.dag.api.GroupInputEdge createEdge(org.apache.tez.dag.api.VertexGroup group,
                                                        org.apache.hadoop.mapred.JobConf vConf,
                                                        org.apache.tez.dag.api.Vertex w,
                                                        TezEdgeProperty edgeProp,
                                                        BaseWork work,
                                                        TezWork tezWork)
                                                 throws IOException

Given a Vertex group and a vertex createEdge will create an Edge between them.

Parameters:

group - The parent VertexGroup

vConf - The job conf of one of the parrent (grouped) vertices

w - The child vertex

edgeProp - the edge property of connection between the two endpoints.

Throws:

IOException

createEdge

public org.apache.tez.dag.api.Edge createEdge(org.apache.hadoop.mapred.JobConf vConf,
                                              org.apache.tez.dag.api.Vertex v,
                                              org.apache.tez.dag.api.Vertex w,
                                              TezEdgeProperty edgeProp,
                                              BaseWork work,
                                              TezWork tezWork)
                                       throws IOException

Given two vertices and the configuration for the source vertex, createEdge will create an Edge object that connects the two.

Parameters:

vConf - JobConf of the first (source) vertex

v - The first vertex (source)

w - The second vertex (sink)

Returns:

Throws:

IOException

getContainerResource

public static org.apache.hadoop.yarn.api.records.Resource getContainerResource(org.apache.hadoop.conf.Configuration conf)

createTezLrMap

public static Map<String,org.apache.hadoop.yarn.api.records.LocalResource> createTezLrMap(org.apache.hadoop.yarn.api.records.LocalResource appJarLr,
                                                                                          Collection<org.apache.hadoop.yarn.api.records.LocalResource> additionalLr)

createPreWarmVertex

public org.apache.tez.dag.api.PreWarmVertex createPreWarmVertex(org.apache.tez.dag.api.TezConfiguration conf,
                                                                int numContainers,
                                                                Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
                                                         throws IOException,
                                                                org.apache.tez.dag.api.TezException

Parameters:

numContainers - number of containers to pre-warm

localResources - additional resources to pre-warm with

Returns:

prewarm vertex to run

Throws:

IOException

org.apache.tez.dag.api.TezException

getDefaultDestDir

public org.apache.hadoop.fs.Path getDefaultDestDir(org.apache.hadoop.conf.Configuration conf)
                                            throws LoginException,
                                                   IOException

Parameters:

conf -

Returns:

path to destination directory on hdfs

Throws:

LoginException - if we are unable to figure user information

IOException - when any dfs operation fails.

localizeTempFilesFromConf

public List<org.apache.hadoop.yarn.api.records.LocalResource> localizeTempFilesFromConf(String hdfsDirPathStr,
                                                                                        org.apache.hadoop.conf.Configuration conf)
                                                                                 throws IOException,
                                                                                        LoginException

Localizes files, archives and jars the user has instructed us to provide on the cluster as resources for execution.

Parameters:

conf -

Returns:

List local resources to add to execution

Throws:

IOException - when hdfs operation fails

LoginException - when getDefaultDestDir fails with the same exception

getTempFilesFromConf

public static String[] getTempFilesFromConf(org.apache.hadoop.conf.Configuration conf)

localizeTempFiles

public List<org.apache.hadoop.yarn.api.records.LocalResource> localizeTempFiles(String hdfsDirPathStr,
                                                                                org.apache.hadoop.conf.Configuration conf,
                                                                                String[] inputOutputJars,
                                                                                String[] skipJars)
                                                                         throws IOException,
                                                                                LoginException

Localizes files, archives and jars from a provided array of names.

Parameters:

hdfsDirPathStr - Destination directory in HDFS.

conf - Configuration.

inputOutputJars - The file names to localize.

Returns:

List local resources to add to execution

Throws:

IOException - when hdfs operation fails.

LoginException - when getDefaultDestDir fails with the same exception

getHiveJarDirectory

public org.apache.hadoop.fs.FileStatus getHiveJarDirectory(org.apache.hadoop.conf.Configuration conf)
                                                    throws IOException,
                                                           LoginException

Throws:

IOException

LoginException

validateTargetDir

public static org.apache.hadoop.fs.FileStatus validateTargetDir(org.apache.hadoop.fs.Path path,
                                                                org.apache.hadoop.conf.Configuration conf)
                                                         throws IOException

Throws:

IOException

getExecJarPathLocal

public String getExecJarPathLocal(org.apache.hadoop.conf.Configuration configuration)
                           throws URISyntaxException

Throws:

URISyntaxException

getBaseName

public static String getBaseName(org.apache.hadoop.yarn.api.records.LocalResource lr)

getResourceBaseName

public String getResourceBaseName(org.apache.hadoop.fs.Path path)

Parameters:

path - - the string from which we try to determine the resource base name

Returns:

the name of the resource from a given path string.

localizeResource

public org.apache.hadoop.yarn.api.records.LocalResource localizeResource(org.apache.hadoop.fs.Path src,
                                                                         org.apache.hadoop.fs.Path dest,
                                                                         org.apache.hadoop.yarn.api.records.LocalResourceType type,
                                                                         org.apache.hadoop.conf.Configuration conf)
                                                                  throws IOException

Localizes a resources. Should be thread-safe.

Parameters:

src - path to the source for the resource

dest - path in hdfs for the resource

type - local resource type (File/Archive)

conf -

Returns:

localresource from tez localization.

Throws:

IOException - when any file system related calls fails.

checkOrWaitForTheFile

public boolean checkOrWaitForTheFile(org.apache.hadoop.fs.FileSystem srcFs,
                                     org.apache.hadoop.fs.Path src,
                                     org.apache.hadoop.fs.Path dest,
                                     org.apache.hadoop.conf.Configuration conf,
                                     Object notifier,
                                     int waitAttempts,
                                     long sleepInterval,
                                     boolean doLog)
                              throws IOException

Throws:

IOException

createConfiguration

public org.apache.hadoop.mapred.JobConf createConfiguration(HiveConf hiveConf)
                                                     throws IOException

Creates and initializes a JobConf object that can be used to execute the DAG. The configuration object will contain configurations from mapred-site overlaid with key/value pairs from the conf object. Finally it will also contain some hive specific configurations that do not change from DAG to DAG.

Parameters:

hiveConf - Current conf for the execution

Returns:

JobConf base configuration for job execution

Throws:

IOException

initializeVertexConf

public org.apache.hadoop.mapred.JobConf initializeVertexConf(org.apache.hadoop.mapred.JobConf conf,
                                                             Context context,
                                                             BaseWork work)

Creates and initializes the JobConf object for a given BaseWork object.

Parameters:

conf - Any configurations in conf will be copied to the resulting new JobConf object.

work - BaseWork will be used to populate the configuration object.

Returns:

JobConf new configuration object

createVertex

public org.apache.tez.dag.api.Vertex createVertex(org.apache.hadoop.mapred.JobConf conf,
                                                  BaseWork work,
                                                  org.apache.hadoop.fs.Path scratchDir,
                                                  org.apache.hadoop.fs.FileSystem fileSystem,
                                                  Context ctx,
                                                  boolean hasChildren,
                                                  TezWork tezWork,
                                                  TezWork.VertexType vertexType,
                                                  Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
                                           throws Exception

Create a vertex from a given work object.

Parameters:

conf - JobConf to be used to this execution unit

work - The instance of BaseWork representing the actual work to be performed by this vertex.

scratchDir - HDFS scratch dir for this execution unit.

fileSystem - FS corresponding to scratchDir and LocalResources

ctx - This query's context

Returns:

Vertex

Throws:

Exception

addCredentials

public void addCredentials(BaseWork work,
                           org.apache.tez.dag.api.DAG dag)
                    throws IOException

Set up credentials for the base work on secure clusters

Throws:

IOException

createTezDir

public org.apache.hadoop.fs.Path createTezDir(org.apache.hadoop.fs.Path scratchDir,
                                              org.apache.hadoop.conf.Configuration conf)
                                       throws IOException

createTezDir creates a temporary directory in the scratchDir folder to be used with Tez. Assumes scratchDir exists.

Throws:

IOException

getTezDir

public org.apache.hadoop.fs.Path getTezDir(org.apache.hadoop.fs.Path scratchDir)

Gets the tez dir that belongs to the hive scratch dir

getInstance

public static DagUtils getInstance()

Singleton

Returns:

instance of this class

createDagName

public String createDagName(org.apache.hadoop.conf.Configuration conf,
                            QueryPlan plan)

getUserSpecifiedDagName

public static String getUserSpecifiedDagName(org.apache.hadoop.conf.Configuration conf)

getResourcesUpdatableForAm

public static Map<String,org.apache.hadoop.yarn.api.records.LocalResource> getResourcesUpdatableForAm(Collection<org.apache.hadoop.yarn.api.records.LocalResource> allNonAppResources)