org.apache.hive.hcatalog.templeton.tool

Class TempletonControllerJob

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool, JobSubmissionConstants

@InterfaceAudience.Private
public class TempletonControllerJob
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool, JobSubmissionConstants

A Map Reduce job that will start another job. We have a single Mapper job that starts a child MR job. The parent monitors the child child job and ends when the child job exits. In addition, we - write out the parent job id so the caller can record it. - run a keep alive thread so the job doesn't end. - Optionally, store the stdout, stderr, and exit value of the child in hdfs files. A note on security. When jobs are submitted through WebHCat that use HCatalog, it means that metastore access is required. Hive queries, of course, need metastore access. This in turn requires delegation token to be obtained for metastore in a secure cluster. Since we can't usually parse the job to find out if it is using metastore, we require 'usehcatalog' parameter supplied in the REST call. WebHcat takes care of cancelling the token when the job is complete.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hive.hcatalog.templeton.tool.JobSubmissionConstants

JobSubmissionConstants.PigConstants, JobSubmissionConstants.Sqoop

Field Summary

Fields inherited from interface org.apache.hive.hcatalog.templeton.tool.JobSubmissionConstants

COPY_NAME, ENABLE_JOB_RECONNECT, ENABLE_LOG, EXIT_FNAME, HADOOP_CLASSPATH, HADOOP_CLASSPATH_EXTRAS, JAR_ARGS_NAME, JOB_TYPE, KEEP_ALIVE_MSEC, MAPREDUCE_JOB_TAGS, MAPREDUCE_JOB_TAGS_ARG_PLACEHOLDER, OVERRIDE_CLASSPATH, POLL_JOBPROGRESS_MSEC, STATUSDIR_NAME, STDERR_FNAME, STDOUT_FNAME, TEMPLETON_JOB_LAUNCH_TIME_NAME, TOKEN_FILE_ARG_PLACEHOLDER, TOKEN_FILE_ARG_PLACEHOLDER_TEZ, WATCHER_TIMEOUT_SECS

Constructor Summary

Constructors

Constructor and Description
TempletonControllerJob(boolean secureMetastoreAccess, AppConfig conf)

Method Summary

Modifier and Type	Method and Description
String	getSubmittedId()
int	run(String[] args) Enqueue the job and print out the job id for later collection.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Constructor Detail

TempletonControllerJob

public TempletonControllerJob(boolean secureMetastoreAccess,
                              AppConfig conf)

Parameters:

secureMetastoreAccess - - if true, a delegation token will be created and added to the job

Method Detail

getSubmittedId

public String getSubmittedId()

run

public int run(String[] args)
        throws IOException,
               InterruptedException,
               ClassNotFoundException,
               org.apache.thrift.TException

Enqueue the job and print out the job id for later collection.

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

IOException

InterruptedException

ClassNotFoundException

org.apache.thrift.TException

See Also:

CompleteDelegator