org.apache.hadoop.mapreduce

类 Job

java.lang.Object

org.apache.hadoop.mapreduce.JobContext

org.apache.hadoop.mapreduce.Job

public class Job
extends JobContext

The job submitter's view of the Job. It allows the user to configure the job, submit it, control its execution, and query the state. The set methods only work until the job is submitted, afterwards they will throw an IllegalStateException.

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	Job.JobState

字段概要

从类继承的字段 org.apache.hadoop.mapreduce.JobContext

CACHE_ARCHIVES_VISIBILITIES, CACHE_FILE_VISIBILITIES, COMBINE_CLASS_ATTR, conf, credentials, INPUT_FORMAT_CLASS_ATTR, JOB_ACL_MODIFY_JOB, JOB_ACL_VIEW_JOB, JOB_CANCEL_DELEGATION_TOKEN, JOB_NAMENODES, MAP_CLASS_ATTR, OUTPUT_FORMAT_CLASS_ATTR, PARTITIONER_CLASS_ATTR, REDUCE_CLASS_ATTR, ugi, USER_LOG_RETAIN_HOURS

构造器概要

构造器

构造器和说明
Job()
Job(Configuration conf)
Job(Configuration conf, String jobName)

方法概要

方法

限定符和类型	方法和说明
void	failTask(TaskAttemptID taskId) Fail indicated task attempt.
Counters	getCounters() Gets the counters for this job.
static Job	getInstance() Creates a new Job A Job will be created with a generic Configuration.
static Job	getInstance(Configuration conf) Creates a new Job with a given Configuration.
static Job	getInstance(Configuration conf, String jobName) Creates a new Job with a given Configuration and a given jobName.
String	getJar() Get the pathname of the job's jar.
TaskCompletionEvent[]	getTaskCompletionEvents(int startFrom) Get events indicating completion (success/failure) of component tasks.
String	getTrackingURL() Get the URL where some job progress information will be displayed.
boolean	isComplete() Check if the job is finished or not.
boolean	isSuccessful() Check if the job completed successfully.
void	killJob() Kill the running job.
void	killTask(TaskAttemptID taskId) Kill indicated task attempt.
float	mapProgress() Get the progress of the job's map-tasks, as a float between 0.0 and 1.0.
float	reduceProgress() Get the progress of the job's reduce-tasks, as a float between 0.0 and 1.0.
void	setCancelDelegationTokenUponJobCompletion(boolean value) Sets the flag that will allow the JobTracker to cancel the HDFS delegation tokens upon job completion.
void	setCombinerClass(Class<? extends Reducer> cls) Set the combiner class for the job.
void	setGroupingComparatorClass(Class<? extends RawComparator> cls) Define the comparator that controls which keys are grouped together for a single call to Reducer.reduce(Object, Iterable, org.apache.hadoop.mapreduce.Reducer.Context)
void	setInputFormatClass(Class<? extends InputFormat> cls) Set the InputFormat for the job.
void	setJarByClass(Class<?> cls) Set the Jar by finding where a given class came from.
void	setJobName(String name) Set the user-specified job name.
void	setMapOutputKeyClass(Class<?> theClass) Set the key class for the map output data.
void	setMapOutputValueClass(Class<?> theClass) Set the value class for the map output data.
void	setMapperClass(Class<? extends Mapper> cls) Set the Mapper for the job.
void	setMapSpeculativeExecution(boolean speculativeExecution) Turn speculative execution on or off for this job for map tasks.
void	setNumReduceTasks(int tasks) Set the number of reduce tasks for the job.
void	setOutputFormatClass(Class<? extends OutputFormat> cls) Set the OutputFormat for the job.
void	setOutputKeyClass(Class<?> theClass) Set the key class for the job output data.
void	setOutputValueClass(Class<?> theClass) Set the value class for job outputs.
void	setPartitionerClass(Class<? extends Partitioner> cls) Set the Partitioner for the job.
void	setReducerClass(Class<? extends Reducer> cls) Set the Reducer for the job.
void	setReduceSpeculativeExecution(boolean speculativeExecution) Turn speculative execution on or off for this job for reduce tasks.
void	setSortComparatorClass(Class<? extends RawComparator> cls) Define the comparator that controls how the keys are sorted before they are passed to the Reducer.
void	setSpeculativeExecution(boolean speculativeExecution) Turn speculative execution on or off for this job.
float	setupProgress() Get the progress of the job's setup, as a float between 0.0 and 1.0.
void	setWorkingDirectory(Path dir) Set the current working directory for the default file system.
void	submit() Submit the job to the cluster and return immediately.
boolean	waitForCompletion(boolean verbose) Submit the job to the cluster and wait for it to finish.

从类继承的方法 org.apache.hadoop.mapreduce.JobContext

getCombinerClass, getConfiguration, getCredentials, getGroupingComparator, getInputFormatClass, getJobID, getJobName, getMapOutputKeyClass, getMapOutputValueClass, getMapperClass, getNumReduceTasks, getOutputFormatClass, getOutputKeyClass, getOutputValueClass, getPartitionerClass, getReducerClass, getSortComparator, getWorkingDirectory

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

Job

public Job()
    throws IOException

抛出:

IOException

Job

public Job(Configuration conf)
    throws IOException

抛出:

IOException

Job

public Job(Configuration conf,
   String jobName)
    throws IOException

抛出:

IOException

方法详细资料

getInstance

public static Job getInstance()
                       throws IOException

Creates a new Job A Job will be created with a generic Configuration.

返回:

the Job

抛出:

IOException

getInstance

public static Job getInstance(Configuration conf)
                       throws IOException

Creates a new Job with a given Configuration. The Job makes a copy of the Configuration so that any necessary internal modifications do not reflect on the incoming parameter.

参数:

conf - the Configuration

返回:

the Job

抛出:

IOException

getInstance

public static Job getInstance(Configuration conf,
              String jobName)
                       throws IOException

Creates a new Job with a given Configuration and a given jobName. The Job makes a copy of the Configuration so that any necessary internal modifications do not reflect on the incoming parameter.

参数:

conf - the Configuration

jobName - the job instance's name

返回:

the Job

抛出:

IOException

setNumReduceTasks

public void setNumReduceTasks(int tasks)
                       throws IllegalStateException

Set the number of reduce tasks for the job.

参数:

tasks - the number of reduce tasks

抛出:

IllegalStateException - if the job is submitted

setWorkingDirectory

public void setWorkingDirectory(Path dir)
                         throws IOException

Set the current working directory for the default file system.

参数:

dir - the new current working directory.

抛出:

IllegalStateException - if the job is submitted

IOException

setInputFormatClass

public void setInputFormatClass(Class<? extends InputFormat> cls)
                         throws IllegalStateException

Set the InputFormat for the job.

参数:

cls - the InputFormat to use

抛出:

IllegalStateException - if the job is submitted

setOutputFormatClass

public void setOutputFormatClass(Class<? extends OutputFormat> cls)
                          throws IllegalStateException

Set the OutputFormat for the job.

参数:

cls - the OutputFormat to use

抛出:

IllegalStateException - if the job is submitted

setMapperClass

public void setMapperClass(Class<? extends Mapper> cls)
                    throws IllegalStateException

Set the Mapper for the job.

参数:

cls - the Mapper to use

抛出:

IllegalStateException - if the job is submitted

setJarByClass

public void setJarByClass(Class<?> cls)

Set the Jar by finding where a given class came from.

参数:

cls - the example class

getJar

public String getJar()

Get the pathname of the job's jar.

覆盖:

getJar 在类中 JobContext

返回:

the pathname

setCombinerClass

public void setCombinerClass(Class<? extends Reducer> cls)
                      throws IllegalStateException

Set the combiner class for the job.

参数:

cls - the combiner to use

抛出:

IllegalStateException - if the job is submitted

setReducerClass

public void setReducerClass(Class<? extends Reducer> cls)
                     throws IllegalStateException

Set the Reducer for the job.

参数:

cls - the Reducer to use

抛出:

IllegalStateException - if the job is submitted

setPartitionerClass

public void setPartitionerClass(Class<? extends Partitioner> cls)
                         throws IllegalStateException

Set the Partitioner for the job.

参数:

cls - the Partitioner to use

抛出:

IllegalStateException - if the job is submitted

setMapOutputKeyClass

public void setMapOutputKeyClass(Class<?> theClass)
                          throws IllegalStateException

Set the key class for the map output data. This allows the user to specify the map output key class to be different than the final output value class.

参数:

theClass - the map output key class.

抛出:

IllegalStateException - if the job is submitted

setMapOutputValueClass

public void setMapOutputValueClass(Class<?> theClass)
                            throws IllegalStateException

Set the value class for the map output data. This allows the user to specify the map output value class to be different than the final output value class.

参数:

theClass - the map output value class.

抛出:

IllegalStateException - if the job is submitted

setOutputKeyClass

public void setOutputKeyClass(Class<?> theClass)
                       throws IllegalStateException

Set the key class for the job output data.

参数:

theClass - the key class for the job output data.

抛出:

IllegalStateException - if the job is submitted

setOutputValueClass

public void setOutputValueClass(Class<?> theClass)
                         throws IllegalStateException

Set the value class for job outputs.

参数:

theClass - the value class for job outputs.

抛出:

IllegalStateException - if the job is submitted

setSortComparatorClass

public void setSortComparatorClass(Class<? extends RawComparator> cls)
                            throws IllegalStateException

Define the comparator that controls how the keys are sorted before they are passed to the Reducer.

参数:

cls - the raw comparator

抛出:

IllegalStateException - if the job is submitted

setGroupingComparatorClass

public void setGroupingComparatorClass(Class<? extends RawComparator> cls)
                                throws IllegalStateException

Define the comparator that controls which keys are grouped together for a single call to Reducer.reduce(Object, Iterable, org.apache.hadoop.mapreduce.Reducer.Context)

参数:

cls - the raw comparator to use

抛出:

IllegalStateException - if the job is submitted

setJobName

public void setJobName(String name)
                throws IllegalStateException

Set the user-specified job name.

参数:

name - the job's new name.

抛出:

IllegalStateException - if the job is submitted

setSpeculativeExecution

public void setSpeculativeExecution(boolean speculativeExecution)

Turn speculative execution on or off for this job.

参数:

speculativeExecution - true if speculative execution should be turned on, else false.

setMapSpeculativeExecution

public void setMapSpeculativeExecution(boolean speculativeExecution)

Turn speculative execution on or off for this job for map tasks.

参数:

speculativeExecution - true if speculative execution should be turned on for map tasks, else false.

setReduceSpeculativeExecution

public void setReduceSpeculativeExecution(boolean speculativeExecution)

Turn speculative execution on or off for this job for reduce tasks.

参数:

speculativeExecution - true if speculative execution should be turned on for reduce tasks, else false.

getTrackingURL

public String getTrackingURL()

Get the URL where some job progress information will be displayed.

返回:

the URL where some job progress information will be displayed.

setupProgress

public float setupProgress()
                    throws IOException

Get the progress of the job's setup, as a float between 0.0 and 1.0. When the job setup is completed, the function returns 1.0.

返回:

the progress of the job's setup.

抛出:

IOException

mapProgress

public float mapProgress()
                  throws IOException

Get the progress of the job's map-tasks, as a float between 0.0 and 1.0. When all map tasks have completed, the function returns 1.0.

返回:

the progress of the job's map-tasks.

抛出:

IOException

reduceProgress

public float reduceProgress()
                     throws IOException

Get the progress of the job's reduce-tasks, as a float between 0.0 and 1.0. When all reduce tasks have completed, the function returns 1.0.

返回:

the progress of the job's reduce-tasks.

抛出:

IOException

isComplete

public boolean isComplete()
                   throws IOException

Check if the job is finished or not. This is a non-blocking call.

返回:

true if the job is complete, else false.

抛出:

IOException

isSuccessful

public boolean isSuccessful()
                     throws IOException

Check if the job completed successfully.

返回:

true if the job succeeded, else false.

抛出:

IOException

killJob

public void killJob()
             throws IOException

Kill the running job. Blocks until all job tasks have been killed as well. If the job is no longer running, it simply returns.

抛出:

IOException

getTaskCompletionEvents

public TaskCompletionEvent[] getTaskCompletionEvents(int startFrom)
                                              throws IOException

Get events indicating completion (success/failure) of component tasks.

参数:

startFrom - index to start fetching events from

返回:

an array of TaskCompletionEvents

抛出:

IOException

killTask

public void killTask(TaskAttemptID taskId)
              throws IOException

Kill indicated task attempt.

参数:

taskId - the id of the task to be terminated.

抛出:

IOException

failTask

public void failTask(TaskAttemptID taskId)
              throws IOException

Fail indicated task attempt.

参数:

taskId - the id of the task to be terminated.

抛出:

IOException

getCounters

public Counters getCounters()
                     throws IOException

Gets the counters for this job.

返回:

the counters for this job.

抛出:

IOException

setCancelDelegationTokenUponJobCompletion

public void setCancelDelegationTokenUponJobCompletion(boolean value)

Sets the flag that will allow the JobTracker to cancel the HDFS delegation tokens upon job completion. Defaults to true.

submit

public void submit()
            throws IOException,
                   InterruptedException,
                   ClassNotFoundException

Submit the job to the cluster and return immediately.

抛出:

IOException

InterruptedException

ClassNotFoundException

waitForCompletion

public boolean waitForCompletion(boolean verbose)
                          throws IOException,
                                 InterruptedException,
                                 ClassNotFoundException

Submit the job to the cluster and wait for it to finish.

参数:

verbose - print the progress to the user

返回:

true if the job succeeded

抛出:

IOException - thrown if the communication with the JobTracker is lost

InterruptedException

ClassNotFoundException