org.apache.hadoop.tools.distcp2.mapred

类 CopyCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

org.apache.hadoop.tools.distcp2.mapred.CopyCommitter

public class CopyCommitter
extends FileOutputCommitter

The CopyCommitter class is DistCp's OutputCommitter implementation. It is responsible for handling the completion/cleanup of the DistCp run. Specifically, it does the following: 1. Cleanup of the meta-folder (where DistCp maintains its file-list, etc.) 2. Preservation of user/group/replication-factor on any directories that have been copied. (Files are taken care of in their map-tasks.) 3. Atomic-move of data from the temporary work-folder to the final path (if atomic-commit was opted for). 4. Deletion of files from the target that are missing at source (if opted for). 5. Cleanup of any partially copied files, from previous, failed attempts.

字段概要

从类继承的字段 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

SUCCEEDED_FILE_NAME, TEMP_DIR_NAME

构造器概要

构造器

构造器和说明
CopyCommitter(Path outputPath, TaskAttemptContext context) Create a output committer

方法概要

方法

限定符和类型	方法和说明
void	abortJob(JobContext jobContext, JobStatus.State state) Delete the temporary directory, including all of the work directories.
void	commitJob(JobContext jobContext) Delete the temporary directory, including all of the work directories.

从类继承的方法 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

abortTask, cleanupJob, commitTask, getWorkPath, needsTaskCommit, setupJob, setupTask

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

CopyCommitter

public CopyCommitter(Path outputPath,
             TaskAttemptContext context)
              throws IOException

Create a output committer

参数:

outputPath - the job's output path

context - the task's context

抛出:

IOException - - Exception if any

方法详细资料

commitJob

public void commitJob(JobContext jobContext)
               throws IOException

从类复制的说明: FileOutputCommitter

Delete the temporary directory, including all of the work directories. This is called for all jobs whose final run state is SUCCEEDED

覆盖:

commitJob 在类中 FileOutputCommitter

参数:

jobContext - the job's context.

抛出:

IOException

abortJob

public void abortJob(JobContext jobContext,
            JobStatus.State state)
              throws IOException

从类复制的说明: FileOutputCommitter

Delete the temporary directory, including all of the work directories.

覆盖:

abortJob 在类中 FileOutputCommitter

参数:

jobContext - the job's context

state - final run state of the job, should be FAILED or KILLED

抛出:

IOException