org.apache.hadoop.hive.ql.exec

Class FileSinkOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.TerminalOperator<FileSinkDesc>

org.apache.hadoop.hive.ql.exec.FileSinkOperator

All Implemented Interfaces:

Serializable, Cloneable, Node

Direct Known Subclasses:

VectorFileSinkOperator

public class FileSinkOperator
extends TerminalOperator<FileSinkDesc>
implements Serializable

File Sink operator implementation.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileSinkOperator.Counter Counters.
class	FileSinkOperator.FSPaths
static interface	FileSinkOperator.RecordWriter RecordWriter.

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	autoDelete
protected boolean	bDynParts
protected Map<Integer,Integer>	bucketMap
protected long	cntr
protected List<String>	dpColNames
protected DynamicPartitionCtx	dpCtx
protected int	dpStartCol
protected List<String>	dpVals
protected List<Object>	dpWritables
protected boolean	filesCreated
protected org.apache.hadoop.fs.FileSystem	fs
protected FileSinkOperator.FSPaths	fsp
protected HiveOutputFormat<?,?>	hiveOutputFormat
protected boolean	isCollectRWStats
protected boolean	isCompressed
protected boolean	isSkewedStoredAsSubDirectories
protected boolean	isTemporary
protected boolean	isUnionDp
protected org.apache.hadoop.mapred.JobConf	jc
protected HiveKey	key
protected ListBucketingCtx	lbCtx
static org.slf4j.Logger	LOG
protected long	logEveryNRows
protected int	maxPartitions
protected boolean	multiFileSpray
protected int	numDynParts
protected long	numRows
protected org.apache.hadoop.fs.Path	parent
protected HivePartitioner<HiveKey,Object>	prtner
protected org.apache.hadoop.io.Writable	recordValue
protected org.apache.hadoop.io.LongWritable	row_count
protected int	rowIndex
protected FileSinkOperator.RecordWriter[]	rowOutWriters
protected Serializer	serializer
protected org.apache.hadoop.fs.Path	specPath
protected boolean[]	statsFromRecordWriter
protected int	totalFiles
protected String	unionPath
protected HashMap<String,FileSinkOperator.FSPaths>	valToPaths

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

abortOp, alias, asyncInitOperations, bucketingVersion, cContext, childOperators, childOperatorsArray, childOperatorsTag, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, HIVE_COUNTER_CREATED_FILES, HIVE_COUNTER_FATAL, id, indexForTezUnion, inputObjInspectors, operatorId, out, outputObjInspector, parentOperators, reporter, runTimeNumRows, state, statsMap

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	FileSinkOperator() Kryo ctor.
	FileSinkOperator(CompilationOpContext ctx)

Method Summary

Modifier and Type	Method and Description
protected boolean	areAllTrue(boolean[] statsFromRW)
void	augmentPlan() Called during semantic analysis as operators are being added in order to give them a chance to compute any additional plan information needed.
void	checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored, org.apache.hadoop.mapred.JobConf job)
void	closeOp(boolean abort) Operator specific close routine.
protected void	createBucketFiles(FileSinkOperator.FSPaths fsp)
protected void	createBucketForFileIdx(FileSinkOperator.FSPaths fsp, int filesIdx)
protected String	generateListBucketingDirName(Object row) Generate list bucketing directory name from a row.
String	getCounterName(FileSinkOperator.Counter counter)
protected FileSinkOperator.FSPaths	getDynOutPaths(List<String> row, String lbDir)
String	getName() Gets the name of the node.
static String	getOperatorName()
org.apache.hadoop.hive.ql.plan.api.OperatorType	getType() Return the type of the specific operator among the types in OperatorType.
protected void	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
void	jobCloseOp(org.apache.hadoop.conf.Configuration hconf, boolean success)
void	process(Object row, int tag) Process the row.
protected boolean	updateProgress() Report status to JT so that JT won't kill this task if closing takes too long due to too many files to close and the NN is overloaded.

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

abort, acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, cleanUpInputFileChanged, cleanUpInputFileChangedOp, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, completeInitializationOp, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, endGroup, flush, flushRecursive, forward, forward, forward, getAdditionalCounters, getBucketingVersion, getChildOperators, getChildren, getColumnExprMap, getCompilationOpContext, getConf, getConfiguration, getCounterName, getDone, getExecContext, getGroupKeyObject, getIdentifier, getIndexForTezUnion, getInputObjInspectors, getIsReduceSink, getMarker, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getReduceOutputName, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, logicalEquals, logicalEqualsTree, logStats, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeParent, removeParents, replaceChild, replaceParent, reset, setAlias, setBucketingVersion, setChildOperators, setColumnExprMap, setCompilationOpContext, setConf, setDone, setExecContext, setGroupKeyObject, setIndexForTezUnion, setInputContext, setInputObjInspectors, setMarker, setNextVectorBatchGroupStatus, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, startGroup, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

valToPaths

protected transient HashMap<String,FileSinkOperator.FSPaths> valToPaths

numDynParts

protected transient int numDynParts

dpColNames

protected transient List<String> dpColNames

dpCtx

protected transient DynamicPartitionCtx dpCtx

isCompressed

protected transient boolean isCompressed

isTemporary

protected transient boolean isTemporary

parent

protected transient org.apache.hadoop.fs.Path parent

hiveOutputFormat

protected transient HiveOutputFormat<?,?> hiveOutputFormat

specPath

protected transient org.apache.hadoop.fs.Path specPath

unionPath

protected transient String unionPath

isUnionDp

protected transient boolean isUnionDp

dpStartCol

protected transient int dpStartCol

dpVals

protected transient List<String> dpVals

dpWritables

protected transient List<Object> dpWritables

rowOutWriters

protected transient FileSinkOperator.RecordWriter[] rowOutWriters

maxPartitions

protected transient int maxPartitions

lbCtx

protected transient ListBucketingCtx lbCtx

isSkewedStoredAsSubDirectories

protected transient boolean isSkewedStoredAsSubDirectories

statsFromRecordWriter

protected transient boolean[] statsFromRecordWriter

isCollectRWStats

protected transient boolean isCollectRWStats

numRows

protected transient long numRows

cntr

protected transient long cntr

logEveryNRows

protected transient long logEveryNRows

rowIndex

protected transient int rowIndex

fs

protected transient org.apache.hadoop.fs.FileSystem fs

serializer

protected transient Serializer serializer

row_count

protected final transient org.apache.hadoop.io.LongWritable row_count

totalFiles

protected transient int totalFiles

multiFileSpray

protected transient boolean multiFileSpray

bucketMap

protected final transient Map<Integer,Integer> bucketMap

prtner

protected transient HivePartitioner<HiveKey,Object> prtner

key

protected final transient HiveKey key

fsp

protected transient FileSinkOperator.FSPaths fsp

bDynParts

protected transient boolean bDynParts

autoDelete

protected transient boolean autoDelete

jc

protected transient org.apache.hadoop.mapred.JobConf jc

filesCreated

protected boolean filesCreated

recordValue

protected org.apache.hadoop.io.Writable recordValue

Constructor Detail

FileSinkOperator

protected FileSinkOperator()

Kryo ctor.

FileSinkOperator

public FileSinkOperator(CompilationOpContext ctx)

Method Detail

initializeOp

protected void initializeOp(org.apache.hadoop.conf.Configuration hconf)
                     throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class Operator<FileSinkDesc>

Throws:

HiveException

getCounterName

public String getCounterName(FileSinkOperator.Counter counter)

createBucketFiles

protected void createBucketFiles(FileSinkOperator.FSPaths fsp)
                          throws HiveException

Throws:

HiveException

createBucketForFileIdx

protected void createBucketForFileIdx(FileSinkOperator.FSPaths fsp,
                                      int filesIdx)
                               throws HiveException

Throws:

HiveException

updateProgress

protected boolean updateProgress()

Report status to JT so that JT won't kill this task if closing takes too long due to too many files to close and the NN is overloaded.

Returns:

true if a new progress update is reported, false otherwise.

process

public void process(Object row,
                    int tag)
             throws HiveException

Description copied from class: Operator

Process the row.

Specified by:

process in class Operator<FileSinkDesc>

Parameters:

row - The object representing the row.

tag - The tag of the row usually means which parent this row comes from. Rows with the same tag should have exactly the same rowInspector all the time.

Throws:

HiveException

areAllTrue

protected boolean areAllTrue(boolean[] statsFromRW)

generateListBucketingDirName

protected String generateListBucketingDirName(Object row)

Generate list bucketing directory name from a row.

Parameters:

row - row to process.

Returns:

directory name.

getDynOutPaths

protected FileSinkOperator.FSPaths getDynOutPaths(List<String> row,
                                                  String lbDir)
                                           throws HiveException

Throws:

HiveException

closeOp

public void closeOp(boolean abort)
             throws HiveException

Description copied from class: Operator

Operator specific close routine. Operators which inherents this class should overwrite this funtion for their specific cleanup routine.

Overrides:

closeOp in class Operator<FileSinkDesc>

Throws:

HiveException

getName

public String getName()

Description copied from interface: Node

Gets the name of the node. This is used in the rule dispatchers.

Specified by:

getName in interface Node

Overrides:

getName in class TerminalOperator<FileSinkDesc>

Returns:

the name of the operator

getOperatorName

public static String getOperatorName()

jobCloseOp

public void jobCloseOp(org.apache.hadoop.conf.Configuration hconf,
                       boolean success)
                throws HiveException

Overrides:

jobCloseOp in class Operator<FileSinkDesc>

Throws:

HiveException

getType

public org.apache.hadoop.hive.ql.plan.api.OperatorType getType()

Description copied from class: Operator

Return the type of the specific operator among the types in OperatorType.

Specified by:

getType in class Operator<FileSinkDesc>

Returns:

OperatorType.*

augmentPlan

public void augmentPlan()

Description copied from class: Operator

Called during semantic analysis as operators are being added in order to give them a chance to compute any additional plan information needed. Does nothing by default.

Overrides:

augmentPlan in class Operator<FileSinkDesc>

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
                             org.apache.hadoop.mapred.JobConf job)
                      throws IOException

Throws:

IOException