org.apache.hadoop.hive.ql.parse.spark

Class SparkPartitionPruningSinkOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<SparkPartitionPruningSinkDesc>

org.apache.hadoop.hive.ql.parse.spark.SparkPartitionPruningSinkOperator

All Implemented Interfaces:

Serializable, Cloneable, Node

Direct Known Subclasses:

VectorSparkPartitionPruningSinkOperator

public class SparkPartitionPruningSinkOperator
extends Operator<SparkPartitionPruningSinkDesc>

This operator gets partition info from the upstream operators, and write them to HDFS. This will later be read at the driver, and used for pruning the partitions for the big table side.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.Counter, Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.io.DataOutputBuffer	buffer
protected static org.slf4j.Logger	LOG
protected Serializer	serializer

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

abortOp, alias, asyncInitOperations, bucketingVersion, cContext, childOperators, childOperatorsArray, childOperatorsTag, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, HIVE_COUNTER_CREATED_FILES, HIVE_COUNTER_FATAL, id, indexForTezUnion, inputObjInspectors, numRows, operatorId, out, outputObjInspector, parentOperators, reporter, runTimeNumRows, state, statsMap

Constructor Summary

Constructors

Constructor and Description
SparkPartitionPruningSinkOperator() Kryo ctor.
SparkPartitionPruningSinkOperator(CompilationOpContext ctx)

Method Summary

Modifier and Type	Method and Description
void	addAsSourceEvent(MapWork mapWork, ExprNodeDesc partKey, String columnName, String columnType) Add this DPP sink as a pruning source for the target MapWork.
void	closeOp(boolean abort) Operator specific close routine.
Operator<?>	getBranchingOp()
String	getName() Gets the name of the node.
static String	getOperatorName()
org.apache.hadoop.hive.ql.plan.api.OperatorType	getType() Return the type of the specific operator among the types in OperatorType.
String	getUniqueId()
void	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
boolean	isWithMapjoin()
void	process(Object row, int tag) Process the row.
void	removeFromSourceEvent(MapWork mapWork, ExprNodeDesc partKey, String columnName, String columnType) Remove this DPP sink from the target MapWork's pruning source.
void	setUniqueId(String uniqueId)

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

abort, acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, cleanUpInputFileChangedOp, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, completeInitializationOp, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, endGroup, flush, flushRecursive, forward, forward, forward, getAdditionalCounters, getBucketingVersion, getChildOperators, getChildren, getColumnExprMap, getCompilationOpContext, getConf, getConfiguration, getCounterName, getDone, getExecContext, getGroupKeyObject, getIdentifier, getIndexForTezUnion, getInputObjInspectors, getIsReduceSink, getMarker, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getReduceOutputName, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logicalEquals, logicalEqualsTree, logStats, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeParent, removeParents, replaceChild, replaceParent, reset, setAlias, setBucketingVersion, setChildOperators, setColumnExprMap, setCompilationOpContext, setConf, setDone, setExecContext, setGroupKeyObject, setIndexForTezUnion, setInputContext, setInputObjInspectors, setMarker, setNextVectorBatchGroupStatus, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, startGroup, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

serializer

protected transient Serializer serializer

buffer

protected transient org.apache.hadoop.io.DataOutputBuffer buffer

LOG

protected static final org.slf4j.Logger LOG

Constructor Detail

SparkPartitionPruningSinkOperator

public SparkPartitionPruningSinkOperator()

Kryo ctor.

SparkPartitionPruningSinkOperator

public SparkPartitionPruningSinkOperator(CompilationOpContext ctx)

Method Detail

initializeOp

public void initializeOp(org.apache.hadoop.conf.Configuration hconf)
                  throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class Operator<SparkPartitionPruningSinkDesc>

Throws:

HiveException

process

public void process(Object row,
                    int tag)
             throws HiveException

Description copied from class: Operator

Process the row.

Specified by:

process in class Operator<SparkPartitionPruningSinkDesc>

Parameters:

row - The object representing the row.

tag - The tag of the row usually means which parent this row comes from. Rows with the same tag should have exactly the same rowInspector all the time.

Throws:

HiveException

closeOp

public void closeOp(boolean abort)
             throws HiveException

Description copied from class: Operator

Operator specific close routine. Operators which inherents this class should overwrite this funtion for their specific cleanup routine.

Overrides:

closeOp in class Operator<SparkPartitionPruningSinkDesc>

Throws:

HiveException

isWithMapjoin

public boolean isWithMapjoin()

getBranchingOp

public Operator<?> getBranchingOp()

getType

public org.apache.hadoop.hive.ql.plan.api.OperatorType getType()

Description copied from class: Operator

Return the type of the specific operator among the types in OperatorType.

Specified by:

getType in class Operator<SparkPartitionPruningSinkDesc>

Returns:

OperatorType.*

getName

public String getName()

Description copied from interface: Node

Gets the name of the node. This is used in the rule dispatchers.

Specified by:

getName in interface Node

Specified by:

getName in class Operator<SparkPartitionPruningSinkDesc>

Returns:

String

getOperatorName

public static String getOperatorName()

getUniqueId

public String getUniqueId()

setUniqueId

public void setUniqueId(String uniqueId)

addAsSourceEvent

public void addAsSourceEvent(MapWork mapWork,
                             ExprNodeDesc partKey,
                             String columnName,
                             String columnType)

Add this DPP sink as a pruning source for the target MapWork. It means the DPP sink's output will be used to prune a certain partition in the MapWork. The MapWork's event source maps will be updated to remember the DPP sink's unique ID and corresponding target columns.

removeFromSourceEvent

public void removeFromSourceEvent(MapWork mapWork,
                                  ExprNodeDesc partKey,
                                  String columnName,
                                  String columnType)

Remove this DPP sink from the target MapWork's pruning source. The MapWork's event source maps will be updated to remove the association between the target column and the DPP sink's unique ID. If the DPP sink has no target columns after the removal, its unique ID is removed from the event source maps.