org.apache.hadoop.hive.ql.exec

Class CommonJoinOperator<T extends JoinDesc>

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

All Implemented Interfaces:

Serializable, Cloneable, Node

Direct Known Subclasses:

AbstractMapJoinOperator, JoinOperator

public abstract class CommonJoinOperator<T extends JoinDesc>
extends Operator<T>
implements Serializable

Join operator implementation.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.Counter, Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected Byte	alias
protected short[]	aliasFilterTags On filterTags ANDed value of all filter tags in current join group if any of values passes on outer join alias (which makes zero for the tag alias), it means there exists a pair for it and safely regarded as a inner join for example, with table a, b something like, a = 100, 10 | 100, 20 | 100, 30 b = 100, 10 | 100, 20 | 100, 30 the query "a FO b ON a.k=b.k AND a.v>10 AND b.v>30" makes filter map 0(a) = [1(b),1] : a.v>10 1(b) = [0(a),1] : b.v>30 for filtered rows in a (100,10) create a-NULL for filtered rows in b (100,10) (100,20) (100,30) create NULL-b with 0(a) = [1(b),1] : a.v>10 100, 10 = 00000010 (filtered) 100, 20 = 00000000 (valid) 100, 30 = 00000000 (valid) ------------------------- sum = 00000000 : for valid rows in b, there is at least one pair in a with 1(b) = [0(a),1] : b.v>30 100, 10 = 00000001 (filtered) 100, 20 = 00000001 (filtered) 100, 30 = 00000001 (filtered) ------------------------- sum = 00000001 : for valid rows in a (100,20) (100,30), there is no pair in b result : 100, 10 : N, N N, N : 100, 10 N, N : 100, 20 N, N : 100, 30 100, 20 : N, N 100, 30 : N, N
protected JoinCondDesc[]	condn
protected int	countAfterReport
protected ArrayList<Object>[]	dummyObj
protected RowContainer<List<Object>>[]	dummyObjVectors
protected int[][]	filterMaps
protected short[]	filterTags
protected Object[]	forwardCache
protected org.apache.hadoop.conf.Configuration	hconf
protected int	heartbeatInterval
protected List[]	intermediate
protected List<ObjectInspector>[]	joinFilterObjectInspectors The ObjectInspectors for join filters.
protected List<ExprNodeEvaluator>[]	joinFilters The filters for join
protected List<ExprNodeEvaluator>[]	joinValues The expressions for join inputs.
protected List<ObjectInspector>[]	joinValuesObjectInspectors The ObjectInspectors for the join inputs.
protected List<ObjectInspector>[]	joinValuesStandardObjectInspectors The standard ObjectInspectors for the join inputs.
protected static org.slf4j.Logger	LOG
protected boolean	needsPostEvaluation Will be true depending on content of residualJoinFilters.
boolean	noOuterJoin
protected static int	NOTSKIPBIGTABLE
protected boolean[]	nullsafes
protected int	numAliases
protected int[]	offsets
protected Byte[]	order
protected List<ExprNodeEvaluator>	residualJoinFilters List of evaluators for conditions which appear on on-clause and needs to be evaluated before emitting rows.
protected List<ObjectInspector>	residualJoinFiltersOIs OIs corresponding to residualJoinFilters.
protected Map<Integer,Object[]>	rowContainerPostFilteredOuterJoin This data structure is used to keep track of rows on which residualFilters evaluated to false.
protected List<ObjectInspector>[]	rowContainerStandardObjectInspectors The standard ObjectInspectors for the row container.
protected boolean[][]	skipVectors
protected TableDesc[]	spillTableDesc
protected int	totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

abortOp, asyncInitOperations, bucketingVersion, cContext, childOperators, childOperatorsArray, childOperatorsTag, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, HIVE_COUNTER_CREATED_FILES, HIVE_COUNTER_FATAL, id, indexForTezUnion, inputObjInspectors, numRows, operatorId, out, outputObjInspector, parentOperators, reporter, runTimeNumRows, state, statsMap

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	CommonJoinOperator() Kryo ctor.
	CommonJoinOperator(CommonJoinOperator<T> clone)
	CommonJoinOperator(CompilationOpContext ctx)

Method Summary

Modifier and Type	Method and Description
protected void	checkAndGenObject()
void	closeOp(boolean abort) All done.
void	endGroup() Forward a record of join results.
protected List<Object>	getFilteredValue(byte alias, Object row)
protected short	getFilterTag(List<Object> row)
String	getName() Gets the name of the node.
protected long	getNextSize(long sz)
static String	getOperatorName()
Map<Integer,Set<String>>	getPosToAliasMap()
protected List<ObjectInspector>	getValueObjectInspectors(byte alias, List<ObjectInspector>[] aliasToObjectInspectors)
protected boolean	hasFilter(int alias)
protected void	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
protected void	internalForward(Object row, ObjectInspector outputOI)
boolean	opAllowedAfterMapJoin()
boolean	opAllowedBeforeMapJoin()
protected void	reportProgress()
void	setPosToAliasMap(Map<Integer,Set<String>> posToAliasMap)
void	startGroup()

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

abort, acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, cleanUpInputFileChangedOp, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, completeInitializationOp, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, flushRecursive, forward, forward, forward, getAdditionalCounters, getBucketingVersion, getChildOperators, getChildren, getColumnExprMap, getCompilationOpContext, getConf, getConfiguration, getCounterName, getDone, getExecContext, getGroupKeyObject, getIdentifier, getIndexForTezUnion, getInputObjInspectors, getIsReduceSink, getMarker, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getReduceOutputName, getSchema, getStatistics, getStats, getType, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logicalEquals, logicalEqualsTree, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, process, processGroup, removeChild, removeChildAndAdoptItsChildren, removeParent, removeParents, replaceChild, replaceParent, reset, setAlias, setBucketingVersion, setChildOperators, setColumnExprMap, setCompilationOpContext, setConf, setDone, setExecContext, setGroupKeyObject, setIndexForTezUnion, setInputContext, setInputObjInspectors, setMarker, setNextVectorBatchGroupStatus, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

numAliases

protected transient int numAliases

joinValues

protected transient List<ExprNodeEvaluator>[] joinValues

The expressions for join inputs.

joinFilters

protected transient List<ExprNodeEvaluator>[] joinFilters

The filters for join

residualJoinFilters

protected transient List<ExprNodeEvaluator> residualJoinFilters

List of evaluators for conditions which appear on on-clause and needs to be evaluated before emitting rows. Currently, relevant only for outer joins. For instance, given the query: select * from t1 right outer join t2 on t1.c1 + t2.c2 > t1.c3; The expression evaluator for t1.c1 + t2.c2 > t1.c3 will be stored in this list.

filterMaps

protected transient int[][] filterMaps

joinValuesObjectInspectors

protected transient List<ObjectInspector>[] joinValuesObjectInspectors

The ObjectInspectors for the join inputs.

joinFilterObjectInspectors

protected transient List<ObjectInspector>[] joinFilterObjectInspectors

The ObjectInspectors for join filters.

residualJoinFiltersOIs

protected transient List<ObjectInspector> residualJoinFiltersOIs

OIs corresponding to residualJoinFilters.

needsPostEvaluation

protected transient boolean needsPostEvaluation

Will be true depending on content of residualJoinFilters.

rowContainerPostFilteredOuterJoin

protected transient Map<Integer,Object[]> rowContainerPostFilteredOuterJoin

This data structure is used to keep track of rows on which residualFilters evaluated to false. We will iterate on this container afterwards and emit rows appending NULL values if it was not done. Key is relation index.

joinValuesStandardObjectInspectors

protected transient List<ObjectInspector>[] joinValuesStandardObjectInspectors

The standard ObjectInspectors for the join inputs.

rowContainerStandardObjectInspectors

protected transient List<ObjectInspector>[] rowContainerStandardObjectInspectors

The standard ObjectInspectors for the row container.

order

protected transient Byte[] order

condn

protected transient JoinCondDesc[] condn

nullsafes

protected transient boolean[] nullsafes

noOuterJoin

public transient boolean noOuterJoin

dummyObj

protected transient ArrayList<Object>[] dummyObj

dummyObjVectors

protected transient RowContainer<List<Object>>[] dummyObjVectors

totalSz

protected transient int totalSz

spillTableDesc

protected transient TableDesc[] spillTableDesc

countAfterReport

protected transient int countAfterReport

heartbeatInterval

protected transient int heartbeatInterval

NOTSKIPBIGTABLE

protected static final int NOTSKIPBIGTABLE

See Also:

Constant Field Values

hconf

protected org.apache.hadoop.conf.Configuration hconf

alias

protected transient Byte alias

forwardCache

protected transient Object[] forwardCache

offsets

protected transient int[] offsets

skipVectors

protected transient boolean[][] skipVectors

intermediate

protected transient List[] intermediate

filterTags

protected transient short[] filterTags

aliasFilterTags

protected transient short[] aliasFilterTags

On filterTags ANDed value of all filter tags in current join group if any of values passes on outer join alias (which makes zero for the tag alias), it means there exists a pair for it and safely regarded as a inner join for example, with table a, b something like, a = 100, 10 | 100, 20 | 100, 30 b = 100, 10 | 100, 20 | 100, 30 the query "a FO b ON a.k=b.k AND a.v>10 AND b.v>30" makes filter map 0(a) = [1(b),1] : a.v>10 1(b) = [0(a),1] : b.v>30 for filtered rows in a (100,10) create a-NULL for filtered rows in b (100,10) (100,20) (100,30) create NULL-b with 0(a) = [1(b),1] : a.v>10 100, 10 = 00000010 (filtered) 100, 20 = 00000000 (valid) 100, 30 = 00000000 (valid) ------------------------- sum = 00000000 : for valid rows in b, there is at least one pair in a with 1(b) = [0(a),1] : b.v>30 100, 10 = 00000001 (filtered) 100, 20 = 00000001 (filtered) 100, 30 = 00000001 (filtered) ------------------------- sum = 00000001 : for valid rows in a (100,20) (100,30), there is no pair in b result : 100, 10 : N, N N, N : 100, 10 N, N : 100, 20 N, N : 100, 30 100, 20 : N, N 100, 30 : N, N

Constructor Detail

CommonJoinOperator

protected CommonJoinOperator()

Kryo ctor.

CommonJoinOperator

public CommonJoinOperator(CompilationOpContext ctx)

CommonJoinOperator

public CommonJoinOperator(CommonJoinOperator<T> clone)

Method Detail

getValueObjectInspectors

protected List<ObjectInspector> getValueObjectInspectors(byte alias,
                                                         List<ObjectInspector>[] aliasToObjectInspectors)

initializeOp

protected void initializeOp(org.apache.hadoop.conf.Configuration hconf)
                     throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class Operator<T extends JoinDesc>

Throws:

HiveException

startGroup

public void startGroup()
                throws HiveException

Overrides:

startGroup in class Operator<T extends JoinDesc>

Throws:

HiveException

getNextSize

protected long getNextSize(long sz)

getFilteredValue

protected List<Object> getFilteredValue(byte alias,
                                        Object row)
                                 throws HiveException

Throws:

HiveException

hasFilter

protected final boolean hasFilter(int alias)

getFilterTag

protected final short getFilterTag(List<Object> row)

endGroup

public void endGroup()
              throws HiveException

Forward a record of join results.

Overrides:

endGroup in class Operator<T extends JoinDesc>

Throws:

HiveException

internalForward

protected void internalForward(Object row,
                               ObjectInspector outputOI)
                        throws HiveException

Throws:

HiveException

checkAndGenObject

protected void checkAndGenObject()
                          throws HiveException

Throws:

HiveException

reportProgress

protected void reportProgress()

closeOp

public void closeOp(boolean abort)
             throws HiveException

All done.

Overrides:

closeOp in class Operator<T extends JoinDesc>

Throws:

HiveException

getName

public String getName()

Description copied from interface: Node

Gets the name of the node. This is used in the rule dispatchers.

Specified by:

getName in interface Node

Specified by:

getName in class Operator<T extends JoinDesc>

Returns:

String

getOperatorName

public static String getOperatorName()

getPosToAliasMap

public Map<Integer,Set<String>> getPosToAliasMap()

Returns:

the posToAliasMap

setPosToAliasMap

public void setPosToAliasMap(Map<Integer,Set<String>> posToAliasMap)

Parameters:

posToAliasMap - the posToAliasMap to set

opAllowedBeforeMapJoin

public boolean opAllowedBeforeMapJoin()

Overrides:

opAllowedBeforeMapJoin in class Operator<T extends JoinDesc>

opAllowedAfterMapJoin

public boolean opAllowedAfterMapJoin()

Overrides:

opAllowedAfterMapJoin in class Operator<T extends JoinDesc>