org.apache.hadoop.hive.ql.exec.persistence

Class MapJoinBytesTableContainer

java.lang.Object

org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer

All Implemented Interfaces:

MemoryEstimate, MapJoinTableContainer, MapJoinTableContainerDirectAccess

public class MapJoinBytesTableContainer
extends Object
implements MapJoinTableContainer, MapJoinTableContainerDirectAccess

Table container that serializes keys and values using LazyBinarySerDe into BytesBytesMultiHashMap, with very low memory overhead. However, there may be some perf overhead when retrieving rows.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	MapJoinBytesTableContainer.DirectKeyValueWriter
static interface	MapJoinBytesTableContainer.KeyValueHelper

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainer

MapJoinTableContainer.ReusableGetAdaptor

Field Summary

Fields

Modifier and Type	Field and Description
static long	DEFAULT_OBJECT_INSPECTOR_MEMORY_SIZE

Constructor Summary

Constructors

Constructor and Description
MapJoinBytesTableContainer(BytesBytesMultiHashMap hashMap)
MapJoinBytesTableContainer(org.apache.hadoop.conf.Configuration hconf, MapJoinObjectSerDeContext valCtx, long keyCount, long memUsage)

Method Summary

Modifier and Type	Method and Description
void	clear() Clears the contents of the table.
MapJoinTableContainer.ReusableGetAdaptor	createGetter(MapJoinKey keyTypeFromLoader) Creates reusable get adaptor that can be used to retrieve rows from the table based on either vectorized or non-vectorized input rows to MapJoinOperator.
void	dumpMetrics()
MapJoinKey	getAnyKey()
static List<Object>	getComplexFieldsAsList(LazyBinaryStruct lazyBinaryStruct, ArrayList<Object> objectArrayBuffer, LazyBinaryStructObjectInspector lazyBinaryStructObjectInspector)
long	getEstimatedMemorySize() Returns estimated memory size based JavaDataModel
static boolean	hasComplexObjects(LazyBinaryStructObjectInspector lazyBinaryStructObjectInspector)
boolean	hasSpill() Checks if the container has spilled any data onto disk.
static boolean	isSupportedKey(ObjectInspector keyOi)
void	put(org.apache.hadoop.io.Writable currentKey, org.apache.hadoop.io.Writable currentValue)
MapJoinKey	putRow(org.apache.hadoop.io.Writable currentKey, org.apache.hadoop.io.Writable currentValue) Adds row from input to the table.
void	seal() Indicates to the container that the puts have ended; table is now r/o.
void	setInternalValueOi(LazyBinaryStructObjectInspector internalValueOi)
void	setNotNullMarkers(byte[] notNullMarkers)
void	setNullMarkers(byte[] nullMarkers)
void	setSerde(MapJoinObjectSerDeContext keyContext, MapJoinObjectSerDeContext valueContext)
void	setSortableSortOrders(boolean[] sortableSortOrders)
int	size() Return the size of the hash table

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_OBJECT_INSPECTOR_MEMORY_SIZE

public static final long DEFAULT_OBJECT_INSPECTOR_MEMORY_SIZE

See Also:

Constant Field Values

Constructor Detail

MapJoinBytesTableContainer

public MapJoinBytesTableContainer(org.apache.hadoop.conf.Configuration hconf,
                                  MapJoinObjectSerDeContext valCtx,
                                  long keyCount,
                                  long memUsage)
                           throws SerDeException

Throws:

SerDeException

MapJoinBytesTableContainer

public MapJoinBytesTableContainer(BytesBytesMultiHashMap hashMap)

Method Detail

setInternalValueOi

public void setInternalValueOi(LazyBinaryStructObjectInspector internalValueOi)

setSortableSortOrders

public void setSortableSortOrders(boolean[] sortableSortOrders)

setNullMarkers

public void setNullMarkers(byte[] nullMarkers)

setNotNullMarkers

public void setNotNullMarkers(byte[] notNullMarkers)

setSerde

public void setSerde(MapJoinObjectSerDeContext keyContext,
                     MapJoinObjectSerDeContext valueContext)
              throws SerDeException

Specified by:

setSerde in interface MapJoinTableContainer

Throws:

SerDeException

putRow

public MapJoinKey putRow(org.apache.hadoop.io.Writable currentKey,
                         org.apache.hadoop.io.Writable currentValue)
                  throws SerDeException

Description copied from interface: MapJoinTableContainer

Adds row from input to the table.

Specified by:

putRow in interface MapJoinTableContainer

Throws:

SerDeException

clear

public void clear()

Description copied from interface: MapJoinTableContainer

Clears the contents of the table.

Specified by:

clear in interface MapJoinTableContainer

getAnyKey

public MapJoinKey getAnyKey()

Specified by:

getAnyKey in interface MapJoinTableContainer

createGetter

public MapJoinTableContainer.ReusableGetAdaptor createGetter(MapJoinKey keyTypeFromLoader)

Description copied from interface: MapJoinTableContainer

Creates reusable get adaptor that can be used to retrieve rows from the table based on either vectorized or non-vectorized input rows to MapJoinOperator.

Specified by:

createGetter in interface MapJoinTableContainer

Parameters:

keyTypeFromLoader - Last key from hash table loader, to determine key type used when loading hashtable (if it can vary).

seal

public void seal()

Description copied from interface: MapJoinTableContainer

Indicates to the container that the puts have ended; table is now r/o.

Specified by:

seal in interface MapJoinTableContainer

put

public void put(org.apache.hadoop.io.Writable currentKey,
                org.apache.hadoop.io.Writable currentValue)
         throws SerDeException

Specified by:

put in interface MapJoinTableContainerDirectAccess

Throws:

SerDeException

hasComplexObjects

public static boolean hasComplexObjects(LazyBinaryStructObjectInspector lazyBinaryStructObjectInspector)

getComplexFieldsAsList

public static List<Object> getComplexFieldsAsList(LazyBinaryStruct lazyBinaryStruct,
                                                  ArrayList<Object> objectArrayBuffer,
                                                  LazyBinaryStructObjectInspector lazyBinaryStructObjectInspector)

isSupportedKey

public static boolean isSupportedKey(ObjectInspector keyOi)

dumpMetrics

public void dumpMetrics()

Specified by:

dumpMetrics in interface MapJoinTableContainer

hasSpill

public boolean hasSpill()

Description copied from interface: MapJoinTableContainer

Checks if the container has spilled any data onto disk. This is only applicable for HybridHashTableContainer.

Specified by:

hasSpill in interface MapJoinTableContainer

size

public int size()

Description copied from interface: MapJoinTableContainer

Return the size of the hash table

Specified by:

size in interface MapJoinTableContainer

getEstimatedMemorySize

public long getEstimatedMemorySize()

Description copied from interface: MemoryEstimate

Returns estimated memory size based JavaDataModel

Specified by:

getEstimatedMemorySize in interface MemoryEstimate

Returns:

estimated memory size in bytes