org.apache.hadoop.hive.serde2.binarysortable

Class BinarySortableSerDe

java.lang.Object

org.apache.hadoop.hive.serde2.AbstractSerDe

org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe

All Implemented Interfaces:

Deserializer, Serializer

Direct Known Subclasses:

BinarySortableSerDeWithEndPrefix

public class BinarySortableSerDe
extends AbstractSerDe

BinarySortableSerDe can be used to write data in a way that the data can be compared byte-by-byte with the same order. The data format: NULL: a single byte (\0 or \1, check below) NON-NULL Primitives: ALWAYS prepend a single byte (\0 or \1), and then: Boolean: FALSE = \1, TRUE = \2 Byte: flip the sign-bit to make sure negative comes before positive Short: flip the sign-bit to make sure negative comes before positive Int: flip the sign-bit to make sure negative comes before positive Long: flip the sign-bit to make sure negative comes before positive Double: flip the sign-bit for positive double, and all bits for negative double values String: NULL-terminated UTF-8 string, with NULL escaped to \1 \1, and \1 escaped to \1 \2 NON-NULL Complex Types: ALWAYS prepend a single byte \1, and then: Struct: one field by one field. List: \1 followed by each element, and \0 to terminate Map: \1 followed by each key and then each value, and \0 to terminate This SerDe takes an additional parameter SERIALIZATION_SORT_ORDER which is a string containing only "+" and "-". The length of the string should equal to the number of fields in the top-level struct for serialization. "+" means the field should be sorted ascendingly, and "-" means descendingly. The sub fields in the same top-level field will have the same sort order. This SerDe takes an additional parameter SERIALIZATION_NULL_SORT_ORDER which is a string containing only "a" and "z". The length of the string should equal to the number of fields in the top-level struct for serialization. "a" means that NULL should come first (thus, single byte is \0 for ascending order, \1 for descending order), while "z" means that NULL should come last (thus, single byte is \1 for ascending order, \0 for descending order).

Field Summary

Fields

Modifier and Type	Field and Description
static Charset	decimalCharSet
static org.slf4j.Logger	LOG
static byte	ONE
static byte	ZERO

Fields inherited from class org.apache.hadoop.hive.serde2.AbstractSerDe

configErrors

Constructor Summary

Constructors

Constructor and Description
BinarySortableSerDe()

Method Summary

Modifier and Type	Method and Description
Object	deserialize(org.apache.hadoop.io.Writable blob) Deserialize an object out of a Writable blob.
static org.apache.hadoop.io.Text	deserializeText(InputByteBuffer buffer, boolean invert, org.apache.hadoop.io.Text r)
byte[]	getNotNullMarkers()
byte[]	getNullMarkers()
ObjectInspector	getObjectInspector() Get the object inspector that can be used to navigate through the internal structure of the Object returned from deserialize(...).
SerDeStats	getSerDeStats() Returns statistics collected when serializing
Class<? extends org.apache.hadoop.io.Writable>	getSerializedClass() Returns the Writable class that would be returned by the serialize method.
boolean[]	getSortOrders()
void	initialize(org.apache.hadoop.conf.Configuration conf, Properties tbl) Initialize the HiveSerializer.
org.apache.hadoop.io.Writable	serialize(Object obj, ObjectInspector objInspector) Serialize an object by navigating inside the Object with the ObjectInspector.
static void	serializeBytes(ByteStream.Output buffer, byte[] data, int length, boolean invert)
static void	serializeBytes(ByteStream.Output buffer, byte[] data, int offset, int length, boolean invert)
static void	serializeDouble(ByteStream.Output buffer, double vd, boolean invert)
static void	serializeFloat(ByteStream.Output buffer, float vf, boolean invert)
static void	serializeHiveDecimal(ByteStream.Output buffer, org.apache.hadoop.hive.common.type.HiveDecimal dec, boolean invert)
static void	serializeHiveDecimal(ByteStream.Output buffer, org.apache.hadoop.hive.common.type.HiveDecimal dec, boolean invert, byte[] scratchBuffer) Decimals are encoded in three pieces:Decimals are encoded in three pieces: Sign: 1, 2 or 3 for smaller, equal or larger than 0 respectively Factor: Number that indicates the amount of digits you have to move the decimal point left or right until the resulting number is smaller than zero but has something other than 0 as the first digit.
static void	serializeHiveDecimal(ByteStream.Output buffer, org.apache.hadoop.hive.serde2.io.HiveDecimalWritable decWritable, boolean invert, byte[] scratchBuffer)
static void	serializeHiveIntervalDayTime(ByteStream.Output buffer, org.apache.hadoop.hive.common.type.HiveIntervalDayTime intervalDayTime, boolean invert)
static void	serializeHiveIntervalYearMonth(ByteStream.Output buffer, HiveIntervalYearMonth intervalYearMonth, boolean invert)
static void	serializeInt(ByteStream.Output buffer, int v, boolean invert)
static void	serializeLong(ByteStream.Output buffer, long v, boolean invert)
static void	serializeOldHiveDecimal(ByteStream.Output buffer, org.apache.hadoop.hive.common.type.HiveDecimalV1 oldDec, boolean invert)
static void	serializeShort(ByteStream.Output buffer, short v, boolean invert)
static void	serializeStruct(ByteStream.Output byteStream, Object[] fieldData, List<ObjectInspector> fieldOis, boolean[] sortableSortOrders, byte[] nullMarkers, byte[] notNullMarkers)
static void	serializeTimestampTZWritable(ByteStream.Output buffer, TimestampLocalTZWritable t, boolean invert)
static void	serializeTimestampWritable(ByteStream.Output buffer, TimestampWritableV2 t, boolean invert)
static void	writeByte(ByteStream.RandomAccessOutput buffer, byte b, boolean invert)

Methods inherited from class org.apache.hadoop.hive.serde2.AbstractSerDe

getConfigurationErrors, initialize, shouldStoreFieldsInMetastore

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

ZERO

public static final byte ZERO

See Also:

Constant Field Values

ONE

public static final byte ONE

See Also:

Constant Field Values

decimalCharSet

public static Charset decimalCharSet

Constructor Detail

BinarySortableSerDe

public BinarySortableSerDe()

Method Detail

initialize

public void initialize(org.apache.hadoop.conf.Configuration conf,
                       Properties tbl)
                throws SerDeException

Description copied from class: AbstractSerDe

Initialize the HiveSerializer.

Specified by:

initialize in interface Deserializer

Specified by:

initialize in interface Serializer

Specified by:

initialize in class AbstractSerDe

Parameters:

conf - System properties. Can be null in compile time

tbl - table properties

Throws:

SerDeException

getSerializedClass

public Class<? extends org.apache.hadoop.io.Writable> getSerializedClass()

Description copied from class: AbstractSerDe

Returns the Writable class that would be returned by the serialize method. This is used to initialize SequenceFile header.

Specified by:

getSerializedClass in interface Serializer

Specified by:

getSerializedClass in class AbstractSerDe

getObjectInspector

public ObjectInspector getObjectInspector()
                                   throws SerDeException

Description copied from class: AbstractSerDe

Get the object inspector that can be used to navigate through the internal structure of the Object returned from deserialize(...).

Specified by:

getObjectInspector in interface Deserializer

Specified by:

getObjectInspector in class AbstractSerDe

Throws:

SerDeException

deserialize

public Object deserialize(org.apache.hadoop.io.Writable blob)
                   throws SerDeException

Description copied from class: AbstractSerDe

Deserialize an object out of a Writable blob. In most cases, the return value of this function will be constant since the function will reuse the returned object. If the client wants to keep a copy of the object, the client needs to clone the returned value by calling ObjectInspectorUtils.getStandardObject().

Specified by:

deserialize in interface Deserializer

Specified by:

deserialize in class AbstractSerDe

Parameters:

blob - The Writable object containing a serialized object

Returns:

A Java object representing the contents in the blob.

Throws:

SerDeException

deserializeText

public static org.apache.hadoop.io.Text deserializeText(InputByteBuffer buffer,
                                                        boolean invert,
                                                        org.apache.hadoop.io.Text r)
                                                 throws IOException

Throws:

IOException

serialize

public org.apache.hadoop.io.Writable serialize(Object obj,
                                               ObjectInspector objInspector)
                                        throws SerDeException

Description copied from class: AbstractSerDe

Serialize an object by navigating inside the Object with the ObjectInspector. In most cases, the return value of this function will be constant since the function will reuse the Writable object. If the client wants to keep a copy of the Writable, the client needs to clone the returned value.

Specified by:

serialize in interface Serializer

Specified by:

serialize in class AbstractSerDe

Throws:

SerDeException

writeByte

public static void writeByte(ByteStream.RandomAccessOutput buffer,
                             byte b,
                             boolean invert)

serializeBytes

public static void serializeBytes(ByteStream.Output buffer,
                                  byte[] data,
                                  int length,
                                  boolean invert)

serializeBytes

public static void serializeBytes(ByteStream.Output buffer,
                                  byte[] data,
                                  int offset,
                                  int length,
                                  boolean invert)

serializeShort

public static void serializeShort(ByteStream.Output buffer,
                                  short v,
                                  boolean invert)

serializeInt

public static void serializeInt(ByteStream.Output buffer,
                                int v,
                                boolean invert)

serializeLong

public static void serializeLong(ByteStream.Output buffer,
                                 long v,
                                 boolean invert)

serializeFloat

public static void serializeFloat(ByteStream.Output buffer,
                                  float vf,
                                  boolean invert)

serializeDouble

public static void serializeDouble(ByteStream.Output buffer,
                                   double vd,
                                   boolean invert)

serializeTimestampWritable

public static void serializeTimestampWritable(ByteStream.Output buffer,
                                              TimestampWritableV2 t,
                                              boolean invert)

serializeTimestampTZWritable

public static void serializeTimestampTZWritable(ByteStream.Output buffer,
                                                TimestampLocalTZWritable t,
                                                boolean invert)

serializeHiveIntervalYearMonth

public static void serializeHiveIntervalYearMonth(ByteStream.Output buffer,
                                                  HiveIntervalYearMonth intervalYearMonth,
                                                  boolean invert)

serializeHiveIntervalDayTime

public static void serializeHiveIntervalDayTime(ByteStream.Output buffer,
                                                org.apache.hadoop.hive.common.type.HiveIntervalDayTime intervalDayTime,
                                                boolean invert)

serializeOldHiveDecimal

public static void serializeOldHiveDecimal(ByteStream.Output buffer,
                                           org.apache.hadoop.hive.common.type.HiveDecimalV1 oldDec,
                                           boolean invert)

serializeHiveDecimal

public static void serializeHiveDecimal(ByteStream.Output buffer,
                                        org.apache.hadoop.hive.common.type.HiveDecimal dec,
                                        boolean invert)

serializeHiveDecimal

public static void serializeHiveDecimal(ByteStream.Output buffer,
                                        org.apache.hadoop.hive.common.type.HiveDecimal dec,
                                        boolean invert,
                                        byte[] scratchBuffer)

Decimals are encoded in three pieces:Decimals are encoded in three pieces: Sign: 1, 2 or 3 for smaller, equal or larger than 0 respectively Factor: Number that indicates the amount of digits you have to move the decimal point left or right until the resulting number is smaller than zero but has something other than 0 as the first digit. Digits: which is a string of all the digits in the decimal. If the number is negative the binary string will be inverted to get the correct ordering. UNDONE: Is this example correct? Example: 0.00123 Sign is 3 (bigger than 0) Factor is -2 (move decimal point 2 positions right) Digits are: 123

Parameters:

buffer -

dec -

invert -

scratchBuffer -

serializeHiveDecimal

public static void serializeHiveDecimal(ByteStream.Output buffer,
                                        org.apache.hadoop.hive.serde2.io.HiveDecimalWritable decWritable,
                                        boolean invert,
                                        byte[] scratchBuffer)

getSerDeStats

public SerDeStats getSerDeStats()

Description copied from class: AbstractSerDe

Returns statistics collected when serializing

Specified by:

getSerDeStats in interface Deserializer

Specified by:

getSerDeStats in interface Serializer

Specified by:

getSerDeStats in class AbstractSerDe

serializeStruct

public static void serializeStruct(ByteStream.Output byteStream,
                                   Object[] fieldData,
                                   List<ObjectInspector> fieldOis,
                                   boolean[] sortableSortOrders,
                                   byte[] nullMarkers,
                                   byte[] notNullMarkers)
                            throws SerDeException

Throws:

SerDeException

getSortOrders

public boolean[] getSortOrders()

getNullMarkers

public byte[] getNullMarkers()

getNotNullMarkers

public byte[] getNotNullMarkers()