org.apache.hadoop.hive.serde2.avro

Class AvroSerDe

java.lang.Object

org.apache.hadoop.hive.serde2.AbstractSerDe

org.apache.hadoop.hive.serde2.avro.AvroSerDe

All Implemented Interfaces:

Deserializer, Serializer

public class AvroSerDe
extends AbstractSerDe

Read or write Avro data from Hive.

Field Summary

Fields

Modifier and Type	Field and Description
static String	AVRO_INT_TYPE_NAME
static String	AVRO_LONG_TYPE_NAME
static String	AVRO_PROP_LOGICAL_TYPE
static String	AVRO_PROP_MAX_LENGTH
static String	AVRO_PROP_PRECISION
static String	AVRO_PROP_SCALE
static String	AVRO_STRING_TYPE_NAME
static String	CHAR_TYPE_NAME
static String	DATE_TYPE_NAME
static String	DECIMAL_TYPE_NAME
static String	LIST_COLUMN_COMMENTS
static String	TABLE_COMMENT
static String	TABLE_NAME
static String	TIMESTAMP_TYPE_NAME
static String	VARCHAR_TYPE_NAME

Fields inherited from class org.apache.hadoop.hive.serde2.AbstractSerDe

configErrors

Constructor Summary

Constructors

Constructor and Description
AvroSerDe()

Method Summary

Modifier and Type	Method and Description
Object	deserialize(org.apache.hadoop.io.Writable writable) Deserialize an object out of a Writable blob.
org.apache.avro.Schema	determineSchemaOrReturnErrorSchema(org.apache.hadoop.conf.Configuration conf, Properties props) Attempt to determine the schema via the usual means, but do not throw an exception if we fail.
ObjectInspector	getObjectInspector() Get the object inspector that can be used to navigate through the internal structure of the Object returned from deserialize(...).
static org.apache.avro.Schema	getSchemaFromCols(Properties properties, List<String> columnNames, List<TypeInfo> columnTypes, String columnCommentProperty)
SerDeStats	getSerDeStats() Returns statistics collected when serializing
Class<? extends org.apache.hadoop.io.Writable>	getSerializedClass() Returns the Writable class that would be returned by the serialize method.
void	initialize(org.apache.hadoop.conf.Configuration configuration, Properties properties) Initialize the HiveSerializer.
void	initialize(org.apache.hadoop.conf.Configuration configuration, Properties tableProperties, Properties partitionProperties) Initialize the SerDe.
org.apache.hadoop.io.Writable	serialize(Object o, ObjectInspector objectInspector) Serialize an object by navigating inside the Object with the ObjectInspector.
boolean	shouldStoreFieldsInMetastore(Map<String,String> tableParams)

Methods inherited from class org.apache.hadoop.hive.serde2.AbstractSerDe

getConfigurationErrors

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TABLE_NAME

public static final String TABLE_NAME

See Also:

Constant Field Values

TABLE_COMMENT

public static final String TABLE_COMMENT

See Also:

Constant Field Values

LIST_COLUMN_COMMENTS

public static final String LIST_COLUMN_COMMENTS

See Also:

Constant Field Values

DECIMAL_TYPE_NAME

public static final String DECIMAL_TYPE_NAME

See Also:

Constant Field Values

CHAR_TYPE_NAME

public static final String CHAR_TYPE_NAME

See Also:

Constant Field Values

VARCHAR_TYPE_NAME

public static final String VARCHAR_TYPE_NAME

See Also:

Constant Field Values

DATE_TYPE_NAME

public static final String DATE_TYPE_NAME

See Also:

Constant Field Values

TIMESTAMP_TYPE_NAME

public static final String TIMESTAMP_TYPE_NAME

See Also:

Constant Field Values

AVRO_PROP_LOGICAL_TYPE

public static final String AVRO_PROP_LOGICAL_TYPE

See Also:

Constant Field Values

AVRO_PROP_PRECISION

public static final String AVRO_PROP_PRECISION

See Also:

Constant Field Values

AVRO_PROP_SCALE

public static final String AVRO_PROP_SCALE

See Also:

Constant Field Values

AVRO_PROP_MAX_LENGTH

public static final String AVRO_PROP_MAX_LENGTH

See Also:

Constant Field Values

AVRO_STRING_TYPE_NAME

public static final String AVRO_STRING_TYPE_NAME

See Also:

Constant Field Values

AVRO_INT_TYPE_NAME

public static final String AVRO_INT_TYPE_NAME

See Also:

Constant Field Values

AVRO_LONG_TYPE_NAME

public static final String AVRO_LONG_TYPE_NAME

See Also:

Constant Field Values

Constructor Detail

AvroSerDe

public AvroSerDe()

Method Detail

initialize

public void initialize(org.apache.hadoop.conf.Configuration configuration,
                       Properties tableProperties,
                       Properties partitionProperties)
                throws SerDeException

Description copied from class: AbstractSerDe

Initialize the SerDe. By default, this will use one set of properties, either the table properties or the partition properties. If a SerDe needs access to both sets, it should override this method. Eventually, once all SerDes have implemented this method, we should convert it to an abstract method.

Overrides:

initialize in class AbstractSerDe

Parameters:

configuration - Hadoop configuration

tableProperties - Table properties

partitionProperties - Partition properties

Throws:

SerDeException

initialize

public void initialize(org.apache.hadoop.conf.Configuration configuration,
                       Properties properties)
                throws SerDeException

Description copied from class: AbstractSerDe

Initialize the HiveSerializer.

Specified by:

initialize in interface Deserializer

Specified by:

initialize in interface Serializer

Specified by:

initialize in class AbstractSerDe

Parameters:

configuration - System properties. Can be null in compile time

properties - table properties

Throws:

SerDeException

getSchemaFromCols

public static org.apache.avro.Schema getSchemaFromCols(Properties properties,
                                                       List<String> columnNames,
                                                       List<TypeInfo> columnTypes,
                                                       String columnCommentProperty)

determineSchemaOrReturnErrorSchema

public org.apache.avro.Schema determineSchemaOrReturnErrorSchema(org.apache.hadoop.conf.Configuration conf,
                                                                 Properties props)

Attempt to determine the schema via the usual means, but do not throw an exception if we fail. Instead, signal failure via a special schema. This is used because Hive calls init on the serde during any call, including calls to update the serde properties, meaning if the serde is in a bad state, there is no way to update that state.

getSerializedClass

public Class<? extends org.apache.hadoop.io.Writable> getSerializedClass()

Description copied from class: AbstractSerDe

Returns the Writable class that would be returned by the serialize method. This is used to initialize SequenceFile header.

Specified by:

getSerializedClass in interface Serializer

Specified by:

getSerializedClass in class AbstractSerDe

serialize

public org.apache.hadoop.io.Writable serialize(Object o,
                                               ObjectInspector objectInspector)
                                        throws SerDeException

Description copied from class: AbstractSerDe

Serialize an object by navigating inside the Object with the ObjectInspector. In most cases, the return value of this function will be constant since the function will reuse the Writable object. If the client wants to keep a copy of the Writable, the client needs to clone the returned value.

Specified by:

serialize in interface Serializer

Specified by:

serialize in class AbstractSerDe

Throws:

SerDeException

deserialize

public Object deserialize(org.apache.hadoop.io.Writable writable)
                   throws SerDeException

Description copied from class: AbstractSerDe

Deserialize an object out of a Writable blob. In most cases, the return value of this function will be constant since the function will reuse the returned object. If the client wants to keep a copy of the object, the client needs to clone the returned value by calling ObjectInspectorUtils.getStandardObject().

Specified by:

deserialize in interface Deserializer

Specified by:

deserialize in class AbstractSerDe

Parameters:

writable - The Writable object containing a serialized object

Returns:

A Java object representing the contents in the blob.

Throws:

SerDeException

getObjectInspector

public ObjectInspector getObjectInspector()
                                   throws SerDeException

Description copied from class: AbstractSerDe

Get the object inspector that can be used to navigate through the internal structure of the Object returned from deserialize(...).

Specified by:

getObjectInspector in interface Deserializer

Specified by:

getObjectInspector in class AbstractSerDe

Throws:

SerDeException

getSerDeStats

public SerDeStats getSerDeStats()

Description copied from class: AbstractSerDe

Returns statistics collected when serializing

Specified by:

getSerDeStats in interface Deserializer

Specified by:

getSerDeStats in interface Serializer

Specified by:

getSerDeStats in class AbstractSerDe

shouldStoreFieldsInMetastore

public boolean shouldStoreFieldsInMetastore(Map<String,String> tableParams)

Overrides:

shouldStoreFieldsInMetastore in class AbstractSerDe

Returns:

Whether the SerDe that can store schema both inside and outside of metastore does, in fact, store it inside metastore, based on table parameters.