org.apache.hadoop.hive.druid

Class DruidStorageHandler

java.lang.Object

org.apache.hadoop.hive.metastore.DefaultHiveMetaHook

org.apache.hadoop.hive.druid.DruidStorageHandler

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, HiveMetaHook, HiveStorageHandler

public class DruidStorageHandler
extends DefaultHiveMetaHook
implements HiveStorageHandler

DruidStorageHandler provides a HiveStorageHandler implementation for Druid.

Field Summary

Fields

Modifier and Type	Field and Description
protected static SessionState.LogHelper	console
static String	INTERMEDIATE_SEGMENT_DIR_NAME
protected static org.slf4j.Logger	LOG
static String	SEGMENTS_DESCRIPTOR_DIR_NAME

Fields inherited from interface org.apache.hadoop.hive.metastore.HiveMetaHook

ALTER_TABLE_OPERATION_TYPE

Constructor Summary

Constructors

Constructor and Description
DruidStorageHandler()
DruidStorageHandler(io.druid.metadata.SQLMetadataConnector connector, io.druid.metadata.MetadataStorageTablesConfig druidMetadataStorageTablesConfig)

Method Summary

Modifier and Type	Method and Description
void	commitCreateTable(org.apache.hadoop.hive.metastore.api.Table table) Called after successfully adding a new table definition to the metastore during CREATE TABLE.
void	commitDropTable(org.apache.hadoop.hive.metastore.api.Table table, boolean deleteData) Called after successfully removing a table definition from the metastore during DROP TABLE.
void	commitInsertTable(org.apache.hadoop.hive.metastore.api.Table table, boolean overwrite) Called after successfully INSERT [OVERWRITE] statement is executed.
void	configureInputJobCredentials(TableDesc tableDesc, Map<String,String> jobSecrets) This method is called to allow the StorageHandlers the chance to populate secret keys into the job's credentials.
void	configureInputJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc).
void	configureJobConf(TableDesc tableDesc, org.apache.hadoop.mapred.JobConf jobConf) Called just before submitting MapReduce job.
void	configureOutputJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc).
void	configureTableJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) Deprecated use configureInputJobProperties/configureOutputJobProperties methods instead.
protected void	deleteSegment(io.druid.timeline.DataSegment segment)
HiveAuthorizationProvider	getAuthorizationProvider() Returns the implementation specific authorization provider
org.apache.hadoop.conf.Configuration	getConf()
static com.metamx.http.client.HttpClient	getHttpClient()
Class<? extends org.apache.hadoop.mapred.InputFormat>	getInputFormatClass()
org.apache.hadoop.hive.metastore.api.LockType	getLockType(WriteEntity writeEntity)
HiveMetaHook	getMetaHook()
Class<? extends org.apache.hadoop.mapred.OutputFormat>	getOutputFormatClass()
Class<? extends AbstractSerDe>	getSerDeClass()
StorageHandlerInfo	getStorageHandlerInfo(org.apache.hadoop.hive.metastore.api.Table table) Used to fetch runtime information about storage handler during DESCRIBE EXTENDED statement
String	getUniqueId()
protected List<io.druid.timeline.DataSegment>	loadAndCommitDruidSegments(org.apache.hadoop.hive.metastore.api.Table table, boolean overwrite, List<io.druid.timeline.DataSegment> segmentsToLoad) Creates metadata moves then commit the Segment's metadata to Druid metadata store in one TxN
protected String	makeStagingName()
void	preAlterTable(org.apache.hadoop.hive.metastore.api.Table table, org.apache.hadoop.hive.metastore.api.EnvironmentContext context) Called before a table is altered in the metastore during ALTER TABLE.
void	preCreateTable(org.apache.hadoop.hive.metastore.api.Table table) Called before a new table definition is added to the metastore during CREATE TABLE.
void	preDropTable(org.apache.hadoop.hive.metastore.api.Table table) Called before a table definition is removed from the metastore during DROP TABLE.
void	preInsertTable(org.apache.hadoop.hive.metastore.api.Table table, boolean overwrite) called before commit insert method is called
void	rollbackCreateTable(org.apache.hadoop.hive.metastore.api.Table table) Called after failure adding a new table definition to the metastore during CREATE TABLE.
void	rollbackDropTable(org.apache.hadoop.hive.metastore.api.Table table) Called after failure removing a table definition from the metastore during DROP TABLE.
void	rollbackInsertTable(org.apache.hadoop.hive.metastore.api.Table table, boolean overwrite) called in case pre commit or commit insert fail.
void	setConf(org.apache.hadoop.conf.Configuration conf)
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

console

protected static final SessionState.LogHelper console

SEGMENTS_DESCRIPTOR_DIR_NAME

public static final String SEGMENTS_DESCRIPTOR_DIR_NAME

See Also:

Constant Field Values

INTERMEDIATE_SEGMENT_DIR_NAME

public static final String INTERMEDIATE_SEGMENT_DIR_NAME

See Also:

Constant Field Values

Constructor Detail

DruidStorageHandler

public DruidStorageHandler()

DruidStorageHandler

public DruidStorageHandler(io.druid.metadata.SQLMetadataConnector connector,
                           io.druid.metadata.MetadataStorageTablesConfig druidMetadataStorageTablesConfig)

Method Detail

getInputFormatClass

public Class<? extends org.apache.hadoop.mapred.InputFormat> getInputFormatClass()

Specified by:

getInputFormatClass in interface HiveStorageHandler

Returns:

Class providing an implementation of InputFormat

getOutputFormatClass

public Class<? extends org.apache.hadoop.mapred.OutputFormat> getOutputFormatClass()

Specified by:

getOutputFormatClass in interface HiveStorageHandler

Returns:

Class providing an implementation of OutputFormat

getSerDeClass

public Class<? extends AbstractSerDe> getSerDeClass()

Specified by:

getSerDeClass in interface HiveStorageHandler

Returns:

Class providing an implementation of AbstractSerDe

getMetaHook

public HiveMetaHook getMetaHook()

Specified by:

getMetaHook in interface HiveStorageHandler

Returns:

metadata hook implementation, or null if this storage handler does not need any metadata notifications

getAuthorizationProvider

public HiveAuthorizationProvider getAuthorizationProvider()

Description copied from interface: HiveStorageHandler

Returns the implementation specific authorization provider

Specified by:

getAuthorizationProvider in interface HiveStorageHandler

Returns:

authorization provider

configureInputJobProperties

public void configureInputJobProperties(TableDesc tableDesc,
                                        Map<String,String> jobProperties)

Description copied from interface: HiveStorageHandler

This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc). Key value pairs passed into jobProperties are guaranteed to be set in the job's configuration object. User's can retrieve "context" information from tableDesc. User's should avoid mutating tableDesc and only make changes in jobProperties. This method is expected to be idempotent such that a job called with the same tableDesc values should return the same key-value pairs in jobProperties. Any external state set by this method should remain the same if this method is called again. It is up to the user to determine how best guarantee this invariant. This method in particular is to create a configuration for input.

Specified by:

configureInputJobProperties in interface HiveStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureInputJobCredentials

public void configureInputJobCredentials(TableDesc tableDesc,
                                         Map<String,String> jobSecrets)

Description copied from interface: HiveStorageHandler

This method is called to allow the StorageHandlers the chance to populate secret keys into the job's credentials.

Specified by:

configureInputJobCredentials in interface HiveStorageHandler

preCreateTable

public void preCreateTable(org.apache.hadoop.hive.metastore.api.Table table)
                    throws org.apache.hadoop.hive.metastore.api.MetaException

Description copied from interface: HiveMetaHook

Called before a new table definition is added to the metastore during CREATE TABLE.

Specified by:

preCreateTable in interface HiveMetaHook

Parameters:

table - new table definition

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

rollbackCreateTable

public void rollbackCreateTable(org.apache.hadoop.hive.metastore.api.Table table)

Description copied from interface: HiveMetaHook

Called after failure adding a new table definition to the metastore during CREATE TABLE.

Specified by:

rollbackCreateTable in interface HiveMetaHook

Parameters:

table - new table definition

commitCreateTable

public void commitCreateTable(org.apache.hadoop.hive.metastore.api.Table table)
                       throws org.apache.hadoop.hive.metastore.api.MetaException

Description copied from interface: HiveMetaHook

Called after successfully adding a new table definition to the metastore during CREATE TABLE.

Specified by:

commitCreateTable in interface HiveMetaHook

Parameters:

table - new table definition

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

loadAndCommitDruidSegments

protected List<io.druid.timeline.DataSegment> loadAndCommitDruidSegments(org.apache.hadoop.hive.metastore.api.Table table,
                                                                         boolean overwrite,
                                                                         List<io.druid.timeline.DataSegment> segmentsToLoad)
                                                                  throws IOException,
                                                                         org.skife.jdbi.v2.exceptions.CallbackFailedException

Creates metadata moves then commit the Segment's metadata to Druid metadata store in one TxN

Parameters:

table - Hive table

overwrite - true if it is an insert overwrite table

Throws:

org.apache.hadoop.hive.metastore.api.MetaException - if errors occurs.

IOException

org.skife.jdbi.v2.exceptions.CallbackFailedException

deleteSegment

protected void deleteSegment(io.druid.timeline.DataSegment segment)
                      throws io.druid.segment.loading.SegmentLoadingException

Throws:

io.druid.segment.loading.SegmentLoadingException

preDropTable

public void preDropTable(org.apache.hadoop.hive.metastore.api.Table table)

Description copied from interface: HiveMetaHook

Called before a table definition is removed from the metastore during DROP TABLE.

Specified by:

preDropTable in interface HiveMetaHook

Parameters:

table - table definition

rollbackDropTable

public void rollbackDropTable(org.apache.hadoop.hive.metastore.api.Table table)

Description copied from interface: HiveMetaHook

Called after failure removing a table definition from the metastore during DROP TABLE.

Specified by:

rollbackDropTable in interface HiveMetaHook

Parameters:

table - table definition

commitDropTable

public void commitDropTable(org.apache.hadoop.hive.metastore.api.Table table,
                            boolean deleteData)

Description copied from interface: HiveMetaHook

Called after successfully removing a table definition from the metastore during DROP TABLE.

Specified by:

commitDropTable in interface HiveMetaHook

Parameters:

table - table definition

deleteData - whether to delete data as well; this should typically be ignored in the case of an external table

commitInsertTable

public void commitInsertTable(org.apache.hadoop.hive.metastore.api.Table table,
                              boolean overwrite)
                       throws org.apache.hadoop.hive.metastore.api.MetaException

Description copied from class: DefaultHiveMetaHook

Called after successfully INSERT [OVERWRITE] statement is executed.

Specified by:

commitInsertTable in class DefaultHiveMetaHook

Parameters:

table - table definition

overwrite - true if it is INSERT OVERWRITE

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

preInsertTable

public void preInsertTable(org.apache.hadoop.hive.metastore.api.Table table,
                           boolean overwrite)

Description copied from class: DefaultHiveMetaHook

called before commit insert method is called

Specified by:

preInsertTable in class DefaultHiveMetaHook

Parameters:

table - table definition

overwrite - true if it is INSERT OVERWRITE

rollbackInsertTable

public void rollbackInsertTable(org.apache.hadoop.hive.metastore.api.Table table,
                                boolean overwrite)

Description copied from class: DefaultHiveMetaHook

called in case pre commit or commit insert fail.

Specified by:

rollbackInsertTable in class DefaultHiveMetaHook

Parameters:

table - table definition

overwrite - true if it is INSERT OVERWRITE

configureOutputJobProperties

public void configureOutputJobProperties(TableDesc tableDesc,
                                         Map<String,String> jobProperties)

Description copied from interface: HiveStorageHandler

This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc). Key value pairs passed into jobProperties are guaranteed to be set in the job's configuration object. User's can retrieve "context" information from tableDesc. User's should avoid mutating tableDesc and only make changes in jobProperties. This method is expected to be idempotent such that a job called with the same tableDesc values should return the same key-value pairs in jobProperties. Any external state set by this method should remain the same if this method is called again. It is up to the user to determine how best guarantee this invariant. This method in particular is to create a configuration for output.

Specified by:

configureOutputJobProperties in interface HiveStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureTableJobProperties

public void configureTableJobProperties(TableDesc tableDesc,
                                        Map<String,String> jobProperties)

Description copied from interface: HiveStorageHandler

Deprecated use configureInputJobProperties/configureOutputJobProperties methods instead. Configures properties for a job based on the definition of the source or target table it accesses.

Specified by:

configureTableJobProperties in interface HiveStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureJobConf

public void configureJobConf(TableDesc tableDesc,
                             org.apache.hadoop.mapred.JobConf jobConf)

Description copied from interface: HiveStorageHandler

Called just before submitting MapReduce job.

Specified by:

configureJobConf in interface HiveStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobConf - jobConf for MapReduce job

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

getLockType

public org.apache.hadoop.hive.metastore.api.LockType getLockType(WriteEntity writeEntity)

Specified by:

getLockType in interface HiveStorageHandler

toString

public String toString()

Overrides:

toString in class Object

getUniqueId

public String getUniqueId()

makeStagingName

protected String makeStagingName()

getHttpClient

public static com.metamx.http.client.HttpClient getHttpClient()

preAlterTable

public void preAlterTable(org.apache.hadoop.hive.metastore.api.Table table,
                          org.apache.hadoop.hive.metastore.api.EnvironmentContext context)
                   throws org.apache.hadoop.hive.metastore.api.MetaException

Description copied from interface: HiveMetaHook

Called before a table is altered in the metastore during ALTER TABLE.

Specified by:

preAlterTable in interface HiveMetaHook

Parameters:

table - new table definition

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

getStorageHandlerInfo

public StorageHandlerInfo getStorageHandlerInfo(org.apache.hadoop.hive.metastore.api.Table table)
                                         throws org.apache.hadoop.hive.metastore.api.MetaException

Description copied from interface: HiveStorageHandler

Used to fetch runtime information about storage handler during DESCRIBE EXTENDED statement

Specified by:

getStorageHandlerInfo in interface HiveStorageHandler

Parameters:

table - table definition

Returns:

StorageHandlerInfo containing runtime information about storage handler OR `null` if the storage handler choose to not provide any runtime information.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException