org.apache.hadoop.hive.metastore.txn

Class TxnUtils

java.lang.Object

org.apache.hadoop.hive.metastore.txn.TxnUtils

public class TxnUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
TxnUtils()

Method Summary

Modifier and Type	Method and Description
static List<Integer>	buildQueryWithINClause(org.apache.hadoop.conf.Configuration conf, List<String> queries, StringBuilder prefix, StringBuilder suffix, List<Long> inList, String inColumn, boolean addParens, boolean notIn) Build a query (or queries if one query is too big but only for the case of 'IN' composite clause.
static List<Integer>	buildQueryWithINClauseStrings(org.apache.hadoop.conf.Configuration conf, List<String> queries, StringBuilder prefix, StringBuilder suffix, List<String> inList, String inColumn, boolean addParens, boolean notIn) Build a query (or queries if one query is too big but only for the case of 'IN' composite clause.
static org.apache.hadoop.hive.common.ValidCompactorWriteIdList	createValidCompactWriteIdList(org.apache.hadoop.hive.metastore.api.TableValidWriteIds tableValidWriteIds) Transform a TableValidWriteIds to a ValidCompactorWriteIdList.
static org.apache.hadoop.hive.common.ValidReaderWriteIdList	createValidReaderWriteIdList(org.apache.hadoop.hive.metastore.api.TableValidWriteIds tableWriteIds) Transform a TableValidWriteIds to a ValidReaderWriteIdList.
static org.apache.hadoop.hive.common.ValidTxnList	createValidReadTxnList(org.apache.hadoop.hive.metastore.api.GetOpenTxnsResponse txns, long currentTxn) Transform a GetOpenTxnsResponse to a ValidTxnList.
static org.apache.hadoop.hive.common.ValidTxnWriteIdList	createValidTxnWriteIdList(Long currentTxnId, List<org.apache.hadoop.hive.metastore.api.TableValidWriteIds> validIds) Transform a GetValidWriteIdsResponse to a ValidTxnWriteIdList.
static String[]	getDbTableName(String fullTableName)
static String	getFullTableName(String dbName, String tableName) Should produce the result as ..
static TxnStore	getTxnStore(org.apache.hadoop.conf.Configuration conf) Get an instance of the TxnStore that is appropriate for this store
static boolean	isAcidTable(org.apache.hadoop.hive.metastore.api.Table table) Should produce the same result as org.apache.hadoop.hive.ql.io.AcidUtils#isAcidTable(org.apache.hadoop.hive.ql.metadata.Table).
static boolean	isTransactionalTable(org.apache.hadoop.hive.metastore.api.Table table) Note, users are responsible for using the correct TxnManager.
static org.apache.hadoop.hive.common.ValidReaderWriteIdList	updateForCompactionQuery(org.apache.hadoop.hive.common.ValidReaderWriteIdList ids)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TxnUtils

public TxnUtils()

Method Detail

createValidReadTxnList

public static org.apache.hadoop.hive.common.ValidTxnList createValidReadTxnList(org.apache.hadoop.hive.metastore.api.GetOpenTxnsResponse txns,
                                                                                long currentTxn)

Transform a GetOpenTxnsResponse to a ValidTxnList. This assumes that the caller intends to read the files, and thus treats both open and aborted transactions as invalid.

Parameters:

txns - txn list from the metastore

currentTxn - Current transaction that the user has open. If this is greater than 0 it will be removed from the exceptions list so that the user sees his own transaction as valid.

Returns:

a valid txn list.

createValidTxnWriteIdList

public static org.apache.hadoop.hive.common.ValidTxnWriteIdList createValidTxnWriteIdList(Long currentTxnId,
                                                                                          List<org.apache.hadoop.hive.metastore.api.TableValidWriteIds> validIds)

Transform a GetValidWriteIdsResponse to a ValidTxnWriteIdList. This assumes that the caller intends to read the files, and thus treats both open and aborted transactions as invalid.

Parameters:

currentTxnId - current txn ID for which we get the valid write ids list

list - valid write ids list from the metastore

Returns:

a valid write IDs list for the whole transaction.

createValidReaderWriteIdList

public static org.apache.hadoop.hive.common.ValidReaderWriteIdList createValidReaderWriteIdList(org.apache.hadoop.hive.metastore.api.TableValidWriteIds tableWriteIds)

Transform a TableValidWriteIds to a ValidReaderWriteIdList. This assumes that the caller intends to read the files, and thus treats both open and aborted write ids as invalid.

Parameters:

tableWriteIds - valid write ids for the given table from the metastore

Returns:

a valid write IDs list for the input table

createValidCompactWriteIdList

public static org.apache.hadoop.hive.common.ValidCompactorWriteIdList createValidCompactWriteIdList(org.apache.hadoop.hive.metastore.api.TableValidWriteIds tableValidWriteIds)

Transform a TableValidWriteIds to a ValidCompactorWriteIdList. This assumes that the caller intends to compact the files, and thus treats only open transactions/write ids as invalid. Additionally any writeId > highestOpenWriteId is also invalid. This is to avoid creating something like delta_17_120 where writeId 80, for example, is still open.

Parameters:

tableValidWriteIds - table write id list from the metastore

Returns:

a valid write id list.

updateForCompactionQuery

public static org.apache.hadoop.hive.common.ValidReaderWriteIdList updateForCompactionQuery(org.apache.hadoop.hive.common.ValidReaderWriteIdList ids)

getTxnStore

public static TxnStore getTxnStore(org.apache.hadoop.conf.Configuration conf)

Get an instance of the TxnStore that is appropriate for this store

Parameters:

conf - configuration

Returns:

txn store

isTransactionalTable

public static boolean isTransactionalTable(org.apache.hadoop.hive.metastore.api.Table table)

Note, users are responsible for using the correct TxnManager. We do not look at SessionState.get().getTxnMgr().supportsAcid() here Should produce the same result as AcidUtils.isTransactionalTable(org.apache.hadoop.hive.ql.metadata.Table).

Returns:

true if table is a transactional table, false otherwise

isAcidTable

public static boolean isAcidTable(org.apache.hadoop.hive.metastore.api.Table table)

Should produce the same result as org.apache.hadoop.hive.ql.io.AcidUtils#isAcidTable(org.apache.hadoop.hive.ql.metadata.Table).

getFullTableName

public static String getFullTableName(String dbName,
                                      String tableName)

Should produce the result as ..

getDbTableName

public static String[] getDbTableName(String fullTableName)

buildQueryWithINClause

public static List<Integer> buildQueryWithINClause(org.apache.hadoop.conf.Configuration conf,
                                                   List<String> queries,
                                                   StringBuilder prefix,
                                                   StringBuilder suffix,
                                                   List<Long> inList,
                                                   String inColumn,
                                                   boolean addParens,
                                                   boolean notIn)

Build a query (or queries if one query is too big but only for the case of 'IN' composite clause. For the case of 'NOT IN' clauses, multiple queries change the semantics of the intended query. E.g., Let's assume that input "inList" parameter has [5, 6] and that _DIRECT_SQL_MAX_QUERY_LENGTH_ configuration parameter only allows one value in a 'NOT IN' clause, Then having two delete statements changes the semantics of the inteneded SQL statement. I.e. 'delete from T where a not in (5)' and 'delete from T where a not in (6)' sequence is not equal to 'delete from T where a not in (5, 6)'.) with one or multiple 'IN' or 'NOT IN' clauses with the given input parameters. Note that this method currently support only single column for IN/NOT IN clauses and that only covers OR-based composite 'IN' clause and AND-based composite 'NOT IN' clause. For example, for 'IN' clause case, the method will build a query with OR. E.g., "id in (1,2,3) OR id in (4,5,6)". For 'NOT IN' case, NOT IN list is broken into multiple 'NOT IN" clauses connected by AND. Note that, in this method, "a composite 'IN' clause" is defined as "a list of multiple 'IN' clauses in a query".

Parameters:

queries - OUT: Array of query strings

prefix - IN: Part of the query that comes before IN list

suffix - IN: Part of the query that comes after IN list

inList - IN: the list with IN list values

inColumn - IN: single column name of IN list operator

addParens - IN: add a pair of parenthesis outside the IN lists e.g. "(id in (1,2,3) OR id in (4,5,6))"

notIn - IN: is this for building a 'NOT IN' composite clause?

Returns:

OUT: a list of the count of IN list values that are in each of the corresponding queries

buildQueryWithINClauseStrings

public static List<Integer> buildQueryWithINClauseStrings(org.apache.hadoop.conf.Configuration conf,
                                                          List<String> queries,
                                                          StringBuilder prefix,
                                                          StringBuilder suffix,
                                                          List<String> inList,
                                                          String inColumn,
                                                          boolean addParens,
                                                          boolean notIn)

Build a query (or queries if one query is too big but only for the case of 'IN' composite clause. For the case of 'NOT IN' clauses, multiple queries change the semantics of the intended query. E.g., Let's assume that input "inList" parameter has [5, 6] and that _DIRECT_SQL_MAX_QUERY_LENGTH_ configuration parameter only allows one value in a 'NOT IN' clause, Then having two delete statements changes the semantics of the inteneded SQL statement. I.e. 'delete from T where a not in (5)' and 'delete from T where a not in (6)' sequence is not equal to 'delete from T where a not in (5, 6)'.) with one or multiple 'IN' or 'NOT IN' clauses with the given input parameters. Note that this method currently support only single column for IN/NOT IN clauses and that only covers OR-based composite 'IN' clause and AND-based composite 'NOT IN' clause. For example, for 'IN' clause case, the method will build a query with OR. E.g., "id in (1,2,3) OR id in (4,5,6)". For 'NOT IN' case, NOT IN list is broken into multiple 'NOT IN" clauses connected by AND. Note that, in this method, "a composite 'IN' clause" is defined as "a list of multiple 'IN' clauses in a query".

Parameters:

queries - OUT: Array of query strings

prefix - IN: Part of the query that comes before IN list

suffix - IN: Part of the query that comes after IN list

inList - IN: the list with IN list values

inColumn - IN: single column name of IN list operator

addParens - IN: add a pair of parenthesis outside the IN lists e.g. "(id in (1,2,3) OR id in (4,5,6))"

notIn - IN: is this for building a 'NOT IN' composite clause?

Returns:

OUT: a list of the count of IN list values that are in each of the corresponding queries