org.apache.hadoop.hive.metastore.txn

Interface TxnStore

All Superinterfaces:

org.apache.hadoop.conf.Configurable

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public interface TxnStore
extends org.apache.hadoop.conf.Configurable

A handler to answer transaction related calls that come into the metastore server.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	TxnStore.MUTEX_KEY
static interface	TxnStore.MutexAPI This is primarily designed to provide coarse grained mutex support to operations running inside the Metastore (of which there could be several instances).

Field Summary

Fields

Modifier and Type	Field and Description
static String	ATTEMPTED_RESPONSE
static String	CLEANING_RESPONSE
static String	FAILED_RESPONSE
static String	INITIATED_RESPONSE
static String	SUCCEEDED_RESPONSE
static int	TIMED_OUT_TXN_ABORT_BATCH_SIZE
static String	WORKING_RESPONSE

Method Summary

Modifier and Type	Method and Description
void	abortTxn(org.apache.hadoop.hive.metastore.api.AbortTxnRequest rqst) Abort (rollback) a transaction.
void	abortTxns(org.apache.hadoop.hive.metastore.api.AbortTxnsRequest rqst) Abort (rollback) a list of transactions in one request.
void	addDynamicPartitions(org.apache.hadoop.hive.metastore.api.AddDynamicPartitions rqst) Add information on a set of dynamic partitions that participated in a transaction.
org.apache.hadoop.hive.metastore.api.AllocateTableWriteIdsResponse	allocateTableWriteIds(org.apache.hadoop.hive.metastore.api.AllocateTableWriteIdsRequest rqst) Allocate a write ID for the given table and associate it with a transaction
boolean	checkFailedCompactions(CompactionInfo ci) Determine if there are enough consecutive failures compacting a table or partition that no new automatic compactions should be scheduled.
org.apache.hadoop.hive.metastore.api.LockResponse	checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest rqst) Check whether a lock has been obtained.
void	cleanEmptyAbortedTxns() Clean up aborted transactions from txns that have no components in txn_components.
void	cleanTxnToWriteIdTable() Clean up entries from TXN_TO_WRITE_ID table less than min_uncommited_txnid as found by min(NEXT_TXN_ID.ntxn_next, min(MIN_HISTORY_LEVEL.mhl_min_open_txnid), min(Aborted TXNS.txn_id)).
long	cleanupMaterializationRebuildLocks(org.apache.hadoop.hive.common.ValidTxnList validTxnList, long timeout)
void	cleanupRecords(org.apache.hadoop.hive.metastore.api.HiveObjectType type, org.apache.hadoop.hive.metastore.api.Database db, org.apache.hadoop.hive.metastore.api.Table table, Iterator<org.apache.hadoop.hive.metastore.api.Partition> partitionIterator) Clean up corresponding records in metastore tables.
void	commitTxn(org.apache.hadoop.hive.metastore.api.CommitTxnRequest rqst) Commit a transaction
org.apache.hadoop.hive.metastore.api.CompactionResponse	compact(org.apache.hadoop.hive.metastore.api.CompactionRequest rqst) Submit a compaction request into the queue.
void	countOpenTxns() Get the count for open transactions.
List<String>	findColumnsWithStats(CompactionInfo ci) Queries metastore DB directly to find columns in the table which have statistics information.
CompactionInfo	findNextToCompact(String workerId) This will grab the next compaction request off of the queue, and assign it to the worker.
Set<CompactionInfo>	findPotentialCompactions(int maxAborted) This will look through the completed_txn_components table and look for partitions or tables that may be ready for compaction.
List<CompactionInfo>	findReadyToClean() Find entries in the queue that are ready to be cleaned.
org.apache.hadoop.hive.metastore.api.Materialization	getMaterializationInvalidationInfo(org.apache.hadoop.hive.metastore.api.CreationMetadata cm, String validTxnList) Get invalidation info for the materialization.
TxnStore.MutexAPI	getMutexAPI()
org.apache.hadoop.hive.metastore.api.GetOpenTxnsResponse	getOpenTxns() Get list of valid transactions.
org.apache.hadoop.hive.metastore.api.GetOpenTxnsInfoResponse	getOpenTxnsInfo() Get information about open transactions.
org.apache.hadoop.hive.metastore.api.GetValidWriteIdsResponse	getValidWriteIds(org.apache.hadoop.hive.metastore.api.GetValidWriteIdsRequest rqst) Gets the list of valid write ids for the given table wrt to current txn
void	heartbeat(org.apache.hadoop.hive.metastore.api.HeartbeatRequest ids) Send a heartbeat for a lock or a transaction
boolean	heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
org.apache.hadoop.hive.metastore.api.HeartbeatTxnRangeResponse	heartbeatTxnRange(org.apache.hadoop.hive.metastore.api.HeartbeatTxnRangeRequest rqst) Heartbeat a group of transactions together
org.apache.hadoop.hive.metastore.api.LockResponse	lock(org.apache.hadoop.hive.metastore.api.LockRequest rqst) Obtain a lock.
org.apache.hadoop.hive.metastore.api.LockResponse	lockMaterializationRebuild(String dbName, String tableName, long txnId)
void	markCleaned(CompactionInfo info) This will remove an entry from the queue after it has been compacted.
void	markCompacted(CompactionInfo info) This will mark an entry in the queue as compacted and put it in the ready to clean state.
void	markFailed(CompactionInfo info) Mark a compaction entry as failed.
int	numLocksInLockTable()
void	onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName, String newCatName, String newDbName, String newTabName, String newPartName)
org.apache.hadoop.hive.metastore.api.OpenTxnsResponse	openTxns(org.apache.hadoop.hive.metastore.api.OpenTxnRequest rqst) Open a set of transactions
void	performTimeOuts() Timeout transactions and/or locks.
void	performWriteSetGC() WriteSet tracking is used to ensure proper transaction isolation.
void	purgeCompactionHistory() For any given compactable entity (partition, table if not partitioned) the history of compactions may look like "sssfffaaasffss", for example.
void	replTableWriteIdState(org.apache.hadoop.hive.metastore.api.ReplTblWriteIdStateRequest rqst) Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
void	revokeFromLocalWorkers(String hostname) This will take all entries assigned to workers on a host return them to INITIATED state.
void	revokeTimedoutWorkers(long timeout) This call will return all compaction queue entries assigned to a worker but over the timeout back to the initiated state.
void	seedWriteIdOnAcidConversion(InitializeTableWriteIdsRequest rqst) Called on conversion of existing table to full acid.
void	setCompactionHighestWriteId(CompactionInfo ci, long highestWriteId) Record the highest write id that the ci compaction job will pay attention to.
void	setHadoopJobId(String hadoopJobId, long id) Once a ThreadPoolExecutor Worker submits a job to the cluster, it calls this to update the metadata.
void	setRunAs(long cq_id, String user) Sets the user to run as.
long	setTimeout(long milliseconds)
org.apache.hadoop.hive.metastore.api.ShowCompactResponse	showCompact(org.apache.hadoop.hive.metastore.api.ShowCompactRequest rqst) Show list of current compactions.
org.apache.hadoop.hive.metastore.api.ShowLocksResponse	showLocks(org.apache.hadoop.hive.metastore.api.ShowLocksRequest rqst) Get information on current locks.
void	unlock(org.apache.hadoop.hive.metastore.api.UnlockRequest rqst) Unlock a lock.

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

INITIATED_RESPONSE

static final String INITIATED_RESPONSE

See Also:

Constant Field Values

WORKING_RESPONSE

static final String WORKING_RESPONSE

See Also:

Constant Field Values

CLEANING_RESPONSE

static final String CLEANING_RESPONSE

See Also:

Constant Field Values

FAILED_RESPONSE

static final String FAILED_RESPONSE

See Also:

Constant Field Values

SUCCEEDED_RESPONSE

static final String SUCCEEDED_RESPONSE

See Also:

Constant Field Values

ATTEMPTED_RESPONSE

static final String ATTEMPTED_RESPONSE

See Also:

Constant Field Values

TIMED_OUT_TXN_ABORT_BATCH_SIZE

static final int TIMED_OUT_TXN_ABORT_BATCH_SIZE

See Also:

Constant Field Values

Method Detail

getOpenTxnsInfo

org.apache.hadoop.hive.metastore.api.GetOpenTxnsInfoResponse getOpenTxnsInfo()
                                                                      throws org.apache.hadoop.hive.metastore.api.MetaException

Get information about open transactions. This gives extensive information about the transactions rather than just the list of transactions. This should be used when the need is to see information about the transactions (e.g. show transactions).

Returns:

information about open transactions

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

getOpenTxns

org.apache.hadoop.hive.metastore.api.GetOpenTxnsResponse getOpenTxns()
                                                              throws org.apache.hadoop.hive.metastore.api.MetaException

Get list of valid transactions. This gives just the list of transactions that are open.

Returns:

list of open transactions, as well as a high water mark.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

countOpenTxns

void countOpenTxns()
            throws org.apache.hadoop.hive.metastore.api.MetaException

Get the count for open transactions.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

openTxns

org.apache.hadoop.hive.metastore.api.OpenTxnsResponse openTxns(org.apache.hadoop.hive.metastore.api.OpenTxnRequest rqst)
                                                        throws org.apache.hadoop.hive.metastore.api.MetaException

Open a set of transactions

Parameters:

rqst - request to open transactions

Returns:

information on opened transactions

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

abortTxn

void abortTxn(org.apache.hadoop.hive.metastore.api.AbortTxnRequest rqst)
       throws org.apache.hadoop.hive.metastore.api.NoSuchTxnException,
              org.apache.hadoop.hive.metastore.api.MetaException,
              org.apache.hadoop.hive.metastore.api.TxnAbortedException

Abort (rollback) a transaction.

Parameters:

rqst - info on transaction to abort

Throws:

org.apache.hadoop.hive.metastore.api.NoSuchTxnException

org.apache.hadoop.hive.metastore.api.MetaException

org.apache.hadoop.hive.metastore.api.TxnAbortedException

abortTxns

void abortTxns(org.apache.hadoop.hive.metastore.api.AbortTxnsRequest rqst)
        throws org.apache.hadoop.hive.metastore.api.NoSuchTxnException,
               org.apache.hadoop.hive.metastore.api.MetaException

Abort (rollback) a list of transactions in one request.

Parameters:

rqst - info on transactions to abort

Throws:

org.apache.hadoop.hive.metastore.api.NoSuchTxnException

org.apache.hadoop.hive.metastore.api.MetaException

commitTxn

void commitTxn(org.apache.hadoop.hive.metastore.api.CommitTxnRequest rqst)
        throws org.apache.hadoop.hive.metastore.api.NoSuchTxnException,
               org.apache.hadoop.hive.metastore.api.TxnAbortedException,
               org.apache.hadoop.hive.metastore.api.MetaException

Commit a transaction

Parameters:

rqst - info on transaction to commit

Throws:

org.apache.hadoop.hive.metastore.api.NoSuchTxnException

org.apache.hadoop.hive.metastore.api.TxnAbortedException

org.apache.hadoop.hive.metastore.api.MetaException

replTableWriteIdState

void replTableWriteIdState(org.apache.hadoop.hive.metastore.api.ReplTblWriteIdStateRequest rqst)
                    throws org.apache.hadoop.hive.metastore.api.MetaException

Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.

Parameters:

rqst - info on table/partitions and writeid snapshot to replicate.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException - in case of failure

getMaterializationInvalidationInfo

org.apache.hadoop.hive.metastore.api.Materialization getMaterializationInvalidationInfo(org.apache.hadoop.hive.metastore.api.CreationMetadata cm,
                                                                                        String validTxnList)
                                                                                 throws org.apache.hadoop.hive.metastore.api.MetaException

Get invalidation info for the materialization. Currently, the materialization information only contains information about whether there was update/delete operations on the source tables used by the materialization since it was created.

Parameters:

cm - creation metadata for the materialization

validTxnList - valid transaction list for snapshot taken for current query

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

lockMaterializationRebuild

org.apache.hadoop.hive.metastore.api.LockResponse lockMaterializationRebuild(String dbName,
                                                                             String tableName,
                                                                             long txnId)
                                                                      throws org.apache.hadoop.hive.metastore.api.MetaException

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

heartbeatLockMaterializationRebuild

boolean heartbeatLockMaterializationRebuild(String dbName,
                                            String tableName,
                                            long txnId)
                                     throws org.apache.hadoop.hive.metastore.api.MetaException

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

cleanupMaterializationRebuildLocks

long cleanupMaterializationRebuildLocks(org.apache.hadoop.hive.common.ValidTxnList validTxnList,
                                        long timeout)
                                 throws org.apache.hadoop.hive.metastore.api.MetaException

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

getValidWriteIds

org.apache.hadoop.hive.metastore.api.GetValidWriteIdsResponse getValidWriteIds(org.apache.hadoop.hive.metastore.api.GetValidWriteIdsRequest rqst)
                                                                        throws org.apache.hadoop.hive.metastore.api.NoSuchTxnException,
                                                                               org.apache.hadoop.hive.metastore.api.MetaException

Gets the list of valid write ids for the given table wrt to current txn

Parameters:

rqst - info on transaction and list of table names associated with given transaction

Throws:

org.apache.hadoop.hive.metastore.api.NoSuchTxnException

org.apache.hadoop.hive.metastore.api.MetaException

allocateTableWriteIds

org.apache.hadoop.hive.metastore.api.AllocateTableWriteIdsResponse allocateTableWriteIds(org.apache.hadoop.hive.metastore.api.AllocateTableWriteIdsRequest rqst)
                                                                                  throws org.apache.hadoop.hive.metastore.api.NoSuchTxnException,
                                                                                         org.apache.hadoop.hive.metastore.api.TxnAbortedException,
                                                                                         org.apache.hadoop.hive.metastore.api.MetaException

Allocate a write ID for the given table and associate it with a transaction

Parameters:

rqst - info on transaction and table to allocate write id

Throws:

org.apache.hadoop.hive.metastore.api.NoSuchTxnException

org.apache.hadoop.hive.metastore.api.TxnAbortedException

org.apache.hadoop.hive.metastore.api.MetaException

seedWriteIdOnAcidConversion

void seedWriteIdOnAcidConversion(InitializeTableWriteIdsRequest rqst)
                          throws org.apache.hadoop.hive.metastore.api.MetaException

Called on conversion of existing table to full acid. Sets initial write ID to a high enough value so that we can assign unique ROW__IDs to data in existing files.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

lock

org.apache.hadoop.hive.metastore.api.LockResponse lock(org.apache.hadoop.hive.metastore.api.LockRequest rqst)
                                                throws org.apache.hadoop.hive.metastore.api.NoSuchTxnException,
                                                       org.apache.hadoop.hive.metastore.api.TxnAbortedException,
                                                       org.apache.hadoop.hive.metastore.api.MetaException

Obtain a lock.

Parameters:

rqst - information on the lock to obtain. If the requester is part of a transaction the txn information must be included in the lock request.

Returns:

info on the lock, including whether it was obtained.

Throws:

org.apache.hadoop.hive.metastore.api.NoSuchTxnException

org.apache.hadoop.hive.metastore.api.TxnAbortedException

org.apache.hadoop.hive.metastore.api.MetaException

checkLock

org.apache.hadoop.hive.metastore.api.LockResponse checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest rqst)
                                                     throws org.apache.hadoop.hive.metastore.api.NoSuchTxnException,
                                                            org.apache.hadoop.hive.metastore.api.NoSuchLockException,
                                                            org.apache.hadoop.hive.metastore.api.TxnAbortedException,
                                                            org.apache.hadoop.hive.metastore.api.MetaException

Check whether a lock has been obtained. This is used after lock(org.apache.hadoop.hive.metastore.api.LockRequest) returned a wait state.

Parameters:

rqst - info on the lock to check

Returns:

info on the state of the lock

Throws:

org.apache.hadoop.hive.metastore.api.NoSuchTxnException

org.apache.hadoop.hive.metastore.api.NoSuchLockException

org.apache.hadoop.hive.metastore.api.TxnAbortedException

org.apache.hadoop.hive.metastore.api.MetaException

unlock

void unlock(org.apache.hadoop.hive.metastore.api.UnlockRequest rqst)
     throws org.apache.hadoop.hive.metastore.api.NoSuchLockException,
            org.apache.hadoop.hive.metastore.api.TxnOpenException,
            org.apache.hadoop.hive.metastore.api.MetaException

Unlock a lock. It is not legal to call this if the caller is part of a txn. In that case the txn should be committed or aborted instead. (Note someday this will change since multi-statement transactions will allow unlocking in the transaction.)

Parameters:

rqst - lock to unlock

Throws:

org.apache.hadoop.hive.metastore.api.NoSuchLockException

org.apache.hadoop.hive.metastore.api.TxnOpenException

org.apache.hadoop.hive.metastore.api.MetaException

showLocks

org.apache.hadoop.hive.metastore.api.ShowLocksResponse showLocks(org.apache.hadoop.hive.metastore.api.ShowLocksRequest rqst)
                                                          throws org.apache.hadoop.hive.metastore.api.MetaException

Get information on current locks.

Parameters:

rqst - lock information to retrieve

Returns:

lock information.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

heartbeat

void heartbeat(org.apache.hadoop.hive.metastore.api.HeartbeatRequest ids)
        throws org.apache.hadoop.hive.metastore.api.NoSuchTxnException,
               org.apache.hadoop.hive.metastore.api.NoSuchLockException,
               org.apache.hadoop.hive.metastore.api.TxnAbortedException,
               org.apache.hadoop.hive.metastore.api.MetaException

Send a heartbeat for a lock or a transaction

Parameters:

ids - lock and/or txn id to heartbeat

Throws:

org.apache.hadoop.hive.metastore.api.NoSuchTxnException

org.apache.hadoop.hive.metastore.api.NoSuchLockException

org.apache.hadoop.hive.metastore.api.TxnAbortedException

org.apache.hadoop.hive.metastore.api.MetaException

heartbeatTxnRange

org.apache.hadoop.hive.metastore.api.HeartbeatTxnRangeResponse heartbeatTxnRange(org.apache.hadoop.hive.metastore.api.HeartbeatTxnRangeRequest rqst)
                                                                          throws org.apache.hadoop.hive.metastore.api.MetaException

Heartbeat a group of transactions together

Parameters:

rqst - set of transactions to heartbat

Returns:

info on txns that were heartbeated

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

compact

org.apache.hadoop.hive.metastore.api.CompactionResponse compact(org.apache.hadoop.hive.metastore.api.CompactionRequest rqst)
                                                         throws org.apache.hadoop.hive.metastore.api.MetaException

Submit a compaction request into the queue. This is called when a user manually requests a compaction.

Parameters:

rqst - information on what to compact

Returns:

id of the compaction that has been started or existing id if this resource is already scheduled

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

showCompact

org.apache.hadoop.hive.metastore.api.ShowCompactResponse showCompact(org.apache.hadoop.hive.metastore.api.ShowCompactRequest rqst)
                                                              throws org.apache.hadoop.hive.metastore.api.MetaException

Show list of current compactions.

Parameters:

rqst - info on which compactions to show

Returns:

compaction information

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

addDynamicPartitions

void addDynamicPartitions(org.apache.hadoop.hive.metastore.api.AddDynamicPartitions rqst)
                   throws org.apache.hadoop.hive.metastore.api.NoSuchTxnException,
                          org.apache.hadoop.hive.metastore.api.TxnAbortedException,
                          org.apache.hadoop.hive.metastore.api.MetaException

Add information on a set of dynamic partitions that participated in a transaction.

Parameters:

rqst - dynamic partition info.

Throws:

org.apache.hadoop.hive.metastore.api.NoSuchTxnException

org.apache.hadoop.hive.metastore.api.TxnAbortedException

org.apache.hadoop.hive.metastore.api.MetaException

cleanupRecords

void cleanupRecords(org.apache.hadoop.hive.metastore.api.HiveObjectType type,
                    org.apache.hadoop.hive.metastore.api.Database db,
                    org.apache.hadoop.hive.metastore.api.Table table,
                    Iterator<org.apache.hadoop.hive.metastore.api.Partition> partitionIterator)
             throws org.apache.hadoop.hive.metastore.api.MetaException

Clean up corresponding records in metastore tables.

Parameters:

type - Hive object type

db - database object

table - table object

partitionIterator - partition iterator

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

onRename

void onRename(String oldCatName,
              String oldDbName,
              String oldTabName,
              String oldPartName,
              String newCatName,
              String newDbName,
              String newTabName,
              String newPartName)
       throws org.apache.hadoop.hive.metastore.api.MetaException

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

performTimeOuts

void performTimeOuts()

Timeout transactions and/or locks. This should only be called by the compactor.

findPotentialCompactions

Set<CompactionInfo> findPotentialCompactions(int maxAborted)
                                      throws org.apache.hadoop.hive.metastore.api.MetaException

This will look through the completed_txn_components table and look for partitions or tables that may be ready for compaction. Also, look through txns and txn_components tables for aborted transactions that we should add to the list.

Parameters:

maxAborted - Maximum number of aborted queries to allow before marking this as a potential compaction.

Returns:

list of CompactionInfo structs. These will not have id, type, or runAs set since these are only potential compactions not actual ones.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

setRunAs

void setRunAs(long cq_id,
              String user)
       throws org.apache.hadoop.hive.metastore.api.MetaException

Sets the user to run as. This is for the case where the request was generated by the user and so the worker must set this value later.

Parameters:

cq_id - id of this entry in the queue

user - user to run the jobs as

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

findNextToCompact

CompactionInfo findNextToCompact(String workerId)
                          throws org.apache.hadoop.hive.metastore.api.MetaException

This will grab the next compaction request off of the queue, and assign it to the worker.

Parameters:

workerId - id of the worker calling this, will be recorded in the db

Returns:

an info element for this compaction request, or null if there is no work to do now.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

markCompacted

void markCompacted(CompactionInfo info)
            throws org.apache.hadoop.hive.metastore.api.MetaException

This will mark an entry in the queue as compacted and put it in the ready to clean state.

Parameters:

info - info on the compaction entry to mark as compacted.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

findReadyToClean

List<CompactionInfo> findReadyToClean()
                               throws org.apache.hadoop.hive.metastore.api.MetaException

Find entries in the queue that are ready to be cleaned.

Returns:

information on the entry in the queue.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

markCleaned

void markCleaned(CompactionInfo info)
          throws org.apache.hadoop.hive.metastore.api.MetaException

This will remove an entry from the queue after it has been compacted.

Parameters:

info - info on the compaction entry to remove

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

markFailed

void markFailed(CompactionInfo info)
         throws org.apache.hadoop.hive.metastore.api.MetaException

Mark a compaction entry as failed. This will move it to the compaction history queue with a failed status. It will NOT clean up aborted transactions in the table/partition associated with this compaction.

Parameters:

info - information on the compaction that failed.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

cleanTxnToWriteIdTable

void cleanTxnToWriteIdTable()
                     throws org.apache.hadoop.hive.metastore.api.MetaException

Clean up entries from TXN_TO_WRITE_ID table less than min_uncommited_txnid as found by min(NEXT_TXN_ID.ntxn_next, min(MIN_HISTORY_LEVEL.mhl_min_open_txnid), min(Aborted TXNS.txn_id)).

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

cleanEmptyAbortedTxns

void cleanEmptyAbortedTxns()
                    throws org.apache.hadoop.hive.metastore.api.MetaException

Clean up aborted transactions from txns that have no components in txn_components. The reson such txns exist can be that now work was done in this txn (e.g. Streaming opened TransactionBatch and abandoned it w/o doing any work) or due to markCleaned(CompactionInfo) being called.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

revokeFromLocalWorkers

void revokeFromLocalWorkers(String hostname)
                     throws org.apache.hadoop.hive.metastore.api.MetaException

This will take all entries assigned to workers on a host return them to INITIATED state. The initiator should use this at start up to clean entries from any workers that were in the middle of compacting when the metastore shutdown. It does not reset entries from worker threads on other hosts as those may still be working.

Parameters:

hostname - Name of this host. It is assumed this prefixes the thread's worker id, so that like hostname% will match the worker id.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

revokeTimedoutWorkers

void revokeTimedoutWorkers(long timeout)
                    throws org.apache.hadoop.hive.metastore.api.MetaException

This call will return all compaction queue entries assigned to a worker but over the timeout back to the initiated state. This should be called by the initiator on start up and occasionally when running to clean up after dead threads. At start up revokeFromLocalWorkers(String) should be called first.

Parameters:

timeout - number of milliseconds since start time that should elapse before a worker is declared dead.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

findColumnsWithStats

List<String> findColumnsWithStats(CompactionInfo ci)
                           throws org.apache.hadoop.hive.metastore.api.MetaException

Queries metastore DB directly to find columns in the table which have statistics information. If ci includes partition info then per partition stats info is examined, otherwise table level stats are examined.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

setCompactionHighestWriteId

void setCompactionHighestWriteId(CompactionInfo ci,
                                 long highestWriteId)
                          throws org.apache.hadoop.hive.metastore.api.MetaException

Record the highest write id that the ci compaction job will pay attention to.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

purgeCompactionHistory

void purgeCompactionHistory()
                     throws org.apache.hadoop.hive.metastore.api.MetaException

For any given compactable entity (partition, table if not partitioned) the history of compactions may look like "sssfffaaasffss", for example. The idea is to retain the tail (most recent) of the history such that a configurable number of each type of state is present. Any other entries can be purged. This scheme has advantage of always retaining the last failure/success even if it's not recent.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

performWriteSetGC

void performWriteSetGC()

WriteSet tracking is used to ensure proper transaction isolation. This method deletes the transaction metadata once it becomes unnecessary.

checkFailedCompactions

boolean checkFailedCompactions(CompactionInfo ci)
                        throws org.apache.hadoop.hive.metastore.api.MetaException

Determine if there are enough consecutive failures compacting a table or partition that no new automatic compactions should be scheduled. User initiated compactions do not do this check.

Parameters:

ci - Table or partition to check.

Returns:

true if it is ok to compact, false if there have been too many failures.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

numLocksInLockTable

int numLocksInLockTable()
                 throws SQLException,
                        org.apache.hadoop.hive.metastore.api.MetaException

Throws:

SQLException

org.apache.hadoop.hive.metastore.api.MetaException

setTimeout

long setTimeout(long milliseconds)

getMutexAPI

TxnStore.MutexAPI getMutexAPI()

setHadoopJobId

void setHadoopJobId(String hadoopJobId,
                    long id)

Once a ThreadPoolExecutor Worker submits a job to the cluster, it calls this to update the metadata.

Parameters:

id - CompactionInfo.id