org.apache.hadoop.hive.ql.txn.compactor

Class Cleaner

java.lang.Object

java.lang.Thread

org.apache.hadoop.hive.ql.txn.compactor.Cleaner

All Implemented Interfaces:

Runnable, org.apache.hadoop.conf.Configurable, MetaStoreThread

public class Cleaner
extends Thread

A class to clean directories after compactions. This will run in a separate thread.

Nested Class Summary

Nested classes/interfaces inherited from class java.lang.Thread

Thread.State, Thread.UncaughtExceptionHandler

Field Summary

Fields

Modifier and Type	Field and Description
protected HiveConf	conf
protected AtomicBoolean	looped
protected RawStore	rs
protected AtomicBoolean	stop
protected int	threadId
protected TxnStore	txnHandler

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor and Description
Cleaner()

Method Summary

Modifier and Type	Method and Description
protected String	findUserToRunAs(String location, org.apache.hadoop.hive.metastore.api.Table t) Determine which user to run an operation as, based on the owner of the directory to be compacted.
org.apache.hadoop.conf.Configuration	getConf()
void	init(AtomicBoolean stop, AtomicBoolean looped) Initialize the thread.
protected org.apache.hadoop.hive.metastore.api.Partition	resolvePartition(CompactionInfo ci) Get the partition being compacted.
protected org.apache.hadoop.hive.metastore.api.StorageDescriptor	resolveStorageDescriptor(org.apache.hadoop.hive.metastore.api.Table t, org.apache.hadoop.hive.metastore.api.Partition p) Get the storage descriptor for a compaction.
protected org.apache.hadoop.hive.metastore.api.Table	resolveTable(CompactionInfo ci) Find the table being compacted
void	run()
protected boolean	runJobAsSelf(String owner) Determine whether to run this job as the current user or whether we need a doAs to switch users.
void	setConf(org.apache.hadoop.conf.Configuration configuration)
void	setThreadId(int threadId) Set the id for this thread.
protected String	tableName(org.apache.hadoop.hive.metastore.api.Table t)

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.metastore.MetaStoreThread

start

Field Detail

conf

protected HiveConf conf

txnHandler

protected TxnStore txnHandler

rs

protected RawStore rs

threadId

protected int threadId

stop

protected AtomicBoolean stop

looped

protected AtomicBoolean looped

Constructor Detail

Cleaner

public Cleaner()

Method Detail

init

public void init(AtomicBoolean stop,
                 AtomicBoolean looped)
          throws org.apache.hadoop.hive.metastore.api.MetaException

Description copied from interface: MetaStoreThread

Initialize the thread. This must not be called until after Configurable.setConf(Configuration) and MetaStoreThread.setThreadId(int) have been called.

Specified by:

init in interface MetaStoreThread

Parameters:

stop - a flag to watch for when to stop. If this value is set to true, the thread will terminate the next time through its main loop.

looped - a flag that is set to true everytime a thread goes through it's main loop. This is purely for testing so that tests can assure themselves that the thread has run through it's loop once. The test can set this value to false. The thread should then assure that the loop has been gone completely through at least once.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

run

public void run()

Specified by:

run in interface Runnable

Overrides:

run in class Thread

setConf

public void setConf(org.apache.hadoop.conf.Configuration configuration)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setThreadId

public void setThreadId(int threadId)

Description copied from interface: MetaStoreThread

Set the id for this thread.

Specified by:

setThreadId in interface MetaStoreThread

Parameters:

threadId - id of the thread

resolveTable

protected org.apache.hadoop.hive.metastore.api.Table resolveTable(CompactionInfo ci)
                                                           throws org.apache.hadoop.hive.metastore.api.MetaException

Find the table being compacted

Parameters:

ci - compaction info returned from the compaction queue

Returns:

metastore table

Throws:

org.apache.hadoop.hive.metastore.api.MetaException - if the table cannot be found.

resolvePartition

protected org.apache.hadoop.hive.metastore.api.Partition resolvePartition(CompactionInfo ci)
                                                                   throws Exception

Get the partition being compacted.

Parameters:

ci - compaction info returned from the compaction queue

Returns:

metastore partition, or null if there is not partition in this compaction info

Throws:

Exception - if underlying calls throw, or if the partition name resolves to more than one partition.

resolveStorageDescriptor

protected org.apache.hadoop.hive.metastore.api.StorageDescriptor resolveStorageDescriptor(org.apache.hadoop.hive.metastore.api.Table t,
                                                                                          org.apache.hadoop.hive.metastore.api.Partition p)

Get the storage descriptor for a compaction.

Parameters:

t - table from resolveTable(org.apache.hadoop.hive.metastore.txn.CompactionInfo)

p - table from resolvePartition(org.apache.hadoop.hive.metastore.txn.CompactionInfo)

Returns:

metastore storage descriptor.

findUserToRunAs

protected String findUserToRunAs(String location,
                                 org.apache.hadoop.hive.metastore.api.Table t)
                          throws IOException,
                                 InterruptedException

Determine which user to run an operation as, based on the owner of the directory to be compacted. It is asserted that either the user running the hive metastore or the table owner must be able to stat the directory and determine the owner.

Parameters:

location - directory that will be read or written to.

t - metastore table object

Returns:

username of the owner of the location.

Throws:

IOException - if neither the hive metastore user nor the table owner can stat the location.

InterruptedException

runJobAsSelf

protected boolean runJobAsSelf(String owner)

Determine whether to run this job as the current user or whether we need a doAs to switch users.

Parameters:

owner - of the directory we will be working in, as determined by findUserToRunAs(String, org.apache.hadoop.hive.metastore.api.Table)

Returns:

true if the job should run as the current user, false if a doAs is needed.

tableName

protected String tableName(org.apache.hadoop.hive.metastore.api.Table t)