org.apache.hadoop.hive.ql.io.orc

Class ExternalCache

java.lang.Object

org.apache.hadoop.hive.ql.io.orc.ExternalCache

All Implemented Interfaces:

OrcInputFormat.FooterCache

public class ExternalCache
extends Object
implements OrcInputFormat.FooterCache

Metastore-based footer cache storing serialized footers. Also has a local cache.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	ExternalCache.ExternalFooterCachesByConf An abstraction for testing ExternalCache in OrcInputFormat.

Field Summary

Fields inherited from interface org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.FooterCache

NO_SPLIT_AFTER_PPD

Constructor Summary

Constructors

Constructor and Description
ExternalCache(org.apache.hadoop.hive.ql.io.orc.LocalCache lc, ExternalCache.ExternalFooterCachesByConf efcf)

Method Summary

Modifier and Type	Method and Description
void	configure(HiveConf queryConfig)
void	getAndValidate(List<HadoopShims.HdfsFileStatusWithId> files, boolean isOriginal, org.apache.orc.impl.OrcTail[] result, ByteBuffer[] ppdResult)
boolean	hasPpd()
boolean	isBlocking()
void	put(OrcInputFormat.FooterCacheKey key, org.apache.orc.impl.OrcTail orcTail)
static void	translateSargToTableColIndexes(org.apache.hadoop.hive.ql.io.sarg.SearchArgument sarg, org.apache.hadoop.conf.Configuration conf, int rootColumn) Modifies the SARG, replacing column names with column indexes in target table schema.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ExternalCache

public ExternalCache(org.apache.hadoop.hive.ql.io.orc.LocalCache lc,
                     ExternalCache.ExternalFooterCachesByConf efcf)

Method Detail

put

public void put(OrcInputFormat.FooterCacheKey key,
                org.apache.orc.impl.OrcTail orcTail)
         throws IOException

Specified by:

put in interface OrcInputFormat.FooterCache

Throws:

IOException

isBlocking

public boolean isBlocking()

Specified by:

isBlocking in interface OrcInputFormat.FooterCache

hasPpd

public boolean hasPpd()

Specified by:

hasPpd in interface OrcInputFormat.FooterCache

configure

public void configure(HiveConf queryConfig)

getAndValidate

public void getAndValidate(List<HadoopShims.HdfsFileStatusWithId> files,
                           boolean isOriginal,
                           org.apache.orc.impl.OrcTail[] result,
                           ByteBuffer[] ppdResult)
                    throws IOException,
                           HiveException

Specified by:

getAndValidate in interface OrcInputFormat.FooterCache

Throws:

IOException

HiveException

translateSargToTableColIndexes

public static void translateSargToTableColIndexes(org.apache.hadoop.hive.ql.io.sarg.SearchArgument sarg,
                                                  org.apache.hadoop.conf.Configuration conf,
                                                  int rootColumn)

Modifies the SARG, replacing column names with column indexes in target table schema. This basically does the same thing as all the shennannigans with included columns, except for the last step where ORC gets direct subtypes of root column and uses the ordered match to map table columns to file columns. The numbers put into predicate leaf should allow to go into said subtypes directly by index to get the proper index in the file. This won't work with schema evolution, although it's probably much easier to reason about if schema evolution was to be supported, because this is a clear boundary between table schema columns and all things ORC. None of the ORC stuff is used here and none of the table schema stuff is used after that - ORC doesn't need a bunch of extra crap to apply the SARG thus modified.