org.apache.hadoop.hive.llap.io.encoded

Class SerDeEncodedDataReader

java.lang.Object

org.apache.tez.common.CallableWithNdc<Void>

org.apache.hadoop.hive.llap.io.encoded.SerDeEncodedDataReader

All Implemented Interfaces:

Callable<Void>, ConsumerFeedback<Reader.OrcEncodedColumnBatch>, TezCounterSource

public class SerDeEncodedDataReader
extends org.apache.tez.common.CallableWithNdc<Void>
implements ConsumerFeedback<Reader.OrcEncodedColumnBatch>, TezCounterSource

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SerDeEncodedDataReader.CacheWriter

Field Summary

Fields

Modifier and Type	Field and Description
static FixedSizedObjectPool<org.apache.hadoop.hive.common.io.encoded.EncodedColumnBatch.ColumnStreamData>	CSD_POOL
static FixedSizedObjectPool<Reader.OrcEncodedColumnBatch>	ECB_POOL

Constructor Summary

Constructors

Constructor and Description

SerDeEncodedDataReader(SerDeLowLevelCacheImpl cache, BufferUsageManager bufferManager, org.apache.hadoop.conf.Configuration daemonConf, org.apache.hadoop.mapred.FileSplit split, List<Integer> columnIds, OrcEncodedDataConsumer consumer, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.mapred.Reporter reporter, org.apache.hadoop.mapred.InputFormat<?,?> sourceInputFormat, Deserializer sourceSerDe, QueryFragmentCounters counters, org.apache.orc.TypeDescription schema, Map<org.apache.hadoop.fs.Path,PartitionDesc> parts)

Method Summary

Modifier and Type	Method and Description
void	cacheFileData(SerDeLowLevelCacheImpl.StripeData sd)
protected Void	callInternal()
org.apache.tez.common.counters.TezCounters	getTezCounters()
void	pause() Pause data production.
protected Void	performDataRead()
boolean	processOneFileSplit(org.apache.hadoop.mapred.FileSplit split, long startTime, Ref<Integer> stripeIxRef, SerDeLowLevelCacheImpl.StripeData slice)
Boolean	readFileWithCache(long startTime)
void	returnData(Reader.OrcEncodedColumnBatch ecb) Returns processed data back to producer; necessary if e.g.
void	startReadSplitFromFile(org.apache.hadoop.mapred.FileSplit split, boolean[] splitIncludes, SerDeLowLevelCacheImpl.StripeData slice)
void	stop() Stop data production, the external operation has been cancelled.
void	unpause() Unpause data production.

Methods inherited from class org.apache.tez.common.CallableWithNdc

call

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CSD_POOL

public static final FixedSizedObjectPool<org.apache.hadoop.hive.common.io.encoded.EncodedColumnBatch.ColumnStreamData> CSD_POOL

ECB_POOL

public static final FixedSizedObjectPool<Reader.OrcEncodedColumnBatch> ECB_POOL

Constructor Detail

SerDeEncodedDataReader

public SerDeEncodedDataReader(SerDeLowLevelCacheImpl cache,
                              BufferUsageManager bufferManager,
                              org.apache.hadoop.conf.Configuration daemonConf,
                              org.apache.hadoop.mapred.FileSplit split,
                              List<Integer> columnIds,
                              OrcEncodedDataConsumer consumer,
                              org.apache.hadoop.mapred.JobConf jobConf,
                              org.apache.hadoop.mapred.Reporter reporter,
                              org.apache.hadoop.mapred.InputFormat<?,?> sourceInputFormat,
                              Deserializer sourceSerDe,
                              QueryFragmentCounters counters,
                              org.apache.orc.TypeDescription schema,
                              Map<org.apache.hadoop.fs.Path,PartitionDesc> parts)
                       throws IOException

Throws:

IOException

Method Detail

stop

public void stop()

Description copied from interface: ConsumerFeedback

Stop data production, the external operation has been cancelled.

Specified by:

stop in interface ConsumerFeedback<Reader.OrcEncodedColumnBatch>

pause

public void pause()

Description copied from interface: ConsumerFeedback

Pause data production.

Specified by:

pause in interface ConsumerFeedback<Reader.OrcEncodedColumnBatch>

unpause

public void unpause()

Description copied from interface: ConsumerFeedback

Unpause data production.

Specified by:

unpause in interface ConsumerFeedback<Reader.OrcEncodedColumnBatch>

callInternal

protected Void callInternal()
                     throws IOException,
                            InterruptedException

Specified by:

callInternal in class org.apache.tez.common.CallableWithNdc<Void>

Throws:

IOException

InterruptedException

performDataRead

protected Void performDataRead()
                        throws IOException,
                               InterruptedException

Throws:

IOException

InterruptedException

cacheFileData

public void cacheFileData(SerDeLowLevelCacheImpl.StripeData sd)

readFileWithCache

public Boolean readFileWithCache(long startTime)
                          throws IOException,
                                 InterruptedException

Throws:

IOException

InterruptedException

processOneFileSplit

public boolean processOneFileSplit(org.apache.hadoop.mapred.FileSplit split,
                                   long startTime,
                                   Ref<Integer> stripeIxRef,
                                   SerDeLowLevelCacheImpl.StripeData slice)
                            throws IOException,
                                   InterruptedException

Throws:

IOException

InterruptedException

startReadSplitFromFile

public void startReadSplitFromFile(org.apache.hadoop.mapred.FileSplit split,
                                   boolean[] splitIncludes,
                                   SerDeLowLevelCacheImpl.StripeData slice)
                            throws IOException

Throws:

IOException

returnData

public void returnData(Reader.OrcEncodedColumnBatch ecb)

Description copied from interface: ConsumerFeedback

Returns processed data back to producer; necessary if e.g. data is locked in cache.

Specified by:

returnData in interface ConsumerFeedback<Reader.OrcEncodedColumnBatch>

getTezCounters

public org.apache.tez.common.counters.TezCounters getTezCounters()

Specified by:

getTezCounters in interface TezCounterSource