org.apache.hadoop.hive.ql.io.parquet.vector

Class BaseVectorizedColumnReader

java.lang.Object

org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader

All Implemented Interfaces:

VectorizedColumnReader

Direct Known Subclasses:

VectorizedDummyColumnReader, VectorizedListColumnReader, VectorizedPrimitiveColumnReader

public abstract class BaseVectorizedColumnReader
extends Object
implements VectorizedColumnReader

It's column level Parquet reader which is used to read a batch of records for a column, part of the code is referred from Apache Spark and Apache Parquet.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	BaseVectorizedColumnReader.NullIntIterator
protected static class	BaseVectorizedColumnReader.RLEIntIterator
protected static class	BaseVectorizedColumnReader.ValuesReaderIntIterator

Field Summary

Fields

Modifier and Type	Field and Description
protected ParquetDataColumnReader	dataColumn
protected int	definitionLevel
protected org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader.IntIterator	definitionLevelColumn
protected org.apache.parquet.column.ColumnDescriptor	descriptor
protected ParquetDataColumnReader	dictionary The dictionary, if this column has dictionary encoding.
protected long	endOfPageValueCount value that indicates the end of the current page.
protected TypeInfo	hiveType
protected boolean	isCurrentPageDictionaryEncoded If true, the current page is dictionary encoded.
protected int	maxDefLevel Maximum definition level for this column.
protected org.apache.parquet.column.page.PageReader	pageReader
protected int	pageValueCount Total values in the current page.
protected int	repetitionLevel
protected org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader.IntIterator	repetitionLevelColumn Repetition/Definition/Value readers.
protected boolean	skipTimestampConversion
protected org.apache.parquet.schema.Type	type
protected long	valuesRead Total number of values read.

Constructor Summary

Constructors

Constructor and Description
BaseVectorizedColumnReader() Used for VectorizedDummyColumnReader.
BaseVectorizedColumnReader(org.apache.parquet.column.ColumnDescriptor descriptor, org.apache.parquet.column.page.PageReader pageReader, boolean skipTimestampConversion, org.apache.parquet.schema.Type parquetType, TypeInfo hiveType)

Method Summary

Modifier and Type	Method and Description
protected void	decimalTypeCheck(org.apache.parquet.schema.Type type) Check the underlying Parquet file is able to parse as Hive Decimal type.
protected void	readPage()
protected void	readRepetitionAndDefinitionLevels()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedColumnReader

readBatch

Field Detail

skipTimestampConversion

protected boolean skipTimestampConversion

valuesRead

protected long valuesRead

Total number of values read.

endOfPageValueCount

protected long endOfPageValueCount

value that indicates the end of the current page. That is, if valuesRead == endOfPageValueCount, we are at the end of the page.

dictionary

protected final ParquetDataColumnReader dictionary

The dictionary, if this column has dictionary encoding.

isCurrentPageDictionaryEncoded

protected boolean isCurrentPageDictionaryEncoded

If true, the current page is dictionary encoded.

maxDefLevel

protected final int maxDefLevel

Maximum definition level for this column.

definitionLevel

protected int definitionLevel

repetitionLevel

protected int repetitionLevel

repetitionLevelColumn

protected org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader.IntIterator repetitionLevelColumn

Repetition/Definition/Value readers.

definitionLevelColumn

protected org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader.IntIterator definitionLevelColumn

dataColumn

protected ParquetDataColumnReader dataColumn

pageValueCount

protected int pageValueCount

Total values in the current page.

pageReader

protected final org.apache.parquet.column.page.PageReader pageReader

descriptor

protected final org.apache.parquet.column.ColumnDescriptor descriptor

type

protected final org.apache.parquet.schema.Type type

hiveType

protected final TypeInfo hiveType

Constructor Detail

BaseVectorizedColumnReader

public BaseVectorizedColumnReader()

Used for VectorizedDummyColumnReader.

BaseVectorizedColumnReader

public BaseVectorizedColumnReader(org.apache.parquet.column.ColumnDescriptor descriptor,
                                  org.apache.parquet.column.page.PageReader pageReader,
                                  boolean skipTimestampConversion,
                                  org.apache.parquet.schema.Type parquetType,
                                  TypeInfo hiveType)
                           throws IOException

Throws:

IOException

Method Detail

readRepetitionAndDefinitionLevels

protected void readRepetitionAndDefinitionLevels()

readPage

protected void readPage()
                 throws IOException

Throws:

IOException

decimalTypeCheck

protected void decimalTypeCheck(org.apache.parquet.schema.Type type)

Check the underlying Parquet file is able to parse as Hive Decimal type.

Parameters:

type -