org.apache.hadoop.hive.ql.io.parquet.vector

Class ParquetDataColumnReaderFactory.DefaultParquetDataColumnReader

java.lang.Object

org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory.DefaultParquetDataColumnReader

All Implemented Interfaces:

ParquetDataColumnReader

Direct Known Subclasses:

ParquetDataColumnReaderFactory.TypesFromBooleanPageReader, ParquetDataColumnReaderFactory.TypesFromDecimalPageReader, ParquetDataColumnReaderFactory.TypesFromDoublePageReader, ParquetDataColumnReaderFactory.TypesFromFloatPageReader, ParquetDataColumnReaderFactory.TypesFromInt32PageReader, ParquetDataColumnReaderFactory.TypesFromInt64PageReader, ParquetDataColumnReaderFactory.TypesFromInt96PageReader, ParquetDataColumnReaderFactory.TypesFromStringPageReader

Enclosing class:

ParquetDataColumnReaderFactory

public static class ParquetDataColumnReaderFactory.DefaultParquetDataColumnReader
extends Object
implements ParquetDataColumnReader

The default data column reader for existing Parquet page reader which works for both dictionary or non dictionary types, Mirror from dictionary encoding path.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.parquet.column.Dictionary	dict
protected int	length
protected org.apache.parquet.column.values.ValuesReader	valuesReader

Constructor Summary

Constructors

Constructor and Description
DefaultParquetDataColumnReader(org.apache.parquet.column.Dictionary dict, int length)
DefaultParquetDataColumnReader(org.apache.parquet.column.values.ValuesReader valuesReader, int length)

Method Summary

Modifier and Type	Method and Description
protected static byte[]	convertToBytes(String value) Method to convert string to UTF-8 bytes.
protected String	enforceMaxLength(String value) Enforce the max legnth of varchar or char.
org.apache.parquet.column.Dictionary	getDictionary()
protected String	getPaddedString(String value) Enforce the char length.
void	initFromPage(int i, org.apache.parquet.bytes.ByteBufferInputStream in) Initialize the reader by page data.
boolean	isValid(double value)
boolean	isValid(float value)
boolean	isValid(long value)
boolean	readBoolean()
boolean	readBoolean(int id)
byte[]	readBytes()
byte[]	readBytes(int id)
byte[]	readChar()
byte[]	readChar(int id)
byte[]	readDecimal()
byte[]	readDecimal(int id)
double	readDouble()
double	readDouble(int id)
float	readFloat()
float	readFloat(int id)
long	readInteger()
long	readInteger(int id)
long	readLong()
long	readLong(int id)
byte[]	readString()
byte[]	readString(int id)
Timestamp	readTimestamp()
Timestamp	readTimestamp(int id)
int	readValueDictionaryId()
byte[]	readVarchar()
byte[]	readVarchar(int id)
void	skip()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

valuesReader

protected org.apache.parquet.column.values.ValuesReader valuesReader

dict

protected org.apache.parquet.column.Dictionary dict

length

protected int length

Constructor Detail

DefaultParquetDataColumnReader

public DefaultParquetDataColumnReader(org.apache.parquet.column.values.ValuesReader valuesReader,
                                      int length)

DefaultParquetDataColumnReader

public DefaultParquetDataColumnReader(org.apache.parquet.column.Dictionary dict,
                                      int length)

Method Detail

initFromPage

public void initFromPage(int i,
                         org.apache.parquet.bytes.ByteBufferInputStream in)
                  throws IOException

Description copied from interface: ParquetDataColumnReader

Initialize the reader by page data.

Specified by:

initFromPage in interface ParquetDataColumnReader

Parameters:

i - value count

in - page data

Throws:

IOException

readBoolean

public boolean readBoolean()

Specified by:

readBoolean in interface ParquetDataColumnReader

Returns:

the next Boolean from the page

readBoolean

public boolean readBoolean(int id)

Specified by:

readBoolean in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the Boolean from the dictionary by id

readString

public byte[] readString(int id)

Specified by:

readString in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the String from the dictionary by id

readString

public byte[] readString()

Specified by:

readString in interface ParquetDataColumnReader

Returns:

the next String from the page

readVarchar

public byte[] readVarchar()

Specified by:

readVarchar in interface ParquetDataColumnReader

Returns:

the next Varchar from the page

readVarchar

public byte[] readVarchar(int id)

Specified by:

readVarchar in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the Varchar from the dictionary by id

readChar

public byte[] readChar()

Specified by:

readChar in interface ParquetDataColumnReader

Returns:

the next Char from the page

readChar

public byte[] readChar(int id)

Specified by:

readChar in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the Char from the dictionary by id

readBytes

public byte[] readBytes()

Specified by:

readBytes in interface ParquetDataColumnReader

Returns:

the next Bytes from the page

readBytes

public byte[] readBytes(int id)

Specified by:

readBytes in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the Bytes from the dictionary by id

readDecimal

public byte[] readDecimal()

Specified by:

readDecimal in interface ParquetDataColumnReader

Returns:

the next Decimal from the page

readDecimal

public byte[] readDecimal(int id)

Specified by:

readDecimal in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the Decimal from the dictionary by id

readFloat

public float readFloat()

Specified by:

readFloat in interface ParquetDataColumnReader

Returns:

the next Float from the page

readFloat

public float readFloat(int id)

Specified by:

readFloat in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the Float from the dictionary by id

readDouble

public double readDouble()

Specified by:

readDouble in interface ParquetDataColumnReader

Returns:

the next Double from the page

readDouble

public double readDouble(int id)

Specified by:

readDouble in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the Double from the dictionary by id

readTimestamp

public Timestamp readTimestamp()

Specified by:

readTimestamp in interface ParquetDataColumnReader

Returns:

the next Timestamp from the page

readTimestamp

public Timestamp readTimestamp(int id)

Specified by:

readTimestamp in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the Timestamp from the dictionary by id

readInteger

public long readInteger()

Specified by:

readInteger in interface ParquetDataColumnReader

Returns:

the next Integer from the page Though the function is looking for an integer, it will return the value through long. The type of data saved as long can be changed to be int or smallint or tinyint. In that case the value returned to the user will depend on the data. If the data value is within the valid range accommodated by the read type, the data will be returned as is. When data is not within the valid range, a NULL will be returned. A long value saved in parquet files will be returned asis to facilitate the validity check. Also, the vectorized representation uses a LongColumnVector to store integer values.

readInteger

public long readInteger(int id)

Specified by:

readInteger in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the Integer from the dictionary by id Though the function is looking for an integer, it will return the value through long. The type of data saved as long can be changed to be int or smallint or tinyint. In that case the value returned to the user will depend on the data. If the data value is within the valid range accommodated by the read type, the data will be returned as is. When data is not within the valid range, a NULL will be returned. A long value saved in parquet files will be returned asis to facilitate the validity check. Also, the vectorized representation uses a LongColumnVector to store integer values.

isValid

public boolean isValid(long value)

Specified by:

isValid in interface ParquetDataColumnReader

Parameters:

value - data to be checked for validity

Returns:

is data valid for the type The type of the data in Parquet files need not match the type in HMS. In that case the value returned to the user will depend on the data. If the data value is within the valid range accommodated by the HMS type, the data will be returned as is. When data is not within the valid range, a NULL will be returned. These functions will do the appropriate check.

isValid

public boolean isValid(float value)

Specified by:

isValid in interface ParquetDataColumnReader

isValid

public boolean isValid(double value)

Specified by:

isValid in interface ParquetDataColumnReader

readLong

public long readLong(int id)

Specified by:

readLong in interface ParquetDataColumnReader

Parameters:

id - in dictionary

Returns:

the Long from the dictionary by id

readLong

public long readLong()

Specified by:

readLong in interface ParquetDataColumnReader

Returns:

the next Long from the page

readValueDictionaryId

public int readValueDictionaryId()

Specified by:

readValueDictionaryId in interface ParquetDataColumnReader

Returns:

the next Dictionary ID from the page

skip

public void skip()

getDictionary

public org.apache.parquet.column.Dictionary getDictionary()

Specified by:

getDictionary in interface ParquetDataColumnReader

Returns:

the underlying dictionary if current reader is dictionary encoded

enforceMaxLength

protected String enforceMaxLength(String value)

Enforce the max legnth of varchar or char.

getPaddedString

protected String getPaddedString(String value)

Enforce the char length.

convertToBytes

protected static byte[] convertToBytes(String value)

Method to convert string to UTF-8 bytes.