org.apache.hadoop.hive.ql.exec.persistence

Class RowContainer<ROW extends List<Object>>

java.lang.Object

org.apache.hadoop.hive.ql.exec.persistence.RowContainer<ROW>

All Implemented Interfaces:

AbstractRowContainer<ROW>, AbstractRowContainer.RowIterator<ROW>

Direct Known Subclasses:

PTFRowContainer

public class RowContainer<ROW extends List<Object>>
extends Object
implements AbstractRowContainer<ROW>, AbstractRowContainer.RowIterator<ROW>

Simple persistent container for rows. This container interface only accepts adding or appending new rows and iterating through the rows in the order of their insertions. The iterator interface is a lightweight first()/next() API rather than the Java Iterator interface. This way we do not need to create an Iterator object every time we want to start a new iteration. Below is simple example of how to convert a typical Java's Iterator code to the LW iterator interface. Iterator itr = rowContainer.iterator(); while (itr.hasNext()) { v = itr.next(); // do anything with v } can be rewritten to: for ( v = rowContainer.first(); v != null; v = rowContainer.next()) { // do anything with v } Once the first is called, it will not be able to write again. So there can not be any writes after read. It can be read multiple times, but it does not support multiple reader interleaving reading.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.ql.exec.persistence.AbstractRowContainer

AbstractRowContainer.RowIterator<ROW>

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
RowContainer(org.apache.hadoop.conf.Configuration jc, org.apache.hadoop.mapred.Reporter reporter)
RowContainer(int bs, org.apache.hadoop.conf.Configuration jc, org.apache.hadoop.mapred.Reporter reporter)

Method Summary

Modifier and Type	Method and Description
void	addRow(ROW t) add a row into the RowContainer
void	clearRows() Remove all elements in the RowContainer.
protected void	close()
void	copyToDFSDirecory(org.apache.hadoop.fs.FileSystem destFs, org.apache.hadoop.fs.Path destPath)
protected boolean	endOfCurrentReadBlock()
ROW	first()
protected int	getAddCursor()
protected int	getBlockSize()
protected int	getCurrentReadBlockSize()
protected org.apache.hadoop.mapred.InputSplit[]	getInputSplits()
protected int	getLastActualSplit()
int	getNumFlushedBlocks()
protected ROW	getReadBlockRow(int rowOffset)
protected FileSinkOperator.RecordWriter	getRecordWriter()
boolean	hasRows()
boolean	isSingleRow()
ROW	next()
protected boolean	nextBlock(int readIntoOffset)
protected void	resetCurrentReadBlockToFirstReadBlock()
protected void	resetReadBlocks()
int	rowCount() Get the number of elements in the RowContainer.
AbstractRowContainer.RowIterator<ROW>	rowIter()
void	setKeyObject(List<Object> dummyKey)
protected org.apache.hadoop.mapred.RecordReader	setReaderAtSplit(int splitNum)
void	setSerDe(AbstractSerDe sd, ObjectInspector oi)
void	setTableDesc(TableDesc tblDesc)
protected void	setupWriter()
protected void	setWriteBlockAsReadBlock()
protected boolean	willSpill()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

Constructor Detail

RowContainer

public RowContainer(org.apache.hadoop.conf.Configuration jc,
                    org.apache.hadoop.mapred.Reporter reporter)
             throws HiveException

Throws:

HiveException

RowContainer

public RowContainer(int bs,
                    org.apache.hadoop.conf.Configuration jc,
                    org.apache.hadoop.mapred.Reporter reporter)
             throws HiveException

Throws:

HiveException

Method Detail

setSerDe

public void setSerDe(AbstractSerDe sd,
                     ObjectInspector oi)

addRow

public void addRow(ROW t)
            throws HiveException

Description copied from interface: AbstractRowContainer

add a row into the RowContainer

Specified by:

addRow in interface AbstractRowContainer<ROW extends List<Object>>

Parameters:

t - row

Throws:

HiveException

rowIter

public AbstractRowContainer.RowIterator<ROW> rowIter()

Specified by:

rowIter in interface AbstractRowContainer<ROW extends List<Object>>

first

public ROW first()
          throws HiveException

Specified by:

first in interface AbstractRowContainer.RowIterator<ROW extends List<Object>>

Throws:

HiveException

next

public ROW next()
         throws HiveException

Specified by:

next in interface AbstractRowContainer.RowIterator<ROW extends List<Object>>

Throws:

HiveException

hasRows

public boolean hasRows()

Specified by:

hasRows in interface AbstractRowContainer<ROW extends List<Object>>

Returns:

whether the row container has at least 1 row. NOTE: Originally we named this isEmpty, but that name conflicted with another interface.

isSingleRow

public boolean isSingleRow()

Specified by:

isSingleRow in interface AbstractRowContainer<ROW extends List<Object>>

Returns:

whether the row container has 1 row.

rowCount

public int rowCount()

Get the number of elements in the RowContainer.

Specified by:

rowCount in interface AbstractRowContainer<ROW extends List<Object>>

Returns:

number of elements in the RowContainer

nextBlock

protected boolean nextBlock(int readIntoOffset)
                     throws HiveException

Throws:

HiveException

copyToDFSDirecory

public void copyToDFSDirecory(org.apache.hadoop.fs.FileSystem destFs,
                              org.apache.hadoop.fs.Path destPath)
                       throws IOException,
                              HiveException

Throws:

IOException

HiveException

clearRows

public void clearRows()
               throws HiveException

Remove all elements in the RowContainer.

Specified by:

clearRows in interface AbstractRowContainer<ROW extends List<Object>>

Throws:

HiveException

setKeyObject

public void setKeyObject(List<Object> dummyKey)

setTableDesc

public void setTableDesc(TableDesc tblDesc)

getAddCursor

protected int getAddCursor()

willSpill

protected final boolean willSpill()

getBlockSize

protected int getBlockSize()

setupWriter

protected void setupWriter()
                    throws HiveException

Throws:

HiveException

getRecordWriter

protected FileSinkOperator.RecordWriter getRecordWriter()

getInputSplits

protected org.apache.hadoop.mapred.InputSplit[] getInputSplits()

endOfCurrentReadBlock

protected boolean endOfCurrentReadBlock()

getCurrentReadBlockSize

protected int getCurrentReadBlockSize()

setWriteBlockAsReadBlock

protected void setWriteBlockAsReadBlock()

setReaderAtSplit

protected org.apache.hadoop.mapred.RecordReader setReaderAtSplit(int splitNum)
                                                          throws IOException

Throws:

IOException

getReadBlockRow

protected ROW getReadBlockRow(int rowOffset)

resetCurrentReadBlockToFirstReadBlock

protected void resetCurrentReadBlockToFirstReadBlock()

resetReadBlocks

protected void resetReadBlocks()

close

protected void close()
              throws HiveException

Throws:

HiveException

getLastActualSplit

protected int getLastActualSplit()

getNumFlushedBlocks

public int getNumFlushedBlocks()