org.apache.hive.hcatalog.pig

Class HCatLoader

java.lang.Object

org.apache.pig.LoadFunc

org.apache.hive.hcatalog.pig.HCatLoader

All Implemented Interfaces:

org.apache.pig.LoadMetadata, org.apache.pig.LoadPushDown

@InterfaceAudience.Public
 @InterfaceStability.Stable
public class HCatLoader
extends org.apache.pig.LoadFunc

Pig LoadFunc to read data from HCat

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.pig.LoadPushDown

org.apache.pig.LoadPushDown.OperatorSet, org.apache.pig.LoadPushDown.RequiredField, org.apache.pig.LoadPushDown.RequiredFieldList, org.apache.pig.LoadPushDown.RequiredFieldResponse

Field Summary

Fields

Modifier and Type	Field and Description
static String	INNER_SIGNATURE
static String	INNER_SIGNATURE_PREFIX
protected static String	PRUNE_PROJECTION_INFO
protected String	signature

Constructor Summary

Constructors

Constructor and Description
HCatLoader()

Method Summary

Modifier and Type	Method and Description
List<org.apache.pig.LoadPushDown.OperatorSet>	getFeatures()
org.apache.hadoop.mapreduce.InputFormat<?,?>	getInputFormat()
org.apache.pig.data.Tuple	getNext()
String[]	getPartitionKeys(String location, org.apache.hadoop.mapreduce.Job job)
org.apache.pig.ResourceSchema	getSchema(String location, org.apache.hadoop.mapreduce.Job job)
protected static long	getSizeInBytes(InputJobInfo inputJobInfo) A utility method to get the size of inputs.
org.apache.pig.ResourceStatistics	getStatistics(String location, org.apache.hadoop.mapreduce.Job job) Get statistics about the data to be loaded.
void	prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader, org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit arg1)
org.apache.pig.LoadPushDown.RequiredFieldResponse	pushProjection(org.apache.pig.LoadPushDown.RequiredFieldList requiredFieldsInfo)
String	relativeToAbsolutePath(String location, org.apache.hadoop.fs.Path curDir)
void	setLocation(String location, org.apache.hadoop.mapreduce.Job job)
void	setPartitionFilter(org.apache.pig.Expression partitionFilter)
void	setUDFContextSignature(String signature)
protected void	storeInUDFContext(String signature, String key, Object value)

Methods inherited from class org.apache.pig.LoadFunc

getAbsolutePath, getCacheFiles, getLoadCaster, getPathStrings, getShipFiles, join, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INNER_SIGNATURE

public static final String INNER_SIGNATURE

See Also:

Constant Field Values

INNER_SIGNATURE_PREFIX

public static final String INNER_SIGNATURE_PREFIX

See Also:

Constant Field Values

PRUNE_PROJECTION_INFO

protected static final String PRUNE_PROJECTION_INFO

See Also:

Constant Field Values

signature

protected String signature

Constructor Detail

HCatLoader

public HCatLoader()

Method Detail

getInputFormat

public org.apache.hadoop.mapreduce.InputFormat<?,?> getInputFormat()
                                                            throws IOException

Specified by:

getInputFormat in class org.apache.pig.LoadFunc

Throws:

IOException

relativeToAbsolutePath

public String relativeToAbsolutePath(String location,
                                     org.apache.hadoop.fs.Path curDir)
                              throws IOException

Overrides:

relativeToAbsolutePath in class org.apache.pig.LoadFunc

Throws:

IOException

setLocation

public void setLocation(String location,
                        org.apache.hadoop.mapreduce.Job job)
                 throws IOException

Specified by:

setLocation in class org.apache.pig.LoadFunc

Throws:

IOException

getPartitionKeys

public String[] getPartitionKeys(String location,
                                 org.apache.hadoop.mapreduce.Job job)
                          throws IOException

Throws:

IOException

getSchema

public org.apache.pig.ResourceSchema getSchema(String location,
                                               org.apache.hadoop.mapreduce.Job job)
                                        throws IOException

Throws:

IOException

setPartitionFilter

public void setPartitionFilter(org.apache.pig.Expression partitionFilter)
                        throws IOException

Throws:

IOException

getStatistics

public org.apache.pig.ResourceStatistics getStatistics(String location,
                                                       org.apache.hadoop.mapreduce.Job job)
                                                throws IOException

Get statistics about the data to be loaded. Only input data size is implemented at this time.

Specified by:

getStatistics in interface org.apache.pig.LoadMetadata

Throws:

IOException

getNext

public org.apache.pig.data.Tuple getNext()
                                  throws IOException

Specified by:

getNext in class org.apache.pig.LoadFunc

Throws:

IOException

prepareToRead

public void prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader,
                          org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit arg1)
                   throws IOException

Specified by:

prepareToRead in class org.apache.pig.LoadFunc

Throws:

IOException

getFeatures

public List<org.apache.pig.LoadPushDown.OperatorSet> getFeatures()

Specified by:

getFeatures in interface org.apache.pig.LoadPushDown

pushProjection

public org.apache.pig.LoadPushDown.RequiredFieldResponse pushProjection(org.apache.pig.LoadPushDown.RequiredFieldList requiredFieldsInfo)
                                                                 throws org.apache.pig.impl.logicalLayer.FrontendException

Specified by:

pushProjection in interface org.apache.pig.LoadPushDown

Throws:

org.apache.pig.impl.logicalLayer.FrontendException

setUDFContextSignature

public void setUDFContextSignature(String signature)

Overrides:

setUDFContextSignature in class org.apache.pig.LoadFunc

storeInUDFContext

protected void storeInUDFContext(String signature,
                                 String key,
                                 Object value)

getSizeInBytes

protected static long getSizeInBytes(InputJobInfo inputJobInfo)
                              throws IOException

A utility method to get the size of inputs. This is accomplished by summing the size of all input paths on supported FileSystems. Locations whose size cannot be determined are ignored. Note non-FileSystem and unpartitioned locations will not report their input size by default.

Throws:

IOException