org.apache.hadoop.hive.ql.io

Class CombineHiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.hadoop.hive.ql.io.HiveInputFormat<K,V>

org.apache.hadoop.hive.ql.io.CombineHiveInputFormat<K,V>

All Implemented Interfaces:

org.apache.hadoop.mapred.InputFormat<K,V>, org.apache.hadoop.mapred.JobConfigurable

public class CombineHiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>
extends HiveInputFormat<K,V>

CombineHiveInputFormat is a parameterized InputFormat which looks at the path name and determine the correct InputFormat for that path name from mapredPlan.pathToPartitionInfo(). It can be used to read files with different input format in the same map-reduce job.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	CombineHiveInputFormat.AvoidSplitCombination This is a marker interface that is used to identify the formats where combine split generation is not applicable
static class	CombineHiveInputFormat.CombineHiveInputSplit CombineHiveInputSplit encapsulates an InputSplit with its corresponding inputFormatClassName.

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat

HiveInputFormat.HiveInputSplit, HiveInputFormat.HiveInputSplitComparator

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat

mrwork, pathToPartitionInfo

Constructor Summary

Constructors

Constructor and Description
CombineHiveInputFormat()

Method Summary

Modifier and Type	Method and Description
Set<Integer>	getNonCombinablePathIndices(org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.fs.Path[] paths, int numThreads) Gets all the path indices that should not be combined
org.apache.hadoop.mapred.RecordReader	getRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter) Create a generic Hive RecordReader than can iterate over all chunks in a CombinedFileSplit.
org.apache.hadoop.mapred.InputSplit[]	getSplits(org.apache.hadoop.mapred.JobConf job, int numSplits) Create Hive splits based on CombineFileSplit.

Methods inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat

canInjectCaches, canWrapForLlap, configure, createConfForMmOriginalsSplit, getInputFormatFromCache, getMmValidWriteIds, getPartitionDescFromPath, init, injectLlapCaches, processPathsForMmRead, pushFilters, pushProjectionsAndFilters, pushProjectionsAndFilters, wrapForLlap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

CombineHiveInputFormat

public CombineHiveInputFormat()

Method Detail

getNonCombinablePathIndices

public Set<Integer> getNonCombinablePathIndices(org.apache.hadoop.mapred.JobConf job,
                                                org.apache.hadoop.fs.Path[] paths,
                                                int numThreads)
                                         throws ExecutionException,
                                                InterruptedException

Gets all the path indices that should not be combined

Throws:

ExecutionException

InterruptedException

getSplits

public org.apache.hadoop.mapred.InputSplit[] getSplits(org.apache.hadoop.mapred.JobConf job,
                                                       int numSplits)
                                                throws IOException

Create Hive splits based on CombineFileSplit.

Specified by:

getSplits in interface org.apache.hadoop.mapred.InputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Overrides:

getSplits in class HiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Throws:

IOException

getRecordReader

public org.apache.hadoop.mapred.RecordReader getRecordReader(org.apache.hadoop.mapred.InputSplit split,
                                                             org.apache.hadoop.mapred.JobConf job,
                                                             org.apache.hadoop.mapred.Reporter reporter)
                                                      throws IOException

Create a generic Hive RecordReader than can iterate over all chunks in a CombinedFileSplit.

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Overrides:

getRecordReader in class HiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Throws:

IOException