org.apache.hadoop.hive.hbase

Class HiveHFileOutputFormat

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<K,V>

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>

org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2

org.apache.hadoop.hive.hbase.HiveHFileOutputFormat

All Implemented Interfaces:

HiveOutputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>, org.apache.hadoop.mapred.OutputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>

public class HiveHFileOutputFormat
extends org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2
implements HiveOutputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>

HiveHFileOutputFormat implements HiveOutputFormat for HFile bulk loading. Until HBASE-1861 is implemented, it can only be used for loading a table with a single column family.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.Counter

Field Summary

Fields

Modifier and Type	Field and Description
static String	HFILE_FAMILY_PATH
static String	OUTPUT_TABLE_NAME_CONF_KEY

Fields inherited from class org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2

DATABLOCK_ENCODING_OVERRIDE_CONF_KEY, LOCALITY_SENSITIVE_CONF_KEY, STORAGE_POLICY_PROPERTY, STORAGE_POLICY_PROPERTY_CF_PREFIX, tableSeparator

Fields inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

BASE_OUTPUT_NAME, COMPRESS, COMPRESS_CODEC, COMPRESS_TYPE, OUTDIR, PART

Constructor Summary

Constructors

Constructor and Description
HiveHFileOutputFormat()

Method Summary

Modifier and Type	Method and Description
void	checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored, org.apache.hadoop.mapred.JobConf jc)
static String	getFamilyPath(org.apache.hadoop.conf.Configuration jc, Properties tableProps) Retrieve the family path, first check the JobConf, then the table properties.
FileSinkOperator.RecordWriter	getHiveRecordWriter(org.apache.hadoop.mapred.JobConf jc, org.apache.hadoop.fs.Path finalOutPath, Class<? extends org.apache.hadoop.io.Writable> valueClass, boolean isCompressed, Properties tableProperties, org.apache.hadoop.util.Progressable progressable) create the final out file and get some specific settings.
org.apache.hadoop.mapred.RecordWriter<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>	getRecordWriter(org.apache.hadoop.fs.FileSystem ignored, org.apache.hadoop.mapred.JobConf job, String name, org.apache.hadoop.util.Progressable progress)

Methods inherited from class org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2

combineTableNameSuffix, configureIncrementalLoad, configureIncrementalLoad, configureIncrementalLoadMap, getRecordWriter, getTableNameSuffixedWithFamily

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

checkOutputSpecs, getCompressOutput, getDefaultWorkFile, getOutputCommitter, getOutputCompressorClass, getOutputName, getOutputPath, getPathForWorkFile, getUniqueFile, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputName, setOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HFILE_FAMILY_PATH

public static final String HFILE_FAMILY_PATH

See Also:

Constant Field Values

OUTPUT_TABLE_NAME_CONF_KEY

public static final String OUTPUT_TABLE_NAME_CONF_KEY

See Also:

Constant Field Values

Constructor Detail

HiveHFileOutputFormat

public HiveHFileOutputFormat()

Method Detail

getFamilyPath

public static String getFamilyPath(org.apache.hadoop.conf.Configuration jc,
                                   Properties tableProps)

Retrieve the family path, first check the JobConf, then the table properties.

Returns:

the family path or null if not specified.

getHiveRecordWriter

public FileSinkOperator.RecordWriter getHiveRecordWriter(org.apache.hadoop.mapred.JobConf jc,
                                                         org.apache.hadoop.fs.Path finalOutPath,
                                                         Class<? extends org.apache.hadoop.io.Writable> valueClass,
                                                         boolean isCompressed,
                                                         Properties tableProperties,
                                                         org.apache.hadoop.util.Progressable progressable)
                                                  throws IOException

Description copied from interface: HiveOutputFormat

create the final out file and get some specific settings.

Specified by:

getHiveRecordWriter in interface HiveOutputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>

Parameters:

jc - the job configuration file

finalOutPath - the final output file to be created

valueClass - the value class used for create

isCompressed - whether the content is compressed or not

tableProperties - the table properties of this file's corresponding table

progressable - progress used for status report

Returns:

the RecordWriter for the output file

Throws:

IOException

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
                             org.apache.hadoop.mapred.JobConf jc)
                      throws IOException

Specified by:

checkOutputSpecs in interface org.apache.hadoop.mapred.OutputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>

Throws:

IOException

getRecordWriter

public org.apache.hadoop.mapred.RecordWriter<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell> getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
                                                                                                                                             org.apache.hadoop.mapred.JobConf job,
                                                                                                                                             String name,
                                                                                                                                             org.apache.hadoop.util.Progressable progress)
                                                                                                                                      throws IOException

Specified by:

getRecordWriter in interface org.apache.hadoop.mapred.OutputFormat<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell>

Throws:

IOException