org.apache.hadoop.hive.ql.optimizer

Class SamplePruner

java.lang.Object

org.apache.hadoop.hive.ql.optimizer.Transform

org.apache.hadoop.hive.ql.optimizer.SamplePruner

public class SamplePruner
extends Transform

The transformation step that does sample pruning.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SamplePruner.AddPathReturnStatus Class used for return value of addPath()
static class	SamplePruner.DefaultPPR DefaultPPR default processor which does nothing.
static class	SamplePruner.FilterPPR FilterPPR filter processor.
static class	SamplePruner.LimitPruneRetStatus
static class	SamplePruner.SamplePrunerCtx SamplePrunerCtx.

Constructor Summary

Constructors

Constructor and Description
SamplePruner()

Method Summary

Modifier and Type	Method and Description
static SamplePruner.AddPathReturnStatus	addPath(org.apache.hadoop.fs.FileSystem fs, String pathPattern, long sizeLeft, int fileLimit, Collection<org.apache.hadoop.fs.Path> retPathList) Try to recursively add files in sub-directories into retPathList until reaching the sizeLeft.
static NodeProcessor	getDefaultProc()
static NodeProcessor	getFilterProc()
static SamplePruner.LimitPruneRetStatus	limitPrune(Partition part, long sizeLimit, int fileLimit, Collection<org.apache.hadoop.fs.Path> retPathList) Try to generate a list of subset of files in the partition to reach a size limit with number of files less than fileLimit
static org.apache.hadoop.fs.Path[]	prune(Partition part, FilterDesc.SampleDesc sampleDescr) Prunes to get all the files in the partition that satisfy the TABLESAMPLE clause.
ParseContext	transform(ParseContext pctx) All transformation steps implement this interface.

Methods inherited from class org.apache.hadoop.hive.ql.optimizer.Transform

beginPerfLogging, endPerfLogging, endPerfLogging

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SamplePruner

public SamplePruner()

Method Detail

transform

public ParseContext transform(ParseContext pctx)
                       throws SemanticException

Description copied from class: Transform

All transformation steps implement this interface.

Specified by:

transform in class Transform

Parameters:

pctx - input parse context

Returns:

ParseContext

Throws:

SemanticException

getFilterProc

public static NodeProcessor getFilterProc()

getDefaultProc

public static NodeProcessor getDefaultProc()

prune

public static org.apache.hadoop.fs.Path[] prune(Partition part,
                                                FilterDesc.SampleDesc sampleDescr)
                                         throws SemanticException

Prunes to get all the files in the partition that satisfy the TABLESAMPLE clause.

Parameters:

part - The partition to prune

Returns:

Path[]

Throws:

SemanticException

addPath

public static SamplePruner.AddPathReturnStatus addPath(org.apache.hadoop.fs.FileSystem fs,
                                                       String pathPattern,
                                                       long sizeLeft,
                                                       int fileLimit,
                                                       Collection<org.apache.hadoop.fs.Path> retPathList)
                                                throws IOException

Try to recursively add files in sub-directories into retPathList until reaching the sizeLeft.

Parameters:

fs -

pathPattern -

sizeLeft -

fileLimit -

retPathList -

Returns:

status of the recursive call

Throws:

IOException

limitPrune

public static SamplePruner.LimitPruneRetStatus limitPrune(Partition part,
                                                          long sizeLimit,
                                                          int fileLimit,
                                                          Collection<org.apache.hadoop.fs.Path> retPathList)
                                                   throws SemanticException

Try to generate a list of subset of files in the partition to reach a size limit with number of files less than fileLimit

Parameters:

part -

sizeLimit -

fileLimit -

retPathList - list of Paths returned

Returns:

the result of the attempt

Throws:

SemanticException