org.apache.hadoop.hive.metastore.utils

Class FileUtils

java.lang.Object

org.apache.hadoop.hive.metastore.utils.FileUtils

public class FileUtils
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	HIDDEN_FILES_PATH_FILTER

Constructor Summary

Constructors

Constructor and Description
FileUtils()

Method Summary

Modifier and Type	Method and Description
static boolean	copy(org.apache.hadoop.fs.FileSystem srcFS, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.FileSystem dstFS, org.apache.hadoop.fs.Path dst, boolean deleteSource, boolean overwrite, org.apache.hadoop.conf.Configuration conf) Copies files between filesystems.
static boolean	equalsFileSystem(org.apache.hadoop.fs.FileSystem fs1, org.apache.hadoop.fs.FileSystem fs2) Determine if two objects reference the same file system.
static String	escapePathName(String path)
static String	escapePathName(String path, String defaultPath) Escapes a path name.
static List<org.apache.hadoop.fs.FileStatus>	getFileStatusRecurse(org.apache.hadoop.fs.Path path, int level, org.apache.hadoop.fs.FileSystem fs) Get all file status from a root path and recursively go deep into certain levels.
static boolean	isDirEmpty(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Utility method that determines if a specified directory already has contents (non-hidden files) or not - useful to determine if an immutable table already has contents, for example.
static void	listStatusRecursively(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus fileStatus, List<org.apache.hadoop.fs.FileStatus> results) Recursively lists status for all files starting from a particular directory (or individual file as base case).
static void	makeDir(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)
static String	makePartName(List<String> partCols, List<String> vals)
static String	makePartName(List<String> partCols, List<String> vals, String defaultStr) Makes a valid partition name.
static org.apache.hadoop.fs.Path	makeQualified(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Variant of Path.makeQualified that qualifies the input path against the default file system indicated by the configuration This does not require a FileSystem handle in most cases - only requires the Filesystem URI.
static boolean	mkdir(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path f) Creates the directory and all necessary parent directories.
static boolean	moveToTrash(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path f, org.apache.hadoop.conf.Configuration conf, boolean purge) Move a particular file or directory to the trash.
static boolean	pathHasSnapshotSubDir(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.FileSystem fs) Check if the path contains a subdirectory named '.snapshot'
static boolean	rename(org.apache.hadoop.fs.FileSystem srcFs, org.apache.hadoop.fs.FileSystem destFs, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path destPath) Rename a file.
static String	unescapePathName(String path)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HIDDEN_FILES_PATH_FILTER

public static final org.apache.hadoop.fs.PathFilter HIDDEN_FILES_PATH_FILTER

Constructor Detail

FileUtils

public FileUtils()

Method Detail

moveToTrash

public static boolean moveToTrash(org.apache.hadoop.fs.FileSystem fs,
                                  org.apache.hadoop.fs.Path f,
                                  org.apache.hadoop.conf.Configuration conf,
                                  boolean purge)
                           throws IOException

Move a particular file or directory to the trash.

Parameters:

fs - FileSystem to use

f - path of file or directory to move to trash.

conf - configuration object

Returns:

true if move successful

Throws:

IOException

copy

public static boolean copy(org.apache.hadoop.fs.FileSystem srcFS,
                           org.apache.hadoop.fs.Path src,
                           org.apache.hadoop.fs.FileSystem dstFS,
                           org.apache.hadoop.fs.Path dst,
                           boolean deleteSource,
                           boolean overwrite,
                           org.apache.hadoop.conf.Configuration conf)
                    throws IOException

Copies files between filesystems.

Throws:

IOException

mkdir

public static boolean mkdir(org.apache.hadoop.fs.FileSystem fs,
                            org.apache.hadoop.fs.Path f)
                     throws IOException

Creates the directory and all necessary parent directories.

Parameters:

fs - FileSystem to use

f - path to create.

Returns:

true if directory created successfully. False otherwise, including if it exists.

Throws:

IOException - exception in creating the directory

rename

public static boolean rename(org.apache.hadoop.fs.FileSystem srcFs,
                             org.apache.hadoop.fs.FileSystem destFs,
                             org.apache.hadoop.fs.Path srcPath,
                             org.apache.hadoop.fs.Path destPath)
                      throws IOException

Rename a file. Unlike FileSystem.rename(Path, Path), if the destPath already exists and is a directory, this will NOT move the sourcePath into it. It will throw an IOException instead.

Parameters:

srcFs - file system src paths are on

destFs - file system dest paths are on

srcPath - source file or directory to move

destPath - destination file name. This must be a file and not an existing directory.

Returns:

result of fs.rename.

Throws:

IOException - if fs.rename throws it, or if destPath already exists.

escapePathName

public static String escapePathName(String path)

escapePathName

public static String escapePathName(String path,
                                    String defaultPath)

Escapes a path name.

Parameters:

path - The path to escape.

defaultPath - The default name for the path, if the given path is empty or null.

Returns:

An escaped path name.

unescapePathName

public static String unescapePathName(String path)

getFileStatusRecurse

public static List<org.apache.hadoop.fs.FileStatus> getFileStatusRecurse(org.apache.hadoop.fs.Path path,
                                                                         int level,
                                                                         org.apache.hadoop.fs.FileSystem fs)
                                                                  throws IOException

Get all file status from a root path and recursively go deep into certain levels.

Parameters:

path - the root path

level - the depth of directory to explore

fs - the file system

Returns:

array of FileStatus

Throws:

IOException

listStatusRecursively

public static void listStatusRecursively(org.apache.hadoop.fs.FileSystem fs,
                                         org.apache.hadoop.fs.FileStatus fileStatus,
                                         List<org.apache.hadoop.fs.FileStatus> results)
                                  throws IOException

Recursively lists status for all files starting from a particular directory (or individual file as base case).

Parameters:

fs - file system

fileStatus - starting point in file system

results - receives enumeration of all files found

Throws:

IOException

makePartName

public static String makePartName(List<String> partCols,
                                  List<String> vals)

makePartName

public static String makePartName(List<String> partCols,
                                  List<String> vals,
                                  String defaultStr)

Makes a valid partition name.

Parameters:

partCols - The partition keys' names

vals - The partition values

defaultStr - The default name given to a partition value if the respective value is empty or null.

Returns:

An escaped, valid partition name.

equalsFileSystem

public static boolean equalsFileSystem(org.apache.hadoop.fs.FileSystem fs1,
                                       org.apache.hadoop.fs.FileSystem fs2)

Determine if two objects reference the same file system.

Parameters:

fs1 - first file system

fs2 - second file system

Returns:

return true if both file system arguments point to same file system

pathHasSnapshotSubDir

public static boolean pathHasSnapshotSubDir(org.apache.hadoop.fs.Path p,
                                            org.apache.hadoop.fs.FileSystem fs)
                                     throws IOException

Check if the path contains a subdirectory named '.snapshot'

Parameters:

p - path to check

fs - filesystem of the path

Returns:

true if p contains a subdirectory named '.snapshot'

Throws:

IOException

makeDir

public static void makeDir(org.apache.hadoop.fs.Path path,
                           org.apache.hadoop.conf.Configuration conf)
                    throws org.apache.hadoop.hive.metastore.api.MetaException

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

isDirEmpty

public static boolean isDirEmpty(org.apache.hadoop.fs.FileSystem fs,
                                 org.apache.hadoop.fs.Path path)
                          throws IOException

Utility method that determines if a specified directory already has contents (non-hidden files) or not - useful to determine if an immutable table already has contents, for example.

Parameters:

fs -

path -

Throws:

IOException

makeQualified

public static org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.Path path,
                                                      org.apache.hadoop.conf.Configuration conf)
                                               throws IOException

Variant of Path.makeQualified that qualifies the input path against the default file system indicated by the configuration This does not require a FileSystem handle in most cases - only requires the Filesystem URI. This saves the cost of opening the Filesystem - which can involve RPCs - as well as cause errors

Parameters:

path - path to be fully qualified

conf - Configuration file

Returns:

path qualified relative to default file system

Throws:

IOException