public abstract class FileSystem extends Configured implements Closeable
All user code that may potentially use the Hadoop Distributed File System should be written to use a FileSystem object. The Hadoop DFS is a multi-machine system that appears as a single disk. It's useful because of its fault tolerance and potentially very large capacity.
The local implementation is LocalFileSystem and distributed
implementation is DistributedFileSystem.
| 限定符和类型 | 类和说明 |
|---|---|
static class |
FileSystem.Statistics |
| 限定符和类型 | 字段和说明 |
|---|---|
static String |
FS_DEFAULT_NAME_KEY |
static org.apache.commons.logging.Log |
LOG |
protected FileSystem.Statistics |
statistics
The statistics for this file system.
|
| 限定符 | 构造器和说明 |
|---|---|
protected |
FileSystem() |
| 限定符和类型 | 方法和说明 |
|---|---|
static void |
addFileSystemForTesting(URI uri,
Configuration conf,
FileSystem fs)
This method adds a file system for testing so that we can find it later.
|
FSDataOutputStream |
append(Path f)
Append to an existing file (optional operation).
|
FSDataOutputStream |
append(Path f,
int bufferSize)
Append to an existing file (optional operation).
|
abstract FSDataOutputStream |
append(Path f,
int bufferSize,
Progressable progress)
Append to an existing file (optional operation).
|
protected void |
checkPath(Path path)
Check that a Path belongs to this FileSystem.
|
static void |
clearStatistics() |
void |
close()
No more filesystem operations are needed.
|
static void |
closeAll()
Close all cached filesystems.
|
static void |
closeAllForUGI(UserGroupInformation ugi)
Close all cached filesystems for a given UGI.
|
void |
completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Called when we're all done writing to the target.
|
void |
concat(Path trg,
Path[] srcs)
Concat existing files together.
|
void |
copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path[] srcs,
Path dst)
The src files are on the local disk.
|
void |
copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path src,
Path dst)
The src file is on the local disk.
|
void |
copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is on the local disk.
|
void |
copyFromLocalFile(Path src,
Path dst)
The src file is on the local disk.
|
void |
copyToLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
void |
copyToLocalFile(Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
static FSDataOutputStream |
create(FileSystem fs,
Path file,
FsPermission permission)
create a file with the provided permission
The permission of the file is set to be the provided permission as in
setPermission, not permission&~umask
It is implemented using two RPCs.
|
FSDataOutputStream |
create(Path f)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
boolean overwrite)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
abstract FSDataOutputStream |
create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
create(Path f,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
FSDataOutputStream |
create(Path f,
short replication)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
short replication,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
boolean |
createNewFile(Path f)
Creates the given Path as a brand-new zero-length file.
|
FSDataOutputStream |
createNonRecursive(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
已过时。
API only for 0.20-append
|
FSDataOutputStream |
createNonRecursive(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
已过时。
API only for 0.20-append
|
abstract boolean |
delete(Path f)
已过时。
Use delete(Path, boolean) instead
|
abstract boolean |
delete(Path f,
boolean recursive)
Delete a file.
|
boolean |
deleteOnExit(Path f)
Mark a path to be deleted when FileSystem is closed.
|
boolean |
exists(Path f)
Check if exists.
|
static FileSystem |
get(Configuration conf)
Returns the configured filesystem implementation.
|
static FileSystem |
get(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority.
|
static FileSystem |
get(URI uri,
Configuration conf,
String user) |
static List<FileSystem.Statistics> |
getAllStatistics()
Return the FileSystem classes that have Statistics
|
long |
getBlockSize(Path f)
已过时。
Use getFileStatus() instead
|
static int |
getCacheSize()
Get the number of entries in the filesystem cache
|
String |
getCanonicalServiceName()
Get a canonical service name for this file system.
|
protected URI |
getCanonicalUri()
Resolve the uri's hostname and add the default port if not in the uri
|
ContentSummary |
getContentSummary(Path f)
Return the
ContentSummary of a given Path. |
long |
getDefaultBlockSize()
已过时。
use
getDefaultBlockSize(Path) instead |
long |
getDefaultBlockSize(Path f)
Return the number of bytes that large input files should be optimally
be split into to minimize i/o time.
|
protected int |
getDefaultPort()
Get the default port for this file system.
|
short |
getDefaultReplication()
已过时。
use
getDefaultReplication(Path) instead |
short |
getDefaultReplication(Path path)
Get the default replication.
|
static URI |
getDefaultUri(Configuration conf)
Get the default filesystem URI from a configuration.
|
Token<?> |
getDelegationToken(String renewer)
Get a new delegation token for this file system.
|
BlockLocation[] |
getFileBlockLocations(FileStatus file,
long start,
long len)
Return an array containing hostnames, offset and size of
portions of the given file.
|
FileChecksum |
getFileChecksum(Path f)
Get the checksum of a file.
|
abstract FileStatus |
getFileStatus(Path f)
Return a file status object that represents the path.
|
Path |
getHomeDirectory()
Return the current user's home directory in this filesystem.
|
long |
getLength(Path f)
已过时。
Use getFileStatus() instead
|
static LocalFileSystem |
getLocal(Configuration conf)
Get the local file syste
|
String |
getName()
已过时。
call #getUri() instead.
|
static FileSystem |
getNamed(String name,
Configuration conf)
已过时。
call #get(URI,Configuration) instead.
|
short |
getReplication(Path src)
已过时。
Use getFileStatus() instead
|
static Map<String,FileSystem.Statistics> |
getStatistics()
已过时。
use
getAllStatistics() instead |
static FileSystem.Statistics |
getStatistics(String scheme,
Class<? extends FileSystem> cls)
Get the statistics for a particular file system
|
abstract URI |
getUri()
Returns a URI whose scheme and authority identify this FileSystem.
|
long |
getUsed()
Return the total size of all files in the filesystem.
|
abstract Path |
getWorkingDirectory()
Get the current working directory for the given file system
|
FileStatus[] |
globStatus(Path pathPattern)
Return all the files that match filePattern and are not checksum
files.
|
FileStatus[] |
globStatus(Path pathPattern,
PathFilter filter)
Return an array of FileStatus objects whose path names match pathPattern
and is accepted by the user-supplied path filter.
|
void |
initialize(URI name,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
boolean |
isDirectory(Path f)
已过时。
Use getFileStatus() instead
|
boolean |
isFile(Path f)
True iff the named path is a regular file.
|
abstract FileStatus[] |
listStatus(Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
FileStatus[] |
listStatus(Path[] files)
Filter files/directories in the given list of paths using default
path filter.
|
FileStatus[] |
listStatus(Path[] files,
PathFilter filter)
Filter files/directories in the given list of paths using user-supplied
path filter.
|
FileStatus[] |
listStatus(Path f,
PathFilter filter)
Filter files/directories in the given path using the user-supplied path
filter.
|
Path |
makeQualified(Path path)
Make sure that a path specifies a FileSystem.
|
static boolean |
mkdirs(FileSystem fs,
Path dir,
FsPermission permission)
create a directory with the provided permission
The permission of the directory is set to be the provided permission as in
setPermission, not permission&~umask
|
boolean |
mkdirs(Path f)
Call
mkdirs(Path, FsPermission) with default permission. |
abstract boolean |
mkdirs(Path f,
FsPermission permission)
Make the given file and all non-existent parents into
directories.
|
void |
moveFromLocalFile(Path[] srcs,
Path dst)
The src files is on the local disk.
|
void |
moveFromLocalFile(Path src,
Path dst)
The src file is on the local disk.
|
void |
moveToLocalFile(Path src,
Path dst)
The src file is under FS, and the dst is on the local disk.
|
FSDataInputStream |
open(Path f)
Opens an FSDataInputStream at the indicated Path.
|
abstract FSDataInputStream |
open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
static void |
printStatistics() |
protected void |
processDeleteOnExit()
Delete all files that were marked as delete-on-exit.
|
abstract boolean |
rename(Path src,
Path dst)
Renames Path src to Path dst.
|
static void |
setDefaultUri(Configuration conf,
String uri)
Set the default filesystem URI in a configuration.
|
static void |
setDefaultUri(Configuration conf,
URI uri)
Set the default filesystem URI in a configuration.
|
void |
setOwner(Path p,
String username,
String groupname)
Set owner of a path (i.e. a file or a directory).
|
void |
setPermission(Path p,
FsPermission permission)
Set permission of a path.
|
boolean |
setReplication(Path src,
short replication)
Set replication for an existing file.
|
void |
setTimes(Path p,
long mtime,
long atime)
Set access time of a file
|
void |
setVerifyChecksum(boolean verifyChecksum)
Set the verify checksum flag.
|
abstract void |
setWorkingDirectory(Path new_dir)
Set the current working directory for the given file system.
|
Path |
startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to.
|
getConf, setConfpublic static final org.apache.commons.logging.Log LOG
protected FileSystem.Statistics statistics
public static void addFileSystemForTesting(URI uri, Configuration conf, FileSystem fs) throws IOException
uri - the uri to store it underconf - the configuration to store it underfs - the file system to storeIOExceptionpublic static FileSystem get(URI uri, Configuration conf, String user) throws IOException, InterruptedException
public static FileSystem get(Configuration conf) throws IOException
IOExceptionpublic static URI getDefaultUri(Configuration conf)
conf - the configuration to accesspublic static void setDefaultUri(Configuration conf, URI uri)
conf - the configuration to alteruri - the new default filesystem uripublic static void setDefaultUri(Configuration conf, String uri)
conf - the configuration to alteruri - the new default filesystem uri@InterfaceAudience.Private public static int getCacheSize()
public void initialize(URI name, Configuration conf) throws IOException
name - a uri whose authority section names the host, port, etc.
for this FileSystemconf - the configurationIOExceptionpublic abstract URI getUri()
protected URI getCanonicalUri()
NetUtils.getCanonicalUri(URI, int)protected int getDefaultPort()
public String getCanonicalServiceName()
SecurityUtil.buildDTServiceName(URI, int)public String getName()
public static FileSystem getNamed(String name, Configuration conf) throws IOException
IOExceptionpublic static LocalFileSystem getLocal(Configuration conf) throws IOException
conf - the configuration to configure the file system withIOExceptionpublic static FileSystem get(URI uri, Configuration conf) throws IOException
IOExceptionpublic static void closeAll()
throws IOException
IOExceptionpublic static void closeAllForUGI(UserGroupInformation ugi) throws IOException
ugi - IOExceptionpublic static FSDataOutputStream create(FileSystem fs, Path file, FsPermission permission) throws IOException
fs - file system handlefile - the name of the file to be createdpermission - the permission of the fileIOExceptionpublic static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission) throws IOException
fs - file system handledir - the name of the directory to be createdpermission - the permission of the directoryIOExceptioncreate(FileSystem, Path, FsPermission)protected void checkPath(Path path)
public BlockLocation[] getFileBlockLocations(FileStatus file, long start, long len) throws IOException
IOExceptionpublic abstract FSDataInputStream open(Path f, int bufferSize) throws IOException
f - the file name to openbufferSize - the size of the buffer to be used.IOExceptionpublic FSDataInputStream open(Path f) throws IOException
f - the file to openIOExceptionpublic FSDataOutputStream create(Path f) throws IOException
IOExceptionpublic FSDataOutputStream create(Path f, boolean overwrite) throws IOException
IOExceptionpublic FSDataOutputStream create(Path f, Progressable progress) throws IOException
IOExceptionpublic FSDataOutputStream create(Path f, short replication) throws IOException
IOExceptionpublic FSDataOutputStream create(Path f, short replication, Progressable progress) throws IOException
IOExceptionpublic FSDataOutputStream create(Path f, boolean overwrite, int bufferSize) throws IOException
f - the file name to openoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.IOExceptionpublic FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, Progressable progress) throws IOException
f - the file name to openoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.IOExceptionpublic FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize) throws IOException
f - the file name to openoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.IOExceptionpublic FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f - the file name to openoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.IOExceptionpublic abstract FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f - the file name to openpermission - overwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - progress - IOExceptionsetPermission(Path, FsPermission)@Deprecated public FSDataOutputStream createNonRecursive(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f - the file name to openoverwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - progress - IOExceptionsetPermission(Path, FsPermission)@Deprecated public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
f - the file name to openpermission - overwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - progress - IOExceptionsetPermission(Path, FsPermission)public boolean createNewFile(Path f) throws IOException
IOExceptionpublic FSDataOutputStream append(Path f) throws IOException
f - the existing file to be appended.IOExceptionpublic FSDataOutputStream append(Path f, int bufferSize) throws IOException
f - the existing file to be appended.bufferSize - the size of the buffer to be used.IOExceptionpublic abstract FSDataOutputStream append(Path f, int bufferSize, Progressable progress) throws IOException
f - the existing file to be appended.bufferSize - the size of the buffer to be used.progress - for reporting progress if it is not null.IOExceptionpublic void concat(Path trg, Path[] srcs) throws IOException
trg - the path to the target destination.psrcs - the paths to the sources to use for the concatenation.IOException@Deprecated public short getReplication(Path src) throws IOException
src - file nameIOExceptionpublic boolean setReplication(Path src, short replication) throws IOException
src - file namereplication - new replicationIOExceptionpublic abstract boolean rename(Path src, Path dst) throws IOException
IOException@Deprecated public abstract boolean delete(Path f) throws IOException
IOExceptionpublic abstract boolean delete(Path f, boolean recursive) throws IOException
f - the path to delete.recursive - if path is a directory and set to
true, the directory is deleted else throws an exception. In
case of a file the recursive can be set to either true or false.IOExceptionpublic boolean deleteOnExit(Path f) throws IOException
f - the path to delete.IOExceptionprotected void processDeleteOnExit()
public boolean exists(Path f) throws IOException
f - source fileIOException@Deprecated public boolean isDirectory(Path f) throws IOException
IOExceptionpublic boolean isFile(Path f) throws IOException
IOException@Deprecated public long getLength(Path f) throws IOException
IOExceptionpublic ContentSummary getContentSummary(Path f) throws IOException
ContentSummary of a given Path.IOExceptionpublic abstract FileStatus[] listStatus(Path f) throws IOException
f - given pathIOExceptionpublic FileStatus[] listStatus(Path f, PathFilter filter) throws IOException
f - a path namefilter - the user-supplied path filterIOException - if encounter any problem while fetching the statuspublic FileStatus[] listStatus(Path[] files) throws IOException
files - a list of pathsIOExceptionpublic FileStatus[] listStatus(Path[] files, PathFilter filter) throws IOException
files - a list of pathsfilter - the user-supplied path filterIOExceptionpublic FileStatus[] globStatus(Path pathPattern) throws IOException
Return all the files that match filePattern and are not checksum files. Results are sorted by their names.
A filename pattern is composed of regular characters and special pattern matching characters, which are:
pathPattern - a regular expression specifying a pth patternIOExceptionpublic FileStatus[] globStatus(Path pathPattern, PathFilter filter) throws IOException
pathPattern - a regular expression specifying the path patternfilter - a user-supplied path filterIOException - if any I/O error occurs when fetching file statuspublic Path getHomeDirectory()
public Token<?> getDelegationToken(String renewer) throws IOException
renewer - the account name that is allowed to renew the token.IOExceptionpublic abstract void setWorkingDirectory(Path new_dir)
new_dir - public abstract Path getWorkingDirectory()
public boolean mkdirs(Path f) throws IOException
mkdirs(Path, FsPermission) with default permission.IOExceptionpublic abstract boolean mkdirs(Path f, FsPermission permission) throws IOException
IOExceptionpublic void copyFromLocalFile(Path src, Path dst) throws IOException
IOExceptionpublic void moveFromLocalFile(Path[] srcs, Path dst) throws IOException
IOExceptionpublic void moveFromLocalFile(Path src, Path dst) throws IOException
IOExceptionpublic void copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
throws IOException
IOExceptionpublic void copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path[] srcs,
Path dst)
throws IOException
IOExceptionpublic void copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path src,
Path dst)
throws IOException
IOExceptionpublic void copyToLocalFile(Path src, Path dst) throws IOException
IOExceptionpublic void moveToLocalFile(Path src, Path dst) throws IOException
IOExceptionpublic void copyToLocalFile(boolean delSrc,
Path src,
Path dst)
throws IOException
IOExceptionpublic Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
IOExceptionpublic void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
IOExceptionpublic void close()
throws IOException
close 在接口中 Closeableclose 在接口中 AutoCloseableIOExceptionpublic long getUsed()
throws IOException
IOException@Deprecated public long getBlockSize(Path f) throws IOException
IOException@Deprecated public long getDefaultBlockSize()
getDefaultBlockSize(Path) insteadpublic long getDefaultBlockSize(Path f)
f - path of file@Deprecated public short getDefaultReplication()
getDefaultReplication(Path) insteadpublic short getDefaultReplication(Path path)
path - of the filepublic abstract FileStatus getFileStatus(Path f) throws IOException
f - The path we want information fromFileNotFoundException - when the path does not exist;
IOException see specific implementationIOExceptionpublic FileChecksum getFileChecksum(Path f) throws IOException
f - The file pathIOExceptionpublic void setVerifyChecksum(boolean verifyChecksum)
verifyChecksum - public void setPermission(Path p, FsPermission permission) throws IOException
p - permission - IOExceptionpublic void setOwner(Path p, String username, String groupname) throws IOException
p - The pathusername - If it is null, the original username remains unchanged.groupname - If it is null, the original groupname remains unchanged.IOExceptionpublic void setTimes(Path p, long mtime, long atime) throws IOException
p - The pathmtime - Set the modification time of this file.
The number of milliseconds since Jan 1, 1970.
A value of -1 means that this call should not set modification time.atime - Set the access time of this file.
The number of milliseconds since Jan 1, 1970.
A value of -1 means that this call should not set access time.IOExceptionpublic static Map<String,FileSystem.Statistics> getStatistics()
getAllStatistics() insteadpublic static List<FileSystem.Statistics> getAllStatistics()
public static FileSystem.Statistics getStatistics(String scheme, Class<? extends FileSystem> cls)
cls - the class to lookuppublic static void clearStatistics()
public static void printStatistics()
throws IOException
IOExceptionCopyright © 2009 The Apache Software Foundation