org.apache.hadoop.hive.llap

Class LlapCacheAwareFs

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.hive.llap.LlapCacheAwareFs

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

public class LlapCacheAwareFs
extends org.apache.hadoop.fs.FileSystem

This is currently only used by Parquet; however, a generally applicable approach is used - you pass in a set of offset pairs for a file, and the file is cached with these boundaries. Don't add anything format specific here.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static String	SCHEME

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
LlapCacheAwareFs()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path arg0, int arg1, org.apache.hadoop.util.Progressable arg2)
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path arg0, org.apache.hadoop.fs.permission.FsPermission arg1, boolean arg2, int arg3, short arg4, long arg5, org.apache.hadoop.util.Progressable arg6)
boolean	delete(org.apache.hadoop.fs.Path arg0, boolean arg1)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path arg0)
URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory()
void	initialize(URI uri, org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path arg0)
boolean	mkdirs(org.apache.hadoop.fs.Path arg0, org.apache.hadoop.fs.permission.FsPermission arg1)
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path path, int bufferSize)
static org.apache.hadoop.fs.Path	registerFile(org.apache.hadoop.hive.common.io.DataCache cache, org.apache.hadoop.fs.Path path, Object fileKey, TreeMap<Long,Long> index, org.apache.hadoop.conf.Configuration conf, String tag)
boolean	rename(org.apache.hadoop.fs.Path arg0, org.apache.hadoop.fs.Path arg1)
void	setWorkingDirectory(org.apache.hadoop.fs.Path arg0)
static void	unregisterFile(org.apache.hadoop.fs.Path cachePath)

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, addDelegationTokens, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, close, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getScheme, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SCHEME

public static final String SCHEME

See Also:

Constant Field Values

Constructor Detail

LlapCacheAwareFs

public LlapCacheAwareFs()

Method Detail

registerFile

public static org.apache.hadoop.fs.Path registerFile(org.apache.hadoop.hive.common.io.DataCache cache,
                                                     org.apache.hadoop.fs.Path path,
                                                     Object fileKey,
                                                     TreeMap<Long,Long> index,
                                                     org.apache.hadoop.conf.Configuration conf,
                                                     String tag)
                                              throws IOException

Throws:

IOException

unregisterFile

public static void unregisterFile(org.apache.hadoop.fs.Path cachePath)

initialize

public void initialize(URI uri,
                       org.apache.hadoop.conf.Configuration conf)
                throws IOException

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path path,
                                                   int bufferSize)
                                            throws IOException

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getUri

public URI getUri()

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path arg0)

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path arg0,
                                                      int arg1,
                                                      org.apache.hadoop.util.Progressable arg2)
                                               throws IOException

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path arg0,
                                                      org.apache.hadoop.fs.permission.FsPermission arg1,
                                                      boolean arg2,
                                                      int arg3,
                                                      short arg4,
                                                      long arg5,
                                                      org.apache.hadoop.util.Progressable arg6)
                                               throws IOException

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path arg0,
                      boolean arg1)
               throws IOException

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path arg0)
                                              throws IOException

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path arg0)
                                             throws FileNotFoundException,
                                                    IOException

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Throws:

FileNotFoundException

IOException

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path arg0,
                      org.apache.hadoop.fs.permission.FsPermission arg1)
               throws IOException

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

rename

public boolean rename(org.apache.hadoop.fs.Path arg0,
                      org.apache.hadoop.fs.Path arg1)
               throws IOException

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Throws:

IOException