org.apache.hadoop.hive.llap.cache

Class SimpleBufferManager

java.lang.Object

org.apache.hadoop.hive.llap.cache.SimpleBufferManager

All Implemented Interfaces:

BufferUsageManager, LlapOomDebugDump, LowLevelCache

public class SimpleBufferManager
extends Object
implements BufferUsageManager, LowLevelCache

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.llap.cache.LowLevelCache

LowLevelCache.Priority

Constructor Summary

Constructors

Constructor and Description
SimpleBufferManager(org.apache.hadoop.hive.common.io.Allocator allocator, LlapDaemonCacheMetrics metrics)

Method Summary

Modifier and Type	Method and Description
String	debugDumpForOom()
void	debugDumpShort(StringBuilder sb)
void	decRefBuffer(org.apache.hadoop.hive.common.io.encoded.MemoryBuffer buffer)
void	decRefBuffers(List<org.apache.hadoop.hive.common.io.encoded.MemoryBuffer> cacheBuffers)
org.apache.hadoop.hive.common.io.Allocator	getAllocator()
org.apache.hadoop.hive.common.io.DiskRangeList	getFileData(Object fileKey, org.apache.hadoop.hive.common.io.DiskRangeList range, long baseOffset, org.apache.hadoop.hive.common.io.DataCache.DiskRangeListFactory factory, LowLevelCacheCounters qfCounters, org.apache.hadoop.hive.common.io.DataCache.BooleanRef gotAllData) Gets file data for particular offsets.
boolean	incRefBuffer(org.apache.hadoop.hive.common.io.encoded.MemoryBuffer buffer)
void	notifyEvicted(org.apache.hadoop.hive.common.io.encoded.MemoryBuffer buffer) Notifies the cache that a particular buffer should be removed due to eviction.
long[]	putFileData(Object fileKey, org.apache.hadoop.hive.common.io.DiskRange[] ranges, org.apache.hadoop.hive.common.io.encoded.MemoryBuffer[] chunks, long baseOffset, LowLevelCache.Priority priority, LowLevelCacheCounters qfCounters, String tag) Puts file data into cache.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleBufferManager

public SimpleBufferManager(org.apache.hadoop.hive.common.io.Allocator allocator,
                           LlapDaemonCacheMetrics metrics)

Method Detail

decRefBuffer

public void decRefBuffer(org.apache.hadoop.hive.common.io.encoded.MemoryBuffer buffer)

Specified by:

decRefBuffer in interface BufferUsageManager

decRefBuffers

public void decRefBuffers(List<org.apache.hadoop.hive.common.io.encoded.MemoryBuffer> cacheBuffers)

Specified by:

decRefBuffers in interface BufferUsageManager

incRefBuffer

public boolean incRefBuffer(org.apache.hadoop.hive.common.io.encoded.MemoryBuffer buffer)

Specified by:

incRefBuffer in interface BufferUsageManager

getAllocator

public org.apache.hadoop.hive.common.io.Allocator getAllocator()

Specified by:

getAllocator in interface BufferUsageManager

getFileData

public org.apache.hadoop.hive.common.io.DiskRangeList getFileData(Object fileKey,
                                                                  org.apache.hadoop.hive.common.io.DiskRangeList range,
                                                                  long baseOffset,
                                                                  org.apache.hadoop.hive.common.io.DataCache.DiskRangeListFactory factory,
                                                                  LowLevelCacheCounters qfCounters,
                                                                  org.apache.hadoop.hive.common.io.DataCache.BooleanRef gotAllData)

Description copied from interface: LowLevelCache

Gets file data for particular offsets. The range list is modified in place; it is then returned (since the list head could have changed). Ranges are replaced with cached ranges. In case of partial overlap with cached data, full cache blocks are always returned; there's no capacity for partial matches in return type. The rules are as follows: 1) If the requested range starts in the middle of a cached range, that cached range will not be returned by default (e.g. if [100,200) and [200,300) are cached, the request for [150,300) will only return [200,300) from cache). This may be configurable in impls. This is because we assume well-known range start offsets are used (rg/stripe offsets), so a request from the middle of the start doesn't make sense. 2) If the requested range ends in the middle of a cached range, that entire cached range will be returned (e.g. if [100,200) and [200,300) are cached, the request for [100,250) will return both ranges). It should really be same as #1, however currently ORC uses estimated end offsets; we do in fact know in such cases that partially-matched cached block (rg) can be thrown away, the reader will never touch it; but we need code in the reader to handle such cases to avoid disk reads for these "tails" vs real unmatched ranges. Some sort of InvalidCacheChunk could be placed to avoid them. TODO

Specified by:

getFileData in interface LowLevelCache

putFileData

public long[] putFileData(Object fileKey,
                          org.apache.hadoop.hive.common.io.DiskRange[] ranges,
                          org.apache.hadoop.hive.common.io.encoded.MemoryBuffer[] chunks,
                          long baseOffset,
                          LowLevelCache.Priority priority,
                          LowLevelCacheCounters qfCounters,
                          String tag)

Description copied from interface: LowLevelCache

Puts file data into cache.

Specified by:

putFileData in interface LowLevelCache

Returns:

null if all data was put; bitmask indicating which chunks were not put otherwise; the replacement chunks from cache are updated directly in the array.

notifyEvicted

public void notifyEvicted(org.apache.hadoop.hive.common.io.encoded.MemoryBuffer buffer)

Description copied from interface: LowLevelCache

Notifies the cache that a particular buffer should be removed due to eviction.

Specified by:

notifyEvicted in interface LowLevelCache

debugDumpForOom

public String debugDumpForOom()

Specified by:

debugDumpForOom in interface LlapOomDebugDump

debugDumpShort

public void debugDumpShort(StringBuilder sb)

Specified by:

debugDumpShort in interface LlapOomDebugDump