HyperLogLog (Hive 3.1.1 API)

java.lang.Object
- org.apache.hadoop.hive.common.ndv.hll.HyperLogLog

All Implemented Interfaces:: NumDistinctValueEstimator

public class HyperLogLog
extends Object
implements NumDistinctValueEstimator

 This is an implementation of the following variants of hyperloglog (HLL)
 algorithm 
 Original  - Original HLL algorithm from Flajolet et. al from
             http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf
 HLLNoBias - Google's implementation of bias correction based on lookup table
             http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf
 HLL++     - Google's implementation of HLL++ algorithm that uses SPARSE registers
             http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf
 
 Following are the constructor parameters that determines which algorithm is
 used
 numRegisterIndexBits - number of LSB hashcode bits to be used as register index.
                        Default is 14. min = 4 and max = 16
 numHashBits - number of bits for hashcode. Default is 64. min = 32 and max = 128
 encoding - Type of encoding to use (SPARSE or DENSE). The algorithm automatically
            switches to DENSE beyond a threshold. Default: SPARSE
 enableBitPacking - To enable bit packing or not. Bit packing improves compression
                    at the cost of more CPU cycles. Default: true
 noBias - Use Google's bias table lookup for short range bias correction.
          Enabling this will highly improve the estimation accuracy for short
          range values. Default: true

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class HyperLogLog.EncodingType

static class HyperLogLog.HyperLogLogBuilder

Nested Classes
Modifier and Type	Class and Description
`static class`	`HyperLogLog.EncodingType`
`static class`	`HyperLogLog.HyperLogLogBuilder`

Field Summary
- Fields inherited from interface org.apache.hadoop.hive.common.ndv.NumDistinctValueEstimator
  LOG

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`add(long hashcode)`
`void`	`addBoolean(boolean val)`
`void`	`addByte(byte val)`
`void`	`addBytes(byte[] val)`
`void`	`addChar(char val)`
`void`	`addDouble(double val)`
`void`	`addFloat(float val)`
`void`	`addInt(int val)`
`void`	`addLong(long val)`
`void`	`addShort(short val)`
`void`	`addString(String val)` Java's default charset will be used for strings.
`void`	`addString(String val, Charset charset)`
`void`	`addToEstimator(double d)`
`void`	`addToEstimator(org.apache.hadoop.hive.common.type.HiveDecimal decimal)`
`void`	`addToEstimator(long v)`
`void`	`addToEstimator(String s)`
`static HyperLogLog.HyperLogLogBuilder`	`builder()`
`boolean`	`canMerge(NumDistinctValueEstimator o)`
`long`	`count()`
`NumDistinctValueEstimator`	`deserialize(byte[] buf)`
`boolean`	`equals(Object obj)`
`long`	`estimateNumDistinctValues()`
`HyperLogLog.EncodingType`	`getEncoding()`
`HLLDenseRegister`	`getHLLDenseRegister()`
`HLLSparseRegister`	`getHLLSparseRegister()`
`int`	`getNumRegisterIndexBits()`
`double`	`getStandardError()`
`int`	`hashCode()`
`int`	`lengthFor(org.apache.hadoop.hive.ql.util.JavaDataModel model)`
`void`	`merge(HyperLogLog hll)` Merge the specified hyperloglog to the current one.
`void`	`mergeEstimators(NumDistinctValueEstimator o)`
`void`	`reset()`
`byte[]`	`serialize()`
`void`	`setCount(long count)`
`void`	`setEncoding(HyperLogLog.EncodingType encoding)`
`void`	`setHLLDenseRegister(byte[] reg)` Reconstruct dense registers from byte array
`void`	`setHLLSparseRegister(int[] reg)` Reconstruct sparse map from serialized integer list
`HyperLogLog`	`squash(int p0)` Reduces the accuracy of the HLL provided to a smaller size
`String`	`toString()`
`String`	`toStringExtended()`

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

- Method Detail
  - builder
```
public static HyperLogLog.HyperLogLogBuilder builder()
```
  - addBoolean
```
public void addBoolean(boolean val)
```
  - addByte
```
public void addByte(byte val)
```
  - addBytes
```
public void addBytes(byte[] val)
```
  - addShort
```
public void addShort(short val)
```
  - addInt
```
public void addInt(int val)
```
  - addLong
```
public void addLong(long val)
```
  - addFloat
```
public void addFloat(float val)
```
  - addDouble
```
public void addDouble(double val)
```
  - addChar
```
public void addChar(char val)
```
  - addString
```
public void addString(String val)
```
    Java's default charset will be used for strings.
    
    Parameters:
    
    val - - input string
  - addString
```
public void addString(String val,
                      Charset charset)
```
  - add
```
public void add(long hashcode)
```
  - estimateNumDistinctValues
```
public long estimateNumDistinctValues()
```
    Specified by:
    
    estimateNumDistinctValues in interface NumDistinctValueEstimator
  - count
```
public long count()
```
  - setCount
```
public void setCount(long count)
```
  - getStandardError
```
public double getStandardError()
```
  - getHLLDenseRegister
```
public HLLDenseRegister getHLLDenseRegister()
```
  - getHLLSparseRegister
```
public HLLSparseRegister getHLLSparseRegister()
```
  - setHLLSparseRegister
```
public void setHLLSparseRegister(int[] reg)
```
    Reconstruct sparse map from serialized integer list
    
    Parameters:
    
    reg - - uncompressed and delta decoded integer list
  - setHLLDenseRegister
```
public void setHLLDenseRegister(byte[] reg)
```
    Reconstruct dense registers from byte array
    
    Parameters:
    
    reg - - unpacked byte array
  - merge
```
public void merge(HyperLogLog hll)
```
    Merge the specified hyperloglog to the current one. Encoding switches automatically after merge if the encoding switch threshold is exceeded.
    
    Parameters:
    
    hll - - hyperloglog to be merged
    
    Throws:
    
    IllegalArgumentException
  - squash
```
public HyperLogLog squash(int p0)
```
    Reduces the accuracy of the HLL provided to a smaller size
    
    Parameters:
    
    p0 - - new p size for the new HyperLogLog (smaller or no change)
    
    Returns:
    
    reduced (or same) HyperLogLog instance
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - toStringExtended
```
public String toStringExtended()
```
  - getNumRegisterIndexBits
```
public int getNumRegisterIndexBits()
```
  - getEncoding
```
public HyperLogLog.EncodingType getEncoding()
```
  - setEncoding
```
public void setEncoding(HyperLogLog.EncodingType encoding)
```
  - equals
```
public boolean equals(Object obj)
```
    Overrides:
    
    equals in class Object
  - hashCode
```
public int hashCode()
```
    Overrides:
    
    hashCode in class Object
  - reset
```
public void reset()
```
    Specified by:
    
    reset in interface NumDistinctValueEstimator
  - serialize
```
public byte[] serialize()
```
    Specified by:
    
    serialize in interface NumDistinctValueEstimator
  - deserialize
```
public NumDistinctValueEstimator deserialize(byte[] buf)
```
    Specified by:
    
    deserialize in interface NumDistinctValueEstimator
  - addToEstimator
```
public void addToEstimator(long v)
```
    Specified by:
    
    addToEstimator in interface NumDistinctValueEstimator
  - addToEstimator
```
public void addToEstimator(String s)
```
    Specified by:
    
    addToEstimator in interface NumDistinctValueEstimator
  - addToEstimator
```
public void addToEstimator(double d)
```
    Specified by:
    
    addToEstimator in interface NumDistinctValueEstimator
  - addToEstimator
```
public void addToEstimator(org.apache.hadoop.hive.common.type.HiveDecimal decimal)
```
    Specified by:
    
    addToEstimator in interface NumDistinctValueEstimator
  - mergeEstimators
```
public void mergeEstimators(NumDistinctValueEstimator o)
```
    Specified by:
    
    mergeEstimators in interface NumDistinctValueEstimator
  - lengthFor
```
public int lengthFor(org.apache.hadoop.hive.ql.util.JavaDataModel model)
```
    Specified by:
    
    lengthFor in interface NumDistinctValueEstimator
  - canMerge
```
public boolean canMerge(NumDistinctValueEstimator o)
```
    Specified by:
    
    canMerge in interface NumDistinctValueEstimator

Class HyperLogLog

Nested Class Summary

Field Summary

Fields inherited from interface org.apache.hadoop.hive.common.ndv.NumDistinctValueEstimator

Method Summary

Methods inherited from class java.lang.Object

Method Detail

builder

addBoolean

addByte

addBytes

addShort

addInt

addLong

addFloat

addDouble

addChar

addString

addString

add

estimateNumDistinctValues

count

setCount

getStandardError

getHLLDenseRegister

getHLLSparseRegister

setHLLSparseRegister

setHLLDenseRegister

merge

squash

toString

toStringExtended

getNumRegisterIndexBits

getEncoding

setEncoding

equals

hashCode

reset

serialize

deserialize

addToEstimator

addToEstimator

addToEstimator

addToEstimator

mergeEstimators

lengthFor

canMerge