HiveStreamingConnection (Hive 3.1.1 API)

java.lang.Object
- org.apache.hive.streaming.HiveStreamingConnection

All Implemented Interfaces:: ConnectionInfo, PartitionHandler, StreamingConnection

public class HiveStreamingConnection
extends Object
implements StreamingConnection

Streaming connection implementation for hive. To create a streaming connection, use the builder API to create record writer first followed by the connection itself. Once connection is created, clients can begin a transaction, keep writing using the connection, commit the transaction and close connection when done. To bind to the correct metastore, HiveConf object has to be created from hive-site.xml or HIVE_CONF_DIR. If hive conf is manually created, metastore uri has to be set correctly. If hive conf object is not specified, "thrift://localhost:9083" will be used as default.

NOTE: The streaming connection APIs and record writer APIs are not thread-safe. Streaming connection creation, begin/commit/abort transactions, write and close has to be called in the same thread. If close() or abortTransaction() has to be triggered from a separate thread it has to be co-ordinated via external variables or synchronization mechanism

Example usage:


 // create delimited record writer whose schema exactly matches table schema
 StrictDelimitedInputWriter writer = StrictDelimitedInputWriter.newBuilder()
                                      .withFieldDelimiter(',')
                                      .build();
 // create and open streaming connection (default.src table has to exist already)
 StreamingConnection connection = HiveStreamingConnection.newBuilder()
                                    .withDatabase("default")
                                    .withTable("src")
                                    .withAgentInfo("nifi-agent")
                                    .withRecordWriter(writer)
                                    .withHiveConf(hiveConf)
                                    .connect();
 // begin a transaction, write records and commit 1st transaction
 connection.beginTransaction();
 connection.write("key1,val1".getBytes());
 connection.write("key2,val2".getBytes());
 connection.commitTransaction();
 // begin another transaction, write more records and commit 2nd transaction
 connection.beginTransaction();
 connection.write("key3,val3".getBytes());
 connection.write("key4,val4".getBytes());
 connection.commitTransaction();
 // close the streaming connection
 connection.close();

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class HiveStreamingConnection.Builder

static class HiveStreamingConnection.TxnState

Nested Classes
Modifier and Type	Class and Description
`static class`	`HiveStreamingConnection.Builder`
`static class`	`HiveStreamingConnection.TxnState`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`abortTransaction()` Manually abort the opened transaction.
`void`	`beginTransaction()` Begin a transaction for writing.
`void`	`close()` Close connection
`void`	`commitTransaction()` Commit a transaction to make the writes visible for readers.
`PartitionInfo`	`createPartitionIfNotExists(List<String> partitionValues)` Creates a partition if it does not exist.
`String`	`getAgentInfo()` Get agent info that is set during streaming connection.
`ConnectionStats`	`getConnectionStats()` Gets stats about the streaming connection.
`HiveConf`	`getHiveConf()` Returns hive configuration object used during connection creation.
`String`	`getMetastoreUri()` Get metastore URI that metastore client uses.
`List<String>`	`getStaticPartitionValues()` Get any static partitions specified during streaming connection creation.
`Table`	`getTable()` Get the table used by streaming connection.
`boolean`	`isDynamicPartitioning()` Get if dynamic partitioning is used.
`boolean`	`isPartitionedTable()` Get if the specified table is partitioned table or not.
`static HiveStreamingConnection.Builder`	`newBuilder()`
`String`	`toString()`
`void`	`write(byte[] record)` Write record using RecordWriter.
`void`	`write(InputStream inputStream)` Write record using RecordWriter.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Method Detail
  - newBuilder
```
public static HiveStreamingConnection.Builder newBuilder()
```
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - createPartitionIfNotExists
```
public PartitionInfo createPartitionIfNotExists(List<String> partitionValues)
                                         throws StreamingException
```
    Description copied from interface: PartitionHandler
    
    Creates a partition if it does not exist.
    
    Specified by:
    
    createPartitionIfNotExists in interface PartitionHandler
    
    Parameters:
    
    partitionValues - - partition values
    
    Returns:
    
    partition location
    
    Throws:
    
    StreamingException - - any metastore related exceptions
  - beginTransaction
```
public void beginTransaction()
                      throws StreamingException
```
    Description copied from interface: StreamingConnection
    
    Begin a transaction for writing.
    
    Specified by:
    
    beginTransaction in interface StreamingConnection
    
    Throws:
    
    StreamingException - - if there are errors when beginning transaction
  - commitTransaction
```
public void commitTransaction()
                       throws StreamingException
```
    Description copied from interface: StreamingConnection
    
    Commit a transaction to make the writes visible for readers.
    
    Specified by:
    
    commitTransaction in interface StreamingConnection
    
    Throws:
    
    StreamingException - - if there are errors when committing the open transaction
  - abortTransaction
```
public void abortTransaction()
                      throws StreamingException
```
    Description copied from interface: StreamingConnection
    
    Manually abort the opened transaction.
    
    Specified by:
    
    abortTransaction in interface StreamingConnection
    
    Throws:
    
    StreamingException - - if there are errors when aborting the transaction
  - write
```
public void write(byte[] record)
           throws StreamingException
```
    Description copied from interface: StreamingConnection
    
    Write record using RecordWriter.
    
    Specified by:
    
    write in interface StreamingConnection
    
    Parameters:
    
    record - - the data to be written
    
    Throws:
    
    StreamingException - - if there are errors when writing
  - write
```
public void write(InputStream inputStream)
           throws StreamingException
```
    Description copied from interface: StreamingConnection
    
    Write record using RecordWriter.
    
    Specified by:
    
    write in interface StreamingConnection
    
    Parameters:
    
    inputStream - - input stream of records
    
    Throws:
    
    StreamingException - - if there are errors when writing
  - close
```
public void close()
```
    Close connection
    
    Specified by:
    
    close in interface StreamingConnection
  - getConnectionStats
```
public ConnectionStats getConnectionStats()
```
    Description copied from interface: StreamingConnection
    
    Gets stats about the streaming connection.
    
    Specified by:
    
    getConnectionStats in interface StreamingConnection
    
    Returns:
    
    - connection stats
  - getHiveConf
```
public HiveConf getHiveConf()
```
    Description copied from interface: StreamingConnection
    
    Returns hive configuration object used during connection creation.
    
    Specified by:
    
    getHiveConf in interface StreamingConnection
    
    Returns:
    
    - hive conf
  - getMetastoreUri
```
public String getMetastoreUri()
```
    Description copied from interface: ConnectionInfo
    
    Get metastore URI that metastore client uses.
    
    Specified by:
    
    getMetastoreUri in interface ConnectionInfo
    
    Returns:
    
    - metastore URI used by client
  - getTable
```
public Table getTable()
```
    Description copied from interface: ConnectionInfo
    
    Get the table used by streaming connection.
    
    Specified by:
    
    getTable in interface ConnectionInfo
    
    Returns:
    
    - table
  - getStaticPartitionValues
```
public List<String> getStaticPartitionValues()
```
    Description copied from interface: ConnectionInfo
    
    Get any static partitions specified during streaming connection creation.
    
    Specified by:
    
    getStaticPartitionValues in interface ConnectionInfo
    
    Returns:
    
    - static partition values
  - getAgentInfo
```
public String getAgentInfo()
```
    Description copied from interface: ConnectionInfo
    
    Get agent info that is set during streaming connection.
    
    Specified by:
    
    getAgentInfo in interface ConnectionInfo
    
    Returns:
    
    - agent info
  - isPartitionedTable
```
public boolean isPartitionedTable()
```
    Description copied from interface: ConnectionInfo
    
    Get if the specified table is partitioned table or not.
    
    Specified by:
    
    isPartitionedTable in interface ConnectionInfo
    
    Returns:
    
    - true if partitioned table else false
  - isDynamicPartitioning
```
public boolean isDynamicPartitioning()
```
    Description copied from interface: ConnectionInfo
    
    Get if dynamic partitioning is used.
    
    Specified by:
    
    isDynamicPartitioning in interface ConnectionInfo
    
    Returns:
    
    - true if dynamic partitioning case else false

Class HiveStreamingConnection

Nested Class Summary

Method Summary

Methods inherited from class java.lang.Object

Method Detail

newBuilder

toString

createPartitionIfNotExists

beginTransaction

commitTransaction

abortTransaction

write

write

close

getConnectionStats

getHiveConf

getMetastoreUri

getTable

getStaticPartitionValues

getAgentInfo

isPartitionedTable

isDynamicPartitioning