Hive Connector

Back Next

Configure session recovery for a task that reads from Kafka

You can configure session recovery for a mapping task that reads from a Kafka source in batch mode and writes to a Hive target. If the task crashes while staging or after staging the data, you can rerun the task to recover the task from the last check point.

When you configure a recovery strategy for the task, the Secure Agent can recover unprocessed messages from a failed mapping. The Secure Agent stores source messages in a recovery file in the staging directory specified in the Hive connection properties. If the mapping task fails, run the mapping task in recovery mode to recover the messages that the Secure Agent did not process.

When you configure a mapping task, you can configure the

Recovery Strategy

property in the

Advanced Session Properties

section on the

Runtime Options

tab of the mapping task and select

Resume from the last checkpoint

. The Secure Agent saves the mapping state of operation and maintains target recovery tables. If the mapping aborts, stops, or terminates, the Secure Agent uses the saved recovery information to resume the mapping from the point of interruption.

Steps to enable message recovery

Verify that you have specified the DFS staging directory in the Hive connection properties and then perform the following tasks to enable message recovery for a mapping task:

In the

Advanced Session Properties

on the

Runtime Options

tab of the mapping task, add the following properties: