Kafka Connector

Kafka Connector

Configure message recovery strategy for a Kafka mapping task

Configure message recovery strategy for a Kafka mapping task

When you configure message recovery for a Kafka mapping task, the Secure Agent can recover unprocessed messages from a failed mapping. When you enable message recovery for a Kafka mapping task, the Secure Agent stores source messages or message IDs in a recovery file. If the mapping task fails, run the mapping task in recovery mode to recover the messages that the Secure Agent did not process. For mappings with Kafka topics, the Secure Agent connects to the recovery topic as a durable subscriber so messages persist even if the mapping task ends.
You can configure message recovery for tasks that read from a Kafka source and writes to a Kafka, Oracle, IBM MQ, or Flat File target.
When you configure a mapping task, you can configure the
Recovery Strategy
property in the
Advanced Session Properties
section on the
Runtime Options
tab of the mapping task and select
Resume from the last checkpoint
. The Secure Agent saves the mapping state of operation and maintains target recovery tables. If the mapping aborts, stops, or terminates, the Secure Agent uses the saved recovery information to resume the mapping from the point of interruption.

Steps to Enable Message Recovery

Complete the following steps to enable message recovery for a Kafka mapping task:
  1. In the
    Advanced Session Properties
    on the
    Runtime Options
    tab of the mapping task, add the following properties:
    Session Property Name
    Session Property Value
    Commit on End of File
    No
    Commit Type
    Source
    Recovery Strategy
    Resume from last checkpoint
  2. Click
    Finish
    .
The Secure Agent stores the messages in the recovery file and a backup of the recovery file in the following directory:
<Informatica Cloud Secure Agent installation directory>\apps\Data_Integration_Server\data
The Secure Agent uses the following format for the recovery file name:
pmgmd_cciFile_<InternalSessionDetails><session_ID><partition_ID>
For example, the Secure Agent creates the recovery file with the following file name:
pmgmd_cciFile_00100s_mtt_0100CO0Z0000000000290300

0 COMMENTS

We’d like to hear from you!