Table of Contents

Search

  1. Preface
  2. Installing Informatica MDM - Relate 360
  3. Configuring Relate 360
  4. Configuring Security
  5. Setting Up the Environment to Process Streaming Data
  6. Configuring Distributed Search
  7. Packaging and Deploying the RESTful Web Services
  8. Troubleshooting

Installation and Configuration Guide

Installation and Configuration Guide

Resetting the Offset Position for a Spark Instance

Resetting the Offset Position for a Spark Instance

When you redeploy an existing Spark instance after a failure, the Spark instance processes the records in the Kafka input topic from the point of failure by default. If you want to process the records from the beginning of the topic or from the latest position of the topic, reset the offset position for the Spark instance before you redeploy it.
To reset the offset position, use the kafka-consumer-groups.sh script located in the following directory: $KAFKA_HOME/bin
To run the kafka-consumer-groups.sh script, use the following command:
kafka-consumer-groups.sh

--bootstrap-server <list_of_kafka_brokers_with_ports>

--group <consumer_group_name>

--topic <kafka_input_topic>

--reset-offsets

--to-earliest|--to-latest

-–execute
The following table describes the options and arguments that you can specify to run the kafka-consumer-groups.sh script:
Option Argument Description
--bootstrap-server list_of_kafka_brokers_with_ports Comma-separated list of Kafka brokers and their port numbers. For example, broker1:9092,broker2:9092.
--group consumer_group_name Name of the Kafka consumer group. The name is based on the Spark instance name and uses the following format:
<Spark instance Name>-spark-bdrm
For example, if the instance name is BDRMRTIngestSpark, then the consumer group name is BDRMRTIngestSpark-spark-bdrm.
--topic kafka_input_topic Name of the topic to which Kafka publishes the input data stream.
--reset-offsets Resets the offset position for the Spark instance.
--to-earliest | --to-latest Indicates the offset position to set.
Use one of the following options:
  • --to-earliest. Resets the offset position to the beginning of the topic.
  • --to-latest. Resets the offset position to the latest position of the topic.
--execute Runs the script.
After you successfully run the kafka-consumer-groups.sh script, redeploy the Spark instance.


Updated June 27, 2019