Table of Contents

Search

  1. Preface
  2. Installing Informatica MDM - Relate 360
  3. Configuring Relate 360
  4. Configuring Security
  5. Setting Up the Environment to Process Streaming Data
  6. Configuring Distributed Search
  7. Packaging and Deploying the RESTful Web Services
  8. Troubleshooting

Installation and Configuration Guide

Installation and Configuration Guide

Resetting the Offset Position for a Spark Instance

Resetting the Offset Position for a Spark Instance

When you redeploy an existing Spark instance after a failure, the Spark instance processes the records in the Kafka input topic from the point of failure by default. If you want to process the records from the beginning of the topic or from the latest position of the topic, reset the offset position for the Spark instance before you redeploy it.
To reset the offset position, use the
kafka-consumer-groups.sh
script located in the following directory:
$KAFKA_HOME/bin
To run the
kafka-consumer-groups.sh
script, use the following command:
kafka-consumer-groups.sh --bootstrap-server <list_of_kafka_brokers_with_ports> --group <consumer_group_name> --topic <kafka_input_topic> --reset-offsets --to-earliest|--to-latest -–execute
The following table describes the options and arguments that you can specify to run the
kafka-consumer-groups.sh
script:
Option
Argument
Description
--bootstrap-server
list_of_kafka_brokers_with_ports
Comma-separated list of Kafka brokers and their port numbers. For example,
broker1:9092,broker2:9092
.
--group
consumer_group_name
Name of the Kafka consumer group. The name is based on the Spark instance name and uses the following format:
<Spark instance Name>-spark-bdrm
For example, if the instance name is
BDRMRTIngestSpark
, then the consumer group name is
BDRMRTIngestSpark-spark-bdrm
.
--topic
kafka_input_topic
Name of the topic to which Kafka publishes the input data stream.
--reset-offsets
Resets the offset position for the Spark instance.
--to-earliest | --to-latest
Indicates the offset position to set.
Use one of the following options:
  • --to-earliest
    . Resets the offset position to the beginning of the topic.
  • --to-latest
    . Resets the offset position to the latest position of the topic.
--execute
Runs the script.
After you successfully run the
kafka-consumer-groups.sh
script, redeploy the Spark instance.


Updated November 20, 2019