Installation and Configuration Guide

10.1 HotFix 1
- 10.1

Back Next

Resetting the Offset Position for a Spark Instance

When you redeploy an existing Spark instance after a failure, the Spark instance processes the records in the Kafka input topic from the point of failure by default. If you want to process the records from the beginning of the topic or from the latest position of the topic, reset the offset position for the Spark instance before you redeploy it.

To reset the offset position, use the

kafka-consumer-groups.sh

script located in the following directory:

$KAFKA_HOME/bin

To run the

kafka-consumer-groups.sh

script, use the following command:

kafka-consumer-groups.sh

--bootstrap-server <list_of_kafka_brokers_with_ports>

--group <consumer_group_name>

--topic <kafka_input_topic>

--reset-offsets

--to-earliest|--to-latest

-–execute

The following table describes the options and arguments that you can specify to run the

kafka-consumer-groups.sh

script:

Option	Argument	Description
--bootstrap-server	list_of_kafka_brokers_with_ports	Comma-separated list of Kafka brokers and their port numbers. For example, broker1:9092,broker2:9092 .
--group	consumer_group_name	Name of the Kafka consumer group. The name is based on the Spark instance name and uses the following format: <Spark instance Name>-spark-bdrm For example, if the instance name is BDRMRTIngestSpark , then the consumer group name is BDRMRTIngestSpark-spark-bdrm .
--topic	kafka_input_topic	Name of the topic to which Kafka publishes the input data stream.
--reset-offsets		Resets the offset position for the Spark instance.
--to-earliest \| --to-latest		Indicates the offset position to set. Use one of the following options: --to-earliest . Resets the offset position to the beginning of the topic. --to-latest . Resets the offset position to the latest position of the topic.
--execute		Runs the script.