When you re-run mappings enabled for incremental load, the mapping does not read records from previous runs.

To enable incremental load, configure the consumer property group ID in the

Consumer Configuration Properties

source advanced property. The group ID consumer property is the name of the consumer group to which the Kafka consumer belongs.

Enter the group ID of the consumer group in the

key1=value1

format.

For example, enter the following group ID of the consumer group:

group.id=consumer-1

.

To configure multiple consumer properties, separate each key-value pair with a comma.

When you configure the consumer property group ID, ensure that you set the

Start Position Offset

property to

Earliest

and the

Connector Mode

property to

Batch

.

The offset of the next record for the configured consumer group is stored after the successful completion of each mapping run. When you re-run the mapping, it starts reading data from the stored offset point.

If you set the

Start Position Offset

property to

Custom

and configure the

Custom Start Position Timestamp

property, the mapping ignores the recorded offset value saved from the previous run and starts reading data from the value set in the

Custom Start Position Timestamp

property. This option helps you customize exactly where in the Kafka log the mapping must start reading the data.

If you do not configure the group ID consumer property, the mapping automatically configures a unique consumer group name and does not incrementally load the data. Each mapping re-run reads all existing records in the source topic.