User Guide

1.2
- 1.3
- 1.1
- 1.0

Back Next

Checkpointing and Guaranteed Delivery

PowerExchange CDC Publisher generates a checkpoint file after it sends the first change operation. As data streaming progresses, the CDC Publisher saves information about the last change operation processed to the checkpoint file. This checkpoint information is used to resume CDC Publisher apply processing after the CDC Publisher is restarted.

The CDC Publisher uses only one checkpoint file for each instance. By default, the file is named "checkpoint" and is created in the "checkpoint" subdirectory of an instance. You can change the file name and directory by specifying the Connector.kafkaFileCheckpointFileName and Connector.kafkaCheckpointFileDirectory properties in the cdcPublisherKafka.cfg configuration file.

The checkpoint file contains checkpoint information only for the last change operation processed. The checkpoint format is specific to the CDC Publisher so do not edit the checkpoint information.

When CDC Publisher writes a checkpoint depends on whether you set the Connector.kafkaProducerGuaranteeDelivery property to true or false in the cdcPublisherKafka.cfg file. With the default value of true, the CDC Publisher uses guaranteed delivery
to help avoid message loss and duplication. CDC Publisher writes a checkpoint after each message containing a change operation is successfully delivered to the target messaging system. CDC Publisher overrides some CDC Publisher connector properties to force a single stream of change operations to be delivered one at a time to a single topic. Also, CDC Publisher sets the retries parameter in the Kafka Streams configuration to 0, unless the Kafka enable.idempotence parameter is set to true in the producer.properties file. When idempotence is enabled, CDC Publisher honors the existing retries parameter value, which must be greater than 0 for exactly-once delivery using the idempotence feature. If you set this property to false and the target messaging system terminates while changes are in flight, CDC Publisher does not skip any change operations after the change stream is restarted but might deliver duplicate messages to the target messaging system.

To send change data to multiple partitions in any single topic on the target messaging system, see Configuring CDC Publisher to Support Multiple Partitions in Individual Topics.

The following considerations pertain to using checkpoints and checkpoint files:

If connectivity to the target messaging system is lost or the CDC Publisher terminates, the CDC Publisher process will restart from the checkpoint position that is recorded in the checkpoint file by default. In this situation, some messages might be duplicated on the target messaging system. To guarantee that messages are not duplicated, ensure that the Connector.kafkaProducerGuaranteeDelivery property is set to true.

If the PowerExchange CDC Publisher process ends abnormally, the checkpoint value in the checkpoint file might not be accurate in the following situations:

The existing checkpoint value does not reflect the last change operation because the change stream terminated after the CDC Publisher sent a message to the target messaging system and before the target acknowledged the message as received. In this case, the CDC Publisher still restarts from the existing checkpoint value. Some messages that were previously sent to the target messaging system might be resent.

The checkpoint file is corrupted. This situation can occur if an attempt to write a checkpoint value to the checkpoint file failed or did not complete. In this case, delete the checkpoint file. Then configure a restart point by setting the Extract.restart1 and Extract.restart2 properties in the cdcPublisherPowerExchange.cfg file. When you restart the CDC Publisher, use the RESTART=FROM_CONFIG parameter. If you do not configure a specific restart point, the CDC Publisher restarts from the oldest point in the log files, as if RESTART=FROM_BEGINNING is specified.

Rename Saved Search

Table of Contents

User Guide

User Guide

Checkpointing and Guaranteed Delivery

Checkpointing and Guaranteed Delivery