The following data-streaming considerations and limitations apply to Kafka targets:
The Kafka producer.properties file controls PowerExchange CDC Publisher connectivity to Kafka.
The PowerExchange CDC Publisher supports Kafka primitive datatypes except FLOAT and DOUBLE. Also, the CDC Publisher tries to convert logical types for dates, times, timestamps, and decimal values on a best effort basis if you set the Formatter.avroUseLogical
type
property to true in the cdcPublisherAvro.cfg configuration file, where
type
is Date, Decimal, TimeMillis, TimeMicros, TimestampMillis, TimestampMicros. The CDC Publisher does not support complex types.
The FLOAT and DOUBLE types are not supported because the CDC Publisher converts all source data to characters in Kafka messages. Any subsequent attempt to convert the character data to FLOAT or DOUBLE would decrease data accuracy and precision.
To minimize the risk of duplicate or missing messages on the Kafka target after a network outage, ensure that the Connector.kafkaProducerGuaranteeDelivery property is set to true, the default setting. Guaranteed delivery provides the highest level of data integrity but might degrade performance. You can disable guaranteed delivery by setting the Connector.kafkaProducerGuaranteeDelivery property to false in the cdcPublisherKafka.cfg configuration file.
If you use Confluent Schema Registry in the Kafka target environment, you can enable CDC Publisher to communicate with Confluent Schema Registry when sending formatted messages. Confluent Schema Registry can intercept and validate the messages against the registered Avro schema. For more information, see
If You Use Confluent Schema Registry on a Kafka Target.
Except for the underscore character (_), PowerExchange CDC Publisher removes non-alphanumeric characters from a topic name by default. You can change this behavior, if needed. For more information, see the Connector.kafkaTopicAllowSpecialCharacters property description in
Connector Configuration Properties.