Table of Contents

Search

  1. Preface
  2. PowerExchange CDC Publisher Overview
  3. Installing and Upgrading PowerExchange CDC Publisher
  4. PowerExchange CDC Publisher Key Concepts
  5. PowerExchange Change Capture Environment
  6. Target Messaging Systems
  7. Configuring PowerExchange CDC Publisher
  8. Streaming Change Data
  9. Monitoring PowerExchange CDC Publisher
  10. Administering PowerExchange CDC Publisher
  11. Appendix A: Command Reference for the Command-Line Utilities
  12. Appendix B: Avro Schema Formats
  13. Appendix C: Custom Pattern Formats
  14. Appendix D: Message Reference

User Guide

User Guide

Apache Kafka Data-Streaming Considerations

Apache Kafka Data-Streaming Considerations

The following data-streaming considerations and limitations apply to Kafka targets:
  • The Kafka producer.properties file controls PowerExchange CDC Publisher connectivity to Kafka.
  • The PowerExchange CDC Publisher supports Kafka primitive datatypes except FLOAT and DOUBLE. Also, the CDC Publisher tries to convert logical types for dates, times, timestamps, and decimal values on a best effort basis if you set the Formatter.avroUseLogical
    type
    property to true in the cdcPublisherAvro.cfg configuration file, where
    type
    is Date, Decimal, TimeMillis, TimeMicros, TimestampMillis, TimestampMicros. The CDC Publisher does not support complex types.
    The FLOAT and DOUBLE types are not supported because the CDC Publisher converts all source data to characters in Kafka messages. Any subsequent attempt to convert the character data to FLOAT or DOUBLE would decrease data accuracy and precision.
  • To minimize the risk of duplicate or missing messages on the Kafka target after a network outage, ensure that the Connector.kafkaProducerGuaranteeDelivery property is set to true, the default setting. Guaranteed delivery provides the highest level of data integrity but might degrade performance. You can disable guaranteed delivery by setting the Connector.kafkaProducerGuaranteeDelivery property to false in the cdcPublisherKafka.cfg configuration file.
  • If you use Confluent Schema Registry in the Kafka target environment, you can enable CDC Publisher to communicate with Confluent Schema Registry when sending formatted messages. Confluent Schema Registry can intercept and validate the messages against the registered Avro schema. For more information, see If You Use Confluent Schema Registry on a Kafka Target.
  • Except for the underscore character (_), PowerExchange CDC Publisher removes non-alphanumeric characters from a topic name by default. You can change this behavior, if needed. For more information, see the Connector.kafkaTopicAllowSpecialCharacters property description in Connector Configuration Properties.

0 COMMENTS

We’d like to hear from you!