Communities
A collaborative platform to connect and grow with like-minded Informaticans across the globe
Product Communities
Connect and collaborate with Informatica experts and champions
Discussions
Have a question? Start a Discussion and get immediate answers you are looking for
User Groups
Customer-organized groups that meet online and in-person. Join today to network, share ideas, and get tips on how to get the most out of Informatica
Get Started
Community Guidelines
Knowledge Center
Troubleshooting documents, product guides, how to videos, best practices, and more
Knowledge Base
One-stop self-service portal for solutions, FAQs, Whitepapers, How Tos, Videos, and more
Support TV
Video channel for step-by-step instructions to use our products, best practices, troubleshooting tips, and much more
Documentation
Information library of the latest product documents
Velocity (Best Practices)
Best practices and use cases from the Implementation team
Learn
Rich resources to help you leverage full capabilities of our products
Trainings
Role-based training programs for the best ROI
Certifications
Get certified on Informatica products. Free, Foundation, or Professional
Product Learning Paths
Free and unlimited modules based on your expertise level and journey
Resources
Library of content to help you leverage the best of Informatica products
Tech Tuesdays Webinars
Most popular webinars on product architecture, best practices, and more
Product Availability Matrix
Product Availability Matrix statements of Informatica products
SupportFlash
Monthly support newsletter
Support Documents
Informatica Support Guide and Statements, Quick Start Guides, and Cloud Product Description Schedule
Product Lifecycle
End of Life statements of Informatica products
Ideas
Events
Change Request Tracking
Marketplace
English
  • PowerExchange CDC Publisher 1.3
  • All Products

Table of Contents

Search

  1. Preface
  2. PowerExchange CDC Publisher Overview
  3. Installing and Upgrading PowerExchange CDC Publisher
  4. PowerExchange CDC Publisher Key Concepts
  5. PowerExchange Change Capture Environment
  6. Target Messaging Systems
  7. Configuring PowerExchange CDC Publisher
  8. Streaming Change Data
  9. Monitoring PowerExchange CDC Publisher
  10. Administering PowerExchange CDC Publisher
  11. Appendix A: Command Reference for the Command-Line Utilities
  12. Appendix B: Avro Schema Formats
  13. Appendix C: Custom Pattern Formats
  14. Appendix D: Message Reference

User Guide

User Guide

Back Next

Publisher-Generated Messages for Data Delivery

Publisher-Generated Messages for Data Delivery

PowerExchange CDC Publisher creates a message for each change record that it receives for a source table or object and sends the message to the target messaging system.
PowerExchange CDC Publisher does not generate messages for DDL operations or for UOW begin records.
The format of all messages is based on the Avro schemas that the PowerExchange CDC Publisher generates. To generate an Avro schema, the PowerExchange CDC Publisher uses the PowerExchange extraction map for the source object.

Message Content

A message contains extracted change data and metadata. The following table describes the fields that are included in a generated message:
Field
Description
DTL__
xxx
The PowerExchange-generated DTL__ columns that have been added to extraction maps by default or by the PowerExchange user. For more information about these columns, see the
PowerExchange Navigator User Guide
.
INFA_OP_TYPE
The DML operation type (INSERT, UPDATE, or DELETE) that was extracted from the source.
INFA_TABLE_NAME
The source
mapname
_
tablename
 from the extraction map name. This value identifies the source object for which change data was extracted.
INFA_SEQUENCE
A sequence number that the CDC Publisher assigns to the change record.
source_column_name
The after image of a source column to which a change operation was applied.
source_column_name
_Present
An indicator of whether the column contains a value from a change operation.
source_column_name
_BeforeImage
The before image value of an updated source column.
source_column_name
_BeforeImage_Present
An indicator of whether the before image of an updated source column is present.
For UPDATE operations, the Avro messages include both the before image and after image. You can set the Formatter.avroIncludeBeforeImage property to false in the cdcPublisherAvro.cfg configuration file to not generate Avro fields for before images.
If you specify CAPTURE_IMAGE=AI in the PowerExchange Logger pwxccl.cfg file to capture after images only, no before image data will be available.

Message Key

PowerExchange CDC Publisher can produce messages that include a key value that consists of a specific character string or the source table name. The target messaging system can use the key value to send messages to a single partition in any target topic. To use this type of key value, specify the Connector.kafkaMessageKey property in the cdcPublisherKafka.cfg configuration file. For more information, see Connector Configuration Properties.
PowerExchange CDC Publisher can also generate a composite key value based on rules that you define for each source table in a rule-configuration file. You must specify the file in the
Formatter.captureColumnValuesFile=
pathFilename
 property in the cdcPublisherAvro.cfg configuration file. For more information, see Formatter Configuration Properties and Generating Composite Message Keys for Source Tables.
If you use the Custom Pattern Formatter, the Formatter.captureColumnValuesFile property also affects custom patterns. For more information, see Formatter Properties and Custom Patterns.

Avro Message Format and Encoding

PowerExchange CDC Publisher produces messages in Avro format with the encoding type that you select.
You control the Avro encoding by setting the Formatter.avroEncodingType configuration property in the cdcPublisherAvro.cfg configuration file. You can specify an encoding type of
JSON
,
binary
, or
none
. The CDC Publisher uses the specified encoding type when serializing records in an Avro format. Set the encoding type to
none
, which indicates no explicit encoding, if you use a third-party encoding schema, such as the Confluent Schema Registry encoding schema.
The PowerExchange CDC Publisher generates an Avro schema when it receives change data for a source table or object. The PowerExchange CDC Publisher can generate Avro schemas in one of the following formats, depending on how you set the Formatter.avroSchemaFormat property in the cdcPublisherAvro.cfg configuration file:
  • Flat
    . Lists all Avro fields in one Avro record. A unique Avro schema is generated for each source object.
  • Nested
    . Organizes each type of information in a separate Avro record. A unique Avro schema is generated for each source object.
  • Generic
    . Generates an Avro schema that accommodates more than one source object. The source column names are included in each record, which allows the generic schema to be independent of any source object. All of the PowerExchange-generated DTL__
    xxx
     columns are included as metadata.
Additionally, you can "wrap" a flat, nested, or generic schema with an Avro schema that acts as a header that contains metadata followed by the underlying Avro schema for the source object. To do so, set the Formatter.avroWrapperSchemaFormat property to avroWrapperSchemaFormatV1 in the cdcPublisherAvro.cfg file. The wrapper schema format contains the following Avro fields:
  • Sequence number of the change record
  • Change operation type
  • Source
    mapname_tablename
     from the extraction map name
  • The "wrapped" Avro schema of the type specified in the Formatter.avroSchemaFormat property
All messages based on the wrapper schema have the same four-field format.
You can use a generic or wrapper schema to represent multiple source tables. The consumer of data in the target topics examines the metadata to determine which source table is represented and discover the source table structure. The consumer can then parse the underlying data and put it into the proper Avro format. Consider using a generic or wrapper schema when you want to send messages with change data from multiple source objects to a single Kafka topic. The topic is identified in the Connector.kafkaTopic property.
A flat or nested schema pertains to a specific source table. It defines the columns in the source table. Consider using a flat or nested schema when you want to send change data from a specific source object to the Kafka topic that is generated for that source object. In this case, set the Connector.kafkaTopic property to USE_TABLE_NAME.

Supported Avro Types

The Avro schemas that the CDC Publisher generates support all Avro primitive types except FLOAT or DOUBLE, which are represented as strings. Also, if you use Avro logical types for dates, decimal values, times, or timestamps, the CDC Publisher makes a best-effort attempt to process the logical types under the following conditions:
  • A Formatter.avroUseLogical
    type
     property is set to true in the cdcPublisherAvro.cfg configuration file, where
    type
     is DateType, DecimalType, TimeMillisType, TimeMicrosType, TimestampMillisType, or TimestampMicrosType.
  • The source field is defined in the extraction map with a compatible data type, scale, and precision.
The CDC Publisher-generated Avro schemas do not convert a timestamp to a date type.
The Publisher-generated Avro schemas do not support complex types.

Avro Schema Usage Considerations

Consumer applications that read data from the messaging system must have a copy of the Avro schema to decode the message. You can use the PwxCDCAdmin utility to generate Avro schemas in a legible format that consumer applications can use. For more information, see Reporting the Avro Format Definitions for Source Tables.
If you change the structure of a source table or object and update the extraction map, or if you change any of the Avro-related properties in the cdcPublisherAvro.cfg configuration file, the PowerExchange CDC Publisher does not automatically update any existing Avro schema. You can use the PwxCDCAdmin utility to clear the existing Avro schema from cache and then regenerate the Avro schema. For more information, see Handling Changes to Source Tables and Extraction Maps.

Custom Pattern Formats

If the Formatter does not meet your message format requirements, you can use the Custom Pattern Formatter, which is an extension of the Formatter to create user-defined patterns that customize the resulting Kafka messages. The Custom Pattern Formatter uses the definitions you create to format the message output. You can create patterns in any text-based form without restrictions, for example, in comma separated (CSV), JSON, or other text-based format. The custom patterns use tags that are similar to HTML tags. The custom patterns describe how the change data is presented in the message output. You can provide a custom pattern to describe the output for source tables, and you can also specify a default pattern for tables for which a specific pattern is not defined. Custom patterns support date, time, and timestamp conversions using masks, and data type overrides. For more information, see Configuring the Custom Pattern Formatter.
0 COMMENTS
We’d like to hear from you!