Table of Contents

Search

  1. Preface
  2. PowerExchange CDC Publisher Overview
  3. Installing and Upgrading PowerExchange CDC Publisher
  4. PowerExchange CDC Publisher Key Concepts
  5. PowerExchange Change Capture Environment
  6. Apache Kafka Targets
  7. Configuring PowerExchange CDC Publisher
  8. Streaming Change Data
  9. Monitoring PowerExchange CDC Publisher
  10. Administering PowerExchange CDC Publisher
  11. Command Reference for the Command-Line Utilities
  12. Avro Schema Formats

User Guide

User Guide

Avro Formatter Configuration Properties

Avro Formatter Configuration Properties

The sample cdcPublisherAvro.cfg file contains configuration properties that define the format of the generated Avro schema and the encoding type to use for serializing the Avro records to be included in messages. You must set the schema format and encoding type.

Property Descriptions

The following properties are in the sample cdcPublisherAvro.cfg file:
Formatter.formatterType
The type of data serialization formatter to use for messages. The only valid value is
Avro
.
Formatter.avroSchemaFormat
Required. The Avro schema format that the PowerExchange CDC Publisher uses to generate the Avro schema that will determine the structure of the message values. Valid values are:
  • avroFlatSchemaFormatV1
    . Structures messages by using a flat Avro schema format, which lists all Avro fields in one Avro record. A unique Avro schema is generated for each source object, which contains the Avro field definitions.
  • avroNestedSchemaFormatV1
    . Structures messages by using a nested Avro schema format, which provides a main Avro record that contains a separate nested record for each type of Avro field.
  • avroGenericSchemaFormatV1
    . Structures messages in a generic manner that accommodates any source object definition. All source columns are represented by an array. Each array entry contains column data and metadata. The source column names are included in each data record, allowing the generic schema to be independent of the source table.
No default value is provided.
You can "wrap" a flat, nested, or generic schema by setting the Formatter.avroWrapperSchemaFormat property to avroWrapperSchemaFormatV1. The schema then consists of four fields for each source object.
Use a generic or wrapper schema to allow a single Avro schema to represent multiple source tables. For more information about the schema formats, see Avro Schema Formats.
Formatter.avroEncodingType
Required. The Avro encoding type that the CDC Publisher Formatter uses to serialize the Avro records to be included in messages. Valid values are:
  • binary
    . Use binary encoding to serialize Avro records.
  • json
    . Use JSON to serialize Avro records.
  • none
    . Do not use any explicit encoding type. Specify this option only if you use Confluent Schema Registry to store and retrieve Avro schemas.
No default value is provided.
The following additional properties can also be included in the cdcPublisherAvro.cfg file at your discretion:
Formatter.avroBinaryAsString
Controls whether change data with a binary datatype is represented as string data in Avro messages. Set this property to true if the data will be consumed by applications that do not support binary data, such as Informatica Intelligent Streaming. The default value is false.
Formatter.avroIncludeBeforeImage
Controls whether the generated Avro schema and messages include a field for before-image data. Set this property to true to include this field. Set this property to false to not include this field.
If you include the before-image field, the field is populated with data for UPDATE operations, if you set the Extract.pwxUpdateImageOption property to enable the extraction of before-image data from the PowerExchange change stream. For DELETE and INSERT operations, the field is not populated with data.
The default value is true.
Formatter.avroBinaryStringRepresentationType
If you set the Formatter.avroBinaryAsString property to true or use a generic Avro format, indicates whether binary data is represented as a hexadecimal string or base64 string. Valid values are:
  • hexadecimal
  • base64
The default value is base64.
Formatter.avroSchemaPrintDefaultFields
Controls whether Avro schemas include the "default" fields. If you need to reduce the schema size, you can set this property to false to exclude the default fields. The default value is true, which includes the default fields.
Formatter.avroSchemaPrintDocFields
Controls whether Avro schemas include the "doc" fields. The doc fields include metadata such as the CDC and PowerExchange datatypes, precision, and scale. If you need to reduce the schema size, you can set this property to false to exclude the doc fields. The default value is true, which includes the doc fields.
Formatter.avroSchemaPrintPretty
Controls whether Avro schemas include spaces and line feeds to improve legibility. If you need to reduce the schema size, you can set this property to false to exclude the spaces and lines feeds. The default is true, which includes the spaces and line feeds.
Formatter.avroWrapperSchemaFormat
Enables the use of an Avro "wrapper" schema format. The wrapper schema can be used to describe any source object. The wrapper, or parent, schema consists of four fields for each source object: the sequence number of the change record, source table name, change operation type, and the "wrapped" Avro child schema expressed as a large string. The consumer application can then parse the underlying data and put it in the proper Avro format for the source object. To use a wrapper schema format, set this property to
avroWrapperSchemaFormatV1
. No default value is provided. For more information, see Avro Wrapper Schema Format.
Formatter.avroUseLogicalDateType
Formatter.avroUseLogicalDecimalType
Formatter.avroUseLogicalTimeMillisType
Formatter.avroUseLogicalTimeMicrosType
Formatter.avroUseLogicalTimestampMillisType
Formatter.avroUseLogicalTimestampMicrosType
If you use Avro logical types for dates, decimal values, times, or timestamps and want the CDC Publisher to make a best-effort attempt to process these logical types, set this property to true. The following sets of properties are mutually exclusive so specify one property or the other but not both:
  • Formatter.avroUseLogicalTimeMillisType and Formatter.avroUseLogicalTimeMicrosType
  • Formatter.avroUseLogicalTimestampMillisType and Formatter.avroUseLogicalTimestampMicrosType
The default value for each of these properties is false.
If you set a property to true, make sure that the source fields are defined in the extraction map with a compatible data type, scale, and precision.

0 COMMENTS

We’d like to hear from you!