You can configure rules to generate a composite message key that includes one or more columns for each source table. PowerExchange CDC Publisher can then include the generated message key in the messages that it sends to the target messaging system. The target messaging system can use the message key to write messages with a specific key value to the same partition in a multi-partitioned topic.
When writing messages with a specific key value to the same partition, the target messaging system can maintain the order in which the change records were received from the CDC Publisher.
If you increase the number of partitions in a target topic, the source data might be written to a different topic partition than before, even if you configured generated message keys.
For the Formatter to generate a message key for each source table, you must create a rule-configuration file in a text editor and then specify the file in the
Formatter.captureColumnValuesFile=
pathFilename
property in the cdcPublisherAvro.cfg file.
Use the following syntax to define rules in the rule-configuration file:
rule=(
xnninstance
.
mapname_tablename
,
column1
,
column2
,
column3
,… )
additional rules...
[delimiter=
character
]
[tableNotFound=ABORT]
[trace={true|false}]
[updateUseImage={before|after}]
To include comments in the file, begin each comment line with the number (#) sign. For example:
#This text is for informational purposes only.
rule
. Defines a rule for generating a composite message key for a source table. In each rule, first identify the extraction map for the source table in the format
xnninstance
.
mapname_tablename
, where
xnninstance
is the schema name that PowerExchange generates for the extraction map,
mapname
is the extraction map name, and
tablename
is the source table name. Then specify the names of one or more table columns that comprise the message key. Ensure that the columns are defined in the extraction-map schema. CDC Publisher compares the specified column names to the extraction-map schema. If any column is not found in the schema, processing ends abnormally and multiple CDCPUB-14019 messages are written to the trace file.
You can define multiple rules in the same rule-configuration file.
When generating the message key, the Formatter uses the character representation of each column value followed by the delimiter character. Each column value and delimiter are appended to the composite key value in the order in which the columns appear in the rule definition. The composite key is then used as the Kafka message key for a record. The position of any column that has empty or null values in the message key is represented by the delimiter character only.
delimiter
.
Optional
. Specifies a single character that will be used as a delimiter after each key column value in generated message keys. You can specify this parameter only once in the rule-configuration file.
Default is the semicolon (;).
tableNotFound
.
Optional
. Set this parameter to ABORT to cause the Formatter to stop processing any source table data when the table does not have a rule definition in the rule-configuration file. Each source table must have a rule definition for the generation of the composite message key to succeed. You can specify this parameter only once in the rule-configuration file.
If you do not specify this parameter, the Connector.kafkaMessageKey property in the cdcPublisherKafka.cfg file can be used to create the message key for any source table that does not have a rule definition. Otherwise, the default rule in the target messaging system parameters determines the key to use for a record.
You can specify this parameter only once in the rule-configuration file.
trace
.
Optional
. Enables or disables tracing for the generation of message keys based on rule definitions. Valid values are:
true
. Enables tracing for message key generation based on rule definitions. If message key generation is successful, CDC Publisher writes message CDCPUB_14018 to the trace data set. This message identifies the generated key value, the source table, and the change stream location of the key generation event.
false
. Disables tracing for message key generation based on rule definitions.
You can specify this parameter only once in the rule-configuration file.
updateUseImage
.
Optional
. For source update operations only, indicates whether to use the before-image or after-image of the update row when generating the message key. Valid values are:
before
. Use the before-image of the data, as it appeared before the update operation.
after
. Use the after-image of the data, as it appeared after the update.
You can specify this parameter only once in the rule-configuration file.
You define the following two rules:
rule=(shipping.address,ADDRESSNO,STREET,CITY)
rule=(hr.employees,EMP,ADDRESSNO,STREET,CITY,COUNTRY)
tableNotFound=ABORT
trace=false
delimiter=;
For the shipping.address table, assume that the ADDRESSNO column value is "123" and the STREET column value is "Park Lane" in the data records. The CITY column is null. Based on these values, the following message key is generated for the table:
123;Park Lane;;
Note that the key columns appear in the same order that they are specified in the rule definition.
For the hr.employees table, assume that the EMP column value is "Fred", the ADDRESSNO column value is "12345", the STREET column value is "Main Street", the CITY column value is "New York", and the COUNTRY column value is "US". Based on these values, the following message key is generated for the table:
Fred;12345;Main Street;New York;US;
If the Formatter tries to process data for any table that does not have a rule definition, processing ends abnormally because the tableNotFound property is set to ABORT.
