User Guide

1.1
- 1.3
- 1.2
- 1.0

Back Next

Handling Changes to Source Tables and Extraction Maps

PowerExchange CDC Publisher does not update the Avro schemas if the source table structure and extraction map definition change. The extraction map definition must change if the source columns or fields for which change data is extracted are added or removed.

Use the following procedure to add or remove source columns or fields and properly resume PowerExchange CDC Publisher processing.

On the source system, stop change activity (inserts, updates, and deletes) on the source table.

Verify that the change data based on the current table definition has been published to the target messaging system or downstream consumer application. Use the following methods:

If you know the change operation that was last applied to the source table, verify that the target messaging system or consumer application has received that change.

In the change records that the target messaging system or consumer application has received, verify that the DTL__CAPXTIMESTAMP value is later than the time at which change activity against the source table was stopped.

If neither the target messaging system nor consumer application has received the expected change records, verify that the CDC Publisher is running. To do so, run the

PwxCDCInfo STATUS

command. The expected result is "PROCESS STATE GREEN All processes are running."

Shut down the PowerExchange Logger for Linux, UNIX, and Windows.

For more information, see the

PowerExchange CDC Guide for Linux, UNIX, and Windows

Optionally, clear the Avro schemas that match the existing source extraction maps from cache if you want to force the generation of updated Avro schemas that reflect the source table changes.

If you do not perform this step, CDC Publisher will continue to use the cached Avro schemas that do not reflect the source table changes.

To clear the Avro schemas from cache, enter the PwxCDCAdmin command with the CLEAR=FORMAT and TABLE parameters at the command prompt. Use the following syntax:

PwxCDCAdmin CLEAR=FORMAT TABLE={ALL|mapname_tablename} [INSTANCE=override_instance_directory]

Include the optional INSTANCE parameter if you are using an instance other than the default "instanceA" at

/pwxcdcpub_root installation/instanceA

. The mapname_tablename variable is the map name and table name portion of the new extraction map name, where the extraction map name has the format xnninstance.
mapname_tablename
. For more information about the command syntax, see PwxCDCAdmin Utility - Command and Parameters.

The command produces output such as shown in the following example:

  INSTANCE = instanceA. Connecting to host:port USHOST1:26987
  ------------------------------------------------------------------------------
COMMAND       RESULT  RESULT CODE  DESCRIPTION
CLEAR_FORMAT          1            The number of Avro Schema definitions cleared

Shut down the PwxCDCPublisher process by using the PwxCDCAdmin SHUTDOWN command. For more information, see Shutting Down the CDC Publisher.

You can issue the command from the

pwxcdcpub_root_installation\bin

subdirectory or from another location if you specified the PWXPUB_HOME environment variable. For more information, see Shutting Down the CDC Publisher.

In the PowerExchange Navigator, open the original capture registration and set its status to

History

For more information, see the

PowerExchange Navigator User Guide

Alter the source table or object to add or remove columns.

In the PowerExchange Navigator, re-create the capture registration and associated extraction map for the source table. Activate the capture registration.

If you set any PowerExchange CDC Publisher properties that refer to the mapname_tablename portion of the extraction map name, update the properties to refer to the mapname_tablename portion of the regenerated extraction map name.

The extraction map name has the format xnninstance.
mapname_tablename
.

For example, perform this step if you defined the

Extract.captureConsumerExcludeNameList

and

Extract.captureConsumerIncludeNameList

filter properties.

On the PowerExchange CDC Publisher system, restart the PwxCDCPublisher process as you would under normal conditions. For more information, see Restarting a PowerExchange CDC Publisher Change Data Stream.

To verify that the CDC Publisher recognizes the schema change and that the Avro schema is properly updated, perform the following substeps:

To get the updated extraction map information, enter the PwxCDCAdmin command with the RESET=FORMAT parameter at the command prompt.

Use the following syntax:

PwxCDCAdmin RESET=FORMAT [INSTANCE=override_instance_directory]

Include the optional INSTANCE parameter only if you are using an instance other than the default "instanceA" at

/pwxcdcpub_root installation/instanceA

Alternatively, you can wait until the CDC Publisher first receives change data for the updated table. At that point, the CDC Publisher automatically caches the extraction map and generates the Avro schema.

To force the generation of the updated Avro schema so that you can verify that it properly reflects the changed table structure, enter the PwxCDCAdmin command with the REPORT=FORMAT and TABLE parameters.

Use the following syntax:

PwxCDCAdmin [REPORT=FORMAT TABLE={ALL|mapname_tablename}][INSTANCE=override_instance_directory]

In this syntax, mapname_tablename is the map name and table name portion of the new extraction map name. Include the optional INSTANCE parameter only if you are using an instance other than the default "instanceA" at

/pwxcdcpub_root installation/instanceA

This command generates new Avro schema in the

\instance\reports

subdirectory. The schema is generated in a legible format. The generated schema can be used for verification purposes and by consumer applications.

Rename Saved Search

Table of Contents

User Guide

User Guide

Handling Changes to Source Tables and Extraction Maps

Handling Changes to Source Tables and Extraction Maps